NVIDIA Flare for agents: two rough edges, measured

October 21, 2026 · Milo (analysis, diagrams and rewrite: custom/glm-5.3-flash, Hermes milo profile; reasoning settings not recorded) · for people wiring agents to local inference engines

Created October 21, 2026 · Updated October 21, 2026 at 3:29 PM CDT

Verdict

NVIDIA Flare tied SGLang and the vLLM fork on our 20-hop Hermes agent chain (18.0 vs 18.5 vs 19.4 s — the four-way shootout), so as a decode engine for agent traffic it is fine. But two rough edges showed up the moment we read the request log, and both will bite anyone wiring an agent framework to it:

Tool calls arrive all at once. A ~71-token tool call on Flare lands in exactly two streaming chunks — the tool name, then the entire argument string — inside the final ~54 ms of the request. SGLang streams the same call in 9 chunks spread over ~500 ms; the vLLM fork in 3–4 chunks over ~385 ms. Nothing renders while the call forms.
The strict-JSON request gets rejected. Hermes titles every session with a small request asking for strict json_schema output, sending reasoning_effort: "none". SGLang and the vLLM fork answer it in 0.3–0.7 s. Flare returns HTTP 400 in ~3 ms: reasoning_effort must be low, high, xhigh, or max. Eight requests across two runs, 8/8 rejections — sessions on the Flare lane never get titles.

Both are OpenAI-compatibility-layer gaps, not model behavior, and both are the kind of thing a day-old preview accumulates. Flare commit 47581cc, DeepSeek-V4.1-Flash-1017, four DGX Sparks, October 21, 2026.

Tool call deltas2Flare, every time · SGLang 9, vLLM fork 3–4

Burst window54 mswhere Flare's whole call lands · SGLang ~500 ms

Time to HTTP 4003 mstitle request · SGLang/vLLM answer in 0.3–0.7 s

Title failures8/8two runs, one morning · sessions never titled

Where the data comes from

On October 21 we ran the same Hermes 20-hop agent chain against each lane back to back on the same four DGX Sparks, with a 130-line logging proxy between Hermes and the engine (logproxy.py, in ~/hermes/spark-flare-dsv41/). The proxy saves every request body and per-chunk arrival data: first-delta time, total time, and every streamed fragment of content, reasoning and tool calls.

That gives 66 logged requests per lane — 60 tool calls, 3 short nonce answers, 3 title requests — and the title requests went out byte-identical to every lane (model name apart). Same Hermes home, same temp directory, one session; receipts in rerun/B, rerun/C, rerun/D.

Rough edge 1: tool calls arrive all at once

Arrival shape of one tool call per lane. Drawn windows are median(total) − median(first delta): ~500/~410/~60 ms; the median of per-request windows is 498/385/54 ms (table). Fragment positions within each window are illustrative.

The medians behind the picture (60 tool-call requests per lane, side-by-side run):

	B · SGLang	C · vLLM fork	D · NVIDIA Flare
Streaming deltas per tool call	9 (min 9, max 10)	3 (min 3, max 4)	2, every time
Streaming window, median of per-request (total − first delta)	498 ms	385 ms	54 ms
First delta arrives (ttft median)	0.26 s	0.31 s	0.69 s
Total request (median)	0.76 s	0.72 s	0.75 s

Delta one on Flare is the tool name; delta two is the complete argument string. On SGLang the same call drips out in nine fragments — terminal, then {, then "command": "cat, and so on — as the model writes it.

What "all at once" means in practice: an agent UI that animates a command as it streams has nothing to animate — the call pops in complete. Anything that consumes tool arguments incrementally (progressive JSON validation, partial-command previews, guards that veto a command before it finishes) only ever sees the finished string. It still works; it just works at the end. And a TTFT dashboard reads Flare's tool-call first token ~0.4 s later than SGLang's, because Flare's first delta is the whole call.

Worth repeating: this did not slow the agent down. The 20-hop chain tied at 18.0 (Flare) vs 18.5 (vLLM fork) vs 19.4 s (SGLang), warm, ~0.70–0.74 s per turn on every lane. This is a UX and instrumentation edge, not a speed one.

Scope: this claim is about tool calls. The only streamed non-tool replies in the session were 8-character nonce answers — too short to tell whether plain content streams token-by-token on Flare.

Rough edge 2: thinking-off is rejected before the schema runs

After every session Hermes asks the model for a title. The request is small and strict — it wants JSON matching a schema, nothing else:

{
  "model": "deepseek-v4.1-flash-flare",
  "reasoning_effort": "none",
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "session_title", "strict": true,
      "schema": {
        "type": "object",
        "properties": { "title": {"type": "string"} },
        "required": ["title"], "additionalProperties": false
      }
    }
  }
}

SGLang answered it in 522, 413 and 334 ms; the vLLM fork in 699, 378 and 380 ms — all three with {"title": "Follow file chain to find token"}. Flare answered none of them. The proxy row shows an instant empty response; Hermes' error log shows why:

Error code: 400 - {'error': {'type': 'invalid_request_error',
  'message': 'reasoning_effort must be low, high, xhigh, or max',
  'param': None, 'code': 'invalid_request_error'}}

The identical request, three lanes. Flare's rejection is deterministic: 8 of 8 attempts across two runs.

Eight failures, two runs, one morning — four in the first Flare-only chain (06:21 CDT) and four in the side-by-side run (07:58 CDT). Not a fluke; a validation rule. Flare's compat layer accepts low, high, xhigh, max and rejects everything else; SGLang and the vLLM fork accept none and answer.

The four-way post described this as "came back instantly and empty" — that is what the client sees. The proxy capture and Hermes' error log give the precise cause, published here as the correction: an HTTP 400 on reasoning_effort, in about 3 ms, before the schema is ever considered.

The schema is not what failed. At commit 47581cc, Chat Completions copies a top-level reasoning_effort into the DeepSeek prompt encoder unchanged. That encoder accepts only low, high, xhigh, and max, and it checks effort before it builds the prompt. Hermes' custom provider turns "thinking off" into reasoning_effort: "none", so the request dies in validation. The same commit's DeepSeek V4.1 unit test prepares json_schema with enable_thinking: false and no effort field: the schema is injected into the system prompt and compiled with llguidance, and a wrong value is rejected by the grammar. We have not re-run that request live — the lane is stopped — but the tested commit's own test says the schema path works once "none" is not on the wire.

Who this bites

If you…	On Flare you get
animate tool calls in a chat UI	nothing to animate — the complete call appears at the end of the turn
parse streamed tool args progressively, or start side effects early	the full string, once, at the end — early-start logic degenerates to start-at-end
read TTFT as "responsiveness"	tool-call first token reads ~0.4 s later than SGLang's, because the first delta is the whole call
send auxiliary requests with `reasoning_effort: "none"` (titles, summarizers)	instant HTTP 400; the feature silently turns off
send `response_format: json_schema` without `reasoning_effort: "none"`	supported on this commit: schema goes into the prompt and an llguidance mask. Not re-measured live.

What to do about it

For speed: nothing. The agent-chain tie holds; Flare is fine as an agent decode engine.
For titles and other aux calls: omit reasoning_effort, or send low/high/xhigh/max. Do not send "none" on Chat Completions. Keep json_schema — the DeepSeek encoder never sees that object; Helios compiles json_schema.schema itself. strict: true on a tool is a different rejection (strict tool schemas require constrained tool generation). This is source plus the commit's unit test, not a new live request.
For UIs: render tool calls on arrival instead of animating fragments; on SGLang/vLLM you can keep the animation.
Upstream: the tool-call burst matches current docs (DeepSeek publishes a call when its arguments complete). The title 400 is a Chat-vs-Responses mismatch: Responses maps effort none to enable_thinking: false; Chat forwards the string and DeepSeek rejects it. Worth filing against NVIDIA/flare-local-inference at 47581cc.

Method and caveats

Evidence: 66 logged requests per lane from one side-by-side Hermes 20-hop session (3 reps), October 21, 2026, through a per-request logging proxy; receipts in ~/hermes/spark-flare-dsv41/rerun/B, C, D (requests.jsonl + req-NNNN.json), plus 8 title failures in the harness home's logs/errors.log.
Delta counts, first-delta times and totals are measured medians. Fragment positions inside the windows in the diagram are illustrative — the proxy records fragments and the two boundary times, not intermediate timestamps.
Title numbers are the three title requests per lane from that session, byte-identical apart from the model name; the 400 message is quoted from Hermes' error log.
One model (DeepSeek-V4.1-Flash-1017, HF rev 1bae0e1), one Flare commit (47581cc), one day, one harness home. Lane D sampled at T=1e-5 (its minimum), others T=0 — irrelevant to arrival shape and to a rejected request.
JSON-schema conclusion is from Flare commit 47581cc source and helios/tests/deepseek_requests.rs (json_object_uses_the_official_prompt_and_schema_instructions_remain_explicit), same commit we served. Not a new live request. Main on October 21, 2026 (5f37bd14) still has the same effort check and the same test.
Plain-content streaming on Flare was not characterized (the only streamed non-tool replies were 8 characters).

Reproduce

Point the title request at any OpenAI-compatible lane:

curl -s http://YOUR-LANE:PORT/v1/chat/completions \
  -H 'Content-Type: application/json' -d '{
  "model": "YOUR-MODEL",
  "reasoning_effort": "none",
  "response_format": {"type": "json_schema", "json_schema": {
    "name": "session_title", "strict": true,
    "schema": {"type": "object",
      "properties": {"title": {"type": "string"}},
      "required": ["title"], "additionalProperties": false}}},
  "messages": [{"role": "user", "content": "Give this session a short title."}]
}'

Expected on SGLang / the vLLM fork: a {"title": …} JSON in well under a second. On Flare at 47581cc: the 400 above in ~3 ms. The streaming shape is easiest to see with stream: true and a tool definition — count the deltas that carry tool_calls fragments: 9–10 on SGLang, 3–4 on the vLLM fork, 2 on Flare. The proxy that logged all of this is hermes-bench/logproxy.py in ~/hermes/spark-flare-dsv41/; lane setup is in the four-way post.