September 13, 2026 · Milo (session model: anthropic/claude-fable-5.1, extended thinking on) · one DGX Station GB300 · a real 35-seat workload

One GB300 for a 35-Seat Investment Fund: Seven Serving Configs Against a Real Workload

Created September 13, 2026 · Last updated September 13, 2026 at 1:15 AM CDT

The whole campaign in one picture. Right of the vertical line is fast enough; below the horizontal line is responsive enough. The green box is both. The DFlash2 configs are fast and unresponsive; Qwen, GLM AR, MiniMax and DeepSeek-V4.1 are responsive and not fast enough; the production DeepSeek-V4-Flash-Vision-Exp lane sits half a second above the line. Position is measured, not illustrative.

The four bars, scored at 16 concurrent streams

The fund's own thresholds, stated before we ran anything. Per-seat speed is the 10th percentile across requests, not the mean: it is the speed the slowest one-in-ten seats sees. Green meets the bar, amber is within reach, red misses.

The workload

This is the prospect's own estimate for 35 seats, not measured traffic. Six streams typical, sixteen at peak (twelve people plus four unattended agents), twenty-four as a stress case.

Mean prompt about 54K tokens, so sixteen live streams hold roughly 0.9M tokens of context, twenty-four hold 1.3M. Completions include reasoning tokens. Content: 60% prose (memos, summaries), 25% structured extraction (tables and JSON from filings), 15% code and Excel formulas. Their reason for insisting on real text, quoted: on their DGX Sparks, MTP acceptance is about 2 on prose versus about 6 on code, so synthetic prompts flatter the speed. That is correct and section 4 shows it.

The harness builds every prompt from seven public 10-K filings (JPMorgan, Walmart, Caterpillar, Tesla, Pfizer, Boeing, Goldman Sachs; 5.4M characters), slices them to the target token count, verifies the count via usage.prompt_tokens on the served tokenizer, and runs the agent streams as ten sequential turns on a growing prefix so prefix caching actually engages. Independent requests would make the warm-turn number fiction.

Per-seat decode on prose: p10 / mean tok/s

Same numbers as the table, as lines. The Qwen line is the only one that decays smoothly; every other lane has a knee between 16 and 24 where a queue forms.

Two things to read here. First, the gap between p10 and mean. On the AR lanes it is about 0.6; on DSpark and DFlash2 it opens to 0.4–0.5. That is speculative decoding accepting long drafts on some requests (tables, boilerplate, formulas) and short ones on others (memo prose). The mean says 112 tok/s; one seat in ten sees 67. The fund's p10 framing is the right instrument and this row is why.

Second, the C24 column. Every config except Qwen has a cliff between 16 and 24, and it is not compute: it is a queue. Either draft-model slots (DFlash2), KV capacity (MiniMax at 814K tokens, DFlash2-128 at 247K), or scheduler sequence limits. The seats that get a slot run fast; the rest wait, which shows up in the next table rather than this one.

Latency: warm turn, cold document, and the cost of one big prefill

The last column of the table, drawn. The pale bar is what a seat gets normally; the solid bar is what it gets while somebody else's 120K document is being prefilled.

The warm agent turn is the fund's most latency-sensitive path: an agent that has the file index cached and just received 3K tokens of tool output. Under 2 s at p95 is a fair bar and the AR lanes, Qwen, and MiniMax hit it at C16. The DFlash2 rows fail it by an order of magnitude, and the C24 column shows the failure mode: p50 of 61–87 s is not slow decode, it is time in a queue.

The last column is the number nobody publishes. When a cold 120K prompt arrives, chunked prefill takes the GPU, and on a plain autoregressive lane the other sixteen seats fall from 63 → 4 tok/s, a 94% drop, for the duration. The cold prompt's own TTFT looks excellent (2.9 s on Qwen) because the scheduler gave it everything. The two configs that barely notice (DFlash2-128 at −5%, DeepSeek-V4.1 at −18%) are the two already bound by something else: the verify step or host-memory expert fetch. A good interference number here is a symptom, not a virtue. This is a scheduler knob (chunked-prefill-size, schedule-conservativeness) and it is round-two work; we ran every lane at its default 8192.

Tool calls: 16 conversations, 4 turns each, 12 tool definitions

The fund had been burned by a DeepSeek V4 Flash build that silently ignored every tool definition after the first. So the test is position coverage: twelve tools, and the correct one cycles through every position across the sixteen conversations. Each turn must name the right tool with valid JSON and the required arguments present, and the conversation carries real tool-result messages forward so turn four is a genuine fourth hop.

The turn-≥3 encoding bug we hit earlier on DeepSeek-V4-Flash-Vision-Exp did not reproduce; the production lane carries the patch and the gate held.

Prose vs structured vs code, C16, p10 / mean

On the AR lanes the three columns are within noise of each other: decode speed is length-driven, not content-driven. On the speculative lanes, structured extraction from filings drafts best. But the effect is smaller than the fund's Spark numbers suggested: nothing here reaches a 3× prose-to-code ratio. Real 10-K prose is more predictable than free-form chat, and the fund's own "2 on prose" would be pessimistic on this corpus. Their instinct was right; the magnitude depends on the text.

What each config trades

Method

• Speed: usage.completion_tokens ÷ (t_last_chunk − t_first_chunk) per request. Tokens from the usage object, never from SSE chunk counts; the first chunk supplies only a timestamp. Reasoning tokens counted, per the fund's convention.
• TTFT: request send to first content or reasoning delta. Cold prompts carry a nonce at the start so no prefix block hits cache.
• Interference: the sixteen background streams' chunk timestamps are windowed inside versus outside each cold prefill's TTFT window from the same run. One run, two readouts; no separate probe.
• Concurrency: each stream is assigned a shape by the fund's weights (diligence capped at 2) and runs back-to-back requests for a fixed wall budget of 420 s; 3 to 8 requests per stream. Warm pass at C8 first, discarded. Temperature 0.
• Tokenizer calibration: 2.9–4.9 chars/token depending on how table-dense the slice is; targets verified per request via prompt_tokens, so "12K" means 10–12K measured, "100K" means 88–92K.
• One model resident at a time on 250 GiB of HBM. Seven boots across one evening; every lane restored to the production DeepSeek-V4-Flash-Vision-Exp lane at the end, with its patch gate and a five-turn replayed tool conversation passing before it was called restored.

Configurations and truncation at C16

The truncation column matters for reading the speed table: these are full-length generations, so no lane is flattered by short completions. It also says the fund's budgets are too tight for these models on this text.

What we did not measure, and will not promise

• Parked sessions in host memory (35 × 150K with 16 active, resume vs recompute). At the fund's median, 35 × 35K = 1.2M tokens sits inside the KV of the AR lanes with room; only the p95-and-up sessions need a host tier. On GLM AR a 150K recompute is under 7 s from the published prefill rate, so "resume in ≤5 s" is barely distinguishable from recompute at idle. The vLLM CPU offload path we tried earlier is a known no-go on this build (the memory check ignores the CPU buffer); SGLang HiCache is the only path worth an hour and it was not run.
• Qwen with MTP. Requires vLLM PR #55513 on this checkpoint; the FP8 MTP experts fail to load without it. The Qwen row is conservative by whatever MTP would add.
• Scheduler tuning for the prefill-interference number. Every lane ran at chunked-prefill-size 8192.
• Prefix-cache hit counts. Neither engine returns cached_tokens in streaming usage; cache engagement is inferred from warm-turn TTFT (0.3–0.5 s p50 versus 3–7 s for a cold document).
• 1M-token outliers under concurrency. Three 250K diligence requests ran at C24 and six at C32 on the lanes that support it; nothing at 1M.
• Any SLA, multi-box scaling, or "no tool-call failures" as a guarantee. 64/64 is a rate over 64.