Step-5-Preview on one GB300: day-one bring-up, quality gates, and where the time goes
- Does it work? Yes. 256K context, needles 30/30 out to 258K tokens, vision 3/3, tools parse, BFCL 559/600, and output that is bit-identical across restarts.
- How fast? 30.1 tok/s single-stream on every prompt type, 70.6 tok/s total at 8 concurrent. With the model’s own MTP head at k=1 that becomes 39–42 tok/s and 86.9, with output bit-identical to no speculation. k=1 is now this lane’s default.
- Can it go faster? A routing trace says moving the most-used experts into GPU memory (the “hotsplit” trick from our MiMo recipe) would buy about +25–40% single-stream. It needs a kernel-level change on this model, and we decided it isn’t worth the work.
The lane
The checkpoint is 367 GiB of tensors. Its routed experts are NVFP4 on 85 layers (3–87) and BF16 on the last three MoE layers (88–90); attention, shared experts and router stay BF16. Each layer has 352 experts and each token picks 8. One GB300 shows about 250 GiB of HBM to CUDA, so something has to live elsewhere.
vLLM’s UVA offload (--offload-backend uva --cpu-offload-params routed_experts) keeps the chosen expert weights in pinned Grace memory and lets the GPU read them directly. We swept the offload amount and took the smallest one that served 128K with a KV pool at least twice the context: 180 GiB on Grace, 183 GiB of weights in HBM. That boot also runs 256K.
vllm/vllm-openai:step-5-preview@sha256:8bc189bb… (TP1, GPU picked by UUID)
--offload-backend uva --cpu-offload-gb 180 --cpu-offload-params routed_experts
--max-model-len 262144 --max-num-seqs 8 --max-num-batched-tokens 8192
--gpu-memory-utilization 0.90 --reasoning-parser step5 --tool-call-parser step5 --enable-auto-tool-choice
--spec-method mtp --spec-tokens 1 # added after the gates below| Offload to Grace | C1 tok/s | KV tokens | Host memory left | Note |
|---|---|---|---|---|
| 160 GiB | 31.8 | 158K at 128K ctx | ~135 GiB | KV under 2× context: rejected |
| 180 GiB | 30.4 | 371K (128K) / 431K (256K) | ~85–92 GiB | chosen |
| 200 GiB | 29.0 | 578K | ~52 GiB | |
| 220 GiB | 27.6 | 795K | ~11–17 GiB | too close to host OOM |
Each step of offload costs about 2 GiB of host memory per GiB moved, which fits power-of-two pinned allocations (inferred). Speed falls about 1.4 tok/s per 20 GiB moved. The model’s whole personality on this box is that trade: KV room versus Grace reads.
Quality gates
Every number here is from one gated session on the lane above. The point of a gate is to catch a serving setup that changes the model’s answers, not to grade the model.
| Gate | Result |
|---|---|
Tokenizer: both engines vs the checkpoint’s tokenizer.json (653 texts) | 653/653 for vLLM and SGLang. The boot warning telling you to set fix_mistral_regex=True is a false alarm; following it changes tokenization on 339 of 653 texts. |
| Logprobs across a restart (797,450 positions, incl. 8 docs of 80K tokens) | bit-identical, Δ 0.000, 0 top-1 flips |
| Greedy output, 20 prompts × 200 tokens, three boots and graphs-off | 20/20 every time |
| Needle ladder, 8K → 258K, single/multi/distractor | 30/30 first try |
| Vision, 3 checkable images | 3/3 |
| Tool calls: 4-turn replay with reasoning carried | pass |
| BFCL v4 simple_python + multiple (our AST grader) | 559/600 (375/400 + 184/200), first measurement, no bar |
| Cross-engine logprobs vs SGLang | not measurable: SGLang’s Step-5 indexer is sharded four ways and divides by zero at one GPU. We accepted the evidence above instead. |
AutoTokenizer.from_pretrained on this checkpoint loads a tokenizer that drops the spaces between words (“up ons uch abase”). It matched the real tokenizer on 0 of 653 texts. The servers are fine; any script that counts tokens or builds training data on the client side is not. The cross-engine gate would never see it, because it feeds token ids.MTP: the free win and the not-quite-free one
Step-5 ships three MTP layers, small heads that guess the next tokens so the big model can verify several at once. We booted k=1, 2 and 3 separately.
| Prompt type | k=0 | k=1 | k=2 | k=3 |
|---|---|---|---|---|
| prose | 30.1 | 39.2 | 39.4 | 36.7 |
| code | 30.1 | 39.6 | 40.8 | 42.7 |
| agent (tool-plan JSON) | 30.1 | 38.9 | 38.9 | 37.7 |
| json | 30.1 | 41.6 | 47.7 | 49.6 |
| math | 30.1 | 41.9 | 48.4 | 52.2 |
| C8 aggregate | 70.6 | 86.9 | 87.5 | 86.9 |
| Greedy output = k=0? | — | 20/20 | 3/20 | 3/20 |
Speculative decoding is supposed to be lossless, and at k=1 it is: bit-identical output, +29–39% single-stream. At k=2 and k=3 the text stays coherent but diverges from k=0 at near-tied tokens, reproducibly. Verifying three or more tokens goes through a numerically different path in the target model, a few BF16 rounding steps away (inferred, not traced). That isn’t a bug in the drafter, but it does mean k≥2 is a different model output, and we haven’t measured whether it’s a better or worse one. So k=1 is the default, and k≥2 stays measured, not promoted. Note also that C8 is flat from k=1 to k=3: a wider verify step reads more unique experts from Grace.
Where the time goes
A profile of one decode token (C1, k=0) says 33.1 ms per step. The MoE expert matrix multiplies are 18.9 ms of it, and they split cleanly in two:
| Expert layers | Count | Per layer | Per token |
|---|---|---|---|
| Experts in Grace memory (layers 3–54) | 52 | ~337 µs | 17.5 ms |
| Experts in HBM (layers 55–87) | 33 | ~45 µs | 1.5 ms |
| Everything else: attention, dense layers, router, launches | ~14 ms |
The same eight expert picks cost about 7× more when they come over C2C. That is the whole case for smarter placement. vLLM’s offloader puts whole early layers on Grace, so it doesn’t care which experts are popular. In our MiMo recipe, re-homing the hottest experts per layer into HBM (“hotsplit”) was the biggest single win.
Would hotsplit pay here? We measured, then said no
We traced every routing decision for 190,283 decode tokens: chat, agent turns with real tool schemas, code, structured output, the benchmark classes and BFCL. Then we asked: with exactly the same HBM budget, what share of the expert picks would land in HBM if the hottest experts went there instead of whole layers?
| Placement (same HBM budget) | Picks served from HBM | Est. C1, k=0 |
|---|---|---|
| Today: whole layers | 38.8% | 30.2 (measured) |
| Hottest experts, scored on the traffic they were chosen from | 75.7% | ~42 |
| Hottest experts, scored on new traffic (BFCL, benchmark classes, second half of the trace) | 57–68% | ~38 |
| Same, with 28.7 GiB more HBM for experts | 83.3% (in-sample) | ~45 |
The cost model behind the right column reproduces the measured step (33.14 ms predicted, 33.06 measured), so it’s anchored, but the speeds are still estimates. Two things make the case weaker than it was on MiMo:
- Routing is spread out and traffic-dependent. The hottest 137 of 352 experts in a layer cover 76% of its picks on average. A hot set chosen from chat covers only 61% of code’s picks. You’d need a broad, maintained ranking.
- The kernel won’t cooperate. MiMo’s split worked because its MoE kernel honoured an
expert_map, so each expert bank could be a separate call. Step-5’s clamped-SwiGLU activation only runs on vLLM’s CUTLASS NVFP4 path in this image, and that path ignoresexpert_map. The other FP4 MoE kernels refuse the activation at boot. Splitting means writing the routing yourself and gating the summation-order change for quality.
Put together: about a day of kernel-level work for roughly +25–40% single-stream, on a preview model, before MTP. That’s a real gain, but not the step change hotsplit was on MiMo, and the ~14 ms outside the experts caps the whole idea near 70 tok/s even at perfect placement. We’re parking it and keeping the lane as above.
What this is not
- Not a model review. BFCL is our own grader on one box, and no long-form quality comparison was run.
- Not a cross-engine check. We couldn’t run SGLang as a reference on one GPU. Bit-identical repeats show the lane is stable, not that it matches a second implementation.
- Not a recipe yet. Everything above is measured; the published recipe comes later.
- Step-5 is under the StepFun Model License 1.0. StepFun doesn’t endorse these numbers, and we’re not redistributing any weights.