This page is the current DS4-F operating note. It tracks what profile is live on the dual DGX Spark route, which trials produced real measurements, and what not to over-claim from the DSpark chase.
July 4 call: DS4-F is now running the Tech2Wild/Tony Patch3-style profile at 350K/12. That profile gave the best single-client C1 result in today’s trial: 56.8 tok/s, beating patched 1M/6 at 54.4 tok/s and the old live 1M/6 baseline at 46.8 tok/s. 1M/6 remains the long-context lane; 200K/16 remains the older high-concurrency fallback.
Running now350K/12
Best C1 today56.8 tok/s
Draft acceptance0.626
KV pool975,835
Long-context lane1M/6
Current running profile
Item
Current value
Status
live route350K/12, selected because James asked to keep whichever profile produced the best C1 performance.
Patch3 scheduler guard bind-mounted; DSpark draft sampling set to probabilistic; old server-side repetition_penalty=1.05 override removed; VLLM_USE_FLASHINFER_SAMPLER=1.
Fresh live proof
/v1/models max_model_len=350000; exact smoke returned DS4F_350K12_LIVE_OK; repeated C1 benchmark best was 56.8 tok/s.
Startup proof
GPU KV cache size: 975,835 tokens; Maximum concurrency for 350,000 tokens per request: 2.79x; Application startup complete.
July 4 Patch3 trial
The interesting change from Tech2Wild’s post was not simply “MTP5” — our live lane already used five speculative tokens. The useful change was the scheduler-level Patch3 guard plus probabilistic DSpark draft sampling and removal of the repetition-penalty override.
Profile
C1 result
Acceptance
Decision
Old live 1M/6
46.8 tok/s
0.483
Baseline only. This was the pre-Patch3 path.
Patched 1M/6
54.4 tok/s
0.591
Works and remains the long-context lane. Startup proof: max_model_len=1048576, KV 1,665,106, full-context concurrency 1.59x.
Patched 350K/12
56.8 tok/s
0.626
selected live Best C1 in this trial, with more agent-friendly slot count.
Gotcha we hit: for MTP5, setting the cudagraph capture cap too low crashed startup with No valid cudagraph sizes after rounding to multiple of 6. Setting MAX_CUDAGRAPH_CAPTURE_SIZE=12 fixed the launch. vLLM logs may later truncate the effective max capture size to 8, but the larger configured value is what let startup pass.
Profile history and fallback map
Profile
Status
Observed result
Operational read
350K/12
running now
Patch3 profile booted, smoked, and repeated C1 best at 56.8 tok/s.
Current best-C1 agent lane.
1M/6
long-context lane
Patch3 profile booted and produced 54.4 tok/s C1; previous isolated live probes showed c1/c4/c6 viability.
Use when the million-token ceiling matters more than the small C1 edge.
200K/16
older fallback
Previously booted, smoked, and benchmarked with strong C16 behavior.
Still the known short/mid-context high-concurrency fallback if the Patch3 lane misbehaves.
1M/12
failed bind
Earlier clean retry had enough KV but failed after warmup in the shared-memory / broadcast path.
Do not retry unchanged. Any new try needs the Patch3/new-compose path and bounded rollback gates.
1.5M/12
failed bind
Earlier run loaded weights and profiled KV, then never bound the API.
Not a production candidate; also beyond the model’s 1,048,576 YaRN-calibrated ceiling.
What changed in the live files
Change
Why it matters
Patch3 scheduler guard
Prevents DSpark spec placeholders from being resized on chunked prefill chunks, the root fix for cold long-prompt garble / prompt echo.
draft_sample_method=probabilistic
Matches the Tech2Wild/Tony recipe and improved acceptance in today’s route trial.
Removed --override-generation-config
Drops the old server-side repetition_penalty=1.05, which is a DSpark spec-decode crash risk rather than a garble fix.
MAX_CUDAGRAPH_CAPTURE_SIZE=12
Needed for MTP5 startup on this image; lower sizing produced the cudagraph rounding failure above.
VLLM_USE_FLASHINFER_SAMPLER=1
Part of the Jul 4 serving recipe now persisted in the default env.
What not to over-claim
Claim
Reality
“350K/12 is globally faster than every other profile.”
No. It won today’s C1 probe. C4/C8/C12 curves still need fresh measurement before making wider throughput claims.
“1M/6 is obsolete.”
No. It is still the million-token lane. The 350K route is a better default when short agent turns and C1 speed matter more.
“MTP5 was the new discovery.”
No. We already had MTP5. The root change was Patch3 + probabilistic draft + no repetition override.
“1M/12 works now.”
Not proven. The old unchanged 1M/12 path failed; a Patch3 1M/12 retry remains a separate experiment.
Next tests
Next step
Why
Pass condition
Run c4/c8/c12 on patched 350K/12
C1 picked the default, but fleet behavior depends on concurrency.
All streams complete, queue returns idle, no garble, and acceptance stays in-family.
Keep patched 1M/6 ready
The 350K lane gives up the 1M ceiling.
Documented env switch and smoke path remain boring.
Only retry 1M/12 with changed variables
Unchanged retries already point at SHM/startup behavior, not raw KV.
Bind, exact smoke, and queue-idle proof under a bounded startup window.
Provenance
Item
Value
Jul 4 live endpoint
http://192.168.1.11:8888/v1, model deepseek-v4-flash-dspark, max_model_len=350000.
Smoke marker
DS4F_350K12_LIVE_OK from an OpenAI-compatible chat request issued after the route switch.
Bench method
benchmarks/keys-concurrency/bench_concurrent.py http://127.0.0.1:8888 1, best of two measured rounds after warmup.
Runtime
vllm-dspark-runtime:dspark-nvfp4-stage-c; DSpark TP=2 across the Spark pair; B12X MoE backend confirmed in logs.
Working directory
/home/milo/ds4-f-wesche-keys-c12
Persisted files
.env.dspark and docker-compose.dspark.yml now carry the patched 350K/12 defaults on Spark1 and Spark2.
