Skip the Transcript: Direct Audio to Inkling

We already run speech-to-speech. Hugging Face speech-to-speech is the Realtime backend for Reachy. The stock conversation app stays on the body. The interesting next experiment is not a new robot stack. It is removing Whisper and letting a multimodal model hear the turn.

What we run today

Reachy is a cascade, not a single speech model. Silero VAD ends the turn. Local Whisper large-v3-turbo writes English. Grok answers in text. ElevenLabs speaks. Kitchen audio stays on the LAN. Only text leaves.

We measured this pipeline on July 27. Soft-end to transcription was a 0.27 s median over 20 turns. Grok time-to-first-token was 2.2–3.1 s, about 68% of the wait to first sound. ElevenLabs first audio was ~0.25 s. Deleting Whisper does not make the robot feel fast. It changes what the brain is allowed to hear.

Hop	Owner	What it emits	Why it exists
VAD	Local	A finished 16 kHz clip	Far-field turn-taking and barge-in
STT	Local Whisper	A transcript	Grok cannot hear audio
LLM	Cloud Grok	Reply text + tools	Identity, memory, short answers
TTS	Cloud ElevenLabs	Streaming PCM	A mouth. Local clone is still parked

Why skip STT

Not for latency. For the class of failure Whisper invents: fluent wrong English, “Big Mile,” a laugh turned into words, a name that only exists in audio. The model never sees the evidence. It only sees the guess.

Inkling is a Thinking Machines multimodal model. Audio, image, and text go in. Text comes out. It is not a talker. Official cards put Inkling-Small at 276B total / 12B active. Hugging Face’s own realtime demo is still cascade: mic → Inkling → Qwen3-TTS. Skipping STT means using the audio input they already trained, not pretending we bought Moshi.

The skip-STT path

Keep the body, the VAD, the Realtime dialect, the memory layer, and the current TTS handler. Delete the transcript. Send the finished VAD clip to a local Inkling-Small server. Get text back. Speak it with the mouth we already have.

Piece	Change
Stock conversation app	None. It already speaks OpenAI Realtime.
VAD / barge-in	None. Still the turn owner.
Whisper	Removed on this path only.
Brain	Local Inkling-Small, thinking off, OpenAI-compatible chat completions.
Memory / identity / no-act fence	Stay in our speech-to-speech handlers. Do not re-litigate.
TTS	Unchanged. Inkling does not synthesize speech.
Audio leaving the house	Still forbidden. Hosted Inkling is a privacy regression.

What the wire looks like

Upstream speech-to-speech now has a first-class switch: --stt none with --llm_backend chat-completions. Each completed VAD segment is posted as audio on the chat-completions message, not as a transcript. Responses-API is the wrong slot; a model can accept audio on /v1/chat/completions and still reject /v1/responses.

speech-to-speech serve \
    --stt none \
    --llm_backend chat-completions \
    --model_name inkling-small \
    --responses_api_base_url http://local-inkling/v1 \
    --responses_api_audio_content_type input_audio

That flag set is conceptual. Our live checkout is a dirty tree with local handlers. It does not have --stt none today. A blind git pull is still forbidden. The first implementation belongs in an isolated worktree, not in the production LaunchAgent.

Safer first experiment, if we only want to test Inkling as a brain: keep Whisper and point the existing chat-completions backend at a local Inkling server. That proves text quality and first-token time without merging upstream. Direct audio is the second experiment, after that server is boring.

Where it can actually run

Two machines are available for the brain: the 512 GB Mac Studio (M3 Ultra) and the two DGX Sparks. The voice Mac is a 64 GB M4 Max and cannot hold this model. Unsloth’s table puts Inkling-Small at 89 GB even in 2-bit and 132–170 GB in 4-bit. Hugging Face lists Inkling-Small NVFP4 at 180 GB — a pair-of-Sparks fit, not a 64 GB box. Full Inkling is still a cluster problem.

Host	Available?	Inkling-Small	Full Inkling
Mac Studio M3 Ultra (512 GB)	Yes	3/4-bit GGUF	2-bit maybe; 4-bit / NVFP4 no
Two DGX Sparks	Yes	NVFP4 180 GB on the pair	No
One Spark (128 GB)	Tight	2-bit maybe	No
Voice Mac (M4 Max, 64 GB)	No	No	No

Voice needs first token with thinking off, not a 1M-context coding model. Official vibe evals show Inkling transcribe-then-reason on audio. If we leave reasoning on, we will pay more than Grok and hear silence. That is an automatic fail.

How we would run it

Two first-party paths fit the machines we actually have. Full Inkling does not. Nothing below has been launched in this lab yet — these are the official recipes, mapped onto our boxes.

Box	Engine	Checkpoint	Status
Mac Studio M3 Ultra (512 GB)	llama.cpp Metal	Unsloth Inkling-Small-GGUF UD-Q3_K_XL (3-bit, 128 GB) or UD-Q4_K_XL (4-bit, 132–170 GB)	Documented. Not launched here.
Two DGX Sparks	SGLang NVFP4 TP=2	thinkingmachines/Inkling-Small-NVFP4 (official 180 GB)	SGLang lists a verified 2× GB10 cell. Not launched here.
Voice Mac (M4 Max, 64 GB)	—	—	Too small.
Full Inkling (975B)	—	NVFP4 600 GB / BF16 2 TB	Neither host.

Studio: llama.cpp + Unsloth GGUF

Unsloth’s documented Inkling-Small quant for a 512 GB Studio is UD-Q3_K_XL. Metal is on by default; their build notes say set -DGGML_CUDA=OFF on a Mac. Inkling reasons by default. For voice, thinking must be off or we will lose to live Grok on first token.

hf download unsloth/Inkling-Small-GGUF \
    --local-dir unsloth/Inkling-Small-GGUF \
    --include "*UD-Q3_K_XL*"

llama-server \
    --model unsloth/Inkling-Small-GGUF/UD-Q3_K_XL/Inkling-Small-UD-Q3_K_XL-00001-of-00004.gguf \
    --temp 1.0 --top-p 1.0 --min-p 0.0 \
    --chat-template-kwargs '{"reasoning_effort":"none"}' \
    --host 127.0.0.1 --port 8080

Unsloth’s published snippet is llama-cli with those same sampling flags. llama-server is the OpenAI-compatible shape speech-to-speech already speaks. Their build also targets llama-mtmd-cli for multimodal; we have not proven audio-in on this GGUF yet. That is a gate, not a given.

Sparks: SGLang NVFP4 TP=2

SGLang’s Inkling-Small cookbook has a dedicated GB10 image and a verified dual-Spark cell: NVFP4, TP=2, one GPU per node, ConnectX-7. Recipes pass --enable-multimodal so the server accepts audio_url as a base64 WAV. Requests that omit reasoning default to high. Voice must send none.

docker pull lmsysorg/sglang:dev-inkling-small-dgx-spark

# Official cell, not a lab-measured launch:
#   thinkingmachines/Inkling-Small-NVFP4
#   TP=2 across two Sparks over ConnectX-7
#   --enable-multimodal
#   --disable-prefill-cuda-graph
#   Triton attention + Marlin FP4/MoE
# Exact sglang serve line comes from their hardware command generator.
# Docker already wants: --ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband

vLLM also publishes an Inkling-Small NVFP4 recipe (180 GB, --tokenizer-mode inkling, audio extra vllm[audio]), but the listed hardware is B300 / GB200 / H200, not GB10. There is a community dual-Spark vLLM yaml in eugr/spark-vllm-docker. We are not treating that as first-party, and we have not run it.

What the voice client then sends

Once either server answers /v1/chat/completions, skip-STT is an audio content part, not a transcript. SGLang’s documented shape:

{"type": "audio_url",
 "audio_url": {"url": "data:audio/wav;base64,..."}}

That is the same clip our VAD already owns. Thinking stays off. The mouth stays the current TTS handler.

How we would stage it

No go implied. If we do this, the order is:

1. Stand up Inkling-Small as a side OpenAI-compatible server: llama.cpp GGUF on the Studio, or SGLang NVFP4 TP=2 on the Spark pair. Text-only smoke first. Do not touch Reachy.
2. Point a disposable speech-to-speech worktree at that server with Whisper still on. Compare short-turn quality and TTFT against live Grok. Keep thinking off.
3. Only then enable --stt none in that worktree. Headset first, kitchen later.
4. Score the same spoken turns both ways: name accuracy, barge-in, first sound, and whether it invents fluent wrong English.
5. If it wins as a fallback lane, say so. If it does not, leave the live stack alone.

Hard fail	Why it stops the experiment
Any household audio to a hosted endpoint	Breaks the LAN contract we already published
Thinking-on first token worse than live Grok	We already know the budget is the LLM, not STT
Kitchen audio worse than turbo Whisper	Then we kept the hard part and lost the easy part
Merge into the dirty production tree	The live robot is the rollback baseline

Companion posts: Reachy Mini: Milo’s Physical Avatar is the live product. One Milo, Many Bodies is the unfinished shared-runtime track. This page is a third thing: a single deleted hop, drawn before anyone types git pull.