Reachy Mini: Milo's Physical Avatar

A Reachy Mini Wireless (Pollen Robotics / Hugging Face, $499 DIY kit) is the body. Milo is the mind. Same identity as desktop Hermes Milo — not a separate chatbot bolted onto a cute shell.

Architecture

Stock path keeps the body. Custom path is everything that makes it Milo.

Layer	Stock (HF / Pollen)	Custom (J&M)
Body / IO	Daemon, motors, media, conversation app	Headless startup and recovery, single-owner frame broker, uploader, head tracker, and volume lock
Brain	Optional hosted realtime rollback	Local far-field STT → realtime language response → streaming speech synthesis
Memory	—	Layered recall + hard relationship facts + durable schema-v2 writeback outbox
Eyes	Body camera and media stack	People/dog identity context; single-owner camera broker; continuous multi-subject gaze
Knowledge	—	Phrase-gated read-only research with spoken filler; no side-effect tools

What's live

Capability	State
Headless appliance	Live. Conversation app, local services, motor wake, health and recovery are automated.
Voice loop	Live. Far-field VAD; local large-v3-turbo STT on Metal; realtime language response; streaming speech.
Memory	Hardened. Layered recall plus serialized schema-v2 durable writeback; ambiguous delivery stops rather than replaying.
Voice authority	Hardened. Literal speech control is loopback-only, authenticated, idle-gated, and fenced against cancellation races.
Read-only research	Live. Only the explicit “ask Big Milo” phrase opens the research path; spoken filler prevents dead air.
Vision and gaze	Live. People/dog context, fresh camera frames, and continuous multi-subject head tracking.
Proactive greeting	P0 live. Deterministic text, quiet hours, cooldowns, unknown-person silence, and decision logs.
Sleep and wake	Implemented. Nap lowers the body without killing the conversation app; wake restores it.
Act-from-voice	Off by design. No email, deployment, purchases, or privileged action path.

Development status at a glance

How the camera stack actually works

One lesson took a while: the camera needs a single owner. Two LOCAL media clients on the Pi both fight the sensor; the second gets empty frames. Stock daemon face-tracking only works when nothing else holds the camera.

Piece	Job
frame_broker	Only process that opens the camera. Publishes fresh local JPEG frames for other components.
frame_uploader	Sends a frame to M4 vision every ~2s for ID
head_tracker	Reads broker frames; tracks faces (+ dog boxes from M4); continuous head goto
M4 vision_service	Names people and dogs. Does not drive the neck anymore
Camera tool	Pulls a fresh broker JPEG when asked “what do you see?”

Early look-at was a LAN hack: M4 bbox every 2s → short head pulse → snap center when the face dropped. It felt like a tick. That loop was the wrong rate for motor control. Tracking lives next to the camera now.

STT: the mic was fine

Far-field VAD needed real tuning (silence thresholds, no live-transcript race). But the “Hey mama / Big Mile / Chirri” class of failures was not the array.

Before	After
faster-whisper tiny.en, greedy	mlx-audio whisper-large-v3-turbo on M4 Metal
No vocabulary bias	Private household and lab vocabulary prompt
Speculative reopen 3s	1.5s (fewer doubled sentences)

Same room, same distance sometimes transcribed clean long sentences under tiny.en — and sometimes invented fluent wrong English. That is a small model + greedy decode, not a dead mic.

First cutover of large-v3-turbo briefly made him “deaf”: a s2s rename_args bug turned gen_kwargs into nested kwargs={} and every STT call crashed. Fixed the same night. Mic and VAD never stopped hearing — the decoder was throwing.

Reliability

Reviewers (Opus / Sonnet / Opus 5): good homebrew; make power-on boring before more toys. That sprint landed.

Item	Status
Cold boot and watchdog	Passed / live. Robot-off health is soft so an unplugged body does not cause a restart loop.
Realtime protocol and cancellation	Hardened. Stale output and post-cancel token races fail closed.
Honcho writeback	Hardened. Private SQLite outbox, shared lock, strict permissions, retention, reconciliation hold, and content-redacted health.
Local operator status	Current. Human-readable deterministic text is available while JSON remains the automation default.
Provider fallback	Not yet proven end to end. This is the largest remaining appliance-grade reliability gap.
Current integrated verification	708 tests passed, plus focused lint and format gates; the existing framework deprecation warning remains.

Knowledge contract

Tier	Allowed	Not allowed
0 Body	Talk, motion tools, camera describe, standing facts	—
1 Read-only	Lookup / Hermes web on ask big Milo	Side effects
2 Act	—	Not wired from voice (email, deploy, purchases…)

Deep research is still first-person Milo thinking harder — not “I asked another agent.”

Build notes worth keeping

• Stock app kept. It speaks OpenAI Realtime dialect; HF speech-to-speech on the M4 is the local backend. No forked robot UI.
• Camera was a ribbon first, then ownership. After the ribbon reseat, the hard problem was multi-consumer IPC, not pixels.
• Look-at tick = control-loop rate. 0.5 Hz sense over LAN cannot drive smooth neck motion.
• VAD first, then STT model size. Far-field needed silence work; name errors needed a real Whisper, not a toy.
• Memory is layered. Small, curated relationship facts applied last beat a large, noisy memory dump. Household facts had to be boringly reliable.
• Degrade toward less authority and worse voice — never silence or guessed identity.

Assembly (short)

One evening + morning, ~50 steps, complete kit. Stewart-platform neck is the party trick.

Next

• Soak local STT, multi-subject gaze, greeting decisions, and nap/wake through normal household use
• Add fuzzy “Big Milo” STT variants while preserving the meta-mention guard
• Suppress the gaze tracker around emotion playback so motion systems do not fight
• Prove language-model and local-TTS fallback through the actual production launch path
• Still parked: guard mode, code red, speaker recognition, private local voice cloning, full duplex, and act-from-voice