How We Built MHR 001: Video Production Pipeline V1

V1 is not an “AI makes the whole video” system. It is a reproducible editorial build that combines real recordings, generated artwork, programmatic graphics, synthetic narration, word-level timing, and ordinary video engineering.

The pipeline at a glance

What each tool actually did

Stage	Tool	Job	Status in Draft 1.2
Research and editorial	James + Milo	Test the product, identify the real story, write the script, and keep James' and Milo's ratings independent.	Used
Presenter artwork	xAI Grok Imagine image model	Generate a reference-locked James-and-Milo frame using character references and a sanitized cartoon lab.	Used
Alternate generated opener	xAI Grok Imagine video	Create a ten-second futuristic MHR studio opener with generated audio.	Saved, not inserted
Narration	ElevenLabs	Render Milo's approved narration voice from the final script.	Used
Transcription	Whisper Large V3 Turbo via MLX	Produce word-level timestamps for captions and timeline alignment.	Used locally
Graphics	Python + Pillow	Render product cards, integration flow, setup pain, telemetry, caveats, verdict, ratings, and economics.	Used
Presenter motion	NumPy + Pillow + FFmpeg	Switch between approved open- and closed-mouth states using the speech envelope. No generative motion model is involved.	Used
Real B-roll	iPhone/iPad capture	Show Apple Home manually controlling the M3 Ultra plug.	Used
Edit and mastering	FFmpeg	Normalize every segment, concatenate visuals, mix audio, loudness-normalize, mux subtitles, and create the MP4.	Used

1. Start with evidence, not a prompt

The episode began with a real five-plug deployment, not a product page. We connected the plugs through Zigbee, the MZ1 bridge, Matter, Home Assistant, Prometheus, and Grafana. That work supplied the claims, caveats, photographs, and verdict.

This ordering matters. The script comes after the test. Generated visuals are illustrations of evidence; they are not substitutes for it.

2. Freeze the spoken structure

The script has four explicit audio blocks: the branded jingle, James' recorded introduction, a 350 ms handoff pause, and Milo's narration. The narration is divided into product, integration, deployment, setup, telemetry, control, measurement caveats, verdict, ratings, and value.

V1 stores the visual timing directly in Python:

shots = [
    ("01-cartoon-james-milo", presenter_intro, 18.953, "video"),
    ("02-cartoon-milo",      milo_closeup,    11.700, "video"),
    ("03-product",           product_card,    12.500, "still"),
    ("08-apple-home-control", real_b_roll,     6.000, "video"),
    # ...caveats, verdict, ratings, economics
]

It is crude but honest: each beat names its source, type, and exact duration. The next version should move this manifest into shots.yaml, but we did not pretend that abstraction already existed.

3. Build a locked voice and timing track

Milo's narration was generated with an approved ElevenLabs voice. Both James' recording and Milo's narration were then transcribed locally with MLX Whisper using word timestamps.

Those timestamps serve two purposes:

• Group spoken words into short subtitle cues.
• Align presenter mouth states to the actual audio instead of guessed timing.

The resulting subtitle track stays optional in the MP4. We mux it as English mov_text rather than permanently burning captions into the picture.

4. Mix three visual classes

5. Animate the presenters deterministically

We did not ask a video model to regenerate James and Milo for every line. That would introduce identity drift, hand errors, background changes, and expensive retakes.

Instead, we made minimal mouth-state edits to the approved frame, feather-composited the patches, and rendered a two-state animation. For each 24 fps frame, the script measures a smoothed RMS envelope from the active speaker's audio and chooses either the open or closed state.

speech = smooth_rms(audio, fps=24) > threshold
for mouth_open in speech:
    frame = open_state if mouth_open else closed_state
    ffmpeg.stdin.write(frame)

This is intentionally simple. It is not phoneme-level lip sync. It is stable, repeatable, and keeps the approved character image intact.

6. Normalize before concatenation

Mixed media fails at the seams. Before concatenation, every still and clip is converted to the same delivery geometry and codec profile:

• 1920×1080 canvas
• 24 frames per second
• Square pixels
• H.264 with yuv420p
• Fixed GOP and disabled scene-cut keyframes
• faststart metadata for web playback

scale=1920:1080:force_original_aspect_ratio=decrease,
pad=1920:1080:(ow-iw)/2:(oh-ih)/2,
setsar=1,fps=24

Once the segments share an exact format, FFmpeg's concat demuxer can join them without re-encoding the complete visual timeline.

7. Master audio separately

The visual and audio timelines are built separately. FFmpeg resamples each audio source to 48 kHz, fades and attenuates the opener, concatenates the four blocks, and applies loudness normalization at −16 LUFS integrated, −1.5 dB true peak, and an 11 LU loudness range target.

The mastered WAV is then encoded to 192 kb/s AAC and muxed with the visual track and subtitles.

8. Treat revisions as builds

Revision	What changed	Why it mattered
Fast draft	Real photos, information cards, James' voice, Milo narration, captions.	Proved the complete editorial and mastering path before spending time on motion.
Draft 1.0	Added independent James and Milo rating narration and rating card.	Made the review format explicit and repeatable.
Draft 1.1	Replaced the opening stills with controlled cartoon presenter clips.	Added character presence without generative-video instability.
Draft 1.2	Inserted six seconds of real Apple Home control B-roll inside the relay-control discussion.	Matched the claim with direct visual evidence.

What V1 gets right

• Rebuildability: one script recreates cards, segments, audio, captions, and the final container.
• Source flexibility: a shot can be a still, generated clip, real recording, or deterministic animation.
• Model independence: motion generation is optional; FFmpeg remains the spine.
• Provenance: generated candidates, selected assets, source links, and revisions stay in the episode workspace.
• Reviewability: each revision produces a concrete MP4 that can be inspected frame-by-frame.

What V1 still gets wrong

• The shot manifest is hard-coded in the episode builder instead of living in shots.yaml.
• The cards are useful but visually conservative; too many long holds still feel like a slideshow.
• The mouth system follows amplitude, not phonemes.
• Transitions are mostly hard cuts. There is no reusable EDL layer yet for crossfades, picture-in-picture, or lower thirds.
• Quality control is a set of scripts and review passes, not a single automated acceptance command.

V2 direction

The next useful abstraction is small:

script.md
shots.yaml          # source, in/out, duration, transition, VO beat
assets/
  real/
  generated/
  voice/
build.py             # deterministic assembly
out/
  review.mp4
  captions.srt
  contact-sheet.jpg

Generated video should remain a swappable shot source—not the editor, timeline, or source of truth. That keeps cloud I2V, local models, real footage, and ordinary graphics interchangeable.

Companion post: MHR 001: ThirdReality Smart Plug Gen3. Telemetry implementation: ThirdReality → Home Assistant → Prometheus recipe.