Architecture note · James Meadlock & Milo

Video Ingestion Pipeline: the architecture we decided not to build

The architecture

A clean pipeline starts by admitting that the input is not clean. A public investor-relations page might expose an ordinary MP4, a Brightcove player with expiring HLS URLs, a registration form, or a replay whose terms prohibit retention. Those are different acquisition problems. They should still produce the same downstream research object.

The end-to-end shape. Open the editable SVG.

There are three design decisions hiding in that diagram:

1. Acquisition is an adapter layer. There is no universal “download webcast” call.
2. The transcript and the clock travel together. Dropping segment times at any handoff destroys citation quality.
3. Every source converges on one node contract. Retrieval should not care whether the words came from official captions, local ASR, or a human transcript.

Why acquisition is the hard boundary

The public examples are more varied than they look. TransUnion exposes investor-day videos from a company page.^[10] Manulife uses Brightcove for its presentation videos and also publishes supporting material.^[11] Restaurant Brands International has used Vimeo-backed event playback.^[12] An Exelon replay on Notified asks for identity fields before playback.^[13] Apple offers an earnings replay while explicitly restricting recording, reproduction, and redistribution.^[9]

That is why browser automation is useful but not magical. A browser can establish JavaScript state, load an embedded player, submit an approved registration, and expose the media requests made by the page.^[14] It cannot turn a prohibited use into a permitted one, and it should not be used to bypass DRM, CAPTCHA, or an access control.

“The page loaded” is not the same as “the video was acquired.” Open the editable SVG.

The timestamp contract

For ordinary document RAG, a citation can point to a page. For video, the useful target is a time range. “This came from the earnings call” is provenance; “34:12–35:07” is a citation.

{
  "text": "...",
  "start_s": 2052.4,
  "end_s": 2107.1,
  "source_id": "event_2026_q3",
  "source_url": "https://company.example/investors/event",
  "speaker": "Chief Financial Officer",
  "transcript_kind": "official | captions | asr | human",
  "artifacts": ["deck:p14", "frame:2078.0"],
  "rights_flag": "owner_supplied | licensed | research_only",
  "payload_hash": "sha256:..."
}

That schema does more than support clickable links. It lets the system explain why a quote may be imperfect, deduplicate changing auto-captions, connect a spoken claim to a slide, and exclude material whose retention terms changed. Hash the transcript payload, not merely the video ID; auto-generated captions can change while the source identifier stays the same.

The quality gate is simple and unforgiving: retrieve a claim, open the source at start_s, and verify that the cited span actually supports the answer. A confident answer attached to the wrong moment is worse than an honest empty result.

Transcript first, visuals second

Investor videos tempt us into full multimodal analysis immediately. That is usually backwards. Most investor events already separate their strongest evidence into three artifacts:

• an official transcript for the words;
• a presentation deck for the charts and tables;
• a recording for tone, Q&A, and material omitted from the first two.

Ingest those official artifacts first. Run ASR only when the official transcript is absent, incomplete, or too slow to arrive. Sample visual frames only at scene changes, slide transitions, or transcript windows where a speaker refers to something visible but unsaid. LlamaIndex’s own multimodal video examples extract frames and build a separate image/text retrieval path rather than treating a video as one native document.^[6] Its VideoDB example likewise delegates video indexing and scene handling to a specialized system.^[7]

Where LlamaIndex and LlamaParse fit

LlamaIndex is useful here, but it is not the downloader and it should not own the canonical corpus by default.

LlamaParse currently lists audio and video formats including MP3, MP4, M4A, WAV, WebM, and several others.^[1] Its pricing model meters audio/video by minute, while its documented limitations still matter for production sizing and unsupported cases.^[2][3]

The stock open-source readers are a sharper warning. VideoAudioReader runs Whisper and returns transcription text as a document, but its implementation does not preserve Whisper’s segment-level timing in the returned document.^[4] YoutubeTranscriptReader similarly joins caption text into a document rather than carrying each caption’s start and duration into retrieval metadata.^[5] Those defaults are fine for “search this video.” They are insufficient for “prove this answer at the right second.”

LlamaCloud Index offers a managed ingestion and retrieval path.^[8] For a small private research library, I would keep Supabase authoritative and treat LlamaIndex as a replaceable orchestration layer. If the framework earns its keep, keep it. If a 200-line ingestion worker is clearer and easier to test, use that instead.

A state ledger, not a Kanban OS

Durable workflow state is valuable. A visible task-management product is not required. Each source needs a machine-readable status, timestamps, attempt count, and a structured failure reason:

This buys resumability without inventing seven agent personalities or making a human drag cards around. A worker claims the next eligible record, performs one bounded transition, writes evidence, and exits. A retry limit prevents one broken webcast from consuming the whole queue. Human review appears only where the machine cannot make an honest access, rights, or quality decision.

The user experience remains boring in the best way: search, ask, receive a sourced answer, click the timestamp, inspect the evidence.

The historical pilot — not approved for build

Do not start by promising every webcast platform. Start with five deliberately different sources:

1. YouTube baseline: captions already exist; prove the timestamp contract.
2. Direct public MP4: prove download, checksum, audio extraction, and ASR.
3. Public embedded player: prove browser-assisted Brightcove or Vimeo acquisition.
4. Registration-gated replay: prove session handling and the human-review boundary.
5. Rights-restricted source: prove the pipeline stops instead of becoming “resourceful.”

For each source, ask the same five tests:

• Can it answer a specific spoken claim with a correct time range?
• Can it retrieve a list or multi-sentence explanation without slicing the thought in half?
• Does the clock remain accurate near the end of a long event?
• Does it decline a question whose answer appears only on screen when visuals were not ingested?
• Does it refuse to manufacture an answer absent from the source?

Only after that works should we add speaker diarization, scene extraction, OCR, multimodal reranking, or hosted workflow machinery. Those may be useful. None is more important than a citation that lands on the right second.

Sources

1. LlamaParse supported document types, accessed August 4, 2026.
2. LlamaParse pricing, accessed August 4, 2026.
3. LlamaParse limitations, accessed August 4, 2026.
4. LlamaIndex VideoAudioReader source, accessed August 4, 2026.
5. LlamaIndex YoutubeTranscriptReader source, accessed August 4, 2026.
6. LlamaIndex multimodal RAG for video processing, accessed August 4, 2026.
7. LlamaIndex multimodal VideoDB example, accessed August 4, 2026.
8. LlamaCloud Index V2 getting started, accessed August 4, 2026.
9. Apple Q3 2026 earnings call replay and usage notice, accessed August 4, 2026.
10. TransUnion Investor Day videos, accessed August 4, 2026.
11. Manulife Investor Day 2024, accessed August 4, 2026.
12. Restaurant Brands International events and presentations, accessed August 4, 2026.
13. Exelon Q4 2025 webcast replay, accessed August 4, 2026.
14. Hermes browser automation documentation, accessed August 4, 2026.