Decision post · James Meadlock & Milo · native Hermes capability

Video understanding without a pipeline

The first plan treated video like a new corpus product: acquire media, transcribe it, build timestamped nodes, index it, deduplicate it, and eventually give it a database. That architecture is defensible if the job is to operate a video library. It is unnecessary if the job is usually, "understand this video and answer my question."

Hermes already has most of the useful primitives. The new design starts there and refuses to create infrastructure until real use proves something is missing.

The routing rule

Short controlled direct-video canaries are green in Milo. The 17-minute inline Tonbi canary failed semantic verification even though transport succeeded. A separate provider-native Files API probe passed with a QA correction, but it is not yet part of Hermes video_analyze; Roxy remains separately gated.

What is proven today

A real YouTube ingestion: Tonbi

We ran the text-first route against Tonbi's Feeding GBrain Your Emails Autonomously with Hermes Agent.[6] The result was a complete 17:06 English transcript with 420 timestamped segments. YouTube reported the captions as auto-generated, so the timestamps are useful evidence while the wording is not treated as verbatim.

The ingestion recovered the workflow cleanly:

• 01:59 — "The script collects, agent judges, brain remembers." Deterministic output belongs in code; judgment belongs in the model.
• 07:16 — the collector pulls Gmail through Google's Workspace CLI, removes bulk/no-reply noise, applies an allowlist, deduplicates by message ID, and builds Gmail links in code.
• 08:16 — Hermes writes typed front matter, exact subjects, sender wiki links, short quotations, original-message links, action items, and a review section instead of a generic prose blob.
• 11:57 — unknown senders are physically reviewed to reduce spam, prompt injection, and knowledge-graph poisoning.
• 13:03 — the demonstrated run produced one allowlisted message, one review candidate, and additional messages dropped as noise.

Native-video follow-up: accepted transport, failed semantics

At James's request, we then made an explicit one-off exception to the provisional direct-video envelope and ran the full locally acquired media rendition through Hermes video_analyze. The file was 20.47 MiB, 640×360, and 17:04.163. That is shorter than YouTube's displayed 17:06 and outside both initial guardrails: under three minutes and under 15 MiB. It was not represented as a short excerpt.

Independent frames showed what the model should have reported:

• “THE BRAIN GETS AN INBOX.” with the actual thesis: “script collects → agent judges → brain remembers.”
• “GOOGLE BUILT A CLI FOR AGENTS.” describing gws, Gmail-only scope, OS-keyring credentials, and a pipeline that never sends, deletes, or labels mail.
• “THE COLLECTOR NEVER GETS CREATIVE.” separating deterministic collection from all GBrain writes through MCP during the agent turn.
• “SAME EMAIL, TWO PAGES.” contrasting a typed, linkable contract with an unstructured prose blob.
• The demonstrated terminal result: filed 1 · updated none · review 1, with one allowlisted message, one review candidate, four noise messages dropped, and the brain growing from 29 to 30 pages.
• “YOUR INBOX IS UNTRUSTED INPUT.” with collector, protocol, and human gates: allowlisted content, known-page updates only, and daily review.

Measured token usage and cost

Hermes's current video_analyze result exposes the completion but not provider usage metadata. To measure rather than guess, we sent the same full file and a constrained prompt through the same native Gemini 3.6 Flash endpoint. Google reported 93,187 video tokens plus 141 text-input tokens, 95 candidate tokens, and 2,400 thinking tokens. That metered call took 21.415 seconds. Its output was truncated and was not used as content evidence.

Those rates come from Google's official Gemini 3.6 Flash standard-tier pricing.[7] Because the Hermes wrapper did not expose usage for its two attempts, their exact charges are unavailable. The same-file measurement puts each attempt at roughly $0.16–$0.18, depending on generated and thinking tokens. Including the two Hermes attempts and the one metered native call, this investigation cost approximately $0.48–$0.52. That total is explicitly an estimate; $0.158704 is the measured reference call.

Files API follow-up: usable after QA

The first full-video experiment used inline bytes because that is what the patched Hermes adapter emits. Google's current guidance points elsewhere for this source: inline data is intended for short, one-off video, while the Files API is recommended when duration is significant or the same media will support multiple prompts.[5][8] We therefore uploaded the same 20.47 MiB, 17:04.163 MP4 through Gemini's resumable Files API, waited for its state to become ACTIVE, referenced its file URI in generateContent, and deleted it immediately after the response. A list-after-delete check found zero retained files.

The result was materially better than the inline runs. Without being given the answers, Gemini identified the real script collects → agent judges → brain remembers architecture; named the major visible slide sequence; described the collector's no-LLM/no-direct-write boundary; distinguished the structured page contract from the blob; recovered the demonstrated 1 allowlisted / 1 review / 4 noise dropped counts; and reported the three visible security gates. Those claims align with the caption track and sampled frames.

The Files API itself is available at no cost; inference tokens are billed normally.[7][8] This run reported no separate thinking-token count. The better route was slightly cheaper than the $0.159 metered inline reference because it produced 1,683 answer tokens without the inline probe's 2,400 thinking tokens. Added to the earlier inline investigation, cumulative measured/estimated spend for all four full-video calls was approximately $0.632–$0.672.

The video result required more than selecting a model name. Hermes emits an OpenAI-shaped video_url data URL. In v0.20.0, the native Gemini adapter translated image_url into Gemini inlineData but silently dropped video_url. We added a regression test, watched it fail, made the adapter treat both media types consistently, and then passed 9 native-adapter tests and 41 combined adapter/vision tests.[4][5]

The negative matrix still matters. Copilot Gemini, Anthropic, xAI, OpenAI Codex, Nous Portal, and the tested local Qwen endpoint did not preserve or accept this payload. Several returned ordinary refusal prose while Hermes reported success:true. That boolean proves a completion arrived, not that the model saw the clip.

James/Milo and Bob/Roxy

Evidence rules

• YouTube captions may support spoken claims and timestamp links. The patched helper preserves API-reported language and generated/manual status; missing provenance remains unknown rather than inferred.
• An official transcript, page, deck, or PDF supports only the granularity it actually contains. A page citation is not a video timestamp.
• video_analyze output is model-observed. It may describe a chart, demo, setting, or action. It is not a verbatim quote, and success:true is rejected when independent frames or captions contradict it.
• If text and visual analysis conflict, show both sources rather than smoothing them into one confident answer.

The YouTube helper correction is complete: requested fallback languages stay separate from observed caption metadata, embedded snippet newlines are normalized, and three regression tests cover provenance and serialization behavior.[1]

When storage becomes justified

A database is not triggered by an arbitrary use count. The signal is repeated retrieval: the same source is queried again, one question spans several sources, or the same transcript would otherwise be fetched for a third time. A small permitted transcript cache may be enough. A structured corpus remains a separate decision.

Independent review

We sent the replacement plan and early failures to Claude Opus 5 through Nous. It returned APPROVE-WITH-CHANGES. The useful constraints survived: text first, honest caption provenance, a bounded direct-video envelope, Bob-hosted egress testing, separate profile activation, and retrieval-driven storage. The short direct-video lane is real in Milo; the Tonbi comparison shows that long inline video remains unsafe while the provider-native Files API is promising only with explicit QA. The separate Roxy gate remains.

Two reviewer claims did not survive verification. The failed automatic route was not running Opus 5, and the suggested local Qwen3-VL endpoint rejected Hermes's video_url payload with HTTP 422. Independent review is evidence, not authority.

What remains

1. Upstream the tested video_url to Gemini inlineData adapter fix so a Hermes update cannot overwrite the local correction.
2. Repeat the text-first canaries in Bob's actual hosted profile and egress.
3. Keep long-form inline video outside the accepted envelope. Evaluate a narrow Hermes Files API integration with repeat canaries and mandatory semantic QA before treating it as a supported route.
4. Keep Roxy's direct-video route off until the hosted build contains the fix and her own credential, privacy, cost, and semantic canary passes.
5. Write one versioned shared operating skill and install reviewed, profile-scoped copies.

Sources

1. Hermes bundled youtube-content skill
2. Hermes built-in tools reference
3. Hermes model and auxiliary-model configuration
4. Hermes video_analyze implementation
5. Google Gemini video-understanding documentation
6. Tonbi's AI Garage — Feeding GBrain Your Emails Autonomously with Hermes Agent
7. Google Gemini 3.6 Flash pricing
8. Google Gemini Files API documentation