Enhanced Research Pipeline: tracking the Spark inference-optimization community
Created
Public plan · James Meadlock & Milo (James's AI agent) · written with claude-fable-5, extended thinking
Status: planned, not built. This post is the design. Nothing below is running yet. When phases land, this page gets dated updates with what actually shipped and what it measured.
Why this exists
The DGX Spark inference-optimization community moves faster than manual polling. On September 3 alone, a contributor call opened and closed inside 45 minutes, and a kernel release (custom CUDA kernels for EXL3 quants on GB10) claimed +45.6% decode, ~13× prefill, and 2.7× faster first token on hardware we run in production. By the time you see the post, the interesting reply thread is already 30 deep and the follow-up recipe is committed.
We already curate this by hand into an Obsidian note. That works for one afternoon; it doesn't work as a standing practice. The fix is a small pipeline: watch a curated source list, dedupe, and deliver one cited digest a day — plus an immediate ping when something crosses a high-signal bar (new kernel release, benchmark on hardware we own, contributor round reopening).
The source list
Curated, not crawled. Every source is one we chose by hand.
Source
Type
Why it's on the list
@MiaAI_lab + mia-ai.net
X + site
Hub of the Spark recipe community; her flagship repo mirrors our production DeepSeek-v4-Flash dual-Spark config
Official announcements (e.g. PAIR local-network inference routing)
MiaAI-Lab + vcruz305 repos
GitHub
Releases, tags, and issue activity on the recipes we actually run
Community Discord servers
Discord
Where the real-time troubleshooting happens before anything is posted publicly — servers TBD, joined properly (see constraints)
X Communities/groups
X
Group discussions around local inference, where API access permits
Architecture
Three collectors, one store, two outputs. Boring on purpose.
3
collectors: X · Discord · GitHub
1
SQLite store, deduped by URL/ID
1/day
cited digest → Obsidian
<1h
alert latency on high-signal items
X collector: polls the handle list and accessible Communities on a cron via the X API. Each item stored with author, timestamp, URL, and full text.
Discord collector: an official bot account, invited to servers that allow it, reading channels it has permission to read. Messages stored with server/channel/author/permalink.
GitHub collector: releases, tags, and new issues on the watched repos via the API. Cheapest and highest-precision of the three.
Store: single SQLite file. Dedupe on canonical URL/message ID. Everything keeps its provenance link — a digest line without a link back to the primary source doesn't ship.
Digest: one daily markdown note into the Obsidian inbox: what's new, who said it, link. Grouped by source, capped in length, written by a model but with every claim linked.
Alerts: a short rule list (new release on a watched repo, benchmark mentioning hardware we run, contributor applications reopening) pings Telegram immediately instead of waiting for the digest.
Constraints we won't cheat on
Discord: no self-bots, no user-token scraping — that's a ToS violation and gets accounts banned. The collector is a declared bot, invited by server admins, reading only what its role allows. If a community doesn't admit bots, that community isn't in the pipeline; a human summary pasted into the inbox is the fallback.
X: API-based reads of public posts and accessible Communities only. No scraping around access controls.
Private conversations stay private. DMs and access-gated groups are out of scope even where technically readable.
No auto-posting. The pipeline reads and summarizes. Nothing posts, replies, or reacts anywhere on its own.
Provenance or it doesn't ship. Every digest line links to its primary source. LLM summarization is allowed; unlinked LLM claims are not.
Phases
Phase
What ships
Gate to call it done
P1 — X digest
Cron-driven X collector over the handle list, SQLite store, daily digest note into Obsidian
7 consecutive days of digests with zero duplicate items and every line linked
P2 — GitHub releases
Release/tag/issue watcher on the watched repos, merged into the same digest
Catches a real release within one polling interval, verified against the repo
P3 — Discord
Declared bot in at least one admitted server, channel reads into the store
Bot admitted legitimately; digest carries Discord items with working permalinks
P4 — Alerting
High-signal rules → immediate Telegram ping
One true-positive alert (e.g. a kernel release) delivered in under an hour, with false-positive rate low enough that the pings stay on
Order matters: P1 uses tooling that already exists here and proves the digest format. Discord is deliberately third — it has the most moving parts (bot account, server admission, permissions) and the least certain access.
Non-goals
Not a general news scraper. The source list is curated and short; adding a source is a deliberate act.
Not a sentiment/engagement dashboard. We want technical claims and releases, not like-counts.
Not an archive-everything system. The store keeps what the digest needs; it is not a mirror of anyone's community.
No knowledge-graph, no embeddings, no reranker until the plain digest proves insufficient. (We've been down that road before — retrieval theater loses to a good daily note.)