GitHubnow
Spark Structured Streaming
Slack aggregation runs on Spark
Trace any rule back to the decision behind it.
Decision lineage is the trail from a rule in the code back to the trigger, the constraint, the alternative that was ruled out, and the decision someone made. Virgo keeps that trail attached. You see why something is the way it is instead of guessing or asking around.
sources#incidentsPR #1244#platformretro
›
⏎ send · esc to interrupt
The why behind a decision goes cold
Months later, all that is left of a decision is the result and a scatter of stray artifacts across tools. The design doc, the incidents, the change that mattered are still out there, but nothing connects them, so you are left guessing.
Today
Design
ConfluenceFeb 2
Pipeline design
Kafka Streams topology
Retro
Google MeetApr 5
Pipeline retro
Kafka Streams too unreliable to keep
Build
GitHubFeb 24
Kafka Streams v1 — PR #1180
First aggregation topology
Trouble
SlackMar 6
#incidents
Dropped messages; rebalancing freezes
Requirements
Google DocsJan 20
Streaming spec
Near-real-time windowed aggregation
Still unreliable
SlackMar 30
#incidents
RocksDB tuning and late records remain
Fix attempt
GitHubMar 18
Custom windows — PR #1244
Hand-rolled TimeWindows aggregation
Why does this run on Spark, not Kafka Streams?
Virgo reconnects the whole trail
Virgo links those scattered artifacts back into one decision history: the idea, the design, the trouble, the constraint, the pivot, and where it landed. It reconstructs the whole evolution, end to end.
- NeedSlackJan 8
#data-eng
Aggregate Slack events for the datasource
- RequirementsGoogle DocsJan 20
Streaming spec
Near-real-time windowed aggregation
- DesignConfluenceFeb 2
Pipeline design
Kafka Streams topology
- TicketsJiraFeb 9
Aggregation pipeline — DATA-88
Cut into tasks
- BuildGitHubFeb 24
Kafka Streams v1 — PR #1180
First aggregation topology
- TroubleSlackMar 6
#incidents
Dropped messages; rebalancing freezes
- Fix attemptGitHubMar 18
Custom windows — PR #1244
Hand-rolled TimeWindows aggregation
- Still unreliableSlackMar 30
#incidents
RocksDB tuning and late records remain
- RetroGoogle MeetApr 5
Pipeline retro
Kafka Streams too unreliable to keep
- ConstraintSlackApr 9
#platform
Stack is Python; Kafka Streams is JVM-only
- DecisionConfluenceApr 16
Move to Apache Spark
PySpark, less low-level tuning
- TodayGitHubnow
Spark Structured Streaming
Slack aggregation runs on Spark
How decision lineage works
Ask why over MCP and Virgo walks the trail as a graph: every milestone returns the piece it holds, and the agent assembles them into the reason it is shaped the way it is.
›
⏎ send · esc to interrupt
The same lineage pays off for people and agents
Traced once, it serves the engineer weighing a change and the agent that must not undo a settled call.
For engineers
Change a system without re-litigating its past.
See why a decision was made, what it ruled out, and what depends on it before you touch it. You stop re-debating settled calls and rediscovering constraints the hard way.
- What was decided: the rule and who set it.
- Why it was decided: the trigger and the constraint behind it.
- What it ruled out: the alternatives already tried and reverted.
For AI agents
The agent respects prior decisions instead of reopening them.
Hand an agent the decision and its constraints and it works within them, rather than reintroducing an approach the team already reverted. Virgo retrieves the lineage at query time, over MCP, so any agent reasons from the real history.
3xfewer iterations
Roughly 3x fewer iterations in code generation when the prior decision and its constraints are on hand, not rediscovered.