Reference-free evaluation: ranking reasoning systems as a tournament
At roughly 97% retrieval accuracy, standard benchmarks stop telling variants apart. With no gold answers to grade against, we rank systems by playing them off each other and scoring the results with Elo.
Virgo is an enterprise reasoning graph for engineering work. We track research in knowledge-graph retrieval and reasoning while developing new retriever variants. That creates a standing evaluation problem. Recent variants reach roughly 97% retrieval accuracy on standard multi-hop benchmarks. At that level, standard benchmarks stop separating systems.
When benchmarks stop benchmarking
Real user queries carry far more nuance than benchmark questions. Standard evaluation depends on gold answers: metrics like F1 and Exact Match require reference labels, and production environments rarely provide them. Internal query logs contain thousands of useful questions and no reference answers.
Without reference answers, evaluation becomes a comparative problem. We can no longer ask “Is this correct?” in an absolute sense. We can only ask:
Given two answers to the same query, which one is better?
Pairwise comparison is a well-established technique. The challenge is scale, since exhaustive comparisons across many system variants are computationally expensive. So we borrow an idea from chess tournaments: a grandmaster does not need to face every other player in the world to be ranked. Strength emerges from selective matches.
Evaluation as a tournament
We model each question as a match and each retriever variant as a competitor. For every query, each variant generates an answer, and we store all outputs in a database before judging. That keeps evaluation consistent and removes the variability of re-running the same system.
We then run pairwise comparisons. An LLM judge evaluates Variant A against Variant B, B against C, and A against C, assigning a win, loss, or draw to each. A single question therefore produces multiple structured comparisons, and across thousands of questions this yields a large dataset of relative-performance signals. Depending on our compute budget and how fast we need to iterate, we run either full round-robin setups or baseline-versus-all comparisons.
Elo works as a skill proxy
We use the Elo rating system to compress pairwise outcomes into a single score per variant, where a higher Elo corresponds to more consistent head-to-head wins. The difference between two ratings maps directly to an expected win probability: if Variant A wins 75% of its comparisons against Variant B, that gap becomes a stable, interpretable measure of relative strength across the dataset.
We do not treat any individual judgment as perfectly reliable. LLM judges are inherently noisy. Their noise diminishes, however, once it is aggregated across a large number of comparisons.
From Bradley-Terry to Elo
Classic Elo updates ratings iteratively after each match. Instead, we estimate ratings in batch using the Bradley-Terry model, which shares the same underlying structure as Elo. Both assume that stronger systems win more often, and that win probability is set by a latent strength parameter.
The Bradley-Terry model defines win probability as:
The difference in strength is:
Elo expresses the same relationship on a base-10 scale:
Take the case where Variant A wins 75% of its comparisons against Variant B, so p = 0.75:
Variant A is therefore about 191 Elo points stronger than Variant B. Under consistent judging conditions, that estimate is stable across datasets and does not depend on any single question. In practice we estimate win probabilities from a finite set of comparisons: more data improves accuracy, but even moderate sample sizes produce useful rankings without excessive computation.
Performance includes latency
Better answers are only part of the problem. A system that improves quality but significantly increases latency is not viable in production, so we measure latency alongside Elo. Each system becomes a point in a two-dimensional space of quality and latency.
Some systems are strictly dominated because they are worse on both dimensions. We discard them. The rest form a Pareto frontier, where each move trades latency for quality. The frontier shows when a 40-point Elo gain costs a threefold increase in latency, and when a faster system nearly matches a slower one.
In summary
Label-free evaluation across many variants needs relative judgment. We turn pairwise LLM judgments into a tournament, then estimate rankings with Bradley-Terry and read them on an Elo-style scale. Latency measurements add the production constraint. Together, the scores show which systems sit on the quality-cost frontier.
As Virgo evolves, new variants keep entering the tournament, and the rankings update automatically as fresh comparisons are collected.