LLM-as-a-judge verdict stability under repeated calls

The problem

On the platform I maintain, a browser agent runs multi-step web tasks and an LLM-as-a-judge classifies each run from the agent's closing text, the step trace and a final screenshot. The judge returns one of five outcomes, which also collapse into completed or not completed, and downstream automation acts on that verdict. It runs at temperature 0 with one call per run, so each verdict looks fixed.

When the judge prompt is unchanged, a gate scores the new judge version (the judge is a DSPy program that an optimizer re-compiles on fresh labels each training cycle; the prompt template stays the same and each re-compile is a version) and the current one on a held-out slice of the analyst-labelled runs, one call per run, and promotes the new one when its accuracy is at least the current one's, so a tie promotes. The slice is a few hundred runs, so one changed verdict moves accuracy by about half a point, and recent promotions were decided by margins of a few rows or by a tie.

A flipped verdict is also visible outside the gate: the latest verdict is sent on to a downstream tracking system as the run's status, and it becomes the label attached to the run in the next labelled set.

At margins like that, two things matter and neither had been measured. Temperature 0 on a hosted model does not guarantee the same answer twice, so how often a run's verdict moves between identical calls decides how much of a margin is noise. And only a minority of runs have the closing text, the step trace and a screenshot together, so how much the verdict leans on each piece decides how far the gate's score can be trusted on partial records.

What I varied

One variable: what the judge sees. The runs span more than a hundred different websites and apps, with different layouts and flows. I took 196 runs that have a complete record: 96 recent ones and 100 drawn from the latest labelled dataset, sampled so that the rarer outcomes are not drowned out by the common one. Flip rate needs no label, so the recent runs are not restricted to the labelled set.

A is the full run record: closing text, step trace and screenshot. This is what production sends.
B is A without the agent's closing text.
C is the screenshot only.
D is closing text and step trace, without the screenshot.

The task context is present in every variant. Every run and input got 5 calls, and wherever those disagreed it got 15 more, so a run that was unanimous in 5 calls may still flip with more calls, and the rates below are lower bounds. A cell is one run under one input variant. Measured: how often the verdicts for one run were not unanimous (flip rate), at the completed / not-completed level and at the five-outcome level. A separate check replays the current judge against itself on the whole held-out slice, 20 passes, once with the input exactly as the gate builds it and once with the input production sends (the gate rebuilds the step trace as plain text, while production sends structured steps; more on that below), and records how much its accuracy moves between passes.

Figure 1: Design. Complete runs are frozen and sent to the judge for every input variant. Flip rate needs no label; the self-replay uses the labelled held-out slice.

Result

With the full record, 2 of 196 runs got mixed completed / not-completed verdicts at temperature 0, and 3 of 196 at the five-outcome level, so the noise is small and not zero. Removing the closing text raised it to 7 of 195 (one run returned no verdict under that variant), screenshot only gave 4 of 196, and text without the screenshot gave 8 of 196 (10 at the five-outcome level). At these counts the 95% Wilson intervals for A and D overlap, so the direction is clear and the size is not.

Figure 2: Flip rate by input variant. Share of runs whose verdicts were not unanimous, by input variant (counts: A 2 / 3, B 7 / 9, C 4 / 6, D 8 / 10 of 196, 195, 196, 196 runs).

Run	A: completed verdicts	B	D
R097	16 of 20	5 of 5	19 of 20
R109	5 of 5	4 of 20	19 of 20
R105	5 of 5	5 of 5	11 of 20
R136	5 of 5	5 of 5	5 of 20
R145	5 of 5	15 of 20	5 of 5

Table 1: Verdict counts per run. Five of the 19 runs with at least one mixed cell; each cell shows the number of completed verdicts out of the calls made (5, or 20 where the first five disagreed).

The self-replay is the number that matters for the pipeline. Across the 190 pairs of identical passes, the accuracy of the current judge on the 218 held-out rows differed by 0 or 1 row in 101 pairs, by 3 rows or fewer in 174 pairs, and by 6 rows in the worst pair. The 95th percentile was 4 rows. A promotion margin of that size is inside the noise.

Figure 3: Accuracy difference between two identical passes. Replaying the current judge against itself: share of the 190 pass pairs (20 passes) by how many rows its accuracy differed, over the 218 held-out rows.

The production-shaped input is tighter: 185 of 190 pairs within 3 rows, a worst case of 5, and its accuracy averages 1.2 rows higher than with the gate's own input.

Key takeaways

Temperature 0 leaves a residual flip rate of 1 to 2% of runs on the full record. A single call is a good verdict and not a certain one.
That residual adds up: two identical passes over the held-out slice differ by up to 4 rows in 95% of pairs and by 6 at most, the size of the margins that decide promotions.
Removing either piece raised the flip rate: 7 of 195 without the closing text and 8 of 196 without the screenshot, against 2 of 196 with the full record. That is roughly fourfold, and counts this small do not separate the two. The closing text is also the agent's own account of what it did, so a judge that leans on it can be talked into a wrong verdict, which makes it the part of the input to keep watching.
The gate and production do not send the same input. The gate rebuilds the step trace as plain text lines that the judge skips, while production sends structured steps. The production-shaped input scores about a row higher and varies less.

What changes in the pipeline

The gate's input is aligned with what production sends. The step trace is parsed back to structured steps before the judge sees it, so the gate scores the judge on the record it actually receives.
A candidate is promoted only when it beats the current judge by more than a noise band. The band is the 95th percentile of the judge's own accuracy difference between identical passes, in rows, from the self-replay: 4 rows here. Inside the band the current judge stays, and a tie no longer promotes.
Both judges are compared on the rows both of them scored. A row on which a judge call errors is dropped from both judges' scores, never from one.

Tradeoffs

Fewer promotions. A candidate that is truly better by less than the band waits a cycle for more evidence.
The band does not cover a changed prompt. A new prompt version is promoted without a comparison today, because prompt changes do not go through this gate, so the rule applies only between versions of the same prompt.
The band has to be re-measured. The slice grows as labels accumulate, so the self-replay is repeated now and then. One self-replay (20 passes over 218 rows) is 4,360 judge calls, about $3.70 at list price, and no extra calls in the gate itself.
The served judge still makes a single call. The residual flip rate stays in production, and the band tells you how much of the gate's score it explains.

What I am exploring next

The noise band makes the gate's decisions more trustworthy, and it leaves a different question open: whether a general-purpose LLM is the right component for this call. I am now looking into decision models such as Jev, a decision-only classifier that returns a label with calibrated probabilities and no written rationale, and into a cascade built on it where the decision model answers when it is confident and passes unsure cases on to a stronger judge or a person.