Reading an LLM judge's verdict from its first token overstates position bias by 42 points

What changed, and when

On 4 September 2026, Gnaneswar Villuri, Hashmath Shaik and Alex Doboli posted The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generating (arXiv:2610.00054). The paper targets a specific measurement practice, not a new judge model: when an LLM compares two candidate answers, labels, or pre-labels and is asked to pick one, a common way to score the comparison cheaply is to look at the probability distribution over the model's very first generated token and take whichever of the two verdict tokens (X or Y, A or B, 1 or 2) has higher probability. This costs one forward pass. Letting the model generate a full response and then parsing its stated verdict costs many more tokens, and at the scale an annotation QA pipeline runs this check (every pairwise comparison, repeated with the two candidates swapped to test for position bias), that cost difference adds up.

The paper's test spans 5,798 comparison pairs from two existing benchmarks, JudgeBench (620 pairs) and LLMBar (419 pairs), run through five judge models (three sizes of Qwen3, plus Llama-3.1-8B and Phi-3.5-mini) across different label schemes, for 11,596 total order-presentations (each pair shown twice, in both orders). Eight presentations produced no verdict even after 1,500 tokens and were dropped from both readout methods. The authors call a pair compliant when the model's first generated token is one of the two valid verdict tokens in both the original and the swapped presentation. Everywhere a pair is non-compliant, forcing a first-token read is not reading a verdict the model intended to give, it is reading whatever token the forcing procedure happens to prefer among two options the model was not actually about to say next.

The evidence table

Compliance (the rate of leading with a verdict token) varies sharply by judge, benchmark and label scheme, from 50.6% to 100.0% across the ten conditions the paper reports in full (two further conditions with a gap of 0.000 are mentioned but not tabulated, for space). The Gap column below is the forced-first-token flip rate minus the full-generation flip rate, computed over every pair in that condition, so it is diluted wherever compliance is high and the two readouts agree trivially.

Source: arXiv:2610.00054, Table 1. Two further conditions with a gap of 0.000 are reported in text but omitted from the table shown here, for a total of ten conditions.

The per-condition gap above mixes compliant pairs, where the two readouts trivially agree, with non-compliant ones, where they can disagree. Isolating just the 924 pairs across the three Qwen3 judges that were non-compliant on JudgeBench or LLMBar gives a cleaner estimate of the problem: the forced first-token read flips the verdict on 89.7% of these pairs (829 of 924) when responses are swapped, against 47.5% (439 of 924) for the full-generation read, a paired difference of +0.422 (95% CI +0.365 to +0.467, McNemar p=1.0x10^-99, which the authors note understates significance). Of the 829 pairs that flipped under the forced read, 827 (99.8%) simply returned whichever verdict token corresponded to the response shown first, in both the original and the swapped presentation, regardless of which response that was. Phi-3.5-mini and Llama-3.1-8B could not be included in this pooled analysis: they produced only 18 and 1 non-compliant pairs respectively, too few to analyze on their own.

A second, smaller failure mode: on a 300-pair sample per judge, the first generated token sometimes disagrees with the verdict the model's full response eventually states, even when that first token was itself a valid verdict token. This happened on 15 of 271 pairs for Qwen3-4B (5.5%), 14 of 257 for Phi-3.5-mini (5.4%), 8 of 285 for Qwen3-8B (2.8%), and 0 of 300 for Llama-3.1-8B (0.0%, Wilson 95% CI 0.0% to 1.3%). The rate of this reversal is uncorrelated with how often a judge is compliant in the first place (r=0.01 across the four judges), so it is a distinct failure from the non-compliance gap above, not a restatement of it.

Why forcing a read on an uncommitted model just returns the first option

When a judge model is about to state a clear verdict, its first generated token already carries that verdict: the logits over the full vocabulary put most of the probability mass on the one correct token, and reading that token costs one forward pass and gets the right answer. The failure shows up only when the model is not about to do that, when it instead opens with a hedge, a restated instruction, or the start of a reasoning chain in prose. Forcing a readout in that case means taking the argmax over just the two candidate verdict-token ids and ignoring that neither one was actually likely next. The paper's finding is that this forced choice is not a noisy coin flip between the two options, it is almost perfectly anchored to the position the response was shown in: 827 of the 829 forced-read flips (99.8%) track the first-shown response, not the response's content, meaning the forced readout on a non-compliant pair measures where a response sat in the prompt, not what the judge thought of it.

That is also why the distortion hits the position-bias measurement specifically and barely touches accuracy. Accuracy asks whether the judge eventually agreed with the correct answer; the full generation still gets to happen even when the first-token shortcut is the thing being evaluated for position bias, so a pipeline that actually uses the judge's final stated verdict is unaffected. It is the audit, a check run specifically to measure how sensitive the judge is to response order, that gets corrupted, because that audit by design relies only on the cheap first-token readout to stay cheap.

What this means for teams auditing a model-in-the-loop labeling pipeline

Position-bias audits measured the shortcut, not just the judge. Any annotation or evaluation pipeline that uses an LLM to compare two candidate labels (two model pre-labels, a model pre-label against a human label, two annotator proposals) and checks the comparison for order sensitivity by reading first-token logits has been measuring, in part, how often its judge opens with a verdict token rather than how biased its judgment actually is. The paper's own framing is precise about where this lands: the distortion "misleads whoever audits a judge rather than whoever uses one," because accuracy itself barely moves.

The fix is proportionate to the problem, not a wholesale switch to full generation. The paper's own selective procedure forces a read only where the judge is compliant and generates fully only where it is not, which in its tests meant generating for 1,678 of 11,596 total order-presentations, an 85.5% reduction in generation cost against always generating, with fidelity to the full-generation result at 94.5% to 100%. That is a direct, usable design for anyone running judge-based QA at the volume a labeling pipeline runs it: check compliance first, since it is one forward pass, and only pay for generation on the pairs that need it.

Our judgement: none of the five models tested (three Qwen3 sizes, Llama-3.1-8B, Phi-3.5-mini) nor either benchmark (JudgeBench, LLMBar) is an annotation-platform-specific tool, and the paper does not test Snorkel, Argilla, Label Studio, Scale AI or Surge AI; we are not claiming any of them use this exact shortcut. The connection we are drawing is architectural: any pipeline that wires an LLM-as-judge step into model-in-the-loop labeling, whatever vendor or open-source tool runs it, inherits this specific measurement risk the moment it reads a judge's verdict from logits without checking whether the judge was about to say that verdict anyway. The paper itself is narrow, two benchmarks, five models, and its own authors caution that reordering judges by position-bias rank was unstable under bootstrap (51.9% of resamples disagreed), so treat the per-judge ranking, not the core mechanism, as provisional.

Check it yourself

JudgeBench and LLMBar are both public benchmarks. The comparison that matters is not any single condition's gap, it is the pooled non-compliant-pairs figure, since that isolates the cases where the two readouts can actually disagree.

open https://arxiv.org/abs/2610.00054          # abstract, authors, submission date
open https://arxiv.org/html/2610.00054         # full text: Table 1, Sections 3 to 5

# the two benchmarks used
# JudgeBench: https://github.com/ScalerLab/JudgeBench (620 pairs used here)
# LLMBar: https://github.com/princeton-nlp/LLMBar (419 pairs used here)

python3 - <<'PY'
# recompute the headline comparison directly from the paper's reported numbers
forced, generated = 0.897, 0.475
print("pooled gap, forced minus generated: %+.3f (%.1f points)" % (forced - generated, 100 * (forced - generated)))

n = 924
forced_n, generated_n = 829, 439
print("forced flip rate check: %.3f" % (forced_n / n))
print("generated flip rate check: %.3f" % (generated_n / n))

locked = 827 / 829
print("share of forced flips locked to first-shown position: %.3f" % locked)

always_generate = 11596
selective_generate = 1678
print("generation-cost reduction under the selective fix: %.1f%%" % (100 * (1 - selective_generate / always_generate)))
PY
# pooled gap, forced minus generated: +0.422 (42.2 points)
# forced flip rate check: 0.897
# generated flip rate check: 0.475
# share of forced flips locked to first-shown position: 0.998
# generation-cost reduction under the selective fix: 85.5%

All four figures reproduce from the numbers the paper itself reports; none of this requires rerunning the judge models. Rerunning the actual judges against JudgeBench and LLMBar, with both a forced first-token read and a full-generation read logged for every presentation, is the only way to check the mechanism on a model or benchmark the paper did not test.

What would prove this wrong

The paper's pooled 42.2-point figure rests on 924 non-compliant pairs drawn from only three Qwen3 judges; Phi-3.5-mini and Llama-3.1-8B could not be pooled at all because they were compliant on nearly every pair (producing only 18 and 1 non-compliant pairs respectively). A model family that is reliably compliant everywhere would never show this failure, so the claim is really about model families prone to hedging or reasoning before committing to a verdict token, not about LLM judges in general.

A dated prediction: by 30 June 2027, an independent group will run the same forced-first-token-versus-full-generation comparison on at least one judge model and one benchmark this paper did not test, and will either confirm a double-digit-percentage-point gap on the pairs where that judge is non-compliant, or will find the gap shrinks to a few points once a different model family and benchmark are used. If it consistently shrinks, the generalizable claim here, that this is a property of forced readouts on non-compliant pairs rather than a quirk of Qwen3 on JudgeBench, is wrong.

Sources

1. Villuri, G., Shaik, H., Doboli, A. The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generating. arXiv:2610.00054, 4 September 2026. Table 1, Sections 3 to 5; HTML version.
2. Benchmarks used in the paper: JudgeBench (620 pairs) and LLMBar (419 pairs), as cited in arXiv:2610.00054 Section 3.
3. BLOMEGA. LLM-judge probes against an LLMBar length baseline.
4. BLOMEGA. What an LLM-judge panel is worth in human-annotator terms, on ChaosNLI.
5. BLOMEGA. Data annotation research: the latest.