On LLMBar's 319 adversarial pairs, picking the shorter answer scores 0.820, and a probe on 50 judges reaches 0.846

What changed, and when

On 26 September 2026 Sourabrata Mukherjee and Sunayana Sitaram (Microsoft Research) posted Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict (arXiv:2609.32407v1, cs.AI). The registry covers 64 open-weight models from 0.36B to 405B parameters in 21 families: 50 generative judges, 9 scalar reward models and 5 read-out-only large judges, run on 14 datasets. Forward-pass interventions (patching, ablation, steering) run on 41 of the 50 judges (Section 3).

The central claim is an "internal-output gap": a judge's verdict can be wrong while a linear read-out of its residual stream still recovers the human label. The flagship case is LLMBar (Zeng et al., ICLR 2024), a benchmark built so the superficially better answer is the worse one. The paper frames the gap as conditional and proposes a pre-read-out diagnostic, cf, the cross-validated accuracy of a length-only classifier, which orders the probe's gain across eight benchmarks at Spearman 0.90 (Section 6).

The paper does not hide the length issue. Section 4 names a "prefer-shorter heuristic (0.799)" on LLMBar-adversarial, and Section 6 says the length-only classifier beats the lens probe there (0.80 against 0.758). What the paper does not give is the pair count for LLMBar-adversarial or a per-subset breakdown, and its reproducibility statement promises released code without a URL. Those are the gaps we checked.

How far above a length rule is the probe?

The table puts the paper's readers (means over 50 judges) next to surface rules we computed on the public release. Our rows use character length on the 319 adversarial pairs; one pair has equal lengths and is scored 0.5.

Paper values: arXiv:2609.32407v1. Ours: github.com/princeton-nlp/LLMBar, Dataset/LLMBar, fetched 30 September 2026. 95 percent normal interval on our 0.820: 0.778 to 0.862 (n = 319).

Three numbers carry the finding, and all three are ours. The zero-cost shorter rule at 0.820 is 0.026 below the full probe and 0.134 above the surface-residualized probe (0.820 minus 0.686). A judge that always chose the longer answer would score 0.180, so the 50 judges' 0.456 sits 0.276 above pure length-following and still 0.364 below its inverse. Lexical overlap with the instruction does worse than chance at 0.445, so it is not a second shortcut.

Which LLMBar subsets carry the length signal?

The public release has 419 pairs: Natural 100, and four adversarial subsets totalling 319 (Neighbor 134, GPTInst 92, GPTOut 47, Manual 46). These counts match the LLMBar paper. The Opening LLM Judges paper lists LLMBar as "natural, adversarial" in its dataset sheet (Table 9) but reports every flagship number on "LLMBar-adversarial" without a count. Our judgement: its 0.799 prefer-shorter figure is the 319-pair pool, because that pool gives 0.820 (0.803 with word counts instead of characters) while adding Natural drops the rule to 0.728. A 50-judge subsample or a token-length rule would explain the remaining 0.02 gap; we cannot tell which.

The shortcut is not uniform. Neighbor (0.881), GPTInst (0.870) and Manual (0.815) are strongly length-confounded. GPTOut, where the dispreferred output was generated by GPT-4 to look good while missing the instruction, sits at 0.553 with an interval of 0.411 to 0.695, which covers chance. On Natural the sign flips: the longer answer wins 0.565 of the time.

This matters for reading the probe. A probe trained within LLMBar-adversarial with cross-validation sees a label that is predictable from length in about 82 percent of pairs, so any activation direction that encodes relative length gets most of the way to 0.846. The paper's three controls exist for exactly this reason: residualization (0.686 against a 0.507 shuffled control), a length-balanced subset (probe 0.684, order-averaged verdict 0.482) and the lens-versus-residual capacity check. Those controls are the evidence for content signal, and they put it roughly 0.18 above chance, not 0.35.

What this means if you run LLM judges on preference data

Report the surface baseline next to every judge score. A number like 0.456 means little without 0.180 and 0.820 beside it. Computing pick-the-shorter and pick-the-longer on a pairwise set takes one pass over the JSON, needs no model, and tells you whether the benchmark measures judgment or length tolerance. The same pattern shows up in our note on human and GPT-4 labels picking the longer answer at 71 percent, where the shortcut runs the other way.

Treat the 0.686, not the 0.846, as the probe's headline on confounded sets. The paper does this in its own controls; readers quoting the abstract often will not. On LLMBar the residualized probe is 0.134 below a rule with zero parameters. The practical value of the read-out is on data where length does not already give the answer, and there the paper's own ladder shows the gain shrinking toward zero (JudgeBench +0.02, hh-rlhf +0.01, Table 2a).

Use cf the way the paper suggests, as a stop sign. A cf of 0.80 says the evaluation set rewards a length heuristic. The paper's rule of thumb is that reading internals pays off once cf is at or above about 0.75. Our judgement is that at that level the first job is to fix or rebalance the evaluation set, because any reader fit on its labels, probe or fine-tune, inherits the length prior. Per-subset cf would also catch mixtures like LLMBar, where one subset (GPTOut) is near chance and another (Natural) has the opposite sign.

For preference-label pipelines, stratify by length before trusting a lift. The DPO result (probe labels recover about 90 percent of the verdict-to-gold gap on an LLMBar cell, n = 80, Section 7) runs on the same confounded distribution. A length-balanced version of that experiment is the one that would tell a data team whether probe labels are worth the activation caching. Related context: per-annotator length preferences in MultiPref.

Check it yourself

Everything below runs in under a minute with Python 3 and curl. Fields in each LLMBar item: input, output_1, output_2, label (1 or 2).

B=https://raw.githubusercontent.com/princeton-nlp/LLMBar/main/Dataset/LLMBar
for s in Natural Adversarial/Neighbor Adversarial/GPTInst Adversarial/GPTOut Adversarial/Manual; do
  curl -s -o "$(echo $s | tr / _).json" "$B/$s/dataset.json"; done

python3 - <<'PY'
import json
subs=['Adversarial_Neighbor','Adversarial_GPTInst','Adversarial_GPTOut','Adversarial_Manual']
adv=[]
for s in ['Natural']+subs:
    d=json.load(open(s+'.json')); adv+=d if s!='Natural' else []
    sh=sum(.5 if len(x['output_1'])==len(x['output_2']) else
           ((1 if len(x['output_1'])<len(x['output_2']) else 2)==x['label']) for x in d)
    print(f"{s:22s} n={len(d):3d} shorter={sh/len(d):.3f}")
sh=sum(.5 if len(x['output_1'])==len(x['output_2']) else
       ((1 if len(x['output_1'])<len(x['output_2']) else 2)==x['label']) for x in adv)
print("adversarial pooled n=%d shorter=%.3f longer=%.3f"%(len(adv),sh/len(adv),1-sh/len(adv)))
PY
# Natural                n=100 shorter=0.435
# Adversarial_Neighbor   n=134 shorter=0.881
# Adversarial_GPTInst    n= 92 shorter=0.870
# Adversarial_GPTOut     n= 47 shorter=0.553
# Adversarial_Manual     n= 46 shorter=0.815
# adversarial pooled n=319 shorter=0.820 longer=0.180

# the paper's promised code: no URL in arXiv v1, nothing found on 30 Sep 2026
curl -s "https://api.github.com/search/repositories?q=opening+llm+judges" | grep total_count
curl -s https://api.github.com/users/sourabrata | grep public_repos

Compare the output with the paper's Section 4 (0.456, 0.483, 0.758, 0.846, 0.799) and Table 2a (cf 0.80 for llmbar). Our overlap baseline and cross-validated threshold rule are extensions of the same loop: overlap counts shared lowercase alphanumeric tokens between the instruction and each output, divided by instruction vocabulary size; the CV rule fits a threshold and sign on the log character-length ratio over five folds shuffled with seed 42.

What would prove this wrong

Our reading assumes the paper's LLMBar-adversarial set is the 319 public adversarial pairs. If the released artifacts show a different item set (for example a subsample, or adversarial plus Natural) on which pick-the-shorter scores well below 0.78, the comparison in our first table loses its footing, and the 0.846 probe would stand further from a length rule than we report.

A dated prediction: if the authors publish per-subset probe accuracy by 31 March 2027, the residual-stream probe's gain over the order-averaged verdict will be smallest on GPTOut (length rule 0.553) and largest on Neighbor and GPTInst (0.881 and 0.870). If GPTOut instead shows a gain as large as Neighbor's, the probe is reading something other than length and our concern is overstated. A second falsifier: on a length-balanced rebuild of LLMBar-adversarial with at least 150 pairs, a probe above 0.75 would show the content signal is larger than the paper's 0.684 balanced-subset figure implies.

Sources

1. Mukherjee, S., Sitaram, S. Opening LLM Judges: Recovering Preference Signals Beyond the Final Verdict. arXiv:2609.32407v1, 26 September 2026. Sections 3, 4, 6, 7; Fig. 2; Tables 2a, 9; App. A.12.
2. Zeng, Z., Yu, J., Gao, T., Meng, Y., Goyal, T., Chen, D. Evaluating Large Language Models at Evaluating Instruction Following. ICLR 2024. The LLMBar benchmark.
3. princeton-nlp. LLMBar repository, MIT license, last pushed 8 July 2024; data files under Dataset/LLMBar, fetched 30 September 2026.
4. GitHub REST API. Repository search for "opening llm judges", queried 30 September 2026, 0 results.
5. BLOMEGA. Human and GPT-4 preference labels pick the longer answer at the same rate.
6. BLOMEGA. BA-DPO and per-annotator length bias on MultiPref.