Swapping GPT-4o Mini for GPT-5 Mini drops 48% of its correct relevance labels, and Gemini 3.7 to 3.8 churns like a rerun

What the Queensland study measured, posted 7 October 2026

Chuting Yu, Guido Zuccon and Teerapong Leelanupab (The University of Queensland) posted The Impact of Backbone Evolution on LLM-Based Relevance Assessments as arXiv:2610.09820v1 (cs.IR) on 7 October 2026, 12 pages of main content. The question: if a team validates a judging prompt on one model version and then changes only the model identifier, does the judge stay as good?

They hold two prompts fixed. UMBRELA, the open reproduction of Bing's relevance assessor (Upadhyay et al., arXiv:2406.06519), asks for a 0 to 3 grade directly. EXAM (Farzi and Dietz, arXiv:2402.00309) generates exam questions per query with the family's earliest model, then has each version rate whether a passage answers them. Both run on the TREC Deep Learning 2019 and 2020 passage qrels, 9,260 and 11,386 human-graded pairs, against 22 versions: Gemini Flash 2.5, 3.5, 3.6, 3.7, 3.8; Gemini Flash Lite 2.5, 3.1, 3.5; GPT Mini 4.1, 4o, 5, 5.4; GPT Nano 4.1, 5, 5.4; Qwen 1.5-7B to 3.5-9B; Llama 2-7B, 3-8B, 3.1-8B. Temperature is 0 throughout.

Their findings: no consistent gain from newer versions (Table 1); a "regression", defined as the old version matching the human grade exactly and the new one not, occurs on 2.3% to 28.0% of pairs for adjacent UMBRELA upgrades (Figure 2); EXAM regresses more on minor updates and less on the big GPT-4o Mini to GPT-5 Mini jump; and a Gemini 3.8 Flash meta-evaluator sorts 4,933 UMBRELA and 6,854 EXAM regressions on DL 2019 into failure modes, led by over-weighting one aspect (31.3%) for UMBRELA and an overly strict relevance threshold (33.3%) for EXAM (Table 2).

The code and per-pair outputs are at ielab/judge-backbone-evolution. We checked it on 8 October 2026: public, created 5 October 2026, five commits all made that morning (UTC), 395 tree entries, about 237 MB. It has no licence file at the root (the GitHub API returns license: null and /LICENSE is a 404); only the bundled rubric-grading-workbench/ carries one. That matters if you want to reuse the labels rather than read them.

Do the released labels reproduce the paper, and what else do they show?

They reproduce it exactly, once you use the right parser. The repo contains two score parsers. The one in analyze_all_pairs.py needs a "final score" string; the one in analyze_stability.py and find_regressions.py falls back to the last digit 0 to 3 in the output. With the strict parser 62 of the 70 UMBRELA cells match; with the fallback, all 70 match to 0.1 point. The difference is mostly GPT-4.1 Nano: 5,030 of its 9,260 DL 2019 outputs contain no "final score" string at all (bare answers such as 1 or 1: 2), and the Table 1 values for it (39.44% and 40.74% EM) only come back with the fallback.

All BLOMEGA rows computed on 8 October 2026 from results/*_annotations.jsonl (DL 2019) and results_2020/*_annotations.jsonl (DL 2020) in ielab/judge-backbone-evolution at its 5 October 2026 head. Regression and change rates use all 20,646 pairs unless marked DL 2019.

The chart answers a question the paper leaves open: is 5% churn a lot? A natural null is two judges with the same accuracies whose mistakes are unrelated. Then the chance that the old model is right and the new one wrong is EM(old) times (1 minus EM(new)), about 20% to 37% here. Every one of the 17 upgrades regresses less than that. Same-family Gemini steps keep most of their errors in common: 3.7 to 3.8 Flash regresses 2.3% against 24.7% expected (0.09 of the null), and 3.5 to 3.6 sits at 0.14. The generational jumps the paper flags behave close to unrelated judges: GPT-4o Mini to GPT-5 Mini at 0.77, Qwen2.5-7B to Qwen3-8B at 0.71, Llama 2 to 3 at 0.70. Our judgement: that ratio is the number to report when deciding whether a "new version" is the same judge.

Why the 28% is really 48%, and why the asymmetry is arithmetic

Every upgrade splits the 20,646 pairs into four cells. For GPT-4o Mini to GPT-5 Mini: both exact on 6,252, only the old model exact on 5,776, only the new one on 1,455, neither on 7,163. The paper's code (analyze_all_pairs.py, the line computing reg_rate) divides 5,776 by all pairs, giving 28.0%. Section 4.2 then describes that number as "losing over a quarter (28.0%) of the correct judgments made by its predecessor". The predecessor made 12,028 correct judgments, so the loss is 48.0%. The same reading applies to the Gemini 3.5 Flash Lite sentence ("5.7% of instances correctly assessed by the older model"), which is 10.7% of the older model's correct labels.

The asymmetry claim follows from the same table. Going forward, the regression count is the "only old right" cell; going backward, it is the "only new right" cell. Their difference, divided by the same N, is the number of correct labels the new model has minus the number the old one has, that is EM(new) minus EM(old). So "upgrading Llama 3 to 3.1 costs 6.7% but downgrading costs 21.8%" restates that Llama 3.1-8B scores 15.2 points higher EM than Llama 3-8B (38.57% against 23.40% combined). We checked the identity on all 17 adjacent steps; it holds to floating-point error. The asymmetry carries no information that Table 1 does not already give. The economic point the authors draw from it, that cheaper old models lose much of what the new one got right, is still fair, and it is larger as a share of correct labels (56.6% for Llama 3.1 to 3) than the 21.8% printed.

How much of the Gemini churn would a rerun produce anyway?

Section 4.1 reports a stability check: three repeats of each Gemini Flash version on DL 2019, with exact-match standard deviation under 0.1% and the conclusion that cross-version changes are "consistent characteristics of the evolving backbones". That check is aggregate. The study's own thesis is that a flat aggregate can hide instance-level churn, so we ran the regression metric on the repeats themselves, which are in the repo.

Rerunning one model against itself already produces regressions. Across the six ordered pairs of three runs, Gemini 3.5 Flash regresses 1.10% to 1.35% of pairs, 3.6 Flash 2.76% to 3.07%, 3.7 Flash 1.73% to 1.98%, and 3.8 Flash 1.84% to 2.20%; 2.5 Flash sits at 2.76% to 3.02%. The share of labels that change between two runs is 2.8% to 8.6%. Against that floor, 2.5 to 3.5 (5.84%) and 3.6 to 3.7 (5.71%) are clearly real. 3.5 to 3.6 (3.38%) is barely above the 3.6 rerun range. 3.7 to 3.8 (1.99%) sits inside the 3.8 range. The paper's reading that Gemini "stabilises" as 3.x matures may be correct, but by 3.8 the measurement is at the noise floor, so the data cannot show it.

One caveat we cannot close: the repeats are in a directory called repeats_no_thinking, and the paper says reasoning was kept where supported. For 3.5 to 3.8 the first repeat differs from the main run on 5.3% to 7.6% of labels, close to the run-to-run range, so the configurations look similar. For 2.5 Flash the main run differs from the repeat on 17.7%, so that version's main file was likely produced differently. Separately, the 3.6 Flash repeats have an EM standard deviation of 0.145 points, above the "<0.1%" the paper states.

What should a team running an LLM relevance judge do before a version swap?

Report regressions as a share of the old model's correct labels, with the four-cell table. Dividing by all pairs makes a bad upgrade look half as bad when the old model's accuracy is near 50%, as it is for most judges here. The four counts (both right, only old right, only new right, both wrong) let anyone derive the rest, including the asymmetry, which needs no separate figure.

Measure the rerun floor first. Before attributing churn to a new version, run the current version twice on the same sample. On Gemini Flash, temperature 0 did not make labels deterministic: 2.8% to 8.6% changed between runs. If your judge sits on a hosted API, any version comparison below that floor is not evidence either way, and a qrels file regenerated next month with the same model id will differ by about that much.

Use the independence ratio to decide whether to re-validate. A ratio near 0.1, as with Gemini 3.7 to 3.8, means the new version mostly makes the old version's mistakes and a spot check may do. A ratio above 0.5, as with GPT-4o Mini to GPT-5 Mini (0.77) or Qwen2.5 to Qwen3 (0.71), means you have a different judge and should re-validate against human labels at the same depth as the first time. That decision needs a human-graded sample: 300 pairs stratified over the 0 to 3 grades pins EM to about plus or minus 6 points (95%, at EM near 50%) and fills the four cells; that is our sizing, not the paper's.

Pin the parser with the prompt. In this study, one model (GPT-4.1 Nano) ignored the output format on more than half the DL 2019 pairs, and two parsers in the same repo disagree on 8 of 70 cells. A judge is model plus prompt plus parser. Version all three, and log the unparsed rate per run, since a format change in a new version can look like a relevance change.

Keep the human labels you validated with. Every number above depends on the TREC assessors' grades existing for the same pairs. Teams that validated a judge once and discarded the human sample cannot run any of these checks after an upgrade. Related notes: five LLM runs agree at 0.995 yet miss the construct, and how many human grades it takes to trust an LLM relevance labeller.

Check it yourself

About 300 MB of JSONL, no API calls, a few minutes on a laptop. Each line in results/<model>_annotations.jsonl and results_2020/... has query_id, corpus_id, original_qrel_score (the human grade) and model_output (repeat files use gemini_output). Llama and Qwen live in llama/ and qwen/ subfolders, with a _trec-dl-2020 suffix in results_2020.

R=https://raw.githubusercontent.com/ielab/judge-backbone-evolution/HEAD
for y in results results_2020; do for m in gpt-4o-mini gpt-5-mini; do
  mkdir -p $y; curl -sL $R/$y/${m}_annotations.jsonl -o $y/${m}_annotations.jsonl; done; done
python3 - <<'PY'
import json,re
def score(t):  # the fallback parser that reproduces all 70 Fig. 2 cells
    m=re.search(r'final\s*score:?\s*[*]*([0-3])[*]*',t,re.I)
    if m: return int(m.group(1))
    m=re.findall(r'\b([0-3])\b',t); return int(m[-1]) if m else None
def load(m):
    d={}
    for y in ('results','results_2020'):
        for l in open(f'{y}/{m}_annotations.jsonl'):
            r=json.loads(l); s=score(r['model_output'])
            if s is not None: d[(y,r['query_id'],r['corpus_id'])]=(s,int(r['original_qrel_score']))
    return d
a,b=load('gpt-4o-mini'),load('gpt-5-mini'); k=set(a)&set(b); n=len(k)
ok=lambda d,x: d[x][0]==d[x][1]
both=sum(ok(a,x) and ok(b,x) for x in k); L=sum(ok(a,x) and not ok(b,x) for x in k)
G=sum(not ok(a,x) and ok(b,x) for x in k)
print(n,both,L,G,n-both-L-G)
print('paper rate',round(100*L/n,1),'share of old correct',round(100*L/(both+L),1),'backward',round(100*G/n,1))
pa,pb=(both+L)/n,(both+G)/n; print('independent-errors rate',round(100*pa*(1-pb),1))
PY
# 20646 6252 5776 1455 7163
# paper rate 28.0 share of old correct 48.0 backward 7.0
# independent-errors rate 36.5

For the rerun floor, fetch results/repeats_no_thinking/gemini-3_8-flash_run{1,2,3}_annotations.jsonl and results/gemini-3_7-flash_annotations.jsonl plus results/gemini-3_8-flash_annotations.jsonl, and apply the same counting on DL 2019 only. Repo metadata: curl -s https://api.github.com/repos/ielab/judge-backbone-evolution (fields created_at, license) and /commits. We did not rerun any model ourselves: the hub gateway we use for small replications exposes no model-version choice and fixes temperature at 0.4, which would confound a version comparison, and the repo's three released repeats already give a cleaner noise floor than a new 150-pair sample would. The EXAM side (data/questions-rate-*.jsonl.gz) we did not recompute.

What would prove this wrong

Our reading is that the paper's smallest Gemini regressions are indistinguishable from rerun noise, and that adjacent same-family steps differ mainly in how unrelated their errors are (ratio 0.09 to 0.77), not in whether they regress at all.

Prediction one: if the authors or anyone else run three repeats of Gemini 3.8 Flash with the exact configuration of the main results (reasoning on, same budget) on TREC DL 2019, the run-to-run regression rate will be at least 1.5% of pairs. If it comes in under 1.0% while 3.7 to 3.8 stays near 2.0%, the upgrade effect is real and our noise-floor reading is wrong.

Prediction two, dated: by 30 June 2027, at least one published version-swap study of LLM relevance judges on TREC DL or TREC RAG will report a same-model rerun regression rate next to its cross-version rates. If none does, practice has not absorbed the point, and we will say so in a follow-up note.

Sources

1. Yu, C., Zuccon, G., Leelanupab, T. The Impact of Backbone Evolution on LLM-Based Relevance Assessments. arXiv:2610.09820v1, The University of Queensland, 7 October 2026. Sections 3, 4.1, 4.2, Table 1, Figure 2, Table 2.
2. ielab. judge-backbone-evolution, code, per-pair UMBRELA and EXAM outputs, created 5 October 2026, no root licence. Accessed 8 October 2026.
3. Upadhyay, S., Pradeep, R., Thakur, N., Craswell, N., Lin, J. UMBRELA, the open-source reproduction of the Bing relevance assessor. arXiv:2406.06519, June 2024. Prompt: castorini/umbrela.
4. Farzi, N., Dietz, L. An Exam-based Evaluation Approach Beyond Traditional Relevance Judgments. arXiv:2402.00309, February 2024.
5. Craswell, N., Mitra, B., Yilmaz, E., Campos, D., Voorhees, E. Overview of the TREC 2019 and TREC 2020 Deep Learning Tracks. TREC proceedings, 2020 and 2021.
6. Dietz, L., et al. Principles and Guidelines for the Use of LLM Judges. ICTIR 2025, pp. 218-229.
7. Chen, L., Zaharia, M., Zou, J. How is ChatGPT's behavior changing over time? arXiv:2307.09009, July 2023.
8. BLOMEGA. LLM relevance labels separate 751 reranker pairs to the qrels' 403.
9. BLOMEGA. Five LLM runs agree at 0.995, and the measure correlates 0.029 with what it was built to measure.

FAQ

Does a newer LLM version make a better relevance judge with the same prompt?

Not reliably. arXiv:2610.09820 (Yu, Zuccon and Leelanupab, 7 October 2026) runs the UMBRELA prompt across 22 versions in six families on TREC DL 2019 and 2020. Exact match rose in 12 of 17 adjacent upgrades and fell in 5; GPT-4o Mini to GPT-5 Mini fell from 58.3% to 37.3% on the 20,646 pairs combined, and Qwen2.5-7B to Qwen3-8B fell from 58.9% to 46.1% (BLOMEGA recomputation from the released labels).

How many relevance labels change when an LLM judge is upgraded to a new version?

From 6.3% of pairs (Gemini 3.7 Flash to 3.8 Flash) to 62.8% (Qwen1.5-7B to Qwen2-7B) under UMBRELA, computed by BLOMEGA from the released per-pair outputs. The paper's regression rate counts pairs the old version had exactly right and the new one did not, divided by all pairs; divided by the old version's correct pairs instead, GPT-4o Mini to GPT-5 Mini loses 48.0% of its correct labels, not 28.0%. Rerunning the same Gemini Flash model three times at temperature 0 already changes 2.8% to 8.6% of labels.

How should I test an LLM judge before swapping its model version?

Run the old and new version on the same human-labelled sample, and also run the new version twice. Report the four-cell table (both right, only old right, only new right, both wrong), the share of the old model's correct labels lost, and the label change rate between the two reruns as a noise floor. On Gemini Flash in this study, the 3.7 to 3.8 upgrade regressed 1.99% of TREC DL 2019 pairs while 3.8 rerun against itself regressed 1.84% to 2.20%, so that upgrade's churn could not be told apart from rerun noise.