LLM relevance labels separate 751 reranker pairs to the qrels' 403, and 7,080 human grades back them

What changed, and when

Fabian David Schmidt, Donato Crisostomi, Carlos Lassance and Nils Reimers (Cohere) posted Rubric-Calibrated Preferences: Cross-Query Calibration of LLM Judgments via Item Response Theory as arXiv:2609.35739v1 on 28 September 2026. Four Hugging Face datasets under the account fabianschmidt-cohere went up the same day (NanoBEIR, BRIGHT and ViDoRe v3 at 21:24 UTC, TREC-DL at 15:15 UTC), and a matching mteb pull request, #5516 adding float relevance gains, has been open since 22 September.

The problem it attacks is familiar to anyone who has bought or built qrels. Benchmark labels are sparse and binary, so nDCG@10 saturates, floors or ties most rerankers on a large share of queries: 45.6% of NanoBEIR queries and 30.7% of BRIGHT queries for the 14 rerankers tested (Section 1). LLM judges can label every pooled document, but pairwise or listwise scores carry no shared scale across queries, and pointwise grades are too coarse to split near neighbours.

RCP takes both. A listwise Bradley-Terry tournament sets the order within a query, and five yes/no criteria shared by every query set the scale. Item response theory, the model used to put test-takers from different exam forms on one scale, does the linking. The authors then paid for an external check: 46 contractors with at least a BSc in the subject, $35 per hour, graded documents blind between 20 July and 25 August 2026 (Appendix D).

How well do the LLM labels match blind human grades?

All values from arXiv:2609.35739v1 except where marked as replayed. Brackets are 95% intervals, clustered by query in the paper.

The document-level result is the one annotation buyers should read twice. The AUC gap is mostly a sparsity effect: binary qrels give about half of the useful versus not-useful pairs the same label, so they cannot order them at all. And they are wrong in both directions, with 27.8% of unlabelled documents rated useful by annotators and 37.5% of labelled positives rated not useful.

How a rubric turns per-query LLM rankings into one scale

Stage A runs listwise reranking over windows of 10 documents, 216 calls per query on NanoBEIR, with the judge scoring each document on a [-5, +5] logit scale. Only the pairwise soft preferences are kept, and Bradley-Terry scores are fitted to them. Those scores order a query well, but a score of 2.0 on one query means nothing on another.

Stage B shows the judge the same documents in 100 windows per query, 6.7 appearances per document, and asks five yes/no questions: topical relevance, information utility, entity or detail match, direct answer, thorough treatment. Pass rates fall from 22.6% to 1.6% (Figure 3). A two-parameter IRT model gives each criterion one difficulty and one discrimination for every query, and each query one scale factor and one offset. On NanoBEIR the per-query scale spans 0.86 to 2.72 between its 5th and 95th percentiles, so without it a one-logit gap would mean three times more on some queries than others.

The calibrated score never reorders a query (Proposition 3.1). What it changes is the size of the gaps and their comparability, which is exactly what nDCG gains need. 79.3% of pool documents sit below the topical criterion's difficulty and 5.1% above the direct-answer criterion's. We recovered both shares from the released theta column: 79.3%, 15.6% and 5.1% of 97,329 judged documents.

One detail matters for anyone copying the protocol. Count-nDCG, the rubric alone with pass shares as gains, separates about as many pairs as RCP-nDCG (Figure 6a). The rubric supplies most of the added resolution. The tournament earns its cost within a query: three blind LLM judges from other families prefer RCP's ideal top 10 over Count-nDCG's on 80.6% to 98.8% of queries with a preference.

Does the released data reproduce the 1.9x claim? Ours: yes, to the decimal

We downloaded the 13 NanoBEIR qrels files, which carry the upstream score plus the calibrated gain and theta, and the reranker runs in provenance/runs. For each of 649 queries and 14 rerankers we computed nDCG@10 with ties broken by document id (the paper's rule), removed each NanoArguAna query's own argument, and ran a paired t-test at 0.05 on all 91 pairs per dataset. Mean over datasets: 63.5% for RCP-nDCG@10 and 34.1% for qrel-nDCG@10, identical to Section 4.4. Summed, that is 751 against 403 of 1,183 tests. The ratio is 1.86, which the abstract rounds to 1.9. All 14 reranker means also match Figure 5, from zerank-2 at 87.2 to Jina Reranker v3 at 70.0 under RCP-nDCG, and from Qwen3-Reranker-8B at 74.7 to zerank-2 at 66.5 under the qrels.

The paper states the direction per dataset (11 of 13 up, ArguAna and HotpotQA down, Appendix G.2) but prints no per-dataset counts. Two things the counts show that the prose does not. On NanoNFCorpus, binary qrel-nDCG@10 separates zero of 91 pairs: the 14 rerankers sit between 45.2 and 49.0 and the smallest paired-test p-value we found is 0.136. RCP-nDCG separates 52 on the same queries. And the HotpotQA reversal is large, 29 against 41, which fits the paper's own limitation that pointwise gains do not penalise a redundant document on a multi-hop query (Appendix F.3).

The released Stage B file is also larger than the paper's headline count. It holds 648,179 per-window rows with five verdicts each, 3,240,895 raw answers, all with finish reason stop. The paper's 486,750 is 649 queries times 150 documents times 5 criteria, the aggregated triples. Both are consistent; practitioners budgeting a replication should plan for the raw number.

What the 72.4% is made of

The headline human number is conditional on a contest being "decided": a majority verdict exists and exactly one metric names that list. Of 311 contests, 22 had no verdict, which leaves 289, and 185 were decided, 59.5% of the sample. The paper's chance line is sound. We recomputed it from Appendix H.3: with 156 disagreements, 27 contests where only RCP-nDCG names a winner and 8 where only qrel-nDCG does, coin-flip verdicts give (156 + 27) / (2 × 156 + 27 + 8) = 183 / 347 = 52.7%.

From Table 21 we can split the 134 wins, which the paper does not do. Disagreements contribute 109 of 156, so the one-sided contests contribute 25 RCP wins and, since 185 minus 156 is 29, only 4 qrel wins. When qrel-nDCG ties and RCP-nDCG names a winner, the annotators side with it 25 times in 27. And 12 of those 25 come from contests in which neither top five contains any positive qrel: removing "contests that qrel-nDCG cannot score" drops the count from 134/185 to 122/173. That is our derivation, not a figure the authors print.

Our judgement: the fairer headline for a skeptic is the disagreement-only rate, 109 of 156 or 69.9% against a clean 50%, because it holds both metrics to the same bar. It is lower than 72.4 and still far from chance. An unclustered Wilson interval on 134/185, which we computed, is 65.6% to 78.4%, almost identical to the paper's query-clustered 66.0% to 78.9%, so the repeated queries barely inflate the variance. A one-sided binomial test against 52.7% gives p of 3 × 10^-8.

How many human grades does it take to trust an LLM relevance labeller?

Fewer contests than this study used, if the effect is this large. With a true rate of 72.4% against 52.7%, about 20 decided contests would already exclude chance at the 95% level, and 47 give 80% power (our normal-approximation arithmetic). The study's 185 decided contests are sized for smaller effects: detecting 65% with 80% power needs about 126, and 60% about 363. At the observed decided share of 59.5%, 363 decided contests means roughly 610 contests, double what was run.

Budget for triple grading. Single grades were not reliable enough to use alone: Krippendorff alpha 0.571, and two annotators land within one grade step of each other only 77.8% of the time. Panel means of at least three reach ICC(1,k) 0.802. The study used 3.12 grades per document (7,080 over 2,268) and 7.6 documents per review. A single-annotator validation would have measured annotator noise as much as label quality.

What the humans cost is not in the paper. Only the $35 hourly rate is given. As arithmetic under our own assumption of 30 seconds to 2 minutes per document grade on four questions, 7,080 grades is 59 to 236 hours, or about $2,100 to $8,300 before recruiting and quality-control staff. The only priced LLM run is TREC-DL with the smaller Qwen3.6-27B, $3.72 per query at public API rates, and replays at 50.2% and 35.0% of the judgments cost $1.87 and $1.33 per query with Kendall 0.987 and 0.969 to the full leaderboard. The 397B runs on eight B200 GPUs are not costed.

The LLM labels one pool; the humans check a sample. That split is the design lesson. RCP labels every one of the 150 pooled documents per query once and then scores any reranker without new calls. The human study graded 2,268 documents across three suites, 723 of them from NanoBEIR, under 1% of its 97,350 pooled documents, and most of them from contests where the two metrics disagree (221 of 311 contests were drawn as disagreements). For an annotation team this is the shape to copy: dense machine labels over the whole pool, a stratified human panel on the disagreements, three grades per item.

Two cautions we would keep in the contract. The annotators' four questions echo the wording of four rubric criteria by design, and the authors say they did not measure whether the two are equivalent, so the human check partly asks whether the LLM applies the rubric the way people read it. And the three zerank rerankers, trained on LLM preferences, rise from ranks 12 to 14 to ranks 1, 2 and 4 under RCP-nDCG; annotators back zerank in 78.8% of the 66 contests where the metrics split on it, but a pool labelled by one judge family should not be used to train and then grade the same model. For related failure modes see our notes on what a judge panel is worth in humans and on rubrics that get gamed.

Check it yourself

The NanoBEIR replay runs in under a minute on a laptop with pandas, pyarrow and scipy. The snippet below is the core of our replay and prints the paper's two means as is. Our full script also applied the NanoArguAna exclusion list in NanoArguAnaRetrieval/excluded.parquet explicitly and got the same per-dataset counts.

# release status, 29 Sep 2026
curl -s -o /dev/null -w "%{http_code}\n" https://github.com/cohere-ai/rcp-ndcg        # 404 that day
curl -s "https://huggingface.co/api/datasets?author=fabianschmidt-cohere" | grep -o 'rcp-ndcg-[a-z0-9-]*'
#   rcp-ndcg-nanobeir, rcp-ndcg-bright, rcp-ndcg-vidore-v3, rcp-ndcg-trecdl (no human-grade files)

B=https://huggingface.co/datasets/fabianschmidt-cohere/rcp-ndcg-nanobeir/resolve/main
for s in ArguAna ClimateFever DBPedia FEVER FiQA2018 HotpotQA MSMARCO NFCorpus NQ Quora SCIDOCS SciFact Touche2020; do
  n=Nano${s}Retrieval; mkdir -p nb/$n
  curl -sL $B/$n/qrels.parquet -o nb/$n/qrels.parquet            # query-id, corpus-id, score, gain, theta
  curl -sL $B/provenance/runs/$n.parquet -o nb/$n/runs.parquet     # model, is_judge, query-id, corpus-id, score
done
curl -sL $B/provenance/item_params.qwen35.json   # gamma 1.169 1.047 0.940 1.157 0.687, beta -2.572 -2.002 -0.251 0.709 4.116

python3 - <<'PY'
import pandas as pd, numpy as np, glob, itertools
from scipy.stats import ttest_rel
def ndcg(rk, g, ideal, k=10):
    dcg = sum(g.get(d, 0)/np.log2(i+2) for i, d in enumerate(rk[:k]))
    idcg = sum(x/np.log2(i+2) for i, x in enumerate(sorted(ideal, reverse=True)[:k]))
    return dcg/idcg if idcg else 0.0
out = {"rcp": [], "qrel": []}
for d in sorted(glob.glob("nb/Nano*")):
    q = pd.read_parquet(d+"/qrels.parquet"); r = pd.read_parquet(d+"/runs.parquet"); r = r[~r.is_judge]
    G = {k: dict(zip(g["corpus-id"], g.gain.fillna(0))) for k, g in q.groupby("query-id")}
    IG = {k: list(g.gain[g.theta.notna()]) for k, g in q.groupby("query-id")}
    P = {k: dict(zip(g["corpus-id"][g.score > 0], g.score[g.score > 0])) for k, g in q.groupby("query-id")}
    S = {"rcp": {}, "qrel": {}}
    for m, gm in r.groupby("model"):
        for qid, gq in gm.groupby("query-id"):
            rk = list(gq.sort_values(["score", "corpus-id"], ascending=False)["corpus-id"])
            S["rcp"].setdefault(m, {})[qid] = ndcg(rk, G[qid], IG[qid])
            S["qrel"].setdefault(m, {})[qid] = ndcg(rk, P[qid], list(P[qid].values()))
    for key in S:
        M = np.array([[v[x] for x in sorted(v)] for _, v in sorted(S[key].items())])
        out[key].append(sum(ttest_rel(M[a], M[b]).pvalue < .05
                            for a, b in itertools.combinations(range(14), 2)) / 91)
print({k: round(100*np.mean(v), 1) for k, v in out.items()})
PY
# output on 29 Sep 2026: {'rcp': 63.5, 'qrel': 34.1}
# the released runs already drop the 21 NanoArguAna self-arguments (build_report: 336 run rows dropped)

# the 52.7% chance line and our split of 134/185, from Appendix H.3 and Table 21
python3 -c "print((156+27)/(2*156+27+8), 109+25, 156+25+4, 134-122, 185-173)"
# 0.5273775216138329 134 185 12 12

Paper: arxiv.org/abs/2609.35739, Section 4.4 for sensitivity, Table 5 for the study composition, Table 21 for the verdict layer, Appendix C.4 for cost. Fields: gain is the calibrated relevance probability used as nDCG gain; theta is null for the 674 human positives outside the judged pool.

What would prove this wrong

The strongest objection is shared framing. The annotators' questions and the rubric use near-identical wording, so an LLM that applies the rubric literally and annotators who read it literally could agree with each other and both miss what a searcher needs. The NIST check on TREC-DL is the independent one: RCP-nDCG@10 favours NIST's winner on all 14 NIST-decisive pairs in 2019 and all 37 in 2020, but those grades separate few pairs, so it is a weak test of the pairs that matter most.

A dated prediction: once the 7,080 human grades are public, which the paper promises and which had not happened on 29 September 2026, an independent recomputation of the main measure will land inside 66.0% to 78.9%, and the disagreement-only rate will stay above 60%. If a recomputation from the released grades gives a disagreement-only rate below 55% by 31 March 2027, the human validation does not hold and the AUC gap is mostly a sparsity artefact of binary qrels.

A second falsifier concerns judge dependence. The paper's GPT-OSS-120B rerun separates 60.4% of NanoBEIR pairs with a mean per-dataset leaderboard Spearman of 0.959 to the primary judge. If an RCP labelling of the same pools by a model family outside Qwen and GPT-OSS, fitted with the released rubric, puts the zerank rerankers outside the top four on NanoBEIR, the zerank rise is a judge-family preference, not a quality signal.

Sources

1. Schmidt, F. D., Crisostomi, D., Lassance, C., Reimers, N. Rubric-Calibrated Preferences: Cross-Query Calibration of LLM Judgments via Item Response Theory. arXiv:2609.35739v1, 28 September 2026. Sections 1 to 4, Tables 1, 4, 5, 21, Appendices C.4, D, G.2, H.3.
2. fabianschmidt-cohere. rcp-ndcg-nanobeir (CC-BY-4.0), last modified 28 September 2026, fetched 29 September 2026. Qrels with gain and theta, 14 reranker runs, per-window Stage B verdicts, item parameters.
3. fabianschmidt-cohere. rcp-ndcg-bright, rcp-ndcg-vidore-v3, rcp-ndcg-trecdl, 28 September 2026. File lists checked for human-grade data on 29 September 2026.
4. Cohere. github.com/cohere-ai/rcp-ndcg, code and data location named in the paper; HTTP 404 on 29 September 2026, 14:31 UTC.
5. embeddings-benchmark/mteb. Pull request #5516: optional float relevance gains for retrieval (ndcg_float_at_k), opened 22 September 2026, open on 29 September 2026.
6. BLOMEGA. One 32-model judge panel, two answers: worth 4.24 humans, or worth 2.30.
7. BLOMEGA. A generic rubric is gamed 64% of the time.