The "anchor bias" AUC in the entity alignment judge paper is scored against the shown label, and a perfect judge would get 0.40

What arXiv:2610.09554 claims, as of 7 October 2026

Vaibhava Lakshmi Ravideshik (University of Michigan) and Mayank Kejriwal (USC Information Sciences Institute) posted Reliability of LLM Judges for Evaluating Entity Alignment as v1 in cs.AI on 7 October 2026. It asks whether an LLM can replace expert annotators when checking entity alignment (EA) output.

The setup crosses three judges (GPT-4o-mini, GPT-4o, Claude Opus 4.7, via OpenRouter at temperature 0), three datasets (OpenEA DBpedia-Wikidata 15K "DW", DBpedia-YAGO 15K "DY", and a new MeSH-SNOMED CT 15K biomedical set "BIO"), and four EA systems (EasyEA, its embedding-only ablation, NeuSymEA and its no-refine ablation). Prompts I and II show the system decision and ask how well justified it is, on 0 to 10; Prompt III hides the decision and asks the model to predict alignment itself. The paper reports 153,000 calls, 238.8M tokens and $577.17 of API spend; the Appendix C sub-totals add up exactly.

The headline: with the label visible, J-ROC-AUC falls to 0.12 to 0.87 (0.12 to 0.53 outside DY), which the authors call anchor bias; hiding the label recovers 0.93 to 1.00. Counterfactual label flips move scores by more than 2 points in 53% to 99% of pairs (FSR), and the strongest judge flips most, the "frontier model paradox". A two-author blinded check on 102 pairs reaches Cohen's kappa 0.902. Their advice to practitioners is to withhold system decisions from LLM judges entirely.

The code is at vaibhavalakshmiravideshik/llm-as-a-judge-entity-alignment: MIT, created 17 May 2026, 2 commits, 21 tree entries (prompts, metrics, perturbations, four runner scripts, three configs). There are no judge scores, EA system outputs or logs in it, although Section 4 says "we release full intermediate logs".

Does J-ROC-AUC measure whether the judge spots wrong decisions?

No. Section 4.2 defines J-ROC-AUC under Prompts I and II as P(score of a correct decision > score of an incorrect one). The released compute_jroc_auc sets y_true = (label == "MATCH"), where label is the decision shown to the judge. The paper's own tables confirm the code, not the definition.

Floors use precision ≥ rP/(rP + 1 - P), with P the share of correct decisions and r the reported recall. Paper numbers from arXiv:2610.09554v1. Rerun details in the check section.

The precision rows are the cleanest test because they need no access to the authors' data. Under the sampling the paper describes, 250 MATCH decisions are drawn from EasyEA's predictions (80.6% right on DW) and 250 NON-MATCH decisions are random cross-graph pairs (essentially all right), so 90.3% of decisions are correct. A judge that accepts 96.4% of correct decisions can then not have precision below 0.900. Table 3 prints 0.491, which is the precision you get when the positive class is "shown as MATCH" at a 50/50 split. Every Prompt I/II precision at t=5 in Tables 3, 4 and 6, and at t* in Table 7, sits between 0.487 and 0.514.

The ideal-validator rows need no LLM. Score correct decisions 10 and wrong ones 0. NON MATCH decisions on random pairs all score 10; MATCH decisions score 10 with probability h. No MATCH outscores a NON MATCH and ties count half, so the coded AUC is h/2: 0.403 on DW, 0.474 on DY, 0.259 on BIO. The Score Gap the paper reads as the "behavioural signature of anchor bias" is -10(1-h), negative for any system that makes mistakes. The DW gaps in Table 13 (-1.36 to +0.23) are smaller in magnitude than the ideal judge's -1.94.

Why a correct judge reads as "inverted" under this sampling

Prompts I and II ask whether a decision is justified, so the decision is the object under test. A random pair shown as NON MATCH is the easiest call in the study; a system MATCH is right about four times in five. A careful judge should give NON MATCH decisions higher justification scores on average. The paper reads exactly that ordering as inversion.

The flip metrics follow the same logic. FSR counts pairs whose score moves by more than 2 points when the shown label is flipped with evidence held fixed. For a validator, flipping a correct decision makes it wrong, so FSR should approach 100%. LFR, as coded, counts a MATCH-to-NON MATCH flip as "following" when the score drops, and a NON MATCH-to-MATCH flip as following when it rises. On a balanced set the ideal judge drops on both, so its LFR is 50%. Claude Opus 4.7's 99% FSR and 50% LFR on DW (Table 2) are the ideal judge's profile. Our judgement: the "frontier model paradox" (stronger judges more label-sensitive) is better read as stronger judges being better validators.

What happens when you rerun it with the label shown and hidden?

We built 120 pairs from OpenEA v1.1 D_Y_15K_V1, 30 per cell: a true pair shown as MATCH (correct), a true pair shown as NON MATCH (wrong), a non-matching pair shown as NON MATCH (correct), a non-matching pair shown as MATCH (wrong). Half of the non-matching pairs are random, half are nearest-name neighbours (top-3 by token-set ratio, excluding the gold partner). Each pair carries up to five attributes and four relations per side. The judge was Claude Sonnet 4.5 through the BLOMEGA LLM gateway, temperature 0.4, one call per pair per condition: a minimal "score the system's decision 0 to 10" prompt modelled on Prompt II, a label-free "same entity, 0 to 10" prompt modelled on Prompt III, and the Prompt II variant again with every label flipped. 360 calls, all parsed after retries.

Mean scores with the label shown: 9.10 for true pairs shown as MATCH, 2.20 for true pairs shown as NON MATCH, 9.93 for non-matches shown as NON MATCH, 0.07 for non-matches shown as MATCH. The judge rejected the wrong NON MATCH decisions on true pairs, which is the opposite of rubber-stamping the visible label. Accuracy at a score threshold of 6 was 97.5% with the label shown and 100% with it hidden.

With the repo's own formulas, the flip gave MAS 8.45, FSR 95.8% and LFR 48.3%. AUC against correctness was 0.998 on the original labels and 0.998 on the flipped ones: the judge tracked correctness both times. By the paper's thresholds this run would be filed as severe anchor bias. Scoring the label-hidden predictions as a validator (score s for a MATCH decision, 10 - s for NON MATCH) also gives 1.000, so on these pairs hiding the label buys nothing.

Limits: one judge, one dataset, temperature 0.4 fixed by the gateway, and easy negatives (label-hidden scores averaged 0.2 on nearest-name negatives, 0.0 on random ones). It tests the metric, not judge behaviour on hard cases.

Do near-identical names explain the DY collapse under Prompt III?

The paper says label-free prediction fails on DY EasyEA (J-ROC-AUC 0.468 to 0.527) because "near-identical entity names provide no discriminative signal". With random negatives, the reverse holds. In OpenEA v1.1 D_Y_15K_V1, all 15,000 gold pairs have identical names once the DBpedia URI is percent-decoded (14,988 before decoding). Name equality alone separates 250 gold pairs from 250 random pairs at AUC 1.000 in all 50 draws we ran, and still reaches 0.970 against nearest-name negatives.

So the DY EasyEA negatives cannot have been random pairs as Section 4 states. The released src/data_loader.py builds them three different ways. For full EasyEA, a NON MATCH row is the source entity paired with the system's wrong prediction. For EasyEA-EmbOnly, a NON MATCH row is the source entity paired with its true partner, labelled NON MATCH because the embedding missed it. run_ea.py falls back to EmbOnly when final_results.jsonl is missing. For NeuSymEA, a NON MATCH row is the second-ranked candidate whenever the top candidate is right. None of the three is the uniform random sampling with rejection described in Section 4 and Appendix I.

The EmbOnly construction fits the DY numbers closely. If every DY pair is a true match and half are shown as NON MATCH, an ideal validator scores a Score Gap of +10 and an ideal predictor a gap of 0. Table 13 reports, for Claude on DY EasyEA-EmbOnly, +9.34 under Prompt II and -0.09 under Prompt III. That is our hypothesis, not a confirmed finding: without the released sample files it cannot be settled. If it holds, DY EasyEA is where the label-shown judges were most right (J-ROC-AUC up to 0.866) and the "collapse" under Prompt III is a predictor correctly saying "match" to 500 true pairs.

On DW the trivial baselines do not explain the near-ceiling Prompt III results: name similarity reaches 0.540 and shared literal values 0.697, against 0.981 to 0.996 for the judges on NeuSymEA. One more detail affects DW. Under the released name extractor, which looks for prefLabel, label or name in the predicate, 0 of the 15,000 Wikidata gold entities get a name (their labels sit under skos:altLabel), so each would be shown as a bare Q-id. Figure 1 shows a Wikidata entity with a natural-language name, so either another extractor was used or the figure is illustrative.

What else in the release does not match the text?

Human evaluation reused scores that do not exist. Appendix C says Experiment 4 "required no additional API calls, as it reused judge scores already collected in Experiments 2-3". Experiment 4's 52 negatives span lexical-neighbour, TF-IDF nearest-neighbour and high-confidence false-positive sampling (Appendix I), while Experiments 2 and 3 used only random negatives by the paper's account. Judge scores for those hard negatives cannot have come from Experiments 2 and 3.

Where the 0.93 to 1.00 comes from. The abstract places "near-ceiling" label-free recovery on distinctive-name datasets. The 1.000 values are NeuSymEA on DY (Table 7), the dataset the paper calls near-identical in name. DW EasyEA and EasyEA-EmbOnly Prompt III AUC cells are printed as dashes in Tables 3 and 4, although the README cites "0.931-0.996 on DW and BIO".

Licence of the biomedical release. The paper says SNOMED CT-derived triples and gold pairs go only to UMLS licence holders. On 8 October 2026 the Hugging Face dataset reported gated: false and licence tag other, and a HEAD request for the SNOMED-derived attr_triples_2 (175,750,112 bytes) returned HTTP 200 without login. We did not download it. The card defers compliance to users; it does not enforce the condition the paper describes.

What this means if you use LLMs to check labels

Score the judge against correctness of the decision, not the class. When an LLM audits annotator or model output, the target variable is "was this label right". Build the evaluation set so correct and wrong decisions occur in both classes; our four-cell design (30 per cell) is the smallest version. An AUC computed against the label class can be driven anywhere by how you sample.

Do not count flip sensitivity as bias in a validation task. If the prompt asks whether a decision is justified, flipping the decision should flip the score. Label anchoring shows up as scores that follow the shown label against the evidence: high scores for wrong decisions. Measure that directly as the mean score on wrong decisions (2.2 and 0.07 out of 10 in our rerun) or the false-accept rate at your operating threshold.

Use hard negatives, and report a trivial baseline. Random cross-graph pairs are so easy that a string match scores 1.000 on DY. Nearest-name candidates, the system's own runners-up and its high-confidence errors are where judges and annotators earn their cost. If a no-LLM baseline matches your judge, the judge is not the thing being measured.

Label-hidden prediction is still a reasonable protocol. Our judgement: it keeps the judge's errors independent of the system's. On our pairs it matched label-shown validation (AUC 1.000 against 0.998); the paper's data do not show it is necessary.

Ask for per-pair outputs before adopting a judge protocol from a paper. Here the repo has no scores, so the metric questions cannot be settled from the release. The per-item score file is the minimum audit artefact, the same standard we apply to judge panels against human raters.

Check it yourself

Everything below ran in about 15 minutes on a laptop, plus 360 LLM calls for the rerun. Files checked on 8 October 2026.

# 1. The repo: file list, commits, licence, and the metric's positive class
R=vaibhavalakshmiravideshik/llm-as-a-judge-entity-alignment
curl -s https://api.github.com/repos/$R | grep -E '"(created_at|pushed_at|spdx_id)"'
curl -s "https://api.github.com/repos/$R/commits?per_page=100" | grep -c '"sha"'   # 2 commits listed
curl -s "https://api.github.com/repos/$R/git/trees/HEAD?recursive=1" | grep -c '"path"'  # 21 entries
curl -s https://raw.githubusercontent.com/$R/main/src/metrics.py | grep -n 'y_true'
#   y_true = (paired[label_col] == "MATCH").astype(int)
curl -s https://raw.githubusercontent.com/$R/main/src/data_loader.py | grep -n '"label":'

# 2. Precision floor if J-Prec were scored against correctness (Table 3, DW EasyEA, 4o-mini, P-II, t=5)
python3 -c "h=.806;P=(250*h+250)/500;r=.964;print(round(r*P/(r*P+1-P),3))"   # 0.9, paper prints 0.491

# 3. OpenEA v1.1 (497,646,801-byte zip on Dropbox); only DW/DY members are needed
#    https://www.dropbox.com/s/nzjxbam47f9yk3d/OpenEA_dataset_v1.1.zip?dl=1
#    fields: ent_links = KG1 uri TAB KG2 uri; attr_triples_* = subject TAB predicate TAB literal
python3 -c "
import urllib.parse as u
L=[l.rstrip('\n').split('\t') for l in open('OpenEA_dataset_v1.1/D_Y_15K_V1/ent_links')]
print(sum(u.unquote(a.split('/')[-1])==b for a,b in L), len(L))"   # 15000 15000

# 4. Licence state of the biomedical release (HEAD only)
curl -s https://huggingface.co/api/datasets/vaibhavalakshmiravideshik/mesh-snomed-entity-alignment-15k | grep -o '"gated":[a-z]*'
curl -sIL https://huggingface.co/datasets/vaibhavalakshmiravideshik/mesh-snomed-entity-alignment-15k/resolve/main/attr_triples_2 | grep -iE '^HTTP|content-length'

Baselines: rapidfuzz token_set_ratio on names, and shared 4+ digit tokens and years across attribute literals; 250 gold vs 250 uniform non-gold pairs per draw, 50 seeds, scikit-learn roc_auc_score. Rerun: seed 2026, cells by index modulo 4, prompts capped at 2,000 characters, the repo's MAS, FSR and LFR formulas. The paper-protocol resample draws 30 correct MATCH, 7 wrong MATCH (rate 1 - 0.806) and 30 correct NON MATCH scores, 200 times.

What would prove this wrong

Our reading is that the label-shown "collapse" is mostly the metric, not the judges. It is testable as soon as per-pair scores exist.

If the authors release the Experiment 2 score files, recompute AUC with y_true = (decision == ground truth). Our prediction: for DW EasyEA under Prompts I and II, at least 4 of the 6 judge-prompt cells will score above 0.75 against correctness. If most cells stay below 0.5 against correctness, the judges really do favour wrong decisions and the paper's conclusion stands despite the metric. We will check any release or revised version posted by 31 March 2027 and update this note.

A second, cheaper test: the DY EasyEA judge_input_n500.csv. If its NON MATCH rows are gold pairs (identical names on both sides), our explanation of the DY Prompt III result holds; if they are random pairs with different names, it is wrong and the DY result needs another account.

Sources

1. Ravideshik, V. L. and Kejriwal, M. Reliability of LLM Judges for Evaluating Entity Alignment. arXiv:2610.09554v1, 7 October 2026. Sections 4, 4.2, 5.2, 5.3, Tables 2, 3, 5, 6, 7, 13, Appendices C and I.
2. Ravideshik, V. L. llm-as-a-judge-entity-alignment (MIT), created 17 May 2026, last push 17 May 2026. Accessed 8 October 2026.
3. Ravideshik, V. L. and Kejriwal, M. MeSH-SNOMED Entity Alignment 15K, Hugging Face, last modified 2 May 2026. Accessed 8 October 2026.
4. Sun, Z., Zhang, Q., Hu, W., et al. A Benchmarking Study of Embedding-based Entity Alignment for Knowledge Graphs (OpenEA). arXiv:2003.07743, 2020. Code at the OpenEA repository; dataset v1.1 zip on Dropbox (link in the check section); v2.0 on figshare 19258760 uses anonymised entity ids.
5. Cheng, J. et al. EasyEA, Findings of ACL 2025; Chen, S. et al. NeuSymEA, Advances in NeurIPS vol. 38 (cited as 2026). As cited in arXiv:2610.09554.
6. BLOMEGA. How many humans is a judge panel worth.

FAQ

What does J-ROC-AUC measure in the entity alignment judge paper?

In the released code the positive class is every pair whose displayed decision is MATCH, not every pair whose decision is correct. Under Prompts I and II an ideal judge then scores h/2, where h is the share of MATCH decisions that are right: 0.403 for EasyEA on DW and 0.259 on BIO. Table 3's precision of 0.491 at recall 0.964 would be impossible against correctness, which forces precision of at least 0.900.

Do LLM judges really rubber-stamp a visible entity alignment label?

Not in our 120-pair DY rerun. Claude Sonnet 4.5 with the label shown scored 0.998 AUC against correctness and gave wrong NON MATCH decisions on true pairs a mean of 2.2 out of 10. The paper's metric on the same scores reads 0.345, or 0.235 resampled to the paper's protocol. Flipping labels moved 95.8% of scores by more than 2 points, as a correct validator should.

Is the paper's code and data released, and under what licence?

The GitHub repo is MIT, created 17 May 2026, with 2 commits and 21 files, but no judge outputs, EA outputs or logs. The MeSH-SNOMED 15K set is on Hugging Face, ungated, licence tag "other"; its SNOMED-derived attr_triples_2 answered an anonymous request with HTTP 200 on 8 October 2026, while the paper limits SNOMED-derived content to UMLS licence holders.