A faithfulness-worded prompt made GPT-5.4 flag 100% of its errors as false positives when judging correct answers as hallucinated

What changed, and when

On 6 October 2026, Jorma Valjakka, Juhani Kivimäki, Juha Mylläri and Jukka K. Nurminen (University of Helsinki) posted The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation (arXiv:2610.08026v1); the paper is accepted for NeurIPS 2026. Open-domain QA hallucination benchmarks typically generate an LLM answer, then apply an automated rule to label it hallucinated or not by comparing it to a dataset's short reference answers. The paper argues this collapses two different criteria into one binary label: reference faithfulness (is everything in the answer entailed by the reference?) versus factual correctness (is the answer free of false or contradicted claims?). The two diverge exactly when an answer is correct but adds a true detail the short reference does not mention, a case a strict faithfulness check will flag as a hallucination.

To measure how often this mismatch actually bites, the authors built a 900-pair human-labeled sample: 300 questions sampled from TriviaQA, HotpotQA and TruthfulQA (100-101 each), each paired with answers from three generator models (Llama-3-8B, Gemma-2-9B, Mistral-7B), with a primary annotator labeling factual correctness under a reference-first protocol and a second annotator independently labeling a 300-pair subset to check reliability (Cohen's kappa 0.808, strong agreement). Hallucination prevalence under the human labels ranged from 38.0% (Gemma-2-9B) to 51.0% (Mistral-7B) by generator, and from 30.3% (TriviaQA) to 72.3% (HotpotQA) by dataset.

Against these human labels, the paper tests lexical metrics (ROUGE-L, BERTScore), a reference-entailment NLI baseline, and seven LLM judges (GPT-5-mini, GPT-5-nano, frontier judges GPT-5.4 and Claude Opus 4.7, and local open-weight judges Gemma-2-9B, Llama-3-8B, Qwen2.5-7B), each run under three prompt variants: p1 (faithfulness-style), p2 (factual-correctness), and p3 (an extended p2).

The evidence table

Cohen's kappa against human labels, bias (automated minus human hallucination rate), and false-positive share of all errors (FP%), for three frontier/near-frontier judges on Mistral-7B's answers, the generator with the highest human-labeled hallucination rate in the sample.

Reproduced from Table 1 of arXiv:2610.08026v1, Section 4.1, judging Mistral-7B generator answers against the 300-pair human-labeled subset for that generator. Bias is positive when the automated labeler over-labels hallucinations relative to humans.

The mechanism: absence is not contradiction

The paper formalizes the split precisely. Let R be the reference answer set and c a claim in a model's answer. A faithfulness labeler flags a hallucination whenever some claim is not entailed by R (written R ⊭ c). A factual-correctness labeler flags one only when a claim directly contradicts R, or is a specific, checkable claim that is simply untrue. The two criteria agree on three cases: a correct bare answer, a correct answer with an added false detail, and a wrong answer that contradicts the reference. They diverge on a fourth: a correct core answer plus a true elaboration the short reference happens not to mention. Faithfulness scores that fourth case as a hallucination; factual correctness does not.

The paper confirms the mechanism with a direct intervention: when elaborative material is manually removed from p1-evaluated answers, the false-positive rate drops from 52.9% to 23.2% (45 answers move from false-positive to true-negative against only 4 moving the other way). Removing the thing the faithfulness criterion penalizes removes most of the false positives, which is strong evidence the elaboration itself, not some other property of the answer, is driving the mislabeling.

What it means for anyone building or buying a hallucination benchmark

The fix is which question you ask, not which model answers it. A controlled factorial test isolates this directly: holding the factual-correctness criterion fixed while adding more prompt structure produced no detectable change in agreement with humans, while changing the criterion (faithfulness to factual-correctness) improved agreement at both levels of structure. Rewriting a judge prompt to be longer or more detailed, without fixing which construct it targets, will not help.

A bigger, newer model is not a safeguard. GPT-5.4, a frontier model, was the single most false-positive-dominant judge in the study under the faithfulness prompt, reaching 97.7-100% FP-dominance across the three generators, while the smaller GPT-5-nano and the other frontier judge, Claude Opus 4.7, were less affected. The authors frame this correctly: which construct an ambiguous criterion defaults to is an evaluation design choice baked into the prompt, not something that scales away with capability.

Non-LLM baselines fail in their own, different direction. Strict reference-entailment NLI was the most faithfulness-biased label source of any tested, 97.7-99.4% false-positive dominance across generators, while ROUGE-L ran the other way, false-negative-prone. A pipeline that swaps an LLM judge for a "simpler, more objective" lexical or NLI metric to avoid prompt-sensitivity is trading one construct-validity problem for a different, equally undocumented one.

Our judgement, specific to anyone licensing or building QA hallucination data for benchmark use: before trusting any automated hallucination label, confirm in writing which construct it operationalizes, faithfulness or correctness, against a small human-labeled holdout, the same audit this paper ran. A label source untested against that distinction is not neutral; based on this paper's numbers, it defaults toward penalizing true, elaborated answers more often than not.

Check it yourself

open https://arxiv.org/abs/2610.08026           # paper, Tables 1-2, Figure 1, Sections 3-5
open https://github.com/jova486/LPHB            # authors' released code and 900-pair dataset
open https://doi.org/10.7910/DVN/PCHISZ          # Harvard Dataverse data release

# what to check without rerunning anything:
# Table 1 (Section 4.1): kappa / bias / FP / FN / FP% per judge, per generator, per prompt
#   GPT-5.4 on Mistral-7B: p1 kappa 0.021 (144 FP, 0 FN, FP% 100.0)
#                          p2 kappa 0.619 (40 FP, 17 FN, FP% 70.2)
# Table 2 (Section 4.2): ROUGE-L / BERTScore / NLI baselines vs. human labels
#   strict NLI FP%: 98.3-99.4% across all three generators
# Section 5.1: elaboration-removal intervention, FP rate 52.9% -> 23.2%
#   (45 FP->TN transitions vs. 4 TN->FP transitions)
# Appendix A: human inter-annotator kappa on 300-pair overlap = 0.808

python3 - <<'PY'
# Table 1: kappa gain from p1 (faithfulness) to p2 (factual correctness), Mistral-7B
judges = {
    "GPT-5.4":         (0.021, 0.619),
    "GPT-5-mini":      (0.194, 0.626),
    "Claude Opus 4.7": (0.693, 0.706),
}
for name, (p1, p2) in judges.items():
    print(f"{name:17s} p1 kappa {p1:.3f}  p2 kappa {p2:.3f}  gain {p2-p1:+.3f}")
PY
# GPT-5.4           p1 kappa 0.021  p2 kappa 0.619  gain +0.598
# GPT-5-mini        p1 kappa 0.194  p2 kappa 0.626  gain +0.432
# Claude Opus 4.7   p1 kappa 0.693  p2 kappa 0.706  gain +0.013

What would prove this wrong

This is an audit of specific judge snapshots (GPT-5-mini, GPT-5-nano, GPT-5.4, Claude Opus 4.7, and three open-weight local models) against one 900-pair human-labeled sample drawn from three specific QA datasets. The authors do not claim faithfulness is always the wrong criterion, only that open-domain QA benchmarks targeting factual correctness need to validate their label source against that specific target.

A dated prediction: by 31 December 2027, an audit of a newly released open-domain QA hallucination benchmark, using the same p1-vs-p2 prompt-ablation method, will again find a faithfulness-style default labeler producing false-positive-dominant errors (FP% above 70%) against a human factual-correctness holdout, because the underlying ambiguity is in how "hallucination" is defined for QA benchmarks generally, not a property specific to the judges tested here. If a new benchmark's default labeling prompt, written without reference to this paper, is independently audited and found balanced between false positives and false negatives, this prediction is wrong.

Sources

1. Valjakka, J., Kivimäki, J., Mylläri, J., Nurminen, J. K. (University of Helsinki). The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation. arXiv:2610.08026v1, 6 October 2026. Accepted, NeurIPS 2026. Tables 1-2; Figure 1; Sections 3-5.
2. BLOMEGA. The 96.5% LLM-judge accuracy is 100% by construction on 429 of 600 items.
3. BLOMEGA. Data annotation research: the latest.