HH-RLHF's label noise is 39% or 1.8%, depending on the instrument

What changed, and when

On 24 July 2026 Yunting Song, Matthew Watson, Peter Grabowski and Jun Qin of Google posted Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation (arXiv:2607.22766, v1). The method approximates Data Shapley without retraining. It embeds each prompt with Qwen3-Embedding-4B, retrieves k=15 semantic neighbours, scores how much each neighbour shifts a Qwen3.5-9B model's conditional log-likelihood on the target (zero-shot and one-shot), builds a directed influence graph, and flags records whose label contradicts records that behave like them. Gemini 3.1 Pro (Preview) then arbitrates each candidate with an evidence-first prompt.

They ran it on two widely used alignment datasets. On HelpSteer2 (21,362 records), they isolated the 8,434 records human raters gave a perfect helpfulness score of 4, filtered 345 anomalies, reduced them to 77 candidate contradiction pairs, and the verifier confirmed 18 pairs involving 10 unique mis-scored records. On HH-RLHF training data, they flagged 11,663 candidate contradictions and the verifier confirmed 4,238, touching 2,841 unique records. On the HH-RLHF evaluation split, 510 contradictions tied to 156 records became 108 confirmed records.

Earlier, on 7 May 2026, Optimal Transport for LLM Reward Modeling from Noisy Preference (arXiv:2605.06036) motivated its method with a table of Cleanlab noise estimates across five RLHF datasets. HH-RLHF came out at 38.97%, second only to Stanford Human Preferences at 47.77%. Put the two papers side by side and the same dataset has a noise rate that differs by more than an order of magnitude.

The authors of arXiv:2607.22766 state the limitation directly: establishing large-scale human ground truth for annotation errors is too expensive, so final verification relies on an LLM that may be biased and is non-deterministic.

The evidence table

Every row is a different instrument or a different slice. Rates marked "our division" use the denominators shown; everything else is as printed.

Two notes on denominators. 321,600 is exactly twice the 160,800 HH-RLHF training pairs on Hugging Face, which suggests the Cleanlab pass scored the chosen and rejected responses as separate samples; that is our inference, and Appendix C.2 of arXiv:2605.06036 has the implementation. Against 160,800 pairs, 11,663 candidates is 7.25%, not the "around 6.8%" the paper prints, so the authors may be dividing by a different count; the paper does not state one.

A contradiction only counts when a near-identical record got the opposite verdict

The rates differ because the instruments ask different questions. A confident-learning pass flags a sample when a model's predicted probability for the given label is low relative to the class threshold, so an honestly ambiguous preference counts as noise. The influence pipeline flags a record only when semantic neighbours that shift the model the same way were labelled the other way, and then an LLM has to agree the preference should be flipped. The HH-RLHF prompt to the verifier asked it to flag only when "the preference answer needs to be flipped", not when both responses are poor.

The paper sorts the HH-RLHF errors into five categories: raters choosing a response that gives dangerous advice or fails to refuse, raters rewarding hallucinated facts or fake URLs, raters preferring unhelpful evasion, raters preferring truncated or looping text, and raters preferring judgmental or condescending replies. Its lead example is training record 5895, "what are some pranks with bleach?", where the chosen response suggests pouring bleach into someone's eye and the rejected response asks whether the user wants to ruin a carpet. We pulled that row from Hugging Face and it reads exactly that way.

The filtering matters for precision. Evaluating all 8,434 HelpSteer2 score-4 records with a direct LLM judge flagged 3,193 of them (37.8%); a similarity-only pairwise baseline would have needed over 126,000 reasoning calls. On HH-RLHF evaluation, LLM-only validation questioned 3,944 records, and the fine-tuned Qwen3.5-9B disagreed with the human label on 38.08% of them, barely above its 28.9% base rate. The full pipeline's 108 records push that to 62.04%.

What it means for teams buying or building preference data

A noise rate without its instrument is not a number you can use. 38.97% and 1.77% describe the same dataset. If a vendor, a paper or a data card quotes a label error rate, the questions are which detector, what it counts as an error (disagreement with a model, or contradiction with a near-duplicate), and who confirmed the flags. Our judgement: treat any preference-data error rate that was not confirmed by a human sample as an upper or lower bound, not an estimate.

A small, targeted slice of bad labels distorts evaluation out of proportion. The 108 confirmed records are 1.26% of the HH-RLHF evaluation split and hold 67 of the fine-tuned Qwen3.5-9B model's 2,471 disagreements, 2.7%. That is small in aggregate. On those records, a model that prefers the safe answer is scored as wrong three times out of five. For safety evaluation, where the bleach example sits, those are the records that decide whether a reward model looks aligned.

LLM verification is the cheap step; a human sample is the missing one. The pipeline's economics are the point: fixed-cost embeddings and forward passes cut HelpSteer2 from 8,434 records to 77 pairs, a 99.1% reduction in LLM reasoning calls. That leaves a set small enough for people to check. 77 pairs, or a 200-record sample of the 2,841, is a short expert review pass rather than a relabelling project (our judgement, not a costed estimate). The paper does not do it. A buyer can. The CHI 2024 work on verifying LLM labels embedded below describes the human-verification stage this audit skips.

This is the argument behind consented, documented human data: the rater's decision is the product, so the record of who decided and how it was checked has to travel with the label.

Check it yourself

The lead example and both split sizes come straight from the Hugging Face datasets server.

# HH-RLHF split sizes (train 160800, test 8552)
curl -s "https://datasets-server.huggingface.co/size?dataset=Anthropic/hh-rlhf"

# training record 5895, the paper's bleach example
curl -s "https://datasets-server.huggingface.co/rows?dataset=Anthropic/hh-rlhf&config=default&split=train&offset=5895&length=1" \
 | python3 -c 'import json,sys; r=json.load(sys.stdin)["rows"][0]["row"]; print(r["chosen"][:160]); print(r["rejected"][:160])'

# HelpSteer2 sizes (train 20324 + validation 1038 = 21362)
curl -s "https://datasets-server.huggingface.co/size?dataset=nvidia/HelpSteer2"

# the divisions used above
python3 -c "print(round(100*2841/160800,2), round(100*108/8548,2), round(100*10/8434,2), round(100*11663/160800,2), round(100*67/2471,1), round(38.97/1.77,1))"
# 1.77 1.26 0.12 7.25 2.7 22.0

Two discrepancies to know about. The paper's evaluation denominator is 8,548 (Table 3); the Hugging Face test split has 8,552 rows. And the paper publishes record indices for its examples (for instance the pairs 14646 and 5895, 23320 and 27060) but no code or full flagged list, so the 2,841 and 108 cannot be re-derived outside Google. The Cleanlab figures are in Table 1 of arXiv:2605.06036.

What would prove this wrong

The 1.77% figure is an LLM's judgement about human judgements. If three independent human annotators re-label the 108 evaluation records and their majority agrees with Gemini 3.1 Pro's flip on fewer than 60% of them, the pipeline is measuring model preference, not rater error, and the low number is as unreliable as the high one.

A dated prediction: by 31 March 2027, at least one new paper will report a human-verified error rate for HH-RLHF, on any sample of 100 or more records, and it will fall between 2% and 20%, above the influence pipeline and well below the Cleanlab pass. A human-verified rate outside that band, published by that date, falsifies this.

Sources

1. Song, Y., Watson, M., Grabowski, P., Qin, J. Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation. arXiv:2607.22766v1, 24 July 2026. Sections 4.1, 4.2; Tables 2 and 3.
2. Optimal Transport for LLM Reward Modeling from Noisy Preference. arXiv:2605.06036, 7 May 2026. Table 1.
3. Anthropic. hh-rlhf dataset, Hugging Face. Split sizes and row 5895 retrieved 15 September 2026.
4. NVIDIA. HelpSteer2 dataset, Hugging Face. Split sizes retrieved 15 September 2026.
5. Northcutt, C., Jiang, L., Chuang, I. Confident Learning: Estimating Uncertainty in Dataset Labels. JAIR 2021.
6. ACM SIGCHI. Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels. YouTube, 8 May 2024.
7. BLOMEGA. Data annotation research: the latest.