RLHF preference labels flip for 18.6% of prompts once inconsistent raters are filtered

What changed, and when

RLHF preference data annotation is the step where a human rater compares two model responses and picks the one they prefer, or scores a single response on a scale, commonly 0 to 100 for harm or helpfulness. Those judgments train a reward model, and the reward model then scores outputs during reinforcement learning so the policy is pushed toward what the reward model predicts people would prefer. The entire pipeline assumes the rating reflects something stable in the annotator's head.

On 31 January 2026, Bijean Ghafouri, Eun Cheol Choi, Priyanka Dey and Emilio Ferrara, all at the University of Southern California's Information Sciences Institute, posted Position: RLHF May Not Reflect Genuine Preferences (arXiv:2604.03238), revised 29 May 2026 and tagged as an ICML 2026 Spotlight position paper. Rather than build a new dataset, the authors took two that already exist, PRISM (Kirk et al., 2024, 1,500 participants recruited through Prolific across 75 countries, 8,011 conversations with 21 LLMs) and PluriHarms (Li et al., ICLR 2026, 150 prompts, 15,000 ratings, 100 US-based annotators), and asked a narrower question: when the same annotator rates two prompts that mean the same thing, do they give the same answer.

They found pairs of near-identical items using sentence-embedding similarity (all-MiniLM-L6-v2, cosine similarity threshold 0.9 for PRISM's temporally repeated items, 0.7 for PluriHarms's deliberately reframed but equivalent prompt pairs), then flagged a pair inconsistent when the same annotator's ratings differed by 15 or more points on the 0 to 100 scale.

The evidence table

The headline numbers come from two different analyses inside the same paper: a temporal-repeat analysis on PRISM (the same annotator rating the same or a near-identical item at two points in time) and a framing analysis on PluriHarms (the same annotator rating two differently worded but equivalent prompts).

Source: arXiv:2604.03238, Sections 6 and Appendix E, F. "Both" rows report the paper's combined filtering result across the two datasets' harm-rating tasks.

The gap between pair-level and per-annotator rates is the part most summaries miss: a 19.91% pair-level rate on PRISM comes from a median per-annotator inconsistency rate of 56.4%, meaning more than half of annotators flip their own rating on at least one repeated item at the median, not a small tail of unreliable raters. The authors also report that annotators below the median inconsistency ratio give systematically higher harm ratings than those above it (t=5.31, p<.001), and that inconsistency ratio correlates with mean harm rating at r=-0.65 across annotators: the more a rater contradicts themselves, the lower they tend to rate harm on average.

How the same annotator's ratings contradict themselves

The paper's diagnostic runs in five steps. First, it finds candidate pairs: items the same annotator rated at two points in time (PRISM, where ratings recur across a longitudinal panel) or two differently worded prompts the researchers judge equivalent (PluriHarms), confirmed by embedding similarity above a dataset-specific threshold. Second, it computes the absolute gap between the annotator's two ratings and flags the pair inconsistent at |Δ| ≥ 15 points. Third, on a 44-case subset of PRISM's identical-item pairs (score gaps ranging 15 to 90, mean 34.96), the authors hand-classify each case into one of four categories drawn from survey-methodology literature: a non-attitude (the rater had no real opinion and answered anyway), a constructed preference (built on the spot from whatever cues were in front of them, unlikely to generalize), a measurement artifact (the instrument itself, not the rater, is the problem), or a genuine but uncrystallized preference (a real but still-forming view). Fourth, annotators are ranked by their own inconsistency ratio and split at the median. Fifth, the high-inconsistency half is dropped and the majority label and mean rating are recomputed on what remains.

What it means for teams buying or building preference data

A reward model trained on unfiltered labels is partly trained on noise the paper can name. 56.8% of the 44 inspected PRISM gaps look like non-attitudes, answers given with no real opinion behind them, which the authors' own policy table says should be excluded from training because they contain no signal. That is a small, hand-coded subset, not a population estimate, but it gives a mechanism for why filtering changes 18.6% of majority labels: some fraction of the disagreement is not noisy signal, it is no signal.

RLHF vs RLAIF is not resolved by this paper, and that gap matters. The established RLAIF comparison (Lee et al., arXiv:2309.00267, Google, ICML 2024) found AI-generated preference labels reach comparable summarization and dialogue performance to human labels and are equally preferred head to head. Neither that paper nor arXiv:2604.03238 tests the two failure modes against each other directly. Our judgement: replacing inconsistent human raters with an AI judge does not obviously fix the measurement-validity problem this paper describes, it relocates it, from a rater who may have no real opinion to a judge model whose score depends on prompt phrasing and its own training data. Whether that relocation nets out as less noise is an open empirical question neither literature has answered together yet.

Commercial annotation vendors are the delivery mechanism for exactly this kind of data, and their market is shifting. Scale AI and Surge AI are the two largest RLHF data vendors by revenue; Prolific is the platform PRISM itself used to recruit its 1,500 participants across 75 countries. An August 2026 Sacra industry analysis reports Surge displacing Scale on some high-profile RLHF accounts as buyers move toward smaller batches of harder judgment tasks, with Surge reporting roughly 50,000 expert contractors and about $1.2B in 2024 revenue against Scale's reported $1.5B 2024 ARR. None of these figures come from audited filings, and none of the three vendors appears in arXiv:2604.03238's own text: the connection is that whichever vendor collects the preference pairs, the annotator-consistency question this paper raises applies to their pipeline too, and none of the public vendor marketing reviewed for this piece reports a comparable inconsistency diagnostic.

Our judgement: the paper's own limitations section says the framing analysis rests on only 7 unique prompt pairs, and that prompt-pair characterizations were made by one researcher without independent validation. The 44-case PRISM classification is a hand-coded illustration of a mechanism, not a measured population rate, and the authors state they only analyzed cases already showing inconsistency, so they cannot estimate how many non-attitudes or constructed preferences hide among responses that looked consistent. The 18.6% flip rate and 13.2-point shift are the paper's strongest, most replicable numbers; the four-category breakdown is suggestive, not a count anyone should cite as a base rate.

Check it yourself

The underlying datasets are public and the paper's headline ratios are simple fractions anyone can recompute from the reported counts.

open https://arxiv.org/abs/2604.03238          # abstract, authors, submission and revision dates
open https://arxiv.org/html/2604.03238         # full text: Sections 5-6, Appendix E (classification), Appendix G (filtering method)
open https://arxiv.org/abs/2404.16019          # PRISM dataset paper (Kirk et al., 2024)
open https://arxiv.org/abs/2601.08951          # PluriHarms dataset paper (Li et al., ICLR 2026)
open https://huggingface.co/datasets/HannahRoseKirk/prism-alignment   # PRISM data and card

python3 - <<'PY'
# recompute the headline ratios directly from the paper's reported counts
prism_inconsistent, prism_total = 136, 683
pluri_inconsistent, pluri_total = 240, 673
pluri_strong, pluri_total2 = 78, 673

print("PRISM pair-level inconsistency: %.2f%%" % (100 * prism_inconsistent / prism_total))
print("PluriHarms pair-level inconsistency: %.2f%%" % (100 * pluri_inconsistent / pluri_total))
print("PluriHarms strongly inconsistent (delta>=30): %.1f%%" % (100 * pluri_strong / pluri_total2))

non_attitude, constructed, artifact, uncrystallized = 25, 14, 4, 1
n = non_attitude + constructed + artifact + uncrystallized
print("PRISM 44-case classification shares: %.1f%% / %.1f%% / %.1f%% / %.1f%%" % (
    100*non_attitude/n, 100*constructed/n, 100*artifact/n, 100*uncrystallized/n))
PY
# PRISM pair-level inconsistency: 19.91%
# PluriHarms pair-level inconsistency: 35.66%
# PluriHarms strongly inconsistent (delta>=30): 11.6%
# PRISM 44-case classification shares: 56.8% / 31.8% / 9.1% / 2.3%

These match the paper's own reported percentages exactly. The 18.6% majority-label flip rate and the 13.2-point mean-rating shift are reported directly in Section 6 as outcomes of the filtering procedure described in Appendix G; they are not independently recomputable from the counts above without the full annotator-level data, which the paper does not publish inline.

What would prove this wrong

The inconsistency thresholds (|Δ| ≥ 15 points for flagging, embedding-similarity cutoffs of 0.9 and 0.7 for calling two prompts equivalent) are the authors' own choices, and the 44-case and 7-pair classifications are small enough that a different threshold or a different pair of datasets could move the headline numbers substantially. The paper itself does not claim the 18.6% flip rate generalizes beyond PRISM and PluriHarms.

A dated prediction: by 31 December 2027, an independent group applying this same repeated-item or reframed-pair diagnostic to a different RLHF preference dataset (not PRISM or PluriHarms, and not run by any of the four original authors) will report a majority-label flip rate, after filtering high-inconsistency annotators, somewhere in a comparable range, roughly 10% to 30% of prompts. If independent replications instead cluster near zero, meaning filtering out inconsistent raters barely changes the majority label on other datasets, the claim that today's RLHF data generally mixes real preference signal with non-attitudes and constructed preferences would be substantially weakened, whatever this specific pair of datasets shows.

Sources

1. Ghafouri, B., Choi, E.C., Dey, P., Ferrara, E. Position: RLHF May Not Reflect Genuine Preferences. arXiv:2604.03238, posted 31 January 2026, revised 29 May 2026. Sections 5-6, Appendix E-H; HTML version.
2. Kirk, H.R., Bartolo, M., Whitefield, A., Röttger, P., et al. The PRISM Alignment Dataset. arXiv:2404.16019, NeurIPS 2024.
3. Li, J.J., Mire, J., Fleisig, E., Pyatkin, V., Collins, A.G.E., Sap, M., Levine, S. PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harm. arXiv:2601.08951, ICLR 2026.
4. Lee, H., et al. RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arXiv:2309.00267, ICML 2024.
5. Sacra. Surge AI research profile, August 2026 (reported contractor count and revenue figures, not audited filings).
6. BLOMEGA. HH-RLHF's label noise is 39% or 1.8%, depending on the instrument.
7. BLOMEGA. Data annotation research: the latest.