Prolific's HUMAINE Audit Finds Pooled Preference Votes Double the Sycophancy Reward

What changed, and when

On 3 February 2026, Prolific researchers Nora Petrova, Andrew Gordon and Enzo Blindow published "Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework" (arXiv:2603.04409), later presented at ICLR 2026. That first HUMAINE run collected multi-turn conversations from 23,404 participants stratified across 22 demographic groups in the US and UK, scored 28 models on five human-centric dimensions with a hierarchical Bayesian Bradley-Terry-Davidson model, and reported that user age was the demographic axis along which a model's rank diverged most.

On 11 June 2026, Prolific's research team published a follow-up post, "State of HUMAINE," on labs.prolific.com. The run behind it had grown to 51 models, 116,536 classified conversations and 50,769 decisive head-to-head votes from 23,707 raters. Instead of asking which model wins for which demographic group, the June post asks what actually predicts how an individual rater judges, vote to vote. That question-level analysis is where the 1% demographic figure and the sycophancy decomposition come from. It is not a reversal of the February paper; the two measure different things. February measured whether model rank shifts by age group, an outcome. June measured what explains an individual rater's judgement style and vote-level biases, a process, and found demographics carry almost none of that signal while unsupervised clustering of rating patterns carries a lot.

The evidence table

Note: the post gives the within-model sycophancy figure once as an odds ratio of 1.12 with a confidence interval, and once elsewhere, without a confidence interval, as 1.13. We report 1.12, the figure given with a confidence interval, and flag the inconsistency rather than silently pick one.

Pooling across models manufactures most of the sycophancy reward

Run a standard audit the way most RLHF pipelines currently do: take every pairwise preference vote across every model in your collection, average them, and estimate how much a more agreeable, confident-sounding answer gets rewarded relative to a plainer, more accurate one. Prolific's pooled estimate across all 51 models in the June run is an odds ratio of 1.33: the more sycophantic answer wins about a third more often than an odds ratio of 1.00, meaning no sycophancy effect at all, would predict.

Split the same votes by source model and estimate the effect only within each model's own pairs, and the ratio falls to 1.12 (95% CI 1.10 to 1.15). Prolific attributes about 60% of the pooled 1.33 to a between-model confound: models that happen to be more sycophantic also tend to be stronger models by other measures, so pooling credits sycophancy with wins the model's actual competence earned. The within-model residual, 1.12, is the number that is actually comparable across models, because it answers "does being more agreeable help a single model win more of its own pairwise comparisons," rather than "do agreeable models also tend to be good models."

Stratified recruiting barely predicts how a rater judges

The February HUMAINE paper recruited 23,404 participants, grown to 23,707 by June, specifically to stratify across 22 demographic groups in the US and UK, on the premise that who is judging changes what wins. The June follow-up tested that premise directly at the level of judgement style rather than final winner, and found almost nothing: age, country, education and ethnicity together explain about 1% of the variance in which of Prolific's roughly 15 rating archetypes a given vote falls into, Cramér's V about 0.05, meaning any demographic group is over- or under-represented in a given style by roughly one percentage point. The taxonomy itself is stable: it reproduces across re-fits and bootstraps about 95% of the time.

What does carry signal is the rater, not their demographics. Clustering the 2,386 raters who logged five or more comparisons produces five recurring evaluator types. Selective Reasoners, who weigh each sub-dimension against the overall pick case by case, are the modal type at about 40%. Decisive Sweepers, who let the overall pick drag every sub-dimension along with it, make up about 27%. Three smaller types split the remaining third, 100% minus 40% minus 27%, each in the 10 to 13% range. A rater's style is sticky: among the 9,681 raters with two or more comparisons, repeating the same archetype on a later vote is about 2.7 times more likely than chance.

What it means for someone doing this work

If a reward model or LLM-judge pipeline estimates a sycophancy or length-bias correction from preference votes pooled across many source models, as most published RLHF pipelines currently do, the correction is likely overstated. Prolific's own numbers put the inflation at close to 2x: an odds ratio of 1.33 pooled versus 1.12 once the between-model confound is removed. Teams building a de-biasing term for a reward model should estimate it within model where the data allows it, or at minimum report both the pooled and within-model figures so a downstream user can tell which one they are getting.

Demographic stratification of an annotator or rater pool is still worth doing for representativeness and defensibility: it lets you claim a result generalizes to a population, not just to whoever showed up. But it is not a substitute for modeling individual rater behavior if the goal is to explain or reduce disagreement. The 1% figure says a well-stratified pool can still be dominated by five or so latent judgement styles that have nothing to do with age, country, education or ethnicity. A team budgeting for annotation quality should spend some of that budget on per-rater calibration or archetype-aware aggregation, not only on demographic quota filling.

The 66% preference-prediction ceiling from content features, next to a single latent factor absorbing 65% of rating variance, says there is a real "taste" component in human preference that off-the-shelf features such as length, hedging language or sentiment do not capture. An LLM-judge trained or prompted only on those surface features will plateau below what a full human panel would give you, by Prolific's own numbers roughly one decision in three.

Our judgement: this is one vendor's internal analysis of its own instrument, not an independent replication, and Prolific sells the rater pool and leaderboard it is describing, so there is an incentive to frame "demographics barely matter, our taxonomy matters" favorably to its own HUMAINE product. The underlying vote-level data has not been released in a form we could re-run the Cramér's V or clustering ourselves, so treat the exact figures, 0.05, 1%, 2.7x, 1.33 versus 1.12, as Prolific-reported rather than independently audited. The sycophancy decomposition is still the most checkable number in the post: it does not require trusting the demographic-variance claim, only the source-model-id field already present in any multi-model preference dataset, and a team with its own pooled cross-model preference data can run the same split-by-model check in an afternoon.

Check it yourself

The post, the original paper and the public leaderboard snapshot are all open. The sycophancy decomposition is also the easiest single claim in this article to test against your own data, since it only needs a model-id column already present in most multi-model preference sets.

# primary source: Prolific's June 2026 follow-up post
curl -s https://labs.prolific.com/posts/state-of-humaine -o state-of-humaine.html
grep -i "odds ratio" state-of-humaine.html

# the original HUMAINE framework paper (3 Feb 2026, ICLR 2026)
curl -s https://arxiv.org/abs/2603.04409

# public leaderboard snapshot (Hugging Face Space)
curl -s https://huggingface.co/spaces/ProlificAI/humaine-leaderboard

# reproduce the sycophancy decomposition on your own pooled preference data:
# 1. group votes by response_model_id (or equivalent source-model field)
# 2. within each group, compute P(more verbose/flattering response wins)
#    -> this is your within-model sycophancy odds ratio
# 3. compute the same statistic on the full pooled set, ignoring model_id
#    -> this is your pooled odds ratio
# 4. compare: if pooled >> within-model, your pipeline carries the same
#    between-model confound Prolific reports (1.33 pooled vs 1.12 within-model)

What would prove this wrong

Prolific describes HUMAINE as a continuously running leaderboard, not a one-off study, with further snapshots planned through 2026. We will check labs.prolific.com and the ProlificAI Hugging Face Space again by 31 January 2027. If that next published update re-runs this decomposition on a larger vote set and finds the between-model share of the pooled sycophancy odds falls outside roughly the 50 to 70% range reported here, or finds Cramér's V between demographics and judgement style above 0.15 once the rater pool exceeds 50,000, the central claim here, that demographics are a weak predictor of judgement style and pooled sycophancy estimates run about twice their true within-model size, would not hold up as stated. An independent replication on a different vendor's multi-rater, multi-model preference pool finding a between-model sycophancy share under 30% would also weigh against the generality of the finding, though it would not contradict Prolific's own numbers on its own data. As of 21 October 2026, neither Scale AI nor Surge AI has published a comparable model-id-level decomposition of pooled preference votes.

Sources

1. Prolific AI Research. "State of HUMAINE." labs.prolific.com, 11 June 2026.
2. Petrova, N., Gordon, A., Blindow, E. "Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework." arXiv:2603.04409, 3 February 2026. Presented at ICLR 2026.
3. Prolific. "Beyond Technical Evals: Why Human-Centered AI Benchmarks Matter." prolific.com/humaine.
4. Hugging Face Space, ProlificAI/humaine-leaderboard, public leaderboard data.
5. BLOMEGA. RLHF preference data distribution mismatch.
6. BLOMEGA. Reward models prefer confident, wrong answers.
7. BLOMEGA. Whose gold: annotator pool composition and leaderboard displacement.