A sparse autoencoder trained on seven RLHF preference datasets found that on LMArena (Chatbot Arena), a response loses 31 percent of its win rate, controlling for length, whenever it refuses the user's request, and loses another 14 percent for avoiding sexual descriptions. The researchers then flipped the chosen/rejected labels on the 1,000 Arena pairs most driven by that refusal feature and retrained a reward model: its accuracy on RewardBench 2's safety subset rose from 8.9 percent, below the benchmark's 25 percent random baseline, to 46.2 percent, with general-capability accuracy unchanged.
Rajiv Movva and Emma Pierson of UC Berkeley, with Smitha Milli of FAIR at Meta and Sewon Min of UC Berkeley, posted "What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data" (WIMHF) to arXiv as 2510.26202 on 30 October 2025. They revised it to version 2 on 11 April 2026, by which point it had been accepted as an oral presentation at ICLR 2026. The paper's own framing is that a chosen/rejected label in an RLHF dataset is a black box: a reward model can predict it, but nobody can say which attributes of the two responses actually drove the vote.
WIMHF's fix is to train a separate sparse autoencoder (SAE) for each dataset on the embedding difference between the chosen and rejected response in every pair, using a BatchTopK architecture with 32 total features, 4 active per input and a Matryoshka prefix of 8. An LLM then writes a short description of what each feature represents (for example, "refuses the user's request"), and the authors regress each feature's activation against the actual preference label to see which named attributes predict the vote. They ran this on seven datasets: LMArena, Community Alignment, Anthropic's HH-RLHF, PRISM, Reddit (via Stanford Human Preferences), PKU-SafeRLHF and the Tulu 3 preference mixture. Code is released under MIT at github.com/rmovva/wimhf, and the processed data is on Hugging Face.
| Dataset | Named feature | Effect | Direction | Source |
|---|---|---|---|---|
| LMArena (Chatbot Arena) | Response refuses the user's request | −31% win rate (length-controlled) | Dispreferred by voters | Movva et al., arXiv:2510.26202 |
| LMArena (Chatbot Arena) | Response avoids sexual descriptions | −14% win rate | Dispreferred by voters | Movva et al., arXiv:2510.26202 |
| Reddit (via Stanford Human Preferences) | Informal, colloquial language or slang | +7% win rate (10% prevalence) | Preferred | Movva et al., arXiv:2510.26202 |
| Reddit (via Stanford Human Preferences) | Witty joke or one-liner instead of advice | +3% win rate (11% prevalence) | Preferred | Movva et al., arXiv:2510.26202 |
| HH-RLHF (Anthropic) | Same informal/slang tone feature | not reported (magnitude) | Dispreferred, opposite of Reddit | Movva et al., arXiv:2510.26202, Fig. 2 |
| PRISM | Same informal/slang tone feature | not reported (magnitude) | Dispreferred, opposite of Reddit | Movva et al., arXiv:2510.26202, Fig. 2 |
| Community Alignment | Paragraphs versus bulleted lists | τ = 0.42 (most subjective feature), ~18% of annotators favor paragraphs | Splits by annotator | Movva et al., arXiv:2510.26202 |
| All 7 datasets, pooled | SAE's 32 interpretable features vs. a fine-tuned Llama-3.2-3B reward model | AUC 0.672 vs. 0.766 (random = 0.5) | Interpretable features close most of the gap | Movva et al., arXiv:2510.26202 |
| Reward model retrained on relabeled Arena pairs | RewardBench 2 safety subset accuracy | 8.9% → 46.2% (+37.3 points) | Improved after relabeling 1,000 pairs | Movva et al., arXiv:2510.26202; benchmark: Malik et al., arXiv:2506.01937 |
The method works because a reward model and a sparse autoencoder are trying to predict the same thing, a preference label, from the same input, an embedding of the two candidate responses. A standard reward model (the paper fine-tunes Llama-3.2-3B for this) does it with a dense, opaque vector. WIMHF instead forces the explanation through a bottleneck of 32 named features, only 4 of which can fire on any single example. Across the seven datasets pooled, those 32 features reach an AUC of 0.672 at predicting the chosen response, against 0.766 for the fine-tuned black-box reward model and 0.5 for random guessing, meaning the interpretable layer recovers roughly two-thirds of the reward model's edge over chance while naming, in English, exactly what it is keying on.
The authors did not stop at naming the feature. They took the 1,000 LMArena pairs where the refusal feature most strongly predicted the human vote, swapped which response was marked chosen and which was marked rejected, and retrained the Llama-3.2-3B reward model on the corrected set. On RewardBench 2's safety subset, a four-way multiple-choice format where random guessing scores 25 percent, the base reward model trained on the original Arena labels scored 8.9 percent, below chance, meaning it had actively learned to reward the less safe option. The version retrained on the 1,000 relabeled pairs scored 46.2 percent, a gain of 37.3 percentage points, while its accuracy on RewardBench 2's non-safety categories stayed inside the base model's confidence interval. Re-scoring the Arena leaderboard itself with the flipped labels moved Claude-3.5-Sonnet up 112 Elo points past Gemini-1.5-Pro, and shifted 16 of the 30 ranked models by at least 50 Elo points.
LMArena's labels come from anonymous site visitors clicking which of two anonymized model outputs they liked better, with no vetting, no pay and no task instructions beyond "pick the better response." That is a different labor pool from the one a lab buys through Surge AI's credentialed experts, Scale AI's crowdworkers or Prolific's demographically screened panel, where annotators are paid, briefed and sometimes required to follow a written rubric. WIMHF's finding is specific to the unpaid, instruction-free pool: it does not show that a Surge or Prolific-sourced preference set carries the same anti-refusal signal, only that an unaudited preference dataset can carry one without anybody noticing until a reward model trained on it scores below chance on a safety benchmark.
For a team buying or building preference data, the actionable split is between knowing a dataset's label rate and knowing what the label rate is made of. Chatbot Arena has been public and widely cited since 2023, and nobody had named this feature in it before WIMHF because standard quality checks (inter-annotator agreement, spot review, demographic balance) do not ask what trait a majority of voters are rewarding. Running a per-dataset sparse autoencoder on a vendor-labeled set before it enters reward model training costs a dataset embedding pass and an SAE fit, both of which the authors' open-source pipeline runs end to end, and surfaces the same kind of answer: a ranked, named list of what the annotators were actually voting on, rather than an assumed proxy for quality.
The code, data and demo are public, and the central relabeling claim can be reproduced from the released artifacts.
git clone https://github.com/rmovva/wimhf.git
cd wimhf
pip install -e .
# dataset configs used in the paper, e.g. configs/community_align.json
ls configs/
python scripts/run_wimhf.py configs/community_align.json --output-dir outputs/community_alignThe claim rests on LMArena's specific voter pool and on one 1,000-pair relabeling run. It would be undercut if, by 30 April 2027, either of two things happens: a replication that runs WIMHF's released pipeline on a current LMArena export finds the anti-refusal feature's win-rate penalty has fallen below 10 percent (rather than the reported 31 percent) as the platform's voter base or UI has changed, or a WIMHF-style audit of a paid, vendor-sourced RLHF preference set from Surge AI, Scale AI or Prolific, run through the same per-dataset sparse autoencoder with no methodology change, fails to surface any comparably strong anti-refusal or anti-safety feature among its top 5 most predictive features. The second outcome would support treating this as a crowd-voting artifact rather than a general property of RLHF preference data.