Active preference-pair selection costs 20.2x more than random sampling in online DPO, for close to no win-rate gain

What changed, and when

Active selection for preference data has a specific lineage. William Muldrew, Peter Hayes, Mingtian Zhang and David Barber introduced Active Preference Learning for Large Language Models (arXiv:2402.08114, ICML 2024), scoring candidate response pairs by predictive uncertainty before sending them into DPO training, on the premise that uncertain pairs teach the policy more per label than random ones. Branislav Kveton and colleagues extended active learning specifically for DPO in Active Learning for Direct Preference Optimization (arXiv:2503.01076, 2025). Both assumed that strategic selection buys real data efficiency over random sampling.

On 3 April 2026 Giyeong Oh, Junghyun Lee, Jaehyun Park, Youngjae Yu, Wonho Bae and Junhyug Noh posted Random Is Hard to Beat: Active Selection in Online DPO with Modern LLMs (arXiv:2604.02766, v1), accepted as a poster at the ICLR 2026 workshop "I Can't Believe It's Not Better" (OpenReview mgJA1v9j9Z), a workshop track, not the peer-reviewed main conference. They re-ran APL against random sampling on current-generation open models, where the original method had been tested on older, weaker base models.

Their stated hypothesis is that strong pretrained priors in modern LLMs leave little room for post-training data selection to help: on-policy generation from an already-capable base model produces diverse, informative response pairs on its own, a kind of "cheap diversity" that an additional selection step has little left to improve on, consistent with the view from Zhou et al.'s LIMA (NeurIPS 2023) that alignment behaves more like style transfer than like teaching new capability.

The evidence table

Numbers are transcribed from Tables 4, 8 and 9 of arXiv:2604.02766v1.

Across all 15 judge-model-dataset rows we pulled from Tables 8 and 9, random sampling wins or ties 10 times and APL wins 5. The clearest APL wins (rows four and six above pattern) happen on the Skywork-v2-Qwen3 judge specifically, and in both cases the random baseline carries high variance itself, ±0.224 in the last row, which is as large as the gap APL is credited with closing. The clearest random wins (rows one, two, four and five) are not close: random beats APL by 4.6 to 36.8 points depending on the row, and does so with an order of magnitude less wall-clock cost.

Where the 20.2x comes from, and why it buys so little

Random sampling's update loop is: sample a prompt, sample a response pair from the current policy, run one DPO gradient update. APL's loop inserts an extra step between sampling and updating: score every candidate pair in the pool by predictive entropy or uncertainty, then filter down to the highest reward-margin pairs, which means running the reward or judge model over the full candidate pool on every single query-update cycle before training on the filtered subset. That extra forward pass over the full pool, not any algorithmic complexity in the filtering itself, is what turns 8.95 seconds into 180.93.

The paper's account of why that expense does not pay off is that modern base models are already strong enough to generate a usefully diverse, informative set of response pairs on their own when sampled on-policy. Older, weaker base models (the ones APL and similar methods were originally validated on) may have needed a selection step to avoid wasting updates on uninformative pairs; a sufficiently capable model's own output distribution already supplies that informativeness for free, which is the "cheap diversity" the authors describe. Scoring on top of that adds cost without adding much the random draw was not already providing.

What it means for anyone building a DPO data pipeline

Benchmark against random before adopting any active-selection method, on your own models. The original APL and later active-DPO work validated the method on different, generally weaker base models. On four current open models here, random sampling won or tied in 10 of 15 measured settings while costing a twentieth of the compute. A result from 2024 does not automatically transfer to 2026-generation base models.

Read win-rate claims against the baseline's own variance, not just its mean. The clearest cases where APL "won" in this paper's tables came against a random baseline with standard deviation as large as the gap being credited to the method (±0.224 to ±0.258 on some UltraFeedback rows). A method that beats a noisy baseline by less than that baseline's own noise has not clearly beaten it.

Treat this as one method, one workshop paper, not a closed case. APL is the only active-selection baseline tested here; no diversity-based or hybrid method is in the comparison, and the venue is an ICLR workshop poster, not a peer-reviewed main-track result. The authors themselves list broader method coverage as future work. The number worth remembering is the cost ratio, 20.2x, which holds regardless of which selection method eventually proves itself; any active-selection pipeline needs to clear that bar to be worth running.

Check it yourself

The cost ratio is a one-line check against Table 4 of arXiv:2604.02766v1:

python3 -c "print(round(180.93/8.95,1))"
# 20.2

HH-RLHF and UltraFeedback are both public preference datasets on Hugging Face, and the four policy models (Llama-3.2-3B, Qwen3-1.7B, Gemma-2B, Qwen2.5-7B) are public weights, so the experiment's inputs are independently reproducible even though the authors have not released a code repository for this paper specifically. The full win-rate tables are in the HTML version at arxiv.org/html/2604.02766v1, Tables 7, 8 and 9.

What would prove this wrong

This is a single workshop-track paper testing one active-selection method. If a peer-reviewed paper, by this team or another, tests APL or a comparable active-selection method against random on a majority of settings across current-generation models and finds a consistent win-rate improvement of at least 3 points while keeping the compute overhead under 5x random's cost, by 31 December 2027, that would undercut "random is hard to beat" as a general claim about post-training data selection for modern LLMs, rather than a property of this specific method and this specific set of models.

Sources

1. Oh, G., Lee, J., Park, J., Yu, Y., Bae, W., Noh, J. Random Is Hard to Beat: Active Selection in Online DPO with Modern LLMs. arXiv:2604.02766v1, 3 April 2026; HTML version; OpenReview, ICLR 2026 workshop.
2. Muldrew, W., Hayes, P., Zhang, M., Barber, D. Active Preference Learning for Large Language Models. arXiv:2402.08114, ICML 2024.
3. Kveton, B., Li, X., McAuley, J., Rossi, R., Shang, J., Wu, J., Yu, T. Active Learning for Direct Preference Optimization. arXiv:2503.01076, 2025.
4. Zhou, C. et al. LIMA: Less Is More for Alignment. arXiv:2305.11206, NeurIPS 2023.
5. BLOMEGA. Active learning vs the random baseline in object detection and Data annotation research: the latest.