Human and GPT-4 preference labels pick the longer answer at the same rate: 71.1% vs 70.9%

What did the paper claim, and when?

On 22 September 2026 David Tsoi and Esra Dönmez (University of Stuttgart) posted Which Objectives Need a Dial? (arXiv:2609.26929v1). They train Multi-Objective DPO (MODPO) models that trade helpfulness against a second objective through a weight w, and ask whether you can tell before training if one model can improve both objectives or must trade them off.

Two measurements are tested. The preference-pair conflict rate is the share of response pairs where the two objectives prefer different responses. The SFT reference-model score correlation is the Pearson correlation of ArmoRM scores for both objectives on 400 reference-model generations. Seven pairs are tested: four from HelpSteer (human scores 0 to 4, 10,459 prompts in the paper's count) and three from UltraFeedback (GPT-4 scores 1 to 5, 63,967 prompts), with verbosity trained in both directions for eight settings.

The headline (Section 6.1): on HelpSteer the measurements work. On UltraFeedback, three pairs with correlations of 0.810 to 0.879 improve both objectives at no weight. The authors trace this to generation length and repetition confounding the reward-model scores, and name as "a plausible contributing factor" that UltraFeedback's GPT-4 scores track length more strongly than HelpSteer's human scores. That last sentence is the one a buyer choosing between human and AI annotation would quote. The paper does not measure it, so we did.

Do the paper's numbers reproduce from the released data?

Yes, closely. We downloaded the HelpSteer train split (35,331 responses, 9,944 prompts) and all of UltraFeedback (255,864 rated completions, 63,966 prompts) from Hugging Face, formed every within-prompt response pair, dropped ties, and recomputed the conflict rate against helpfulness and the response-level score correlation the paper calls "Data corr." (Table 4).

Paper columns from arXiv:2609.26929v1 Table 1 (2048-token limit) and Table 4. "Ours" columns computed by BLOMEGA on 27 September 2026 from the full released datasets, before any length filter or train/validation split. The verbosity (−) conflict rate is 1 minus the (+) rate by construction. The last column counts weights in Table 5's repetition-control block where both mean scores beat the re-decoded reference; it is our count on point estimates, because the paper publishes no intervals for that block.

Joint pair counts line up too. We get 21,409 joint helpfulness-correctness pairs on HelpSteer; the paper reports 21,184 after a 2048-token filter that keeps 0.996 of pairs. For UltraFeedback honesty we get 203,509 on the full data; 90% of that (the paper's random train split) times its 0.995 retention is about 182,245, against the reported 182,000. The pre-training measurements in this paper are what the data says.

One discrepancy we could not resolve. Table 1 credits HS complexity with improving both objectives at "2 of 5" weights, but in Table 5 every greedy 2048-token complexity model has both means above the reference (helpfulness 0.626 to 0.653 against 0.621; complexity 0.440 to 0.456 against 0.382). Table 1 therefore seems to use a significance criterion that its caption does not state. Our repetition-control counts use point estimates, so the two columns are not on identical footing.

Where does the length bias actually sit?

This is our main finding. Measured across all responses, the paper's intuition holds: length correlates with helpfulness at Spearman 0.157 in HelpSteer and 0.527 in UltraFeedback, more than three times higher. But DPO and MODPO never compare a response to one from another prompt. They compare two responses to the same prompt. Measured there, the longer response is the preferred one on helpfulness in 71.1% of 26,771 decided HelpSteer pairs and 70.9% of 283,875 decided UltraFeedback pairs.

The cross-prompt gap is a composition effect, in our judgement: UltraFeedback pools completions from many models of very different quality (median 135 words against HelpSteer's 68), so short, weak answers drag the pooled correlation up. That variance never reaches a preference pair.

The second objectives cut against the paper's story. UltraFeedback honesty, instruction-following and truthfulness prefer the longer response in 60.4%, 60.6% and 53.8% of pairs. HelpSteer correctness and coherence do so in 68.3% and 65.3%. The human-labelled "aligned" objectives are the more length-hungry ones.

And "human" is not one number. HelpSteer2, NVIDIA's later human-annotated release (20,324 train responses), prefers the longer response on helpfulness in only 56.7% of 7,196 pairs, 14.4 points below HelpSteer. The protocol moved the length preference more than the choice between human and GPT-4 raters did.

Length does shape conflict on HelpSteer. Restricting to pairs whose lengths are within 1.2x of each other, the helpfulness-verbosity conflict rate rises from 0.227 to 0.361 and helpfulness-complexity from 0.203 to 0.292, while UltraFeedback moves from 0.181 to 0.190 (honesty), 0.164 to 0.168 and 0.193 to 0.166. In HelpSteer, length is part of what makes helpfulness agree with verbosity; in UltraFeedback the conflict rate barely depends on it.

So the confound the paper found is real, but it lives downstream of the labels. The greedy 2048-token UltraFeedback models write 599 to 972 words per answer, with a repeated 4-gram in 88.5% to 97.3% of responses (Table 6), against a reference of 234 words. Re-decoding with repetition penalty 1.10 and 4-gram blocking cuts UltraFeedback length by 58.9% and lifts helpfulness by 0.137 and the second objective by 0.125 on average; HelpSteer moves by −0.001 and −0.008 (Table 2). The paper itself says the re-decode cannot separate an ArmoRM bias against repetition from genuinely better answers.

Counting Table 5's repetition-control block, which the paper summarises only as "substantially more consistent," UltraFeedback goes from 0 of 15 settings improving both objectives to 10 of 15. Honesty and instruction-following flip to 5 of 5 each, though honesty's smallest gain is 0.013 and no intervals are published. Truthfulness stays at 0 of 5: it sits below the re-decoded reference (0.595) at every weight, by 0.021 to 0.052, despite a reference correlation of 0.810. On the HelpSteer side, complexity rises from 2 to 5 of 5 and verbosity toward length from 1 to 6 of 6.

Our reading, marked as a hypothesis: truthfulness is the UltraFeedback attribute whose labels reward length least (53.8% longer-wins, against 70.9% for helpfulness, a 17.1-point gap, the widest of the three UltraFeedback pairs). Since MODPO's preference pairs come from helpfulness, the policy learns to lengthen answers, which the truthfulness signal does not pay for. The gap in length preference between the two objectives, rather than the rater type, may be the cheaper predictor.

What should a team buying preference labels do with this?

Do not treat RLAIF labels as the length-biased option by default. At the pair level, GPT-4 UltraFeedback helpfulness (70.9%) matches human HelpSteer (71.1%), and human HelpSteer2 is at 56.7%. What a length audit returns depends on the rubric and collection protocol. Run the audit on your own pairs before paying for either.

Measure length bias within prompts, not across the pool. A pooled Spearman of 0.527 against 0.157 would push most teams toward human annotation. The within-prompt longer-wins rate is the number that enters a DPO loss, and it costs one groupby to compute. Report it per attribute, next to the conflict rate.

Use the conflict rate; it is reproducible. Every one of the paper's eight conflict rates came back within 0.001 from the public data. It is a stable, cheap pre-training statistic. The correlation that failed is the one that depends on a reward model scoring generations.

Check generation before you conclude two objectives conflict. The UltraFeedback verdict flipped for two of three pairs once repetition was controlled. Log words per answer and the repeated-4-gram share for every checkpoint; greedy decoding at 4,096 tokens produced 900-word loops here. For comparison with other preference-data audits, see our notes on HH-RLHF label noise and on preference-data distribution mismatch.

Limits. The paper uses one seed per model, one 7B base per dataset, and ArmoRM as sole evaluator, which was itself trained on data that includes HelpSteer and UltraFeedback (Limitations iv). Our check covers the labels only; we did not rerun training or scoring. The code repository cited in the paper, github.com/esradonmez/modpo-objective-trade-off, returned 404 from the GitHub API on 27 September 2026.

Check it yourself

Two parquet downloads and about 25 seconds of pandas reproduce every "ours" number above. Files are public on Hugging Face; UltraFeedback's per-attribute score is completions[i].annotations.<aspect>.Rating, with 5.36% of honesty ratings missing.

curl -sL -o hs.parquet  https://huggingface.co/api/datasets/nvidia/HelpSteer/parquet/default/train/0.parquet
curl -sL -o uf0.parquet https://huggingface.co/api/datasets/openbmb/UltraFeedback/parquet/default/train/0.parquet
curl -sL -o uf1.parquet https://huggingface.co/api/datasets/openbmb/UltraFeedback/parquet/default/train/1.parquet

python3 - <<'PY'
import pandas as pd, numpy as np, itertools
def longer_wins(df, a):              # within-prompt pairs, ties on score or length dropped
    w = n = 0
    for _, g in df.groupby('prompt'):
        for x, y in itertools.combinations(g.to_dict('records'), 2):
            if x[a] == y[a] or x['words'] == y['words'] or np.isnan(x[a]) or np.isnan(y[a]): continue
            n += 1; w += np.sign(x[a]-y[a]) == np.sign(x['words']-y['words'])
    return round(w/n, 3), n
h = pd.read_parquet('hs.parquet'); h['words'] = h.response.str.split().str.len()
rows = []
for f in ['uf0.parquet', 'uf1.parquet']:
    for i, c in enumerate(pd.read_parquet(f).completions):
        for r in c:
            d = {'prompt': f+str(i), 'words': len(str(r['response']).split())}
            for k in ['helpfulness', 'truthfulness']:
                try: d[k] = float(r['annotations'][k]['Rating'])
                except Exception: d[k] = np.nan
            rows.append(d)
u = pd.DataFrame(rows)
print('HS helpfulness', longer_wins(h, 'helpfulness'))    # (0.711, 26771)
print('UF helpfulness', longer_wins(u, 'helpfulness'))    # (0.709, 283875)
print('UF truthfulness', longer_wins(u, 'truthfulness'))  # (0.538, 214541)
print('HS corr(help,correctness)', round(h.helpfulness.corr(h.correctness), 3))  # 0.852, paper 0.853
PY

The conflict rate is the same loop over pairs where both helpfulness and the second attribute are untied, counting sign disagreements: 0.027 for HelpSteer correctness, 0.193 for UltraFeedback truthfulness. For HelpSteer2 swap in nvidia/HelpSteer2; helpfulness gives 0.567 on 7,196 pairs. The repetition-control count is read directly off Table 5 of the PDF, lower block, against each pair's re-decoded SFT row.

What would prove this wrong?

Our claim is narrow: at the preference-pair level, GPT-4 UltraFeedback labels are not more length-biased than human HelpSteer labels, so rater type is not what broke the paper's prediction. It is wrong if a within-prompt length audit that also controls for the responding model (UltraFeedback pairs usually mix completions from different models) shows UltraFeedback helpfulness preferring the longer answer at least 10 points more often than HelpSteer's.

A dated prediction: by 30 June 2027, if the authors or anyone else score the released UltraFeedback MODPO outputs with an independent, length-controlled evaluator (human raters or a length-controlled LLM judge), honesty and instruction-following will improve jointly with helpfulness at a majority of weights, and truthfulness will still trade off. If truthfulness also improves jointly, our length-preference-gap hypothesis for it is wrong. If honesty and instruction-following trade off under an independent evaluator, the re-decoding result was an ArmoRM artifact and the paper's measurements genuinely fail on UltraFeedback.

Sources

1. Tsoi, D., Dönmez, E. Which Objectives Need a Dial? Predicting Objective Conflict and Covering Trade-offs in Steerable Pluralistic Alignment. arXiv:2609.26929v1, 22 September 2026. Tables 1, 2, 4, 5, 6, 7; Sections 5 and 6.1; Limitations.
2. NVIDIA. HelpSteer dataset, train split, 35,331 rows. Downloaded 27 September 2026.
3. OpenBMB. UltraFeedback dataset, 63,966 prompts, 255,864 rated completions. Downloaded 27 September 2026.
4. NVIDIA. HelpSteer2 dataset, train split, 20,324 rows. Downloaded 27 September 2026.
5. Zhou, Z. et al. MODPO: Multi-Objective Direct Preference Optimization. ACL Findings 2024. The training method the paper builds on.
6. Singhal, P. et al. A Long Way to Go: Investigating Length Correlations in RLHF. COLM 2024. Cited by the paper on length confounds.
7. BLOMEGA. HH-RLHF's label noise is 39% or 1.8%, depending on the instrument.
8. BLOMEGA. RLHF's exponential penalty on disagreeing users is a preference-data sourcing problem.