BA-DPO halves DPO's length gain on MultiPref, and the 227 annotators it models pick the longer answer 48% to 96% of the time

What did the BA-DPO paper claim, and when?

On 28 September 2026 Antonio Ferrara, Alberto Rumi and Francesco Bonchi (Intesa Sanpaolo AI Research) posted BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment (arXiv:2609.35044v1, cs.LG). The code went up the same day at github.com/Ambress92/BA-DPO under CC BY 4.0, repository created 2026-09-28T12:27:41Z according to the GitHub API.

The method takes the BARP model of Ferrara et al. (2024), which lets evaluator k see item quality plus a scalar bias theta_k on items carrying a group attribute, and moves it inside DPO. The logit of a vote becomes u + theta_k times the attribute difference between winner and loser (Eq. 8), where u is the usual DPO margin. Because the bias is additive to the reward, the partition function still cancels (Proposition 2), so there is no reward model and no second network: 227 extra scalars on MultiPref, 60 on the synthetic corpus.

The theory result that matters in practice is Theorem 3 with Corollary 4. If the annotator pool is connected through shared cross-group comparisons, the votes fix every difference theta_k minus theta_k', but not the mean. Raising every annotator's bias while lowering the policy's own tilt toward the attribute changes no likelihood. The authors turn that degeneracy into a control knob: anchor to the reference model's attribute rate (the default in every experiment) or to a stated target such as 0.5 for statistical parity (Appendix C.4).

Two corpora test it. Signed-UltraFeedback appends a gender-coded and race-coded first-name signature to UltraFeedback responses and has 60 simulated annotators in three classes of 20 vote with planted biases (Section 5.1). MultiPref (Miranda et al., ACL 2025) supplies real annotators: 227 of them, four per comparison, with length (one side at least 1.5 times the words of the other) or Markdown declared as the attribute. Qwen2.5-0.5B-Instruct is fully fine-tuned, Llama-3.1-8B-Instruct trained with LoRA, and Mistral-7B-Instruct-v0.3 repeats the core comparison.

How much of the bias does BA-DPO actually remove?

On planted bias, almost all of it. On real length bias, about half. Every row is a mean over seeds 42, 123 and 456.

Percentages are the paper's "bias removed": the share of the gap between DPO and the reference that a method closes, computed per seed. The held-out gap is implicit-reward accuracy on held-out pairs that differ on the attribute minus pairs that do not. All from arXiv:2609.35044v1.

The baselines fail in instructive ways. R-DPO at alpha 0.005 cuts more tokens than BA-DPO (77% at 0.5B, 89% at 8B) but drives the held-out gap to minus 0.040 and minus 0.038: the policy now prefers the shorter response on attribute-carrying pairs. At the published alpha of 0.02 it collapses answers to 138 tokens and pushes the Markdown rate to 0.70, above DPO (Appendix F.2), the length-to-format substitution Lamparth et al. (2026) described. SamPO leaves the policy longer than DPO (minus 38% at 0.5B). Group-DRO removes 4 to 18% on Signed-UF and nothing on length, because a bias every annotator shares is not disagreement.

The paper is careful about what the length number means. At 8B the Skywork judge ranks methods in order of their length, 4.9 for the reference up to 6.7 for SamPO, so BA-DPO's 6.1 against DPO's 6.5 cannot be read as quality. The held-out gap does not use the judge, which is why the authors lead with it.

We checked the MultiPref preprocessing against the public release. Using the repository's own src/attributes.py length rule on allenai/multipref, we get 41,844 raw votes, 10,997 ties (26.3%), 30,847 kept, 227 evaluators (89 crowd, 138 expert, no overlap), 58.66% cross-group, the longer side winning 72.88% for a log-odds of 0.989, and for Markdown 26.43% cross-group at log-odds 0.980. Every figure matches Section 5.1 and Appendix D.1 to the reported precision.

Why the per-annotator parameters do not change the policy

Split theta_k into a shared mean and a deviation. Proposition 5 shows the policy's implicit reward can carry the shared part, because it is a function of the response, but can never carry the deviation, because the policy does not know who voted. So only the mean competes with the policy for the bias signal, and only the mean moves the policy's attribute rate. That is why the pooled variant, one scalar for everyone, gives the same policy as the per-annotator variant to within about 0.01 on Signed-UF (Section 5.2).

Parameterisation decides how fast that mean moves. With 227 free parameters, each updated only by its own annotator's cross-group votes, the mean moves 227 times slower under gradient descent than a single shared parameter (Proposition 6), roughly 3.5 times slower under Adam by the authors' estimate. Table 9 shows the result: the free-theta arm on MultiPref length ends with its mean at 0.42 and removes 17%, and shuffling the annotator ids gives 0.44 and 16%. The shared arms end at 0.79 to 0.84 and remove 51 to 54%. The offline estimate, which counts every win of the longer response as bias, is 0.99.

Where the per-annotator deviations earn their keep is description. On Signed-UF the learned theta_k correlate with the planted values at r = 0.95 (gender-coded) and 0.98 (race-coded), and they predict held-out votes at the planted ceiling, 0.752 at 0.5B, including the class of 20 that leans against the race-coded name, where the pooled scalar predicts at chance (0.500, Table 13). The paper's conclusion proposes using exactly these estimates to flag biased annotators while a pool is still being labelled. It reports that audit only for the synthetic corpus, with about 400 judgments per annotator. It does not report per-annotator estimates for MultiPref. That is the gap we measured.

What does per-annotator length bias look like in the real MultiPref pool?

This section is ours: computed by BLOMEGA on 29 September 2026 from the allenai/multipref parquet on Hugging Face, with the paper's length rule. For each annotator we counted cross-group votes (exactly one side at least 1.5 times longer, the only votes that identify theta_k) and the share won by the longer side.

The pool is uniformly long-leaning and widely spread. Among the 180 annotators with at least 30 cross-group votes, the share for the longer response runs from 47.7% to 95.9%, quartiles 64.5% and 80.6%, median 72.7%. On the log-odds scale the observed standard deviation is 0.58 against a mean sampling error of 0.28, so a method-of-moments estimate of the true spread is about 0.51. A chi-square test of one common rate gives 1,032 on 179 degrees of freedom, and 72 of the 180 have a 95% interval that excludes the pooled 72.9%.

Raw shares mix bias with pair quality, since annotators saw different comparisons and a long answer is sometimes the better one. So we ran a same-pair control. For every cross-group vote we took the annotator's choice (1 if longer) minus the mean choice of the other annotators on the same comparison, and averaged per annotator. The standard deviation of that excess across the 180 annotators is 0.103. Shuffling which annotator cast which vote within each comparison, 200 times, gives a null of 0.055 (99th percentile 0.062, maximum 0.063). 23 annotators sit more than 15 points from their co-annotators, against 2.4 expected by chance. The spread is not an artefact of who got which pairs. Subtracting the null in quadrature leaves about 0.087 in probability, roughly 0.44 in log-odds at a 73% base rate (0.087 divided by 0.73 times 0.27), which is our approximation, not a fitted model.

Two things the planted corpus had are missing here. There is no counter-leaning class: no annotator with 30 or more votes is significantly below 50%, so a pooled scalar already points the right way for everyone, which fits the paper's finding that pooled and per-annotator give the same policy. And crowd workers lean harder than experts: on the 4,946 comparisons where both pools cast cross-group votes, crowd workers pick the longer answer 74.4% of the time and experts 71.1%, and the pool means of the same-pair excess are plus 0.021 and minus 0.020.

Identifiability holds for the pool, thinly for some members. The graph linking annotators who judged a cross-group comparison in common is one connected component of all 227, so Theorem 3 applies. But the median annotator has 64 cross-group votes over the full release (about 72 on average in the paper's 27,812-vote training split: 27,812 / 227 x 58.7%), against about 157 per attribute in Signed-UF (25,840 / 60 x 36.4%). 47 annotators have fewer than 30, and 36 of those are experts, whose median is 47 votes against the crowd's 101. Five annotators have every cross-group vote on one side (5 of 5, 13 of 13, 16 of 16 and 2 of 2 for longer, 0 of 3), which is exactly the condition under which Section 4 says their theta_k diverges. Under Markdown, where only 26.4% of votes are cross-group, every one of these counts shrinks by more than half.

What this means if you collect or train on preference data

Keep annotator ids and attribute flags in the release. The paper notes MultiPref is one of the few public corpora with annotator identities, and it could not find any with identified annotators and a demographic marker, which is why it built a synthetic one. Pseudonymous ids like MultiPref's (mystifying_fermi) cost nothing at collection time and are the precondition for every per-annotator result here. The authors recommend treating per-annotator bias estimates as personal data, which is a reason to plan access, not to drop the ids.

To debias the policy, the pooled scalar is enough. On both corpora and three models it matches the per-annotator variant, and on MultiPref our count finds no counter-leaning group that would break it. Use the shared-mean form if you also want per-annotator estimates; avoid free theta_k, which reached 17% removal against 51%.

Choose the anchor on purpose. The default keeps the reference's attribute rate, including any bias the SFT model already learned from majority-chosen responses. A target rate removes it, but someone has to state the target. For length there is no natural one, as the authors say. Our judgement: for length, the held-out gap near zero is the more defensible stopping signal than any token count.

To audit annotators, budget cross-group votes, not total votes. Only pairs that differ on the attribute identify theta_k. On MultiPref 58.7% of votes are cross-group for length and 26.4% for Markdown, so an annotator with 100 votes has about 59 and 26 respectively. If you want the audit the paper describes, assign pairs so that every annotator reaches a floor (the synthetic corpus had about 157), and flag annotators whose votes are all on one side instead of reporting an infinite bias.

Declare every attribute you care about. BA-DPO removes only what it is told. Appendix F.1 shows an undeclared attribute stays in the policy, and R-DPO's shift from length to Markdown shows a fixed single-axis correction can move the pressure elsewhere. We discussed dataset-level length preference, and why human and GPT-4 labels look alike on it, in our HelpSteer and UltraFeedback recount; the per-annotator view here is the part that note could not see. For how much of a preference label set is noise rather than bias, see HH-RLHF's label noise.

Limits. Both real attributes are surface features of one corpus, the judge is one reward model, and the authors name a corpus with a social attribute and real identified annotators as the missing test. Our heterogeneity numbers are descriptive and treat each vote as independent; they are not BA-DPO's fitted theta_k, which also condition on the policy's reward.

Check it yourself

The data is public (ODC-BY) and the length rule is in the authors' repository. This reproduces the paper's MultiPref counts and our per-annotator table in under a minute.

curl -sL -o mp.parquet \
  https://huggingface.co/api/datasets/allenai/multipref/parquet/default/train/0.parquet
curl -s https://raw.githubusercontent.com/Ambress92/BA-DPO/main/src/attributes.py | grep -n "LENGTH_RATIO ="

python3 - <<'PY'
import pandas as pd, numpy as np
d = pd.read_parquet("mp.parquet")
W = {"A-is-clearly-better":"a","A-is-slightly-better":"a",
     "B-is-clearly-better":"b","B-is-slightly-better":"b"}
rows = []
for r in d.itertuples():
    la, lb = len(r.completion_a.split()), len(r.completion_b.split())
    cross = max(la, lb) >= 1.5 * max(1, min(la, lb))
    for pool, anns in (("crowd", r.normal_worker_annotations),
                       ("expert", r.expert_worker_annotations)):
        for a in anns:
            w = W.get(a["overall_pref"])
            if w is None: continue                      # tie, dropped as in the paper
            rows.append((a["evaluator"], pool, cross,
                         (la > lb) if w == "a" else (lb > la)))
R = pd.DataFrame(rows, columns=["ann", "pool", "cross", "longer_won"])
X = R[R.cross]
print(len(R), R.ann.nunique(), round(R.cross.mean(), 4), round(X.longer_won.mean(), 4))
# 30847 227 0.5866 0.7288
T = X.groupby("ann").longer_won.agg(nx="size", wins="sum")
T["share"] = T.wins / T.nx
print(T.nx.median(), (T.nx < 30).sum(), ((T.wins == T.nx) | (T.wins == 0)).sum())
# 64.0 47 5
print(T[T.nx >= 30].share.describe().round(3))
# count 180, min 0.477, 25% 0.645, 50% 0.727, 75% 0.806, max 0.959
PY

Fields that matter: normal_worker_annotations and expert_worker_annotations hold two votes each per comparison; each vote has evaluator (the pseudonymous id) and overall_pref. The same-pair control and the 200-shuffle null take a few more lines: subtract the mean of the other voters on the same comparison_id, then permute evaluator within each comparison.

What would prove this wrong

Our heterogeneity claim is falsifiable with the authors' own code. If the shared-mean arm of BA-DPO is trained on MultiPref length and its deviations epsilon_k are published, then for annotators with at least 30 cross-group votes the fitted epsilon_k should have a standard deviation of at least 0.3 in log-odds and correlate with our same-pair excess at r of 0.6 or more. We expect someone, the authors or a replication, to be able to report this by 30 June 2027. If the fitted spread is below 0.15 or the correlation below 0.3, the variation we measured is mostly pair quality that our co-annotator control did not remove, and the audit use of BA-DPO on real data is weaker than the synthetic corpus suggests.

The paper's own central claim has a clear falsifier too. On a corpus with real identified annotators and a social attribute (a persona or a language variety, as the authors propose), BA-DPO should remove well over half of DPO's shift in that attribute, as it does on planted names. If it removes 30% or less while the held-out gap stays open, the planted-corpus numbers owe more to the corpus being generated from the model's own likelihood (Eq. 7) than to the method.

Sources

1. Ferrara, A., Rumi, A., Bonchi, F. BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment. arXiv:2609.35044v1, 28 September 2026. Sections 3 to 5, Tables 1 to 3, Appendix B.5, D.1, F.2, G, Tables 5, 6, 9, 12, 13.
2. Ambress92/BA-DPO. Source code and configs, created 28 September 2026, CC BY 4.0. Length and Markdown rules in src/attributes.py, loader in src/data_loading.py. Accessed 29 September 2026.
3. Allen Institute for AI. allenai/multipref on Hugging Face, ODC-BY, last modified 12 March 2025; parquet index. Downloaded 29 September 2026.
4. Miranda, L. J. V. et al. Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback. ACL 2025. The MultiPref paper.
5. Ferrara, A., Bonchi, F., Fabbri, F., Karimi, F., Wagner, C. Bias-aware ranking from pairwise comparisons. Data Mining and Knowledge Discovery 38, 2024. The BARP model.
6. Lamparth, M. et al. Reward bias substitution: single-axis bias mitigations redirect optimization pressure. arXiv:2605.27996, 2026.
7. BLOMEGA. Human and GPT-4 preference labels pick the longer answer at the same rate: 71.1% vs 70.9%. 27 September 2026.
8. BLOMEGA. HH-RLHF's label noise is 39% or 1.8%, depending on the instrument.