The LLM annotation source that cost 78x more produced no better quality, a budgeted-allocation paper found

What changed, and when

On 7 October 2026, Biao Xiang (Penn State), Ali Eshragh and Yuexing Li (Johns Hopkins Carey Business School), and Kai Wang (Georgia Tech) posted Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation (arXiv:2610.10974v1).

Off-policy evaluation (OPE) estimates how well a target policy would perform using only data logged under a different, already-deployed policy. When the logged data barely covers the actions the target policy would take, the estimator is forced into high-variance reweighting. Counterfactual annotations, a domain expert or an LLM predicting what would have happened under an unobserved action, can add evidence, but every source has its own cost and its own error profile. The paper's question: given a fixed budget and several annotation sources of known cost and quality, which context-action pairs and which sources should you actually buy?

They formulate this as an integer allocation problem that minimizes the annotation-dependent part of estimator variance, and solve the multi-source version with a majorization-minimization (MM) algorithm paired with dynamic-programming subroutines. The headline numbers come from two experiments: a synthetic clinical bandit, and ASSISTments, a real tutoring-interaction dataset annotated by four actual LLMs.

The evidence table

The ASSISTments experiment asks each of four LLMs to predict a student's response time on a problem, given the student's ability profile and historical response times for similar student-problem pairs: 20 independent predictions per context-action pair, 20,000 predictions total per model.

Reproduced from Table 6 of arXiv:2610.10974v1, Appendix E.2.4. Bias and variability computed from 20 predictions per context-action pair across 100×10 pairs, after converting predicted response times to rewards.

Reproduced from Table 8 of arXiv:2610.10974v1, Appendix E.4. Mean relative reduction in empirical MSE over 30 pilot replications, each evaluated on 50 independent final value estimates, against the no-annotation baseline.

How the allocator spends a fixed budget across sources

The paper excludes GPT-o3-mini from its ASSISTments allocation experiment once Table 6 shows it offers no quality advantage at a far higher price. Under a budget of κ = $0.50, their MM algorithm allocates annotations across the three remaining sources (Gemini-2.5-Flash, Claude-Haiku-4.5, GPT-4o-mini), purchasing 6,000 annotations and reducing the variance objective from J(0) = 88.61 to J = 79.06, a 10.77% reduction. The convergence trace shows the algorithm leaning heavily on GPT-4o-mini for its low bias and cost-efficiency, while using the pricier Claude-Haiku-4.5 more sparingly.

What multi-source buys over single-source narrows once you optimize properly. Under uniformly random allocation, single-source variance objectives range from 80.79 to 83.72 while every multi-source subset drops below 80. Once the same MM optimization is applied to single-source allocation too, those single-source bars improve to 79.32 to 80.82, closing most of the gap to the three-source optimum of 79.06. The paper's own reading: a well-optimized single cheap source gets most of the benefit; multi-sourcing adds a smaller but real remaining gain.

What it means for anyone buying LLM annotation at scale

Price and quality are not coupled by default. This is one dataset and one task, but the direction is stark: the cheapest of four sources had the lowest bias, and the most expensive had the highest. Any pipeline that defaults to "the most capable model" for annotation without measuring bias against ground truth on its own task is paying for a correlation that did not hold here.

Measuring source quality is cheap relative to what it buys. The bias and variability estimates in Table 6 came from 20,000 predictions per model, a small, one-time cost (their total across all four sources was under $47) that then informed which sources to exclude entirely before any budget allocation ran.

Optimized allocation, not source count, is most of the gain. The jump from random single-source allocation (worst case 83.72) to optimized single-source (worst case 80.82) is larger than the jump from optimized single-source (79.32 best case) to optimized multi-source (79.06). Buying access to more sources is a second-order improvement once you are already allocating well within one.

A caveat the paper is explicit about: the clinical environment is synthetic, built to exercise the allocation theory, not drawn from real patient data. Only the ASSISTments numbers above come from real annotations on real logged interactions; the clinical MSE reductions (20.58%, and the 19.13%/18.67% figures in Table 8) describe a constructed environment and should be read as a demonstration of the method, not a clinical-deployment result.

Check it yourself

open https://arxiv.org/abs/2610.10974                 # paper, Tables 6 and 8, Figures 7, 8, 9
open https://sites.google.com/view/assistmentsdata     # ASSISTments platform data releases

# what to check without rerunning anything:
# Table 6 (Appendix E.2.4): bias, variability and total API cost per LLM source
#   GPT-4o-mini  mean|bias|=0.83  cost=$0.48
#   GPT-o3-mini  mean|bias|=0.96  cost=$37.26   (78x the cost, worse bias)
# Section E.2.4 text: budget kappa=$0.50, 6,000 annotations purchased,
#   J(0)=88.61 -> J=79.06, a 10.77% reduction
# Table 8 (Appendix E.4): mean relative MSE reduction over 30 pilot replications,
#   MM (greedy init.) = 8.47% on ASSISTments, 19.13% on the synthetic clinical env.

python3 - <<'PY'
sources = {
    "GPT-4o-mini":      {"bias": 0.83, "cost": 0.48},
    "Claude-Haiku-4.5": {"bias": 0.86, "cost": 8.11},
    "GPT-o3-mini":      {"bias": 0.96, "cost": 37.26},
    "Gemini-2.5-Flash":{"bias": 1.09, "cost": 1.14},
}
cheapest = min(sources.items(), key=lambda kv: kv[1]["cost"])
costliest = max(sources.items(), key=lambda kv: kv[1]["cost"])
markup = costliest[1]["cost"] / cheapest[1]["cost"]
print(f"cheapest: {cheapest[0]}  ${cheapest[1]['cost']:.2f}  bias={cheapest[1]['bias']}")
print(f"costliest: {costliest[0]}  ${costliest[1]['cost']:.2f}  bias={costliest[1]['bias']}")
print(f"markup: {markup:.1f}x, bias change: {costliest[1]['bias']-cheapest[1]['bias']:+.2f}")
PY
# cheapest: GPT-4o-mini  $0.48  bias=0.83
# costliest: GPT-o3-mini  $37.26  bias=0.96
# markup: 77.6x, bias change: +0.13

What would prove this wrong

The cost-quality disconnect is specific to one task (predicting tutoring response times) on one dataset (ASSISTments) with four specific model snapshots as of October 2026. It would be wrong as a general claim if the pattern reverses on a task where the more expensive model's extra reasoning genuinely helps.

A dated prediction: by 31 December 2027, a published replication of this bias-and-cost measurement on a different counterfactual-prediction task will still find at least one low-cost LLM source with bias statistically indistinguishable from, or better than, the most expensive source tested, because the mismatch here traces to reasoning-heavy pricing tiers not being matched to this task's actual difficulty, which is a pricing-versus-task-fit problem likely to recur across tasks. If a replication instead finds a clean, monotonic cost-to-quality relationship, that explanation is wrong.

Sources

1. Xiang, B. (Penn State), Eshragh, A., Li, Y. (Johns Hopkins Carey Business School), Wang, K. (Georgia Tech). Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation. arXiv:2610.10974v1, 7 October 2026. Tables 6, 8; Figures 7, 8, 9; Section 7, Appendix E.
2. ASSISTments. Tutoring-platform interaction logs, the real-world dataset behind the paper's LLM-annotated education bandit experiment.
3. BLOMEGA. Data annotation research: the latest.
4. BLOMEGA. Decision models for text annotation cost and accuracy.