A blind 50/50 split of the annotation budget missed the near-optimal region up to 80% of the time. A $2 proxy run did not.
A paper posted 1 September 2026 swept how a fixed annotation budget should divide between demonstrations and preference data, across three model families, four tasks and two RL objectives. Allowing 10% off peak, most tasks accepted splits covering 55% to 75% of the feasible range. Ratios transferred from a 1B proxy landed inside a 3B-to-8B target's 5% near-optimal region 0.90 to 0.95 of the time; a fixed 0.5 split landed inside it 0.20 to 0.60 of the time. The proxy sweep costs about $2.10.
What changed, and when
On 1 September 2026, Jingtan Wang, Arun Verma, Xiaoqiang Lin, Zhengyuan Liu, Nancy F. Chen, Daniela Rus and Bryan Kian Hsiang Low posted Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs (arXiv:2609.01573v1, accepted to EMNLP 2026).
The framing is what is new. Earlier work asked which SFT-to-RL ratio is optimal and reported broad trends, for example that SFT dominates in low-data regimes. This paper asks instead for the near-optimal region: the set of allocation ratios whose performance stays within a stated tolerance of the best observed performance at that budget and model size. Formally, for a total budget B and ratio r, rB samples go to SFT and (1 − r)B to the RL stage, and the region is every r whose score is at least (1 − ε) times the peak.
Budget is counted in annotated training samples, not GPU hours, and the paper is explicit about why: annotation dominates the cost of post-training. Even against cheap synthetic annotation at $10-3 per example, it reports compute staying below annotation, which is what makes the ratio a procurement question rather than a scheduling one.
The sweep runs the Cartesian product of model size, ratio, budget, task, algorithm and model family, on the order of hundreds to thousands of post-training runs, which is why every run uses LoRA rather than full fine-tuning. The ratio grid is {0.00, 0.25, 0.50, 0.75, 1.00}, budgets sit in [0, 15k] samples with the analysis restricted to B ≥ 5k, and a denser 9-point grid adding {0.125, 0.375, 0.625, 0.875} reproduces the same qualitative behaviour.
The evidence table
The headline comparison is Table 6 of the paper: how often each recommendation lands inside the target model's near-optimal region, averaged over budgets and over target sizes from 3B to 8B. "Proxy" means running the 5-point ratio grid on the family's smallest model (1B for Llama, 1.5B for Qwen 2.5) and transferring the region. "Fixed 0.5" means splitting the budget down the middle without measuring anything.
| Setting | Tolerance | Proxy hit rate | Fixed r = 0.5 hit rate | Gap | Source |
|---|---|---|---|---|---|
| Llama, HelpSteer | 5% | 0.95 | 0.20 | +0.75 | Table 6 |
| Llama, instruction following | 5% | 0.95 | 0.40 | +0.55 | Table 6 |
| Qwen 2.5, instruction following | 5% | 0.90 | 0.60 | +0.30 | Table 6 |
| Llama, HelpSteer | 10% | 1.00 | 0.70 | +0.30 | Table 6 |
| Llama, instruction following | 10% | 0.95 | 0.70 | +0.25 | Table 6 |
| Qwen 2.5, instruction following | 10% | 1.00 | 1.00 | 0.00 | Table 6 |
The four tasks and where their data comes from matter, because the result is a claim about ratios and not about any one dataset. Every task keeps SFT and RL-stage data from a consistent source, either human-annotated or machine-generated, so the comparison across ratios is not confounded by provenance.
| Task | SFT-stage data | RL-stage data | Evaluation | Source |
|---|---|---|---|---|
| Math | GSM8K | Tülu3 Grade School Math | GSM8K test accuracy | Table 1 |
| Instruction following | Tülu3 Persona IF | Tülu3 Persona IF (DPO) / Tülu3 RLVR IF (GRPO) | IFEval accuracy | Table 1 |
| Summarization | Reddit TL;DR | Reddit Comparison | ROUGE-L F1 | Table 1 |
| Helpfulness | HelpSteer | HelpSteer2 | Reward model score | Table 1 |
The region is what transfers; the single best ratio is not
Here is the shape of the result. Performance as a function of the allocation ratio is not a peak with steep sides. It is a plateau. At a 10% tolerance most tasks admit near-optimal ratios spanning 55% to 75% of the allocation space, and per-ratio hit-rate heatmaps show the admitted ratios are contiguous on the grid rather than scattered, which is the evidence that the plateau is real and not a sampling artefact.
That plateau generally widens with model size at a fixed tolerance. The paper is careful about why, and we are repeating its caution rather than its headline: part of the widening is mechanical, because a larger model has a higher absolute peak, so the same relative tolerance admits more absolute slack. Under an absolute tolerance anchored to the smallest model, the widening is dampened for Llama and reverses outright on Llama math. What survives both definitions is transfer.
Cost asymmetry moves the region, in a direction that is convenient. The paper fixes a DPO preference example at $0.001 and varies ρ, the ratio of SFT cost to DPO cost, noting that synthetic SFT demonstrations from frontier models run roughly 1 to 2 times the cost of preference-style annotation. As ρ rises, the near-optimal region widens at the same tolerance: when demonstrations get expensive, the split matters less and annotation logistics can drive it.
The opposite regime is the one to watch, and it is the case where a team budgets in GPU hours rather than in labels. DPO takes roughly 2 to 3 times the GPU time of SFT, which puts ρ near 0.5, and there the plateau disappears.
| Near-optimal region width (Math, Llama, ρ = 0.5, GPU-hour budget) | ε = 2% | ε = 5% | ε = 10% | Source |
|---|---|---|---|---|
| 1B | 0.00 | 0.00 | 0.00 | Table 7 |
| 3B | 0.00 | 0.00 | 0.06 | Table 7 |
| 8B | 0.00 | 0.13 | 0.25 | Table 7 |
| Transfer 1B → 3B, 1B → 8B, 3B → 8B | 1.00 | 1.00 | 1.00 | Table 8 |
A width of 0.00 means one ratio and no flexibility. In that regime transfer is trivially perfect, because the single near-optimal ratio is the same at every scale, so the proxy's recommendation is always right and also always the only option. The paper says so. It is the honest reading, and it is a useful boundary: the plateau is a property of counting budget in labels, not a universal property of post-training.
What it means if you are buying the labels
Buy the proxy sweep before you buy the labels. The asymmetry is the whole argument. A 5-point ratio sweep on a 1B model is about 100 GPU-minutes and about $2.10; the annotation budget it is allocating is 5,000 to 15,000 samples, which at even $0.001 per preference pair and a dollar per human demonstration is the part with real money in it. Spending two dollars to place tens of thousands of dollars of annotation is not a close call.
A 50/50 default is a coin flip dressed as a policy. It was inside the 5% region 0.20 of the time on Llama HelpSteer and 0.40 on Llama instruction following. It costs exactly as much as the proxy run and returns no information about where the region actually sits. Our judgement: the reason 0.5 persists is that it is the only ratio nobody has to defend, and this paper removes that excuse for about the price of a coffee.
Ask for the tolerance, not the ratio. If a vendor or an internal team quotes "we use a 70/30 split", the useful follow-up is what tolerance that was chosen under and on what proxy. The gap between the proxy and the blind default collapses to zero at 10% tolerance on Qwen 2.5 instruction following and is 0.75 at 5% tolerance on Llama HelpSteer. The same decision is either free or expensive depending entirely on the quality bar.
The plateau is permission to optimise for something else. If 55% to 75% of the allocation space is within 10% of peak, then inside that band you can choose the ratio on grounds the paper does not model: which annotation is easier to source consented, which supplier has capacity, which data you can reuse across projects. That is a real operational freedom and it only exists once you know where the band is.
Check which budget you are actually spending. The plateau is measured against a budget counted in annotated samples. Counted in GPU hours, with DPO at 2 to 3 times the cost of SFT, the region collapsed to a single ratio at 1B and 3B. Teams whose constraint is a compute allocation rather than a labeling contract should not expect the flexibility.
Check it yourself
The paper publishes no code or trained artefacts we could find, so the check here is on the arithmetic of the recommendation, not on the sweep. Everything below runs in under a second.
# the paper (HTML full text is available, which is where Tables 6 to 8 live)
curl -s -o /dev/null -w "%{http_code}\n" https://arxiv.org/abs/2609.01573 # 200
curl -s -o /dev/null -w "%{http_code}\n" https://arxiv.org/html/2609.01573 # 200
# what the proxy buys, per Table 6, in expected wasted post-training runs
python3 - <<'PY'
tbl = { # (setting, tolerance): (proxy hit rate, fixed r=0.5 hit rate)
("Llama HelpSteer", "5%"): (0.95, 0.20),
("Llama Inst", "5%"): (0.95, 0.40),
("Qwen 2.5 Inst", "5%"): (0.90, 0.60),
("Llama HelpSteer", "10%"): (1.00, 0.70),
("Llama Inst", "10%"): (0.95, 0.70),
("Qwen 2.5 Inst", "10%"): (1.00, 1.00),
}
GPU_HR = 1.25 # CoreWeave L40, as reported in the paper
proxy_min, target_min = 100, 100
for (name, tol), (pr, fx) in tbl.items():
miss = fx - pr # extra failure probability of guessing
print(f"{name:16} eps={tol:>3} proxy {pr:.2f} fixed {fx:.2f} "
f"extra miss {abs(miss):.2f} proxy cost ${proxy_min/60*GPU_HR:.2f}")
print()
print(f"proxy route : {(proxy_min+target_min)/60*GPU_HR:.2f} USD, "
f"{(proxy_min+target_min)} GPU-min")
print(f"8B full sweep: {500/60*GPU_HR:.2f} USD, 500 GPU-min "
f"({500/(proxy_min+target_min):.1f}x)")
PY
# Llama HelpSteer eps= 5% proxy 0.95 fixed 0.20 extra miss 0.75 proxy cost $2.08
# Llama Inst eps= 5% proxy 0.95 fixed 0.40 extra miss 0.55 proxy cost $2.08
# Qwen 2.5 Inst eps= 5% proxy 0.90 fixed 0.60 extra miss 0.30 proxy cost $2.08
# Llama HelpSteer eps=10% proxy 1.00 fixed 0.70 extra miss 0.30 proxy cost $2.08
# Llama Inst eps=10% proxy 0.95 fixed 0.70 extra miss 0.25 proxy cost $2.08
# Qwen 2.5 Inst eps=10% proxy 1.00 fixed 1.00 extra miss 0.00 proxy cost $2.08
#
# proxy route : 4.17 USD, 200 GPU-min
# 8B full sweep: 10.42 USD, 500 GPU-min (2.5x)
# the grid the whole result is measured on, and the denser check
python3 -c "
G=[0.00,0.25,0.50,0.75,1.00]
D=sorted(G+[0.125,0.375,0.625,0.875])
print('5-point grid :', G)
print('9-point check:', D)
print('10% region spanning 55-75% of the axis covers',
[r for r in G if 0.125 <= r <= 0.875], 'to',
[r for r in G if 0.0 <= r <= 0.75])
"Four limits the paper states about its own numbers, which we repeat because they bound the recommendation. Every run uses LoRA, treated as an approximation of full fine-tuning. Most of the sweep is single-seed, with a 3-seed validation only on math and summarization for the Llama family. The ratio r = 0 is excluded from the cost-asymmetry analysis because it consistently underperforms and needs disproportionately more data when ρ > 1. And the widening of the region with scale is partly a consequence of the relative tolerance definition, visible in the Llama math case where the absolute-tolerance slope is negative.
What would prove this wrong
The claim we are taking from the paper is operational: for a budget counted in annotated samples, a cheap proxy sweep identifies a near-optimal allocation region that transfers to a much larger target, and it beats a fixed 50/50 split by enough to pay for itself many times over.
A dated prediction. By 31 December 2027, a study that runs the same protocol on a model family not in this paper (Llama 3, Qwen 2.5 and Qwen 3 are the three tested), at a 5% tolerance, with budgets at or above 5,000 annotated samples, will report a proxy hit rate above 0.80 on at least two of its tasks. A result showing proxy transfer at or below the fixed-0.5 baseline at 5% tolerance, on a sample-counted budget, falsifies the reading here. The most likely way it breaks is the one the paper flags itself: post-training algorithm rankings can flip across scale, and a family whose small model is far weaker than its large one (the Llama math pattern) is where the region stops behaving.
Sources
- Wang, J., Verma, A., Lin, X., Liu, Z., Chen, N. F., Rus, D., Low, B. K. H. Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs. arXiv:2609.01573v1, 1 September 2026, EMNLP 2026. Tables 1, 6, 7 and 8, Sections 2.1, 3.1 to 3.4 and Appendices E.4 to E.7; HTML full text.
- Lambert, N., et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124. Source of the Persona IF, RLVR IF and Grade School Math RL-stage data.
- Wang, Z., et al. HelpSteer2: Open-source dataset for training top-performing reward models. arXiv:2406.08673. The RL-stage data for the helpfulness task.
- Cobbe, K., et al. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168. GSM8K, the SFT-stage source and evaluation for the math task.
- BLOMEGA. Two audits of HH-RLHF disagree on how much of it is mislabelled, on the quality side of the same preference-data purchase.
- BLOMEGA. Data annotation research: the latest.