onPanda cuts median annotation time 52% against post-editing and 2% against ranking, on 21 prompts
On 21 September 2026 StepFun posted onPanda, an annotation tool where the annotator fixes the first bad token and the model regenerates everything after it. In a controlled study of 3 annotators and 21 image-description prompts, median time per prompt was 330 s against 681 s for post-editing (51.5% less) and against 336 s for four-way ranking (1.8% less). The data stays close to the rollout model, perplexity +0.86% against +36.31% for post-edits. Our check of the Hugging Face release on 23 September finds 1,609 annotation files against the 7,491 sessions the paper describes.
What StepFun released on 21 September 2026
Lei Yang, Mengyin Liu, Xiangyu Zhang and nine co-authors (StepFun, with Xiamen University) posted onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction (arXiv:2609.24983v1, cs.CL, 12 pages) on 21 September 2026. The first author announced it on X the same week (Lei Yang, @diyerxx).
The interaction is old and the packaging is new. Correcting a prefix and regenerating the suffix goes back to interactive machine translation, Predictive Translation Memory in 2014 and INMT in 2019, which the paper cites. What onPanda adds is the alignment-data framing: the annotator hovers the first inappropriate token, gets the model's top-20 candidates with probabilities, clicks one or double-clicks to type a replacement, and the server truncates and continues generation from that prefix. Every intermediate response is kept as a node in an annotation tree, and each correction is logged with its position, the rejected token and the chosen one.
Four things shipped alongside the paper: the web tool (github.com/on-panda/on-panda, MIT, 32 stars on 23 September), the onpanda Python parser (version 0.1.7 on PyPI), a 21-prompt multi-reference set called Panda-MultiRef-21, and the Panda-CVL vision-language dataset with a correction benchmark. The project page summarises the time result as "experiments show that onPanda reduces annotation time by 52%". The abstract is more careful: "a small controlled study suggests".
What did the 21-prompt study actually measure?
The design is a Latin square: 21 prompts split into 3 groups of 7, each annotator uses each method on a different group, no annotator sees the same prompt twice. So every cell of Table 1 rests on 21 annotations, 7 per annotator. All rollouts came from Qwen3.5-35B-A3B in instruct mode at temperature 0.7, top-p 0.8, and the onPanda, POTATO and Argilla first candidate shared one rollout. The annotators are full-time staff of the authors' institution, none of them onPanda developers.
| Metric | onPanda (token correction) | POTATO (post-editing) | Argilla (rank 4 rollouts) | Source |
|---|---|---|---|---|
| Median time per prompt | 330 s | 681 s | 336 s | Table 1 |
| Mean time per prompt | 515.6 s | 711.1 s | 684.5 s | Section 3.1 |
| Prompts with a qualified SFT response | 21 of 21 | 21 of 21 | 11 of 21 | Table 1 |
| Mean time per qualified SFT response | 515.6 s | 711.1 s | 1,306.8 s | Ours: mean x 21 / qualified |
| Pairwise win rate, GPT-5.5 judge, both orders | 66.7% | 54.8% | 28.6% | Table 1 |
| Human preference, onPanda vs POTATO | 54.8% | 45.2% (complement) | not reported | Section 3.1 (pair count not reported) |
| Perplexity under the rollout model (baseline 1.171) | 1.181 (+0.86%) | 1.596 (+36.31%) | 1.161 (-0.83%) | Table 1 |
| Preference pairs per prompt | 7.43 | 0.95 | 6.00 | Table 1 |
| NASA-TLX workload, adapted 0 to 10 | 3.1 | 6.8 | 5.4 | Section 3.2 |
| Confidence interval or significance test on time | not reported | not reported | not reported | Section 3.1 |
All values from arXiv:2609.24983v1 except the per-qualified-SFT row, which is our arithmetic. The win rates sum to 150.1%, which is what 21 prompts x 3 pairs x 2 orders = 126 judgments over 84 appearances per method gives with no ties (56, 46 and 24 wins).
Read the table the way a buyer of annotation hours would. The median gap to ranking is 6 seconds. The mean gap is 168.9 seconds, 24.7%, and the paper attributes Argilla's long right tail to reviewing four lengthy responses on hard prompts. Ranking also failed to produce a usable SFT response on 10 of 21 prompts, so if SFT data is what you are buying, our arithmetic puts ranking at about 1,307 s per qualified response against 515.6 s, a 60.5% saving. If preference pairs are what you are buying, ranking gave 6.00 per prompt and onPanda 7.43, a much smaller difference.
The quality numbers are thinner than the table makes them look. The 66.7% win rate is judged by GPT-5.5. The human comparison, 54.8% for onPanda over POTATO, has no pair count in the paper. Without that count, 54.8% cannot be tested against a coin flip, and on 21 prompts it could easily be one. The authors list the scale limits themselves: 3 annotators, 21 prompts, one rollout model, in-house participants, no controlled agent study, and no training experiment on the correction data.
Why regeneration from a corrected prefix keeps data on-policy
Post-editing produces text the model would not have written: every human sentence after the first fix is off-distribution, and the error often recurs downstream, so the editor fixes it several times. Continuation fixes it once. The model conditions on the corrected token and writes the rest itself, which is the paper's explanation for part of the time gap and for the perplexity result, 1.181 for onPanda against a four-rollout baseline of 1.171 and 1.596 for post-edits. The paper puts per-prompt perplexity noise across rollouts at about plus or minus 2.8%, so +0.86% is inside noise and +36.31% is not.
The preference-data claim follows from the tree. A negative and its positive descendant share an identical prefix and diverge at exactly one logged position, so the pair carries a rejected token and a chosen token at the same index under any tokenizer. That is closer to a controlled contrast than a pair of independently sampled responses, where the two texts differ everywhere and a reward model has to infer which difference mattered. Our judgement: this is the most useful property in the paper, and it is the one with no training evidence yet. The Limitations section says so directly.
The mechanism has two dependencies. The inference API must continue from an assistant prefix (vLLM exposes this as continue_final_message) and return top-k logprobs; many closed APIs do neither, and the paper names vLLM, SGLang, llama.cpp, ollama and several vendor APIs that do. And the error rate must be sparse. When the rollout model is far from the task and corrections become dense, the typed share rises and both the speed and the on-policy advantage shrink. On-policy also means on-policy to the checkpoint that generated the rollout, which decays once that model is retrained.
How much of a final response is human-written, by domain?
The paper's production aggregate is 97.0% model tokens, 2.1% candidate picks and 0.9% typed, across 131,996 sessions from 32 vision, 42 audio and 24 agentic annotators and 22,952.2 human-hours, which yielded about 388K logged corrections (Table 2 and Section 3.2; sessions x corrections per session gives 387,516 on our arithmetic). We recomputed the split per domain from Table 2's per-response averages. The result is ours.
Three facts drop out of that arithmetic. Humans supply roughly the same number of tokens per qualified response in every domain, 2.6 for vision, 2.3 for audio and 2.1 for agentic, so the human share is set almost entirely by response length. Audio responses average 37.0 model tokens, which makes audio 6.22% human-supplied, five times vision's 1.24%. And the pooled figure reproduces only if sessions are the weight: weighting Table 2 by sessions gives 96.9%, 2.1% and 1.0%, while weighting by qualified responses (sessions x SFT samples per session) gives 97.3%, 1.8% and 0.9%. The paper says "across all qualified responses"; its numbers match the session weighting.
For a team deciding whether to adopt this, the practical reading is this. On-policy fidelity is a per-token property, and a short-response task such as audio transcription review will carry several percent human tokens even when annotators intervene only twice. Typed tokens matter more than picks: a picked candidate is by construction something the model ranked in its top 20. Typed share is 0.48% for vision, 1.89% for audio and 0.21% for agentic on our arithmetic.
The time profile has an oddity worth asking the authors about. Table 2's human-hours divided by sessions gives a mean of 8.34 minutes per vision session and 10.77 per audio session, above the reported 75th percentiles of 4.48 and 3.81 minutes. Either a small tail of sessions consumes most of the hours or human-hours counts time outside sessions. The paper does not say which, and it changes what "cheap" means in a staffing plan.
What we found in the released data changes what you can build on today. The table below is our inventory, pulled from the GitHub and Hugging Face APIs on 23 September 2026.
| Artifact | What the paper says | What is shipped | License | Source |
|---|---|---|---|---|
| Panda-CVL test | 652 sessions; 652 good + 1,474 not-good = 2,126 eval instances | 626 .panda.json files, 2,139 dialogs (599 is_good true, 1,309 false, 231 unset); every file parses to exactly 1 SFT sample with onpanda 0.1.7 | CC0 | Section 4; ours, HF diyer22/Panda-CVL-test |
| Panda-CVL train | 6,839 sessions | 983 files, 0.913 GB; 36 sampled files across all 9 folders each parse to 1 SFT sample | CC0 | Section 4; ours, HF diyer22/Panda-CVL-train |
| Panda-CVL rollout model | step-1o-turbo, 32B dense VLM | test-set chat configs name 7 checkpoints; 490 of 2,595 operations that carry a chat config (18.9%) are Qwen2.5-7B-Instruct GPTQ, Qwen3.5-35B-A3B or Qwen3.5-2B AWQ | n/a | Section 4; ours, chat_config.model |
| Panda-MultiRef-21 | 21 prompts, 4 independent annotations each, 84 total | 84 files as stated, English, Qwen3.5-35B-A3B rollouts; one of the 4 sets is converted post-editing data; password-zipped (not-train-corpora) | CC0 | Appendix B; ours, GitHub |
| Controlled-study logs (Table 1) | 63 annotations across 3 methods | not released; no POTATO or Argilla files, no method-to-annotator assignment | n/a | ours, GitHub org listing |
| onpanda Python parser | "provided in the accompanying onpanda repository" | 33 files, benchmark and correction-verifier code present | no license file | ours, on-panda/on-panda-python |
| onPanda web tool | open-sourced | 125 files, Vue and JS | MIT | ours, on-panda/on-panda |
Counts are a snapshot: the Hugging Face repos were created on 22 September 2026 and the train split was last modified at 18:14 UTC that day, so an upload may still be in progress. 1,609 of 7,491 is 21.5%; 626 of 652 is 96.0% of the test split.
Two consequences follow. Table 3's benchmark scores (best F1 17.09% for GPT-5.5, best correction accuracy 15.83% for GPT-6) were computed on 652 test sessions, and the public split has 26 fewer, so an exact reproduction is not possible from what is posted. And the Panda-MultiRef-21 data is the only released window into the study conditions. Its 3 onPanda sets are complete 21-prompt passes by single annotators, not the 7-prompt Latin-square cells, so they cannot reproduce the 330 s median. They do show how much annotators differ: corrections on the path to the final response average 7.10 per prompt, 8.43 and 3.19 for the three onPanda annotators on identical starting rollouts, and the median span from first to last correction is 147 s, 498 s and 32 s. Across those 63 files, 354 of 393 corrections (90.1%) were candidate clicks, and typed text is 152 characters out of 124,038 in the final responses, 0.12%. These are our counts.
Check it yourself
Everything above is reproducible from public URLs. The first block counts the released files, the second parses them with the authors' own package, the third redoes our Table 1 and Table 2 arithmetic.
# 1. Count what is on Hugging Face (paper: 652 test, 6,839 train sessions)
for s in test train; do
curl -s "https://huggingface.co/api/datasets/diyer22/Panda-CVL-$s/tree/main?recursive=true" |
python3 -c "import json,sys; d=json.load(sys.stdin); print('$s', sum(x['path'].endswith('.panda.json') for x in d))"
done
# test 626
# train 983
# 2. Parse the test split with the authors' parser: one SFT sample per file
python3 -m venv v && v/bin/pip install -q onpanda==0.1.7 huggingface_hub
v/bin/python - <<'PY'
import glob, onpanda
from huggingface_hub import snapshot_download
root = snapshot_download("diyer22/Panda-CVL-test", repo_type="dataset", local_dir="cvl-test")
files = glob.glob(root + "/**/*.panda.json", recursive=True)
sfts = sum(len(onpanda.PandaTree(f, onpanda.utf8_tokenizer).build_legacy_data_v1()["sfts"]) for f in files)
print(len(files), "files,", sfts, "SFT samples") # 626 files, 626 SFT samples
PY
# 3. Paper arithmetic (Table 1, Section 3.1, Table 2)
python3 - <<'PY'
print("vs POTATO median %.1f%%, vs Argilla median %.1f%%" % (100*(1-330/681), 100*(1-330/336)))
print("Argilla s per qualified SFT %.1f" % (684.5*21/11)) # 1306.8
S = {"vision": 25596, "audio": 105143, "agentic": 1257} # sessions
sft = {"vision": 1.001, "audio": 1.318, "agentic": 6.018} # SFT per session
tok = {"vision": (207.0, 1.6, 1.0), "audio": (34.7, 1.6, 0.7), "agentic": (557.0, 0.9, 1.2)}
for k, (m, c, t) in tok.items(): print(k, "human %.2f%%" % (100*(c+t)/(m+c+t)))
for name, w in (("per session", S), ("per response", {k: S[k]*sft[k] for k in S})):
M, C, T = (sum(w[k]*tok[k][i] for k in S) for i in range(3)); X = M+C+T
print(name, "model %.1f%% picks %.1f%% typed %.1f%%" % (100*M/X, 100*C/X, 100*T/X))
PY
# per session model 96.9% picks 2.1% typed 1.0%
# per response model 97.3% picks 1.8% typed 0.9%
# 4. Multi-reference set (4 annotators x 21 prompts)
curl -sLO "https://github.com/on-panda/Panda-MultiRef-21/raw/main/Panda-MultiRef-21_password(not-train-corpora).zip"
unzip -q -P 'not-train-corpora' 'Panda-MultiRef-21_password(not-train-corpora).zip'
# per dialog: operations[-1].operator in {continue_with_chosen, continue_with_input},
# continue_with_input.input_patch = typed text, operations[].time = epoch msFields to know in a .panda.json: dialogs[k].operations[-1].parent is the tree edge, common_prefix_length is the fork point in characters, rejected_token.logprobs holds the top-20 list at the rejected position, and annotate.is_good is the verdict. When is_good is unset the parser treats the highest-numbered dialog as the positive, which is how every MultiRef file (all 506 verdicts unset) still yields an SFT sample.
What would prove this wrong
Our release count is a snapshot. If diyer22/Panda-CVL-train reaches 6,839 files and the test split 652 by 31 October 2026, the gap was an upload in progress and only the rollout-model discrepancy remains. If the counts are unchanged by then, the paper's dataset description and its benchmark scores refer to data the public cannot fully obtain.
The efficiency claim has a sharper test. Our prediction: the first independent timing study with at least 10 annotators and a second rollout model, published by 30 June 2027, will find onPanda faster than post-editing by more than 30% at the median and within 15% of four-way ranking at the median. If it finds onPanda more than 30% faster than ranking at the median, our reading that the headline is a post-editing result is wrong. If it finds less than 30% against post-editing, the 21-prompt result did not generalise.
The on-policy claim is falsifiable with released data. Train one reward model or DPO run on onPanda's same-prefix pairs and one on independently sampled pairs with matched count from the same prompts. The paper's Future Work section says the authors are building exactly this. If same-prefix pairs do not beat independent pairs on a held-out preference set, position-exact supervision is a data-format property and not a training advantage.
Sources
- Yang, L., Liu, M., Wang, J., et al. (StepFun, Xiamen University). onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction. arXiv:2609.24983v1, 21 September 2026. Sections 2 to 4, Tables 1 to 3, Appendix B, Limitations.
- onPanda project page, on-panda.github.io/research, accessed 23 September 2026. Source of the "experiments show ... 52%" wording.
- Panda-CVL dataset page, on-panda.github.io/Panda-CVL, accessed 23 September 2026. States 7,491 sessions, 6,839 train, 652 test.
- Hugging Face datasets diyer22/Panda-CVL-test and diyer22/Panda-CVL-train, created 22 September 2026, CC0, file trees read 23 September 2026.
- GitHub on-panda/Panda-MultiRef-21 (CC0, pushed 26 July 2026), on-panda/on-panda-python (commit b780221, 28 August 2026), on-panda/on-panda (MIT). Read via the GitHub API on 23 September 2026.
- Lei Yang (@diyerxx), announcement post on X, September 2026.
- Green, S., Chuang, J., Heer, J., Manning, C. D. Predictive Translation Memory: a mixed-initiative system for human language translation. UIST 2014, pages 177 to 187. The prefix-correction interaction onPanda descends from.
- Jurgens, D., Chen, M., Iyer, L. Potato 2.0: a comprehensive annotation platform with AI-in-the-loop support. ACL 2026 System Demonstrations. The post-editing baseline.