CHORUS shipped its own audit, and it says the 33.8% is measured on a different clock

The most useful file in this translation system is the authors' audit of their own paper

CHORUS is a seven-agent workspace for professional translators, accepted at ACM ICMI 2026, with a within-subject study of 30 licensed English-Chinese translators reporting 33.8% faster completion, lower workload, and BLEU up 3.08 and COMET up 0.015. The code is public at github.com/iamgeorge/CHORUS-OSS.

Inside that release, at docs/PAPER_CODE_AUDIT.md, is a claim-by-claim comparison of the paper against the code. Forty-five atomic system claims, each verified with file and line evidence on both sides: 22 MATCH, 16 PARTIAL, 4 MISMATCH, 3 NOT_IMPLEMENTED, with 12 graded high severity. Its own header says it is an internal review aid and invites the authors to remove it before release. They left it in.

We checked the audit against the code ourselves. It is accurate, and it is the single most useful artifact any HCI translation paper has shipped this year. One of its findings lands directly on the headline: the timing measure behind 33.8% is not the quantity the paper says it is.

What was published

arXiv:2602.19016, CHORUS: Designing Human-AI Multi-Agent Collaboration for Professional Translators, by George Xi Wang, Jiaqian Hu, Guande Wu and Jing Qian. Submitted 22 February 2026, accepted at the 28th ACM International Conference on Multimodal Interaction, Napoli.

The system decomposes revision into seven agents aligned to MQM dimensions (accuracy, terminology, style, linguistic conventions, locale conventions, audience appropriateness, design and markup), each with its own prompt, each proposing token-level edits the translator accepts or rejects. Behind them sits a memory pipeline that weights past micro-edits by an inferred effort score, and a Live Style Guide that renders MQM coverage as a radar chart.

The study: 30 licensed English-Chinese translators, 21 to 50 years old, 3 to 21 years of experience, each assigned two of four WMT24 domains, 10 sentences per condition, 40 trials each, roughly 100 minutes per session, compensated 100 yuan. Baseline was an in-app chat workspace on the same model. Both conditions started from a GPT-5.3 draft.

Reported results from arXiv:2602.19016. The 22 of 30 conversion and the 95% time-saving range of 22.8% to 43.3% implied by the confidence interval are our arithmetic.

The effects are real and the tests are appropriate. A paired t-test on log completion time with t(59) = -5.35 is not a marginal result, and the confidence interval on the ratio implies a saving between 22.8% and 43.3%. Both quality metrics move in the same direction with medium effect sizes. Note the other side of the 73.3%: eight of the 30 translators were slower or worse under CHORUS on each metric.

What the audit found

The audit's method is stated plainly: 45 atomic claims extracted from the active LaTeX and from a camera_ready_edits.md file, each verified independently against the repository with file and line evidence, plus a reverse pass enumerating implemented behaviours absent from the paper. Verdicts are MATCH, PARTIAL, MISMATCH or NOT_IMPLEMENTED, with a separate severity grade for how wrong a reader would consider the description.

Claim text and verdicts from docs/PAPER_CODE_AUDIT.md in the release. The final column is our own independent check against the shipped code, 22 September 2026.

We verified the code-side column independently. Backend/app.py:64 sets STYLE_GUIDE_SUMMARY_MODEL to gpt-4o-mini, not GPT-5.3. app.py:66 sets LLM_MAX_TOKENS to 2,048, against the paper's "default max tokens". A repository-wide grep for a temperature literal returns one hit, "0", on the transcription call, against the paper's "temperature 0.7". app.py:179 contains the line "manual": "consistency". The Prompts/ directory holds 13 markdown files and none contains a bad-example or avoid list.

Two of the four high-severity mismatches carry a post-audit note dated 20 July 2026 saying they were fixed in code after the audit and before release. We confirmed both. Frontend/components/AgentSidebar.tsx:1605 now posts to /dimension-router, so the LLM ranking the paper describes is live rather than dead code, and Prompts/stress_inference.md now carries 14 references to Scherer's appraisal dimensions. The note claims the restructured effort score is numerically identical to the old one over 20,000 random signal cases, maximum delta 0.0, which we could not check without running the server.

The headline number and the clock it was measured on

This is the finding that matters outside the repository. The paper's method section says timed performance was "computed from active editing duration, excluding idle periods". The audit reports, and the code confirms, that total_edit_time_ms is wall-clock time from the first input to Confirm. Mid-editing pauses are included. Trailing idle before Confirm is included. Long pauses over 2.5 seconds are detected, and they only add to the effort score; they are never subtracted from the duration. The one thing excluded is the pre-first-keystroke reading period, tracked separately as initial_pause_ms.

This does not make 33.8% wrong. Both conditions are timed the same way, so the comparison is fair, and a wall-clock saving is arguably the number a translation buyer cares about anyway. What it changes is the interpretation. "Active editing duration excluding idle periods" claims the tool made the work faster. Wall clock from first keystroke to Confirm also improves if the tool made translators deliberate less, or commit sooner, or pause less because suggestions arrived unprompted. The paper's own Live Effort result and its task-order analysis, where the CHORUS slope is significantly steeper (b = -0.146, z = -6.66), are consistent with either reading.

A judgement, marked as one: this is a wording problem in the paper, not a validity problem in the study, and the fix is one sentence. It is also exactly the kind of thing that never surfaces when a system paper ships no code.

What we counted in the bundled profile

The release ships a demo memory profile at memory/demo_user/. We counted its events. There are 1,121 across seven buckets, and 1,107 of them are in consistency.

This is the concrete shape of the audit's fifth high-severity finding. The paper describes a general memories bucket for non-agent edits. There is no eighth bucket. Manual typing, undo and redo are tagged agent_dim: "manual" and the alias table sends them to consistency, which is also where the Audience Appropriateness agent's history goes. In the bundled profile the ratio is 1,100 manual to 7 agent events in that store.

Since the memory pipeline retrieves the top-ranked events from a bucket and injects them into the agent's prompt, a bucket that is 99.4% raw keystroke history is a different input than a bucket of curated agent interactions. Whether that helps or hurts is untested. It is simply not what the paper describes.

One more thing we measured rather than read. The four WMT24++ English-Chinese task pools in Tasks/ hold 155 literary, 130 news, 227 social and 111 speech segments, 623 in total, plus a combined file of 998 rows with 993 distinct sources. The study used 80 of them. Which 80 is not recorded in the release, so the exact stimuli are not reproducible from these files alone.

What this means for localization teams and for anyone publishing a translator tool

Take the system result first, because it is good. Seven inspectable MQM-aligned agents beat one chat window, for 30 certified translators, on speed, self-reported workload and two automatic quality metrics, with the translator deciding every edit and nothing auto-applying. That is the strongest evidence we have seen that decomposing LLM assistance by quality dimension is worth the engineering. The qualitative finding underneath it is the transferable one: translators said dimension-specific suggestions were easier to inspect than a broad rewrite, and that they stopped re-prompting to restate the same priorities.

• Buy on the decomposition, not on the 33.8%. The speed number is a wall-clock saving on 10 sentences per condition with a purpose-built interface against a chat box, and the interface and the agent architecture are confounded by the paper's own admission. The part that generalises is one quality lens at a time with token-level accept and reject.
• BLEU on 10 sentences per participant is a noisy instrument. A 3.08-point BLEU gain with Hedges g = 0.56 across 30 translators is a defensible group-level result, but 8 of 30 went the other way. If you are piloting a tool, expect a third of your translators to get no benefit.
• The prompts are the reusable asset. Thirteen prompt files, one per MQM dimension plus a router and a style-guide summariser, tracing to the Unbabel MQM Typology 3.0. That is a ready-made dimension taxonomy for any post-editing QA workflow, independent of the rest of the system.
• Ship the audit. This is the argument this release makes best. A 45-claim paper-against-code check cost these authors some embarrassment and bought their readers the ability to use the system correctly. Nobody could have found the timing-measure gap from the PDF.

The reverse pass is worth reading on its own. It lists behaviours that shaped the studied system and appear nowhere in the paper: voice input in both conditions, segment-level translation-memory injection into every agent prompt, an undocumented stress-derived boost inside the reported Live Effort scores, hidden canned intent suffixes on four of the agents, and heuristic fallback text that some participants may have read as personalised feedback. Any of those could move a result. None of them would be visible without the code.

Check it yourself

Everything above is in one clone and takes under a minute.

git clone https://github.com/iamgeorge/CHORUS-OSS && cd CHORUS-OSS

# the audit itself, 335 lines
sed -n '1,40p' docs/PAPER_CODE_AUDIT.md

# the implementation-paragraph checks
grep -n 'STYLE_GUIDE_SUMMARY_MODEL\|LLM_MAX_TOKENS\|CHATGPT_MODEL' Backend/app.py
grep -rn 'temperature' Backend/app.py          # one hit: "0", for transcription
grep -n '"manual"' Backend/app.py              # manual -> consistency

# the ranking endpoint, dead at audit time, live now
grep -n 'dimension-router' Frontend/components/AgentSidebar.tsx

# our bucket count
python3 -c '
import json, glob, collections, os
for f in sorted(glob.glob("memory/demo_user/*/events.jsonl")):
    ev = [json.loads(l) for l in open(f) if l.strip()]
    print(os.path.basename(os.path.dirname(f)), len(ev),
          dict(collections.Counter(e.get("agent_dim") for e in ev)))'

# the task pools
wc -l Tasks/*.csv

Expected: app.py:64 gives gpt-4o-mini, app.py:66 gives 2048, the temperature grep returns add_text_field("temperature", "0") at line 6753, and the bucket count prints consistency 1107 {'audience_appropriateness': 7, 'manual': 1100} against six buckets totalling 14. The task pools are 156, 131, 228 and 112 lines including headers.

The underlying data is WMT24++ English to Simplified Chinese, so the source segments are independently checkable even though the study's 80-sentence selection is not recorded.

What would prove this wrong

Our claim about the headline is narrow: that the reported 33.8% is a wall-clock saving including thinking time, not a saving in active editing time. The test is to re-derive it from the logs with idle trimming applied, subtracting every detected pause over the code's own 2.5-second threshold from both conditions.

We predict that by 31 March 2027, if that re-derivation is published, the idle-trimmed saving will be smaller than 33.8% by at least 5 percentage points, because CHORUS's suggestion cards remove deliberation pauses rather than typing. If the idle-trimmed saving comes back within 2 points of 33.8%, the descriptions are interchangeable and the audit's finding is cosmetic.

A second, independent test of the system claim rather than the measure: run the same seven-agent decomposition inside a plain chat interface, so the agent architecture and the purpose-built UI are no longer confounded. The paper says explicitly that it cannot separate them. If the decomposition alone reproduces even half the speed-up, the transferable finding is the MQM split; if it reproduces none, the finding is the interface.

Sources

1. George Xi Wang, Jiaqian Hu, Guande Wu, Jing Qian. CHORUS: Designing Human-AI Multi-Agent Collaboration for Professional Translators. arXiv:2602.19016, submitted 22 February 2026. Accepted at ACM ICMI 2026. Sections 5 and 6.
2. github.com/iamgeorge/CHORUS-OSS, the released system. docs/PAPER_CODE_AUDIT.md (335 lines, audit dated 20 July 2026), Backend/app.py, Frontend/components/AgentSidebar.tsx, Prompts/ (13 files), Tasks/ (4 domain pools) and memory/demo_user/. Last pushed 20 July 2026, retrieved 22 September 2026.
3. Daniel Deutsch et al. WMT24++. The English to Simplified Chinese segments used as study stimuli.
4. Arle Lommel et al. Multidimensional Quality Metrics. The error taxonomy the seven agents are aligned to, via Unbabel Typology 3.0 as cited in the prompts.
5. BLOMEGA verification, 22 September 2026: independent checks of the audit's code-side claims against app.py, AgentSidebar.tsx and Prompts/; event counts across the seven bundled memory buckets; row counts for the four task pools; and the arithmetic on the reported confidence interval and participant conversion.

FAQ

What is CHORUS?

A seven-agent workspace for professional translators, each agent aligned to one MQM quality dimension, proposing token-level edits the translator accepts or rejects. Published as arXiv:2602.19016 and accepted at ACM ICMI 2026, with code at github.com/iamgeorge/CHORUS-OSS.

What did the study find?

In a within-subject study with 30 licensed English-Chinese translators, CHORUS cut completion time by 33.8% against a chat-interface baseline (geometric mean ratio 0.662, 95% CI 0.567 to 0.772), lowered Live Effort from 65.84 to 54.85, and raised BLEU from 34.90 to 37.98 and COMET from 0.837 to 0.852. On each quality metric, 22 of 30 participants scored higher with CHORUS.

What is in the paper-code audit?

A claim-by-claim comparison of the paper against the shipped code, dated 20 July 2026, in docs/PAPER_CODE_AUDIT.md. Of 45 atomic claims: 22 MATCH, 16 PARTIAL, 4 MISMATCH and 3 NOT_IMPLEMENTED, with 12 graded high severity. Its header describes it as an internal review aid the authors could remove before release.

Does the audit undermine the results?

No. Both conditions were run the same way, so the comparisons stand. What it changes is interpretation, mostly on the timing measure: the paper describes completion time as active editing duration excluding idle periods, while the code records wall clock from the first keystroke to Confirm, with every mid-editing pause included.

What is the general memories bucket problem?

The paper says non-agent edits are stored in a general memories bucket. There is no eighth bucket. Manual edits are aliased to the consistency bucket, which is also where the Audience Appropriateness agent writes. In the profile bundled with the release, 1,100 of that bucket's 1,107 events are manual edits, and the other six buckets hold 14 events between them.

What is worth reusing from this release?

The 13 prompt files, one per MQM dimension plus a router and a style-guide summariser, traceable to Unbabel MQM Typology 3.0. They are a ready-made dimension taxonomy for a post-editing QA workflow independent of the rest of the system.