One open model wrapped 97% of its translations in text that was not the translation
TransClean, published 10 September 2026 by the Language Technology Group at the University of Oslo, ran 792,000 translations through 12 LLMs across 22 language pairs with one identical prompt, then measured how often the output contained something other than the translation. The rate ranged from 3.30% for Qwen3-30B-A3B-Thinking-2507 to 97.32% for gemma-3-27b-it. Detecting that a segment is contaminated is 98.27% accurate. Recovering exactly the translation from it is 52.18% accurate. That gap is the part of an LLM localization pipeline nobody budgets for.
The noise rate got measured at scale for the first time on 10 September
10 September 2026. Shenbin Qian and Yves Scherrer, Language Technology Group, Department of Informatics, University of Oslo, published TransClean: A Benchmark for Detecting and Extracting Clean Translations from Large Language Model Outputs (arXiv:2609.11399). Code and data at github.com/shenbinqian/TransClean.
The generation run: 22 language pairs at varying resource levels and in both directions, 3,000 sentence pairs sampled per pair from four parallel corpora (the TED Multilingual Parallel Corpus, the WMT20 Quality Estimation dataset, the SwissAdmin corpus and a Chinese-Korean parallel corpus), 12 models, 3 prompt templates. The noise analysis uses the 792,000 outputs produced by Prompt 0, the template that induced the most noise.
The authors call the extra material translation noise and split it into content noise, which changes what the segment says, and formatting noise, which changes only how it is presented. Claude Opus 4.6 proposed the pattern taxonomy over the full output set and the authors verified it by hand; an independent pass with gemma-4-31B produced frequency estimates correlating at Spearman rho = 0.84. The benchmark itself is 9,900 paired noisy and clean outputs: 8,800 synthetically noised and 1,100 authentic examples curated by hand with silver clean translations.
Noise rate is a property of the model, and it varies by a factor of thirty
Table 1 is the headline measurement. Every model saw the same prompts on the same segments. Noise% is the share of outputs containing anything beyond the translation. Expl% isolates the subset that carries an explanation. WrongL% is the share where the model produced the wrong target language entirely.
| Model | Noise% | Explanation% | Wrong target language% | Source |
|---|---|---|---|---|
| gemma-3-27b-it | 97.32 | 97.32 | 31.82 | Table 1 |
| DeepSeek-V3.2-Exp-671B-chat | 60.72 | 59.17 | 11.65 | Table 1 |
| Qwen2.5-32B-Instruct | 55.62 | 53.93 | 15.22 | Table 1 |
| t5gemma-xl-xl-prefixlm-it | 46.32 | 33.88 | 25.41 | Table 1 |
| Llama-3.2-3B-Instruct | 40.58 | 29.47 | 21.67 | Table 1 |
| DeepSeek-R1-Distill-Qwen-32B | 26.75 | 22.22 | 11.92 | Table 1 |
| aya-expanse-32b | 16.63 | 15.03 | 4.93 | Table 1 |
| Tower-Plus-72B | 5.15 | 3.34 | 2.05 | Table 1 |
| Qwen3-4B-Instruct-2507 | 4.65 | 3.03 | 1.77 | Table 1 |
| Qwen3-30B-A3B-Instruct-2507 | 4.02 | 2.95 | 1.18 | Table 1 |
| Qwen3-4B-Thinking-2507 | 3.51 | 2.84 | 0.99 | Table 1 |
| Qwen3-30B-A3B-Thinking-2507 | 3.30 | 2.81 | 0.65 | Table 1 |
gemma-3-27b-it is not slightly chattier. Under one of the three prompts it reached 99.73%, meaning a clean translation was the exception at a rate of roughly one in four hundred. Manual inspection confirms it is explanatory text, not truncation or refusal. Its wrong-target-language rate of 31.82% is separately alarming: on nearly a third of segments the model produced text in a language other than the one requested.
Note the two 3B and 4B models on that list. Llama-3.2-3B-Instruct sits at 40.58% and Qwen3-4B-Instruct-2507 at 4.65%. Size does not predict this. Post-training does. The four Qwen3 models cluster between 3.30% and 4.65% regardless of parameter count or thinking mode, and Tower-Plus-72B, which is purpose-built for translation, sits at 5.15%.
Table 2 is the anatomy of the noise, so you can tell which of it a script can remove.
| Pattern | Share of noisy outputs | Category | What it looks like |
|---|---|---|---|
| explanation | ~33% | content | the translation, then **Explanation:** and a gloss of each phrase |
| alternative translations | ~15% | content | the translation, then Alternative translation: and a second one |
| off-topic response | ~12% | content | "I am an AI assistant designed to be helpful..." |
| verbose preamble | ~9% | formatting | Here is the translation of "..." from English to Chinese: |
| bilingual output | ~7% | content | source language label, the source, then the target |
| JSON-style quoting | ~7% | content | "translation": "..." |
| language prefix | ~7% | formatting | Arabic: then the text |
| extra punctuation | ~3% | formatting | trailing !!!! |
| code block | ~3% | formatting | the translation inside triple backticks |
| special formatting | ~2% | formatting | [["Australian Shepherd"]] |
| translation prefix | ~1% | formatting | Translation: then the text |
| cultural note | ~1% | content | the translation, then a parenthetical "(Note: ...)" |
Add the content rows: roughly 75% of noisy outputs carry content noise. Those are the ones a regex cannot safely remove, because removing them requires deciding which of two candidate translations was meant, or where a gloss stops and the translation resumes.
Finding the noise is easy, cutting it out is not, and the second number is the one that hits production
The paper evaluates two cleanup strategies. The span-based method uses a translation quality estimation model to locate the span that is not translation, then cuts it. The LLM-based method prompts a model, either Qwen or Aya, to isolate the translation. Accuracy is exact match after whitespace stripping and Unicode NFC normalisation, which is strict and also exactly what a translation memory needs.
| Method | Detection, synthetic | Extraction, synthetic | Detection, authentic | Extraction, authentic | Source |
|---|---|---|---|---|---|
| Span-based (quality estimation) | 99.68 | 50.59 | 96.36 | 15.82 | Table 4 |
| Qwen extractor | 98.38 | 54.07 | 96.73 | 52.18 | Table 4 |
| Aya extractor | 89.94 | 36.69 | 99.73 | 51.36 | Table 4 |
| Rule-based detector | 87.26 | not applicable | 98.27 | not applicable | Table 4 |
The span-based method collapses from 50.59 to 15.82 the moment it meets noise it was not designed for. That is the generalisation warning: a cleanup rule tuned on the noise you have seen does not survive the noise you have not. The LLM extractors hold roughly steady across the two subsets, 54.07 to 52.18 and 36.69 to 51.36, which is the argument for using a model rather than a pattern here, even though a model costs a second inference per segment.
Read the "already clean" column. The span-based method scores 100 there because it leaves untouched anything it does not flag. Aya scores 59.91, because it paraphrased already-correct translations about 40% of the time, 892 of 2,200 cases, altering wording, punctuation or structure. A cleanup stage that silently rewrites 40% of your good segments is worse than no cleanup stage, and it is invisible to a detector-only metric.
On combined noise, where a preamble and an explanation and a stray code fence arrive together, the best method reaches 14.95%.
What a localization engineer should change on Monday
Measure your own noise rate before you compare models. The TransClean rule-based detector reaches 98.27% on authentic outputs, which is good enough for a dashboard. If you are benchmarking two LLMs for a translation step and one of them is at 40% noise, your quality comparison is partly measuring formatting habits. The paper's framing is that cleanliness and quality are separate axes, and most internal evaluations conflate them.
Treat the extraction stage as a lossy step with a published loss rate. 52.18% exact match on authentic noise means roughly 47.82% of contaminated segments reach the next stage still contaminated. If those segments are going into a translation memory, they are permanent: an explanation stored as a target segment will be fuzzy-matched and re-proposed to a human translator for years.
Prefer a model whose noise rate is already low over a cleanup stage. This is the cheapest available fix and the measurement is unambiguous. Four Qwen3 variants and Tower-Plus-72B all sit between 3.30% and 5.15%. Choosing one of those over gemma-3-27b-it removes 94 percentage points of the problem without writing any code. If a model must be used for other reasons, constrained decoding or a structured output schema moves the problem to the formatting category, which strips at 89.91% rather than 17.68%.
Instrument the wrong-language rate separately. It is not a subset of chattiness. gemma-3-27b-it at 31.82%, t5gemma-xl at 25.41% and Llama-3.2-3B-Instruct at 21.67% mean roughly a quarter of segments are in the wrong language, which no translation-quality metric will flag as catastrophic and which every end user will notice immediately. A language identifier on the output costs microseconds.
A judgement, marked as one: the reason this went unmeasured until September 2026 is that the people running LLM translation in production strip the junk with a regex on day one, decide it works because the obvious cases disappear, and never sample the remainder. The authentic-noise extraction column, 15.82% for the pattern-based approach, is what that regex actually scores once the noise stops matching the shapes you happened to see first.
Check it yourself
The benchmark, the noise taxonomy and the detector are public. The fastest reproduction is to measure your own stack rather than theirs.
git clone https://github.com/shenbinqian/TransClean
cd TransClean
# 1. the benchmark itself: 8,800 synthetic + 1,100 authentic noisy/clean pairs
ls data/
# 2. run the rule-based detector over YOUR last 10k production outputs
# and report the share flagged. That is your Noise%.
# 98.27% accuracy on authentic noisy outputs (Table 4).
# 3. the strict metric the paper uses, so your numbers are comparable:
python3 - <<'PY'
import unicodedata
def norm(s): return unicodedata.normalize("NFC", s.strip())
def acc_ext(pred, gold): return sum(norm(p)==norm(g) for p,g in zip(pred,gold))/len(gold)
PYThe claim worth testing on your own data is the 12-pattern taxonomy in Table 2. Sample 200 flagged outputs, label them against those twelve, and see whether your distribution matches the published one, roughly 33% explanation and 15% alternatives. If your mix is mostly formatting, a deterministic cleaner is defensible. If it is mostly content, it is not, and the 17.68% content-noise extraction accuracy is the number to plan around.
One caveat the authors state plainly: the pattern frequencies in Table 2 are LLM-estimated over a set too large to hand-count, cross-checked at Spearman rho = 0.84 against a second model. Treat them as a distribution, not as counts.
What would prove this wrong
The claim under test is that extraction, not detection, is the binding constraint, and that it is hard because content noise is semantically entangled with the translation. It is wrong if, by 10 September 2027, a published method reaches 80% or better exact-match extraction accuracy on the TransClean curated subset of 1,100 authentic examples. The 2026 best is 52.18%. The benchmark is public and the metric is exact match after NFC normalisation, so this is directly checkable rather than a matter of interpretation.
A second prediction, marked as judgement: no major commercial translation API will publish a per-model noise rate before 10 September 2027. The measurement is trivial, the numbers are unflattering for at least one widely deployed open model, and nobody is currently required to report it. If a vendor publishes one, this reading of the incentives was wrong.
FAQ
How often does an LLM add extra text to a translation?
Between 3.30% and 97.32% of outputs, depending entirely on the model, measured over 792,000 outputs from 12 LLMs on 22 language pairs under an identical prompt. DeepSeek-V3.2-Exp measured 60.72%, Qwen2.5-32B-Instruct 55.62%, Tower-Plus-72B 5.15%.
What kinds of junk do LLMs add?
Twelve patterns in two groups. Content: explanations (~33%), alternative translations (~15%), off-topic assistant replies (~12%), bilingual output (~7%), JSON-style quoting (~7%), cultural notes (~1%). Formatting: verbose preambles (~9%), language prefixes (~7%), extra punctuation (~3%), code blocks (~3%), special bracketing (~2%), "Translation:" prefixes (~1%).
Can you just strip it with a regex?
You can detect it that way, at 98.27% accuracy on authentic outputs. You cannot reliably cut it out that way: the span-based pattern approach scored 15.82% exact match on authentic noise, against 52.18% for an LLM extractor. Formatting noise strips at 89.91%, content noise at 17.68%.
Does translation noise distort benchmark scores?
Yes. Extra text is scored as part of the translation by overlap and neural metrics alike, so a model that translates correctly but prefixes "Here is the translation of" is penalised for a habit rather than an error. That is the motivation the authors give for separating cleanliness from quality.
Which models are safest for an unattended translation step?
On this measurement, the four Qwen3-2507 variants (3.30% to 4.65%) and Tower-Plus-72B (5.15%). The riskiest tested was gemma-3-27b-it at 97.32%, reaching 99.73% under one of the three prompts.
Sources
- Shenbin Qian and Yves Scherrer, University of Oslo, TransClean: A Benchmark for Detecting and Extracting Clean Translations from Large Language Model Outputs, arXiv:2609.11399, submitted 10 September 2026. Table 1 (noise, explanation and wrong-language rates for 12 models across 3 prompts); Table 2 (twelve noise patterns and frequencies); Table 3 (annotator agreement, 40.73% unanimous); Table 4 (detection and extraction accuracy); Table 5 (extraction by noise category). PDF.
- shenbinqian/TransClean on GitHub. The 9,900-pair benchmark, the rule-based detector and the extraction baselines.
- Source corpora for the generation run: the TED Multilingual Parallel Corpus (Kulkarni, 2015), the WMT20 Quality Estimation dataset (Barrault et al., 2020), the SwissAdmin corpus (Scherrer et al., 2014) and the Chinese-Korean parallel corpus (Park and Zhao, 2019).
Related BLOMEGA guides: Multilingual LLM judges and translationese bias · The per-language token ceiling · Localization is the new default.