Reasoning models ignore the language you tell them to think in, and the ones that obey translate better

What Google published on 18 September 2026

Yuxiang Liu (University of Illinois at Urbana-Champaign, during a Google internship), Jiaming Luo, Eleftheria Briakou and Colin Cherry of Google DeepMind posted arXiv:2609.21247 on 18 September 2026, accepted to EMNLP 2026 Main. They prompted six large reasoning models to think in a named language before translating, then measured whether the model obeyed.

The setup: three datasets (WMT24++, CultureMT, DRT-Literature), nine dataset-language-pair combinations, six models, 16 samples per source at temperature 0.6 and top-p 0.95, scored with COMET-22, MetricX and MetricX-QE. The adherence measure is Reasoning-Language Instruction-Adherence, the fraction of traces a language-ID detector assigns to the instructed language. The prompt is one line: "Please always think in {reasoning language}."

Target-language reasoning barely works at all. For Qwen-32B, mean adherence to a target-language instruction is 28.5%, against 71.0% for English and 97.3% for Chinese. The paper restricts every later analysis to English and Chinese for that reason.

Five models, one rule

RLIA is the fraction of reasoning traces detected as the instructed language, on WMT24++ English-Chinese. COMET and Norm.MetricX are reported for the same setting. Source: arXiv:2609.21247, Tables 3 and 4. "Better reasoning language" is read off the COMET column.

The rule holds in all five cases: the language a model follows more reliably is the language it translates better in. The paper names this cognitive consistency and is careful not to call it causation, since underlying language proficiency plausibly drives both.

The direction transfers. The magnitude does not, and that is our finding rather than the paper's. gpt-oss-20B obeys a Chinese instruction 2.4% of the time and loses 0.0026 COMET for it. Gemma-4-E4B obeys 2.1% of the time and loses 0.1888, which is 22.4% of its English score. Two models that ignore the instruction at an indistinguishable rate, and a factor of 73 between what ignoring it costs them. Adherence is a direction test, not a damage estimate.

The metrics also disagree about the size of the Gemma-4-E4B collapse. COMET reads it as -0.1888. Norm.MetricX reads -0.0626 and Norm.MetricX-QE -0.0603. COMET puts the failure at roughly three times the size the MetricX family does. If you are running a language-selection sweep, which metric you picked decides whether this looks like a catastrophe or a bad afternoon.

Adherence is the cheap half of the experiment

This is the practical shape of the result. Measuring RLIA needs a language-ID pass over sampled traces. It needs no references, no COMET server, no reference-free metric, and it does not depend on having target-language evaluation data at all. Measuring quality needs the full scored run. The paper's own framing is that RLIA is an inexpensive diagnostic for identifying promising reasoning languages and not a causal predictor of quality, and our five-of-five sign agreement is exactly as strong as five models allow.

Where the metrics part company

Qwen-32B is the model the paper profiles in most detail. Instructing it in Chinese raises the mean across nine dataset-pair combinations on all three metrics. Per pair, it is less uniform.

Qwen-32B, COMET-22 by instructed reasoning language across the nine dataset-language-pair combinations of Table 2, arXiv:2609.21247. Bold marks the pairs where reasoning in English scored higher.

On COMET, Chinese wins 7 of 9. The two exceptions are WMT24++ English-German (-0.0171) and CultureMT English-French (-0.0040), the two pairs where neither the source nor the target is Chinese and the target is a European language. On Norm.MetricX and Norm.MetricX-QE, Chinese wins all nine. So the reference-based and reference-free MetricX variants agree with each other and disagree with COMET on two pairs out of nine, and both disagreements fall on the pairs furthest from the reasoning language.

Length is non-monotonic, and the better language costs more tokens

The second finding is about how much reasoning, not which language. The authors sample 16 traces per source, rank them by length, bin adjacent ranks into eight groups and score each bin. This compares long and short traces for the same source sentence, so source difficulty is controlled. Confidence comes from a 10,000-replicate, domain-stratified, document-cluster bootstrap. Quality is flat or improves modestly from short to intermediate lengths and falls off at the longest bins.

There is a controlled version too. Turning gpt-oss-20B's reasoning effort up nearly doubles its average trace length and produces only small, metric-dependent quality changes.

Put the two findings together and the decision has a price on it. For Qwen-32B, reasoning in Chinese is worth +0.0066 COMET, +0.0069 Norm.MetricX and +0.0076 Norm.MetricX-QE on WMT24++ English-Chinese. It costs 442 tokens in the shortest bin against 300 for English, a 47% premium, and a mean of 696 against 554 across the eight bins, +25.8%. That is a real inference bill for under one COMET point.

Domain cuts the other way from intuition. Social and literary inputs average 15.69 and 38.06 words; news and speech average 54.05 and 72.98. The short inputs elicit the longer traces. Whatever the models are budgeting reasoning against, it is not input length.

What the traces are actually doing

The structural half of the paper introduces Hierarchical Meta-Summarization. Gemini 2.5 Flash in non-thinking mode segments each trace into atomic steps and summarises each one; WMT24 traces average about 450 tokens and roughly 7 steps, giving nearly 860,000 step-level summaries. Those are subsampled into K = 128 subsets of 500, each summarised into candidate patterns, then embedded, clustered and consolidated.

What comes out is a three-stage organisation: Understanding and Planning, Translating and Drafting, Refining and Verifying. Gemma-4-E4B and Llama-8B recover it cleanly. gpt-oss-20B merges drafting and refinement. DRT-14B, the only MT-specialised model in the set, does not recover it at all: its clusters merge initial translation with problem identification and then emphasise refinement, revision, evaluation and critique. Fine-tuning a model on translation reasoning reorganises the stages around diagnosis and correction.

On allocation, most of the token budget goes to Translating and Drafting, and the ratios are stable across news, social, speech and literary even though absolute budgets vary. Chinese reasoning shifts relatively more budget to Understanding and Planning and less to the other two. On effectiveness, longer Understanding and Planning is the most consistently positive signal across domains; longer Translating and Drafting helps mainly on speech; longer Refining and Verifying is mixed, and the paper offers both readings, that hard examples trigger more verification and that excessive revision over-edits.

What to do with this in a localization pipeline

Run the language-ID check before you run the benchmark. One sampling pass plus langid per candidate reasoning language, then score only the winner. On this evidence the direction transfers every time, and the check costs a fraction of a scored evaluation.

Do not assume a reasoning language generalises across models. Chinese is right for Qwen-14B, Qwen-32B and Llama-8B and wrong for gpt-oss-20B and Gemma-4-E4B. Swapping the model in a pipeline means redoing the check, not inheriting the setting.

Instructing the target language is close to useless. 28.5% mean adherence for Qwen-32B, and as low as 4.6% on CultureMT English-Spanish. If your instinct is "think in the target language so it sounds native", the model is not doing it.

Cap reasoning length rather than encouraging it. Quality is flat to modestly better into the middle bins and degrades at the longest, and doubling gpt-oss-20B's effort bought almost nothing. A cap is cheaper than a nudge and the evidence points the same way.

Caveat that limits all of it: every quality number here comes from a learned metric, not a human. The authors say so. These are metric-aligned trends. Given that COMET and MetricX already disagree by a factor of three on the size of the largest effect in the paper, a human evaluation could reorder any of it.

Check it yourself

The adherence half reproduces on open weights without a metric server.

# the prompt the paper used, verbatim in shape
PROMPT = (
  'Please always think in {reasoning_language}.
'
  'Translate the following text from {source_language} to {target_language}:
'
  '{source_text}'
)

# 1. sample 16 traces per source at temperature 0.6, top_p 0.95
# 2. split the trace off the translation, run language ID on the TRACE
# 3. RLIA = fraction of traces detected as the instructed language

pip install langid
python3 - <<'EOF'
import langid, collections
def rlia(traces, instructed):
    c = collections.Counter(langid.classify(t)[0] for t in traces)
    return 100 * c[instructed] / len(traces)
# paper's values to check against, WMT24++ en-zh:
# Qwen-32B      en 64.2  zh 99.2
# gpt-oss-20B   en 98.3  zh  2.4
# Gemma-4-E4B   en 95.3  zh  2.1
EOF

Datasets: WMT24++ (Deutsch et al., 2025), CultureMT (Yao et al., 2023) and DRT-Literature (Wang et al., 2024). Models: DeepSeek-R1-Distill-Qwen-14B and 32B, DeepSeek-R1-Distill-Llama-8B, gpt-oss-20B, gemma-4-E4B-it and DRT-14B. The paper uses an in-house language-ID detector and notes it; any public langid will give a slightly different number, so compare the gap between languages rather than the absolute rate.

What would prove this wrong

A sixth model that breaks the sign rule. Five models is five models. A model with higher Chinese adherence that nonetheless translates better when instructed in English would kill the diagnostic outright. Our prediction, testable on any open reasoning model released before the end of 2026: the sign will hold and the magnitude will again be unpredictable from the adherence rate.

Human evaluation of the Gemma-4-E4B collapse. COMET says -0.1888 and MetricX says -0.0626. If a human MQM pass on the same outputs lands near the MetricX figure, then COMET is over-reading language-mixed output and the headline number in this article is the wrong one to quote.

The token premium. Our 25.8% is a mean over the eight bin lengths printed on Figure 1 for one model on one dataset. If per-sample token counts are released and the mean premium is under 10%, the cost side of the trade is smaller than we have described and switching reasoning language becomes close to free for Qwen-32B.

Sources

1. Yuxiang Liu, Jiaming Luo, Eleftheria Briakou, Colin Cherry, When Does Reasoning Help in Machine Translation? A Hierarchical Analysis of LRM Reasoning Traces, arXiv:2609.21247, 18 September 2026. Accepted to EMNLP 2026 Main. University of Illinois at Urbana-Champaign and Google DeepMind.
2. Daniel Deutsch et al., WMT24++: Expanding the language coverage of WMT24 to 55 languages and dialects, 2025, the source of the five language pairs used here.
3. Ricardo Rei et al., COMET-22, and Juraska et al., MetricX-QE, the three learned metrics all quality numbers in the paper are computed with.
4. Eleftheria Briakou, Jiaming Luo, Colin Cherry, Markus Freitag, Translating step-by-step: decomposing the translation process for improved translation quality of long-form texts, 2024, the pre-drafting / drafting / refinement / proofreading pipeline the induced three-stage structure resembles.

Related BLOMEGA guides: LLM translation noise rates · When human post-editing wins · Cohere North Small Translate, per task

FAQ

Should a reasoning model think in English or the target language?

Neither by default. In arXiv:2609.21247, the best reasoning language is model-specific and tracks which language instruction the model actually follows. Chinese is better for Qwen-14B, Qwen-32B and Llama-8B; English is better for gpt-oss-20B and Gemma-4-E4B. Instructing the target language works worst of all: mean adherence for Qwen-32B is 28.5%, against 71.0% for English and 97.3% for Chinese.

How do I pick a reasoning language without running a full evaluation?

Sample traces with the instruction in each candidate language and run language identification on the trace, not the translation. That gives the Reasoning-Language Instruction-Adherence rate, and across all five models in the paper the higher-adherence language is the better-translating one. It needs no references and no quality metric. It predicts the direction only, never the size of the effect.

Does longer reasoning produce better translations?

No, the relationship is non-monotonic. Ranking 16 samples per source by length and binning them, quality is flat or modestly better from short to intermediate traces and degrades at the longest bins. Separately, turning gpt-oss-20B's reasoning effort up nearly doubled its average trace length for only small, metric-dependent quality changes.

What does the reasoning trace of a translation model actually contain?

Applying Hierarchical Meta-Summarization to roughly 860,000 step-level summaries, general-purpose reasoning models share a three-stage organisation: Understanding and Planning, Translating and Drafting, Refining and Verifying. Most of the token budget goes to drafting. DRT-14B, the MT-specialised model in the set, does not recover the three stages and reorganises around diagnosis, revision and critique.

How much does switching reasoning language cost in tokens?

For Qwen-32B on WMT24++, Chinese traces run 442 tokens in the shortest rank bin against 300 for English, and a mean of 696 against 554 across the eight bins printed on Figure 1. That is a 25.8% token premium for a gain of 0.0066 COMET. The mean and percentage are our arithmetic on the paper's figure.