The English-forcing tax is 30.6 points for Hindi, and about 18 once you pay for the extra hop

Three agents are compared against two, and the extra agent is charged to English

A paper posted to arXiv on 14 September 2026 and accepted at the 2nd Workshop for Research on Agent Language Models at EMNLP 2026 measures what it calls the English-Forcing Tax: routing the internal handoffs of a multi-agent pipeline through English costs 13.0 percentage points of Exact Match for Spanish and 30.6 points for Hindi, against an otherwise identical native-language pipeline. We reproduced every number in its Table 1 from the reported discordant pairs, to the item. All four McNemar p-values recompute exactly.

The comparison also buys an extra agent. The native condition runs two agents; the English-forced condition runs three, because it adds a back-translator. The paper measures the price of the second agent directly, at 8.3 to 12.7 points. Charge the third agent the same price and the residual that is actually attributable to English is 4.7 points for Spanish, 5.0 for Arabic, 16.0 for Chinese and 18.0 for Hindi. The direction and the typological ordering survive. The headline magnitude does not.

What was published, and by whom

arXiv:2609.15079, Translating the Translator: Decomposing the Cost of English-Forced Inter-Agent Communication, by Kushagra Agrawal (Åbo Akademi University), Yuming Feng (Chongqing Sanxia University of Science and Technology) and Man-Fai Leung (Anglia Ruskin University). Submitted 14 September 2026, CC BY 4.0, supported by the Natural Science Foundation of Chongqing (CSTB2024NSCQ-LZX0083).

The setup: XQuAD, 300 questions each in Spanish, Arabic, Chinese and Hindi. The model is aya-23-8b-8bit running under MLX on a 16 GB MacBook Air, greedy decoding at temperature 0. The whole study is 3,600 pipeline executions and roughly 7,200 generations, about 8 hours sequential. No code or output release is named in the paper.

Three conditions. S is one agent that reads the native passage and answers natively. N adds a second agent: Agent A extracts facts natively, Agent B answers from those facts. E takes the same extracted facts, translates them to English, answers in English, and adds Agent C to back-translate the answer into the user's language. Everything is scored against the native-language gold answer.

The table, and our recomputation of it

Exact Match is a count out of 300, so every reported rate pins down an integer. Spanish E at 0.423 can only be 127/300. From those counts, b minus c has to equal the accuracy difference in items, and it does in all four languages: 39, 50, 83, 92. We then computed the exact two-sided binomial p-value from the reported b and c.

Table 1 is internally consistent to the item. One thing is not. Table 2 of the same paper reports the chrF diagnostic split by outcome and gives 322 successes and 878 failures over the 1,200 Condition E executions. Summing the four Condition E Exact Match counts gives 127 + 79 + 70 + 41 = 317. The two tables disagree by 5 items, 0.4% of the sample. It does not move the conclusion, and we could not resolve which figure is the corrected one without the raw logs, which are not released.

The third agent is not free, and the paper already priced it

The paper's own S-to-N comparison is a clean measurement of what one extra natural-language handoff costs when nothing about the language changes: 8.3 points for Spanish, 11.7 for Arabic, 11.7 for Chinese, 12.7 for Hindi. Those are large. Condition E adds a third agent on top of the two in Condition N. Nothing in the design isolates that third hop from the English pivot.

So we built the most conservative correction available from the paper itself. Assume the third agent costs exactly what the second one cost in that language, and subtract it from the reported tax. This is an assumption, and we mark it as one: back-translating a short answer is probably easier than extracting facts from a passage, which makes 100% of the second hop's price an upper bound on the third hop's price, and therefore makes our residual a lower bound.

Percentage points out of 300 questions per language. The paper prints the taxes as 13.0, 16.7, 27.7 and 30.6; the underlying item counts give 13.00, 16.67, 27.67 and 30.67, so our Hindi residual is 18.00 points against 17.9 if you subtract from the printed value.

Two things follow. For Spanish the honest number to plan around is closer to 5 points than 13, which changes whether a native-routing rewrite pays for itself. For Hindi it is closer to 18 than 31, which does not change the decision at all: 18 points of Exact Match is still the largest single lever in that pipeline.

The discordant pairs support the direction independently of any hop accounting. Hindi has 106 questions the native pipeline gets right and the English pipeline gets wrong, against 14 the other way. That 7.6:1 asymmetry is not noise, whatever share of it the third agent owns.

The typological gradient is four points

The paper reports that the tax grows with typological distance from English: Spanish 13.0, Arabic 16.7, Chinese 27.7, Hindi 30.6. The ordering is clean and it survives our hop correction (4.7, 5.0, 16.0, 18.0). It is also four data points. Under a permutation null, the chance that four languages fall in a pre-specified order by accident is 1 in 24, or 4.2%. That is one coin-flip short of the conventional threshold, from a single model, on a single benchmark.

The paper says so itself, and names the confound plainly: language distance is entangled with how much of each language Aya-23 saw. Our view, marked as a judgement: the sign is well established here, the ranking is suggestive, and the specific per-language magnitudes should not be carried into a business case without a second model.

One more number worth keeping. For Chinese the Exact Match penalty is 27.7 points but the token-F1 penalty is 18.1, a 9.6-point gap that appears in no other language. Chinese failures keep partial overlap with the reference. Hindi failures do not (30.6 EM against 27.9 F1). If you score a Chinese pipeline on Exact Match you will read a harsher verdict than a human would.

What this means if you are building multilingual agents

The default in LangChain, AutoGen and CrewAI is English for tool schemas, system prompts and inter-agent messages, regardless of the user's language. This paper is the first measurement we have seen that isolates the cost of that default rather than the cost of being non-English in general. Even at the conservative end it is a double-digit penalty for Hindi and Chinese.

Three practical reads:

• Count your hops before you blame the language. Every natural-language handoff costs 8 to 13 Exact Match points in this setup, in any language. If your pipeline has five agents, the orchestration bill is likely larger than the translation bill. Shortening the chain is a cheaper fix than changing its language, and you can test it in an afternoon.
• Native routing is a config change, not a research programme. Agent A already reads the native passage successfully. The loss starts when its output is forced into English. Removing the forward translation removes the back-translation too, which is where two of the paper's four named failure modes live.
• Exact Match is the wrong acceptance metric for Chinese. The 9.6-point EM-minus-F1 gap means a Chinese pipeline that looks 28 points worse is about 18 points worse by token overlap. Pick the metric before you pick the architecture.

The failure examples are the most portable part of the paper. "Kony Ealy" became "Be Like Eli", and the English-side agent then confidently produced "Eli Manning". "Sierra Freeway" became "Sierra Highway". Named entities and local administrative terms are where a forward translation silently destroys the answer, and they are exactly the content a glossary or a do-not-translate list already covers in a normal localization pipeline. Nobody had wired one into the agent layer here.

Check it yourself

The paper releases no code or data, so the checkable part is the internal consistency of its tables. Everything below runs on the published numbers in about a second.

from math import comb

# Table 1 of arXiv:2609.15079: (S, N, E) correct out of 300, and (b, c)
D = {'es': (191,166,127,66,27), 'ar': (164,129,79,67,17),
     'zh': (188,153,70,102,19), 'hi': (171,133,41,106,14)}

for lang, (S, N, E, b, c) in D.items():
    p = 2*sum(comb(b+c, i)*0.5**(b+c) for i in range(min(b, c)+1))
    hop, tax = (S-N)/300, (N-E)/300
    print(f"{lang}  b-c={b-c:3d}  N-E items={N-E:3d}  "
          f"p={p:.3e}  tax={100*tax:.1f}pp  residual={100*(tax-hop):.1f}pp")

print("Condition E successes summed:", sum(v[2] for v in D.values()))  # 317
print("Table 2 reports:", 322)

Expected output: b minus c equals N minus E in every language; the p-values are 6.467e-05, 3.496e-08, 6.365e-15 and 1.157e-18, matching the paper's 6.47e-05, 3.50e-08, 6.37e-15 and 1.16e-18; the residuals are 4.7, 5.0, 16.0 and 18.0 points; and the Condition E success count is 317 against Table 2's 322.

To replicate the study rather than the table: XQuAD is at github.com/google-deepmind/xquad, 1,190 question-answer pairs per language across 11 languages. The model is CohereLabs/aya-23-8B. The paper's Appendix A carries the three prompt templates. The missing fourth condition, and the one that would settle this, is N plus a no-op third agent in the native language.

What would prove this wrong

Our claim is that the third generation hop, not the English pivot, accounts for roughly two thirds of the reported Spanish and Arabic tax and roughly 42% of the Chinese and Hindi tax. The test is a four-condition design: add Condition N3, identical to N but with a third native-language agent that paraphrases the answer, and compare N3 against E.

We predict that by 31 March 2027, a study running that control on XQuAD or a comparable QA set will report an N3-versus-E gap under 8 percentage points for Spanish and above 12 points for Hindi. If the N3-versus-E gap comes back within 2 points of the published N-versus-E gap for Spanish, our hop correction is wrong and the full 13.0 points belong to English routing.

A second, cheaper falsification: run the same three conditions on an unquantized model. The paper uses an 8-bit Aya-23-8B and says the direction of the quantization effect cannot be determined without a control. If the tax shrinks by more than half at full precision, the number being measured is partly a quantization artifact.

Sources

1. Kushagra Agrawal, Yuming Feng, Man-Fai Leung. Translating the Translator: Decomposing the Cost of English-Forced Inter-Agent Communication. arXiv:2609.15079, submitted 14 September 2026. Accepted at the 2nd Workshop for Research on Agent Language Models, EMNLP 2026. Tables 1 and 2, Sections 3 to 6.
2. Mikel Artetxe, Sebastian Ruder, Dani Yogatama. On the Cross-lingual Transferability of Monolingual Representations. ACL 2020. The XQuAD benchmark used here.
3. Viraat Aryabumi et al. Aya 23: Open Weight Releases to Further Multilingual Progress. arXiv:2405.15032, 2024. The model under test.
4. Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. WMT 2015. The diagnostic metric in the paper's Table 2.
5. Thomas G. Dietterich. Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation 10(7), 1998. The McNemar procedure we recomputed.
6. BLOMEGA measurement, 22 September 2026: recomputation of Table 1's exact binomial p-values and item counts, the Table 1 versus Table 2 discrepancy of 5 items, and the same-price third-hop decomposition.

FAQ

What is the English-Forcing Tax?

The drop in Exact Match accuracy when a multi-agent LLM pipeline routes its internal handoffs through English instead of the user's language. In arXiv:2609.15079 it is 13.0 percentage points for Spanish, 16.7 for Arabic, 27.7 for Chinese and 30.6 for Hindi, measured on 300 XQuAD questions per language with Aya-23-8B.

Does the measurement isolate English routing?

Not completely. The English-forced condition runs three agents and the native condition runs two, because English-forcing adds a back-translator. The paper measures the cost of adding the second agent at 8.3 to 12.7 points. If the third agent costs the same, the residual attributable to English is 4.7 points for Spanish, 5.0 for Arabic, 16.0 for Chinese and 18.0 for Hindi. That is our arithmetic, not the paper's.

Do the paper's statistics hold up?

Table 1 is internally consistent to the item: b minus c equals the accuracy difference in questions for all four languages, and the exact binomial p-values recompute to 6.467e-05, 3.496e-08, 6.365e-15 and 1.157e-18. Table 2 reports 322 Condition E successes where Table 1's counts sum to 317, a 5-item disagreement we could not resolve.

Should I route my agents in the user's language?

For typologically distant languages the evidence points that way, and it is a configuration change rather than a research programme. But count your generation hops first: every extra natural-language handoff costs 8 to 13 Exact Match points in this setup regardless of language, so a shorter chain may buy more than a different language.

How strong is the typological-distance claim?

It is four languages. The ordering Spanish, Arabic, Chinese, Hindi is clean and survives our correction, but the chance of four points falling in a pre-specified order by accident is 1 in 24. The paper also notes the confound with how much of each language the model saw in training.