BabelArena: medium-resource languages break agent language, low-resource ones break the task
BabelArena, posted on 20 September 2026 by Alibaba Token Hub, runs five frontier agents through 702 tasks in 23 languages (16,146 instances). We recomputed its tier table: averaged over the five models, 71% of the Pass^3 loss lands between the medium and low tiers, while 74% of the language-consistency loss lands between the high and medium tiers. Medium-resource languages break the agent's language; low-resource languages break the task.
Pass^3 falls 3.68 points at the low tier but only 1.52 at the medium tier
BabelArena groups its 23 languages into 7 high-, 8 medium- and 8 low-resource languages (Table 2) and reports every model's Pass^1, Pass^3 and language consistency (LC) per group (Table 5). The paper describes the decline as general. Splitting it into two steps, which the paper does not do, shows the two metrics fail at different places.
Our arithmetic on Table 5, averaged over the five models: Pass^3 goes 30.84, 29.32, 25.64 across high, medium and low, so the steps are 1.52 and 3.68 points. LC goes 93.34, 87.74, 85.82, so the steps are 5.6 and 1.92. Reliability loss is back-loaded into the low tier; language drift is front-loaded into the medium tier.
For a team deciding which markets come next, that is two different gates, not one ladder. Our judgement from the numbers: the eight medium languages can ship as a batch if you police output language, and the eight low languages need a per-language task-success gate and, for some stacks, a different model.
What Alibaba Token Hub published on 20 September 2026
arXiv:2609.23490, BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents, by Kuang Peng, Yuchun Fan, Jiangnan Li, Minghao Wu and six co-authors (Alibaba Token Hub, UIUC, Northeastern, Monash, Ant International). A workflow called BabelFlow adapts four existing agent benchmarks, VitaBench, DeepPlanning, the text part of tau3-Bench and ClawEval, into 22 more languages, translating prompts, tasks, tool descriptions and environment content together and keeping tool names, IDs and enum codes byte-identical.
The tiers: high is English, Chinese, Japanese, French, Russian, Arabic and Spanish; medium is Korean, Indonesian, Hindi, Thai, Hebrew, Turkish, Vietnamese and Malay; low is Belarusian, Eastern Panjabi, Kazakh, Khmer, Burmese, Lao, Tamil and Telugu. The five agents (Claude-Opus-4.8, GPT-5.6-Terra, Gemini-3.1-Pro, Qwen-3.7-Max, Qwen-3.8-Max) run at medium reasoning effort, three trials per task, with Gemini-3.5-Flash as the simulated user and GPT-5.6-Sol as judge on three of the four families.
The instance count is internally consistent: 702 tasks times 23 languages is exactly 16,146, so every task exists in every language and the tier comparisons are on the same task set. No data is out yet. On 24 September 2026, GitHub repository search for "BabelArena" returned 0 results, Hugging Face dataset search returned none, and the Hugging Face paper API answered "Paper not found". Per-language scores therefore cannot be checked; the tier table is all anyone outside the authors has.
Every model loses reliability late and language early, except Qwen-3.7-Max
| Model | Pass^1 high / med / low | Pass^3 high / med / low | LC high / med / low | Pass^3 loss in low step | Relative Pass^1 loss, high to low | Source |
|---|---|---|---|---|---|---|
| Claude-Opus-4.8 | 51.3 / 48.1 / 44.8 | 34.3 / 33.9 / 29.6 | 90.4 / 87.2 / 86.7 | 91% | 12.7% | Table 5; last two columns ours |
| GPT-5.6-Terra | 45.3 / 43.9 / 36.8 | 26.3 / 25.1 / 21.9 | 98.8 / 90.9 / 89.6 | 73% | 18.8% | Table 5; last two columns ours |
| Gemini-3.1-Pro | 45.9 / 42.4 / 36.4 | 27.4 / 25.8 / 21.0 | 98.6 / 87.5 / 83.8 | 75% | 20.7% | Table 5; last two columns ours |
| Qwen-3.7-Max | 45.8 / 42.9 / 40.1 | 32.4 / 29.5 / 27.6 | 89.0 / 86.9 / 84.9 | 40% | 12.4% | Table 5; last two columns ours |
| Qwen-3.8-Max | 49.8 / 46.4 / 43.7 | 33.8 / 32.3 / 28.1 | 89.9 / 86.2 / 84.1 | 74% | 12.2% | Table 5; last two columns ours |
| Mean of 5 | 47.62 / 44.74 / 40.36 | 30.84 / 29.32 / 25.64 | 93.34 / 87.74 / 85.82 | 71% | 15.2% | our arithmetic |
| Any single language | not reported | not reported | not reported | not reported | not reported | paper reports tiers only |
Loss share = (medium minus low) divided by (high minus low). Relative loss = (high minus low) divided by high. All 15 model-metric series fall monotonically across the three tiers.
Claude-Opus-4.8 is the sharpest case: its Pass^3 moves 0.4 points from high to medium (34.3 to 33.9) and then 4.3 points to low, so 91% of its reliability loss is in the low tier. GPT-5.6-Terra loses only 1.4 Pass^1 points at the medium step and 7.1 at the low step, 84% of its Pass^1 decline. The Pass^3 steps for the other two are Gemini-3.1-Pro 1.6 then 4.8 and Qwen-3.8-Max 1.5 then 4.2. Qwen-3.7-Max is the one model with an even slope, 2.9 then 1.9, and on LC its steps are 2.1 and 2.0; Claude's LC steps are 3.2 and 0.5, Qwen-3.8-Max's 3.7 and 2.1.
LC runs the other way. GPT-5.6-Terra drops 7.9 LC points from high to medium and 1.3 from medium to low; Gemini-3.1-Pro drops 11.1 then 3.7. One caveat pushes against us and we checked its size: the high tier includes English, where a switch to English cannot happen, so English inflates the high-tier LC. If English LC is 100, the other six high languages average (7 x 93.34 minus 100) / 6 = 92.23, the medium step shrinks to 4.49, and the medium share is still 70%.
The paper's own switch annotations agree. In 2,000 language-inconsistent DeepPlanning and ClawEval trajectories, the expected language was high-tier in 24.3%, medium in 39.5% and low in 35.7% (Figure 3a). Per language, since each has the same 702 tasks, that is 4.05% for each non-English high language, 4.94% for each medium language and 4.46% for each low language: medium languages produce the most switches per language. That division is ours.
The model you pick for French is not the one to pick for Khmer
We counted pairwise order swaps in Table 5 across the ten model pairs. On Pass^1, 3 of 10 pairs swap between the high and medium tiers and 2 of 10 between high and low. The top two never move: Claude-Opus-4.8 first and Qwen-3.8-Max second in all three tiers.
The swaps matter because of what happens to the spread. In the high tier, places three to five sit within 0.6 points (Gemini 45.9, Qwen-3.7 45.8, GPT 45.3), so the ranking there is noise. In the low tier they spread to 3.7 points: Qwen-3.7-Max 40.1, GPT-5.6-Terra 36.8, Gemini-3.1-Pro 36.4. The spread from first to last is 6.0 points in the high tier, 5.7 in the medium tier and 8.4 in the low tier. A model shortlist built on English or French evaluation cannot see a gap that only opens in Tamil or Lao.
On LC, GPT-5.6-Terra is first in every tier (98.8, 90.9, 89.6), but 4 of 10 pairs swap between high and low, mostly around Gemini-3.1-Pro, which goes from second at 98.6 to last at 83.8. On Pass^3, only one pair swaps (GPT and Gemini, in the low tier).
The paper's text names the wrong leader for every tier
Section 4.2 says "Qwen-3.8-Max leads both task-completion metrics across all three groups". Table 5 says otherwise: Claude-Opus-4.8 is ahead on Pass^1 in every tier (51.3 vs 49.8, 48.1 vs 46.4, 44.8 vs 43.7) and on Pass^3 in every tier (34.3 vs 33.8, 33.9 vs 32.3, 29.6 vs 28.1). The leads are 0.5 to 1.7 points. Qwen-3.8-Max does lead VitaBench and DeepPlanning in Table 4, which is probably where the sentence came from. That is our reading, not the authors'.
Two smaller checks. The stated LC drop range "3.8 to 14.8 points" starts at 3.7 on Table 5 (Claude, 90.4 to 86.7). And if Table 5's tiers are weighted 7/8/8 by language count, four models reproduce their Table 4 family average within 0.54 Pass^1 points (Claude 47.93 vs 47.88, GPT 41.86 vs 41.85), but Qwen-3.8-Max comes out at 46.50 against 47.78. The paper does not state how either table is aggregated, so this is an open question for the authors, not an error we can prove.
Why the low tier breaks tasks and the medium tier breaks language
The low-tier task loss has a traceable cause in the paper's appendix. In DeepPlanning's shopping domain, product search ranks localized catalog records by lexical overlap, and 53.7% to 68.7% of low-resource searches come back empty against 6.1% to 17.1% in English (Table 11). One Telugu trajectory for Qwen-3.8-Max needed 158 product searches, 126 of them empty, to buy the same four products that English found in five. Low-tier total input tokens run 1.69 to 2.06 times English on VitaBench with turns at 0.97 to 1.16 times (Table 6).
The failure mix shifts too. In 1,600 labelled VitaBench failures, Thai (a medium language) and Tamil (low) show smaller outcome-error shares than English and Chinese and larger tool-use and control-flow shares, with looping prominent (Figure 2). Tamil's tool-use language failures are 7.8% of its sampled failures against 2.8% for English (Figure 6): the agent passes a localized value where the tool expects a canonical key.
The medium-tier language loss is a habit, not a comprehension problem. English is the destination of 91.2% of annotated switches; 13 of 20 inspected DeepPlanning responses open the <plan> block with English such as "Let me compile a travel plan now", and ClawEval agents preface tool calls with English lines like "Let me check the weather now". A single such line sets LC to 0 for the whole trajectory (Appendix A), which is why LC is sensitive where task success is not.
What it means for the order you localize an agent in
These are our judgements from a tier-level table; per-language data would sharpen or overturn them.
- Batch the medium tier, gate it on output language. Korean, Indonesian, Hindi, Thai, Hebrew, Turkish, Vietnamese and Malay cost 1.52 Pass^3 points on average against the high tier. The visible defect users will report is English leaking into replies and tool-call prefaces. A language check on every trajectory, including text before tool calls and inside structured outputs, is the release gate. Our note on reasoning-language adherence covers the same drift in translation models.
- Treat each low-tier language as its own launch. Pass^3 drops a further 3.68 points on average and up to 4.8 (Gemini-3.1-Pro). Run Pass^3 on your own tasks per language, and measure empty-search rates if the agent retrieves from a catalog: that was the largest cost driver in the paper.
- Re-run model selection for the low tier. A stack on GPT-5.6-Terra or Gemini-3.1-Pro loses 18.8% and 20.7% of its Pass^1 from high to low; Claude-Opus-4.8 and both Qwen models lose 12.2% to 12.7%. Choosing on English scores hides a 3.7-point gap among the bottom three.
- Budget per solved task, not per call. A rough proxy, ours, mixing Table 6 (VitaBench tokens) with Table 5 (all-family success) and using the high tier as the English stand-in: low-tier input tokens per solved task run about 1.9 to 2.4 times the high tier (Qwen-3.7-Max 1.69 x 45.8 / 40.1 = 1.93; Gemini 1.90 x 45.9 / 36.4 = 2.40).
- Fix retrieval before translation polish. Empty searches, not prose quality, drove the low-tier call explosion. Indexing native-language synonyms or cross-lingual retrieval addresses the cause directly.
A different lever is to route the whole agent through English and translate at the edges; our reproduction of the English-forcing tax in agent handoffs measures what that costs, so it is not repeated here.
Two confounds keep this provisional. Translation quality falls with the tier (4.28, 4.02, 3.91 on a 5-point scale), and the high tier was scored by human experts while medium and low were scored by GPT-5.6-Sol, so some of the low-tier gap may be benchmark translation, not agent capability. And the simulated user is Gemini-3.5-Flash in every language, so a weaker simulator in Lao or Khmer would depress every agent's low-tier score.
Check it yourself
The tier recomputation needs only Table 5 of the PDF (https://arxiv.org/pdf/2609.23490, page 7).
python3 - <<'PY'
T5 = { # (Pass^1, Pass^3, LC) for high, medium, low: arXiv:2609.23490 Table 5
"Claude-Opus-4.8": [(51.3,34.3,90.4),(48.1,33.9,87.2),(44.8,29.6,86.7)],
"GPT-5.6-Terra": [(45.3,26.3,98.8),(43.9,25.1,90.9),(36.8,21.9,89.6)],
"Gemini-3.1-Pro": [(45.9,27.4,98.6),(42.4,25.8,87.5),(36.4,21.0,83.8)],
"Qwen-3.7-Max": [(45.8,32.4,89.0),(42.9,29.5,86.9),(40.1,27.6,84.9)],
"Qwen-3.8-Max": [(49.8,33.8,89.9),(46.4,32.3,86.2),(43.7,28.1,84.1)]}
for k, name in [(0,"Pass^1"),(1,"Pass^3"),(2,"LC")]:
g = [sum(v[i][k] for v in T5.values())/5 for i in range(3)]
print(name, [round(x,2) for x in g], "low-step share %.0f%%" % (100*(g[1]-g[2])/(g[0]-g[2])))
PYExpected: Pass^3 [30.84, 29.32, 25.64] with a 71% low-step share; LC [93.34, 87.74, 85.82] with 26% in the low step, so 74% in the medium step.
Release check, as run on 24 September 2026:
curl -s "https://api.github.com/search/repositories?q=BabelArena" | grep total_count # 0
curl -s "https://huggingface.co/api/datasets?search=BabelArena" # []
curl -s "https://huggingface.co/api/papers/2609.23490" # Paper not found
python3 -c "print(702*23)" # 16146What would prove this wrong
Our claim is that the medium tier costs agents language and the low tier costs them tasks. It rests on tier averages; per-language results can break it.
Prediction: when BabelArena or an equivalent per-language table is published, and by 31 March 2027, at least 6 of the 8 medium-tier languages will sit within 3 Pass^3 points of the mean of the six non-English high-tier languages for the same model, and at least 6 of the 8 low-tier languages will sit more than 3 points below it. If three or more medium languages fall more than 3 points below the high tier on Pass^3, or if low-tier LC falls by more than medium-tier LC relative to high for a majority of models, the two-gate reading is wrong and a single ladder describes the data better.
A shorter test: if a rerun with native-quality human translation in the low tier (replacing GPT-5.6-Sol-scored translations averaging 3.91) closes more than half of the 3.68-point low-step Pass^3 loss, the low-tier cliff is a benchmark translation artefact, not an agent one.
Sources
- Kuang Peng, Yuchun Fan, Jiangnan Li, Minghao Wu, Jialong Tang, Haoran Wei, Weixuan Wang, Jianhong Tu, Baosong Yang, Tong Xiao. BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents. arXiv:2609.23490, submitted 20 September 2026. Tables 1, 2, 4, 5, 6, 7, 10, 11; Figures 2, 3, 6; Appendices A and B.
- Release checks, 24 September 2026: GitHub repository search for BabelArena (0 results); Hugging Face dataset search (empty); Hugging Face paper API ("Paper not found").
- BLOMEGA recomputation, 24 September 2026: tier steps and loss shares, pairwise rank swaps, English-adjusted LC step, per-language switch shares from Figure 3a, Table 4 versus Table 5 aggregation check, tokens-per-solved-task proxy.
FAQ
What is BabelArena?
A multilingual agent benchmark posted to arXiv on 20 September 2026 by Alibaba Token Hub and co-authors: 702 canonical tasks from VitaBench, DeepPlanning, tau3-Bench and ClawEval, each adapted into 23 languages, for 16,146 instances. Five frontier models are scored on Pass^1, Pass^3 and language consistency.
Which languages hurt agent performance most?
The paper reports results by resource tier only, not per language. Averaged over five models, Pass^3 falls 1.52 points from the high tier to the medium tier and 3.68 more to the low tier (Belarusian, Eastern Panjabi, Kazakh, Khmer, Burmese, Lao, Tamil, Telugu). Language consistency falls 5.6 points at the first step and only 1.92 at the second.
Is the BabelArena data released?
Not as of 24 September 2026. The paper describes a release covering 702 task identities, but GitHub repository search for BabelArena returns 0 results, Hugging Face dataset search is empty, and the Hugging Face paper page reports the paper is not linked from any repository.