A language lookup table matches the pooled probe router in arXiv 2610.06216, on the paper's own numbers

The routing gain in arXiv 2610.06216 is almost all available from the language tag alone

The calibration result is sound and we reproduced it from the paper's figures. The English-trained Qwen3-4B probe has a macro expected calibration error (ECE) of 0.125 over nine non-English languages; the mean of the nine per-language values printed on Figure 2 is 0.1251. The pooled probe's nine values average 0.0434 against a reported 0.043. Across all 8 model configurations in Table 10, pooling cuts ECE by 37.7% to 68.6%, which rounds to the "38 to 69%" in the introduction.

The routing claim is where our check diverges. Table 9 gives each model's mean test success in each language. Taking those 80 numbers and Table 7's relative costs, a lookup router that sends English, Chinese, Russian and Swahili prompts to Qwen3.5-9B and the other six languages to gpt-oss-high scores 86.6% success at 12.8% saving. The paper's pooled probe router, at its validation-selected maximum-success point, scores 86.6% at 13.0% (Table 4). At a higher cost weight the lookup reaches 86.0% at 22.4%, against the probe router's anchor-matching point of 85.5% at 22.9%.

The lookup is in-sample: it is fitted on the same test means it is scored on, so it flatters itself. To price that optimism we re-ran it 5,000 times with every (language, model) success rate perturbed by the sampling noise of a 450-problem validation split, chose on the noisy table, and scored on Table 9. It averages 86.3% success at 11.2% saving at the lambda 0 setting (5th to 95th percentile 85.9% to 86.6%), and 85.4% at 25.2% at lambda 0.10. Our judgement: on this benchmark, per-prompt probes add at most about half a point of success over knowing the language, and the paper reports no language-only baseline that would show it either way.

What the University of Queensland team published on 5 October 2026

arXiv:2610.06216, Cross-lingual Calibration of Pre-Generation Success Probes for Multilingual LLM Routing, by Andrea Paganelli (University of Queensland and Politecnico di Milano), Stefano Civelli, Pietro Bernardelle and Gianluca Demartini (University of Queensland). The arXiv API lists v1 only, submitted 2026-10-05T12:26:05Z, cs.CL. It extends Lugoloobi et al. (arXiv:2602.09924), who showed that linear probes on pre-generation activations predict a model's success and can drive cost-aware routing, from English to ten languages.

The setup: 3,000 MATH problems, translated by DeepL into Arabic, Chinese, French, Italian, Russian, Swahili, Telugu, Thai and Turkish with math notation protected, split 70/15/15 by underlying problem (2,100 train, 450 validation, 450 test). Eight open-weight configurations (Qwen3 0.6B, 1.7B, 4B, 8B; Qwen3.5 4B, 9B; gpt-oss-20b at low and high reasoning) each produce 5 samples per prompt at temperature 0.7, so each label is a success fraction in steps of 0.2. Ridge probes read the residual stream at the last prompt token, before any output token. Three supervision regimes are compared: English only (2,100 English prompts), pooled (210 prompts in each of ten languages, 2,100 total) and a same-language oracle (2,100 in the target language).

Routing uses an encoder-target decoupled design from Varshney et al. (2026): one Qwen3-4B prefill, layer 23, feeds eight ridge heads that each predict one target model's success, and the router maximises predicted success minus lambda times cost over a 10-point lambda grid from 0 to 1. Costs are a proxy, not measured: $0.10 per GB of deployed memory footprint per million output tokens, input at one sixth of that, multiplied by each model's average token counts (Appendix D).

The paper's numbers next to our recomputations

All paper values from arXiv:2610.06216v1. Means, percentages, lookup routing, Monte Carlo and price sensitivity are BLOMEGA computations, 8 October 2026, reproduced by headline.py (section h-check).

Every arithmetic chain we could rebuild holds. Table 7's expected cost per thousand generations equals input price times input tokens plus output price times output tokens, to the fourth decimal for all eight models, and the Qwen3-4B prefill share of 0.40% in Table 8 follows from it. Table 9's column means match to rounding. The confidence interval on the pooled router's 86.6% is a level interval and contains the 85.9% anchor; the paper does not print a paired interval on the 0.7-point difference, so the success gain over always using gpt-oss-high is not shown to be distinguishable from zero. The 13.0% saving interval does exclude zero.

The English-trained ETD heads are weakest exactly where the router needs them. For the two gpt-oss targets, English-trained heads score AUROC 0.545 and 0.584 (near chance) with NLL of 1.350 and 2.110, against 0.693 and 0.712 when pooled (Table 11). That explains the English raw router's 80.8% success, 5.1 points under the single-model anchor: it cannot tell when the expensive model is needed.

Miscalibration does not follow the success gap. Chinese has the highest non-English success for Qwen3-4B (0.630) and the second-worst English-probe ECE (0.187); Telugu has the second-lowest success (0.322) and one of the better ECEs (0.071). For Qwen3-0.6B the spread is wider still, 0.033 in Swahili to 0.329 in Chinese, a factor of 10.0 by our division of the Figure 4 labels. A team cannot predict from a language's difficulty how far an English-fit score will drift there.

How a pre-generation router chooses, and what a lookup table already knows

Both routers decide before decoding. The ETD router pays for one Qwen3-4B prefill (0.40% of the anchor's cost) and gets a per-prompt estimate for each of 8 models. The lookup pays nothing per prompt and gets one estimate per language and model. The paper's routing gain over the anchor comes from exploiting differences between models, and on this benchmark the largest of those differences sit at the language level, which a table with 80 cells captures.

The language-level structure is visible directly in Table 9. Qwen3.5-9B, at 0.679 of gpt-oss-high's modelled cost, is ahead in English (0.935 vs 0.925), Chinese (0.898 vs 0.886), Russian (0.889 vs 0.884) and Swahili (0.683 vs 0.647). It is far behind in Telugu (0.580 vs 0.849) and Thai (0.715 vs 0.826). Any router that sends the four winning languages to the cheaper model and keeps the rest on gpt-oss-high earns both a small success gain and a 12.8% saving with no model internals at all.

The cost side carries a second dependency. Dividing each output price in Table 7 by the stated $0.10 per GB gives footprints of 0.60, 1.70, 4.00, 8.00 and 9.00 GB for the Qwen models, equal to their parameter counts in billions, which corresponds to about one byte per parameter (8-bit weights), and 12.24 GB for gpt-oss-20b in its native quantised format. The paper says the proxy uses "the precision used for inference" without naming the Qwen precision. If those models were served at BF16, every Qwen price doubles, Qwen3.5-9B becomes 1.358 times as expensive as gpt-oss-high, and the same lambda 0 routes cost 14.3% more than the anchor. That is our sensitivity test on the lookup router; the paper's own frontier would shift the same direction, by an amount we cannot compute without its per-prompt choices.

What this means for teams routing or scoring multilingual traffic

• Never threshold an English-fit score in another language without target-language labels. The paper's clearest result transfers beyond math: ranking survived (AUROC 0.729 vs 0.758) while probability scale did not (ECE 0.125 vs 0.031). The same caution applies to quality-estimation scores, toxicity scores and any confidence gate in a localisation pipeline.
• A small labelled set per language beats a large English one. 210 labelled prompts in each of ten languages beat 2,100 English prompts on ECE for all 8 models and on AUROC for all 8. For a data budget, that argues for breadth across languages over depth in English, which is a collection-planning decision, not a modelling one.
• Build the language-level baseline first. Our recomputation shows it captures nearly all of the measured routing gain here. A per-prompt router earns its prefill cost and its engineering only if it beats a table indexed by language (and, in a dubbing or support setting, by content type).
• Pin the cost model before trusting a saving. The headline 13.0% depends on a memory-footprint proxy whose Qwen precision is unstated; at BF16 the cheaper-model routes stop being cheaper. Measure tokens per language on your own traffic, since Table 7 averages token counts across all ten languages.
• Machine-translated test sets are a ceiling on what you learn. Nine of ten languages here are DeepL output (mean COMET-Kiwi 0.8065), and the authors list this as a limitation. COMET-Kiwi's rank correlation with mean model success across the nine languages is 0.58 by our calculation, with Turkish an outlier (third-highest COMET-Kiwi at 0.8284, sixth of nine on success). The Unbabel/wmt22-cometkiwi-da checkpoint is gated and licensed CC BY-NC-SA 4.0 on Hugging Face, which matters for commercial QA use.

For how English-first evaluation hides the same Swahili gap in agent tooling, see our note on agent skills that score zero in Hindi and Swahili; for judge drift across languages, multilingual LLM judges and translationese bias.

Check it yourself

Release state, one command each (results on 8 October 2026 in comments):

curl -s "https://export.arxiv.org/api/query?id_list=2610.06216" | grep -E "<(published|updated)>"
# published 2026-10-05T12:26:05Z, v1 only (abs/2610.06216v2 returns 404)
curl -s https://huggingface.co/api/papers/2610.06216
# {"error":"Paper not found. ..."}
curl -s "https://api.github.com/search/repositories?q=pre-generation+success+probes+multilingual" | grep total_count
# "total_count": 0
curl -sL https://arxiv.org/pdf/2610.06216 -o p.pdf
python3 -c "import pypdf;print('\n'.join(p.extract_text() for p in pypdf.PdfReader('p.pdf').pages))" > p.txt
grep -c -i -E "github\.com|huggingface\.co" p.txt
# 0

The recomputation needs only the paper's tables. headline.py holds Table 5, Table 7, Table 9, Table 10 and the Figure 2 and Figure 4 ECE labels as literals and prints every number in this note:

python3 -I headline.py
# == 2. ECE reduction ... range 37.7% to 68.6%
# == 3. Qwen3-4B eng (Fig2) mean of 9 = 0.1251 (Table 10: 0.125)
# == 5. lam 0     success 86.6%  saving 12.8%
#       lam 0.1   success 86.0%  saving 22.4%
# == 5b. lam 0    mean success 86.3% (5th-95th pct 85.9-86.6)  mean saving 11.2%
#        lam 0.1  mean success 85.4% (5th-95th pct 84.3-86.2)  mean saving 25.2%
# == 8. BF16 lookup lam 0  success 86.6%  saving -14.3%

The core of the lookup router is four lines. T9[lang] is the Table 9 row for a language (8 success rates) and rel[m] the "Relative Exp. cost" column of Table 7:

for lam in [0, .01, .02, .05, .10, .20, .35, .50, .75, 1]:
    picks = {l: max(range(8), key=lambda j: T9[l][j] - lam*rel[M[j]]) for l in L}
    success = mean(T9[l][picks[l]] for l in L)
    saving  = 1 - mean(rel[M[picks[l]]] for l in L)

Two limits on our check: Table 9 reports means, so the lookup cannot model per-language token counts (it uses each model's ten-language average cost, as the paper's Table 7 does), and the Monte Carlo treats model estimates as independent, which overstates noise in the differences because the same 450 problems are shared.

What would prove this wrong

Our claim: on this benchmark a probe-free router keyed on prompt language matches the pooled probe router within one point of test success at matched saving, and the reported savings depend on 8-bit Qwen pricing.

It would be wrong if the authors, or anyone with the rollouts, publish a language-only baseline chosen on the validation split and evaluated on the same test split, and the pooled probe router beats it by more than 1.0 point of success at the same saving (or by more than 5 points of saving at the same success), with a paired interval excluding zero. We predict that by 30 June 2027 no such result will appear, and that any released per-prompt routing logs will show the pooled router assigning Qwen3.5-9B mainly to English, Chinese, Russian and Swahili prompts. A second falsifier: if the authors state the Qwen inference precision as BF16 with the Table 7 prices unchanged, our reading of the cost proxy is wrong.

Sources

1. Andrea Paganelli, Stefano Civelli, Pietro Bernardelle, Gianluca Demartini. Cross-lingual Calibration of Pre-Generation Success Probes for Multilingual LLM Routing. arXiv:2610.06216v1, 5 October 2026, University of Queensland and Politecnico di Milano. Sections 3 and 4, Tables 1 to 11, Figures 1, 2 and 4, Appendices A, D, E, G.
2. arXiv API record, export.arxiv.org/api/query?id_list=2610.06216, checked 8 October 2026: v1 only.
3. William Lugoloobi, Thomas Foster, William Bankes, Chris Russell. LLMs encode their failures: Predicting success from pre-generation activations. arXiv:2602.09924, 2026; code at github.com/KabakaWilliam/llms_know_difficulty.
4. Stefano Civelli et al. A shared geometry of difficulty in multilingual language models. ACL 2026 (short papers), pages 796 to 807; related repository Stefano-Civelli/multilingual-difficulty-interpretability, created 29 April 2026, no licence file, checked 8 October 2026.
5. Dan Hendrycks et al. Measuring mathematical problem solving with the MATH dataset. NeurIPS Datasets and Benchmarks, 2021.
6. Ricardo Rei et al. CometKiwi, WMT 2022; model card Unbabel/wmt22-cometkiwi-da, gated, CC BY-NC-SA 4.0, checked via the Hugging Face API on 8 October 2026.
7. BLOMEGA computations, 8 October 2026: Table 7 cost arithmetic, Table 9 means, Table 10 ECE reductions, Figure 2 and 4 per-language means, language-lookup routing frontier, 5,000-draw validation-noise simulation, BF16 price sensitivity, COMET-Kiwi rank correlation.

FAQ

Do English-trained success probes work on other languages?

They still rank prompts, but their probabilities drift. In arXiv 2610.06216 the English-trained Qwen3-4B probe keeps a rollout AUROC of 0.729 on nine non-English languages (oracle 0.758), while its expected calibration error is 0.125 against 0.031 for same-language probes, and as high as 0.247 in Swahili. Spreading the same 2,100-prompt budget over ten languages (210 each) cuts ECE to 0.043; across 8 models the cut is 38% to 69% by our recomputation of Table 10.

Does probe-based routing beat routing by language alone?

Not by much on this benchmark. The paper's pooled router reaches 86.6% test success at 13.0% modelled saving and 85.5% at 22.9%. Picking one model per language from the paper's own Table 9 gives 86.6% at 12.8% and 86.0% at 22.4% in-sample, or 85.4% at 25.2% when the table is estimated on a noisy 450-problem split (our Monte Carlo). The paper does not report a language-only baseline, so the per-prompt value of the probe is not isolated.

Can I reproduce the results, and are the savings real money?

Not from a release: as of 8 October 2026 the paper links no code, translations, rollouts or probe weights, Hugging Face has no page for it and a GitHub search finds nothing. The savings are modelled with a proxy price of $0.10 per GB of memory footprint per million output tokens, which by our arithmetic assumes about 1 GB per billion Qwen parameters (8-bit). Priced at BF16, the lambda-0 lookup routes would cost 14.3% more than always using gpt-oss-high instead of 12.8% less.