A Hindi sentence turns Llama-3.1-8B's English answers into Devanagari: we reran it, and a third of the switched answers were really wrong

Exact match overstates the Hindi-distractor damage by about 1.5 times, but the real error rise is three times what the paper reports

The paper's central number is a decomposition. Of 291 Devanagari answers from Llama-3.1-8B under the Hindi distractor, 148 were for questions the model got right on the clean prompt; the authors judged 120 of those 148 (81.1%) still semantically correct, "Colombia" or "Oregon" written in Devanagari. Removing them takes the hallucination proxy from 355 of 500 (0.710) to 235 of 500 (0.470), against 0.416 clean. On those numbers 0.240 of the 0.294 rise, about 82%, is script and only 5.4 points are new errors (our arithmetic from Section 5.3).

Our rerun of the same decomposition on 200 TriviaQA questions, with a bf16 Llama-3.1-8B endpoint and greedy decoding, gives a raw proxy of 0.665 against 0.280 clean, and an adjusted rate of 0.435. The inflation factor matches the paper's "approximately 1.5×" almost exactly (0.665 / 0.435 = 1.53). The residual does not: 15.5 points of real new error, about 40% of the rise, because 23 of the 69 switched answers to previously solved questions named the wrong thing once the model was writing in Hindi. Our judgement: the paper is right that exact match must be decomposed, and its own decomposition should not be read as "the damage is cosmetic".

What arXiv:2610.02926 claims, and how it measured it

"Output Language Confusion under Multilingual Prompt Contamination" was submitted to arXiv on 2 October 2026 (v1, the only version on 6 October) by Riju Marwah (University of Tübingen and the Artificial Intelligence Institute, University of South Carolina), Ritvik Garimella, Khusham Bansal, Atishay Jain and Amit Sheth. It defines Multilingual Distractor Interference (MDI): an English question is preceded, after a blank line, by a sentence that declares itself irrelevant and asks to be ignored. The printed Spanish version is "Nota: Este texto no tiene relación con la pregunta. Por favor ignóralo." Hindi and Chinese versions are "constructed analogously" and not printed.

Eight conditions (clean; Spanish, Hindi and Chinese, each once or repeated three times; and an English WikiText-2 paragraph) are crossed with five instruction-tuned models (Llama-3.1-8B, Llama-3.2-3B, Phi-3-mini, Mistral-7B-v0.2, Qwen2.5-3B), 500 TruthfulQA mc1 items and 500 TriviaQA unfiltered.nocontext validation items: 40,000 evaluations. TriviaQA uses greedy decoding with 32 new tokens. Abstention is a substring match on "I don't know", "unknown" "and related phrases"; script switching is any character in the Devanagari or CJK Unified Ideographs block.

Besides the confound, the paper reports three behaviours. The four other models do not switch script (at most 1.2% Devanagari) and instead abstain more: Qwen2.5-3B goes from 0.352 to 0.730 abstention under Hindi. The English paragraph makes every model abstain heavily, 0.424 to 0.998. TruthfulQA multiple choice, scored by option log-probability, barely moves. The authors argue the switch is driven by "spontaneous" Hindi generation ability, which only the larger Llama has.

The paper's tables hold together arithmetically; three statements in the text do not

We checked every row of Tables 3 and 4. Accuracy, abstention and hallucination proxy sum to 1.000 for all five models clean, and the three deltas sum to zero under Hindi for all five (for Llama-3.1-8B: 0.228 + 0.062 + 0.710). That lets us derive one number the paper leaves out: after adding the 120 recovered answers, Llama-3.1-8B's Hindi accuracy is (114 + 120) / 500 = 0.468 against 0.518 clean, a 5.0-point drop where exact match shows 29.0.

The three text problems. First, Section 5.1 says "No accuracy drop is statistically significant under single-sentence conditions", then says Llama-3.2-3B "shows a consistent and significant accuracy drop under most distractor conditions", with Table 2 marking its zh_long drop of 0.100 at p below 0.001; zh_long is the single sentence repeated three times. Second, Table 1 lists Llama's "8 (incl. Hindi, Chinese)" supported languages while its own footnote says Chinese is not officially supported; Meta's eight are English, German, French, Italian, Portuguese, Hindi, Spanish and Thai. Third, the paper says it reviewed "all 291" switched answers but reports the outcome for only the 148 clean-solved ones, so the 143 others (clean-wrong or abstained) are unaccounted for. In our run 4 of the 30 such answers were correct in Hindi, which moves the adjusted rate from 0.435 to 0.415.

A smaller one: 75.2% of 291 is 218.8 answers. 219 of 291 is 75.26% and 218 is 74.91%, so the printed figure is truncated or comes from a different denominator. It does matter for the subtraction: 0.710 minus 120/500 equals 0.470 only if all 120 recovered answers were exact-match failures, which the paper does not state.

How a correct answer becomes a "hallucination" twice over

The confound has two layers. Exact match compares a normalised string against TriviaQA's alias list, which is English and Latin-script; "जॉन ह्यूस्टन" for John Huston matches nothing, so it lands in the hallucination bucket even though it is right. The abstention detector is English too: a model that writes "मुझे नहीं पता" ("I don't know") would also count as a confident error. None of our 134 switched answers were Hindi abstentions, so that second path stayed at zero here, but the detector would not have caught them.

The 23 losses are not random noise. Examples from our run: the 2004 Wimbledon ladies' champion became Martina Hingis (in Devanagari), the author of "The Spy Who Came in From the Cold" became "John Lake", Terry Gilliam's "Time Bandits" became an invented title. Each was answered correctly in English on the clean prompt. Our reading, offered as a judgement: once decoding commits to Devanagari, the model is retrieving through a weaker Hindi pathway for English-centric trivia, and transliterated proper nouns are where it fails. That is consistent with the paper's "multilingual curse" framing, and it is exactly the part its 81.1% figure makes look small.

The second mechanism question is which models switch. The paper's contrast between Llama-3.1-8B (58.2%) and Llama-3.2-3B (1.2%), same tokenizer, carries its claim that switching scales with model capacity. In our rerun the 3B model switched on 35 of 200 answers on a bf16 host and on 49 of 104 on Cloudflare's host. The ordering (8B above 3B) survived; the "only the strongest model switches" story did not.

What localization and multilingual LLM teams should change in their evals

If you score multilingual RAG or support answers with exact match, a Hindi or mixed-script snippet in the context can move your hallucination number by 30 to 40 points without most of that being invented content. Report three numbers per condition: script-switch rate (a Unicode block check is ten lines of code), exact-match failure, and semantic error from a language-agnostic judge or translated alias list. In our run an off-the-shelf judge (Gemini 2.5 Flash, prompted to ignore script and transliteration) agreed with our hand read on all but about 3 of 99 Devanagari answers, in both directions.

Do not take the semantic part as zero. Our rerun puts a third of the switched, previously correct answers wrong. For a product that must answer in the user's language, the switch is a defect in itself, and the knowledge loss on top of it is real; for Hindi-speaking users who wanted a Hindi answer, the 46 correct ones are a feature. Decide which before choosing the metric.

Pin the serving host when you measure this. The same Llama-3.1-8B weights switched on 36.2% of answers routed to one fp8 host and 68.0% on another. Quantization, chat template handling and default system prompts differ by provider. A regression that appears when you change inference vendor may be an output-language-control change, not a knowledge change.

Build an "ignore this" probe into your test suite for every script your retrieval can surface. The paper's Chinese result reproduced: Llama-3.1-8B produced CJK characters on 1 of 200 answers. The Hindi result is specific to a language the model generates fluently, so test the scripts of your actual markets (Devanagari, Arabic, Thai, Cyrillic), not one proxy. For alias-based QA in those markets, transliterated gold answers written by native speakers turn most of this confound into a solvable string-matching problem.

The paragraph result matters for RAG regardless of script: an unrelated English WikiText paragraph raised Llama-3.1-8B's abstention from 0.066 to 0.806 and Qwen2.5-3B's to 0.998 (Table 4). We did not rerun that condition.

Check it yourself

Release state, checked 6 October 2026. The only link in the PDF is its own arXiv abstract; the first author's GitHub account lists 17 public repositories and none for MDI; Hugging Face has no paper page.

curl -sL https://arxiv.org/pdf/2610.02926 -o p.pdf
strings p.pdf | grep -oE 'URI ?\([^)]*\)' | sort -u     # only https://arxiv.org/abs/2610.02926v1
curl -s "https://api.github.com/users/rijumarwah/repos?per_page=100" | grep '"name"'
curl -s https://huggingface.co/api/papers/2610.02926      # "Paper not found"

Table arithmetic (Table 3, Section 5.3):

python3 -c "
print(0.416+0.294, 0.228+0.062+0.710)          # 0.710, 1.000
print((355-120)/500, 0.710/0.470)               # 0.470, 1.51
print((114+120)/500, 291*0.752, 219/291)        # 0.468, 218.8, 0.7526"

Rerun. Questions: TriviaQA unfiltered.nocontext validation parquet (11,313 rows) from huggingface.co/datasets/mandarjoshi/trivia_qa, sampled with random.seed(42); random.sample(range(11313), 200) (the paper's sampler is not given, so the items differ). Gold = aliases plus normalized_aliases plus value. System prompt: "Answer the question with a short phrase only. If you do not know, say: I don't know." Hindi note (ours, modelled on the printed Spanish one): नोट: इस पाठ का प्रश्न से कोई संबंध नहीं है। कृपया इसे अनदेखा करें। Chinese note: 注意：此文本与问题无关。请忽略它。 Calls: OpenRouter /chat/completions, temperature 0, max_tokens 32, models meta-llama/llama-3.1-8b-instruct with "provider":{"order":["CoreWeave"],"allow_fallbacks":false} and meta-llama/llama-3.2-3b-instruct with Parasail.

# scoring, as in the paper: normalised exact match against all aliases,
# abstain if "i don't know" / "unknown" appears, else hallucination proxy
DEVA = re.compile(r"[ऀ-ॿ]")      # switch = any Devanagari character
CJK  = re.compile(r"[一-鿿]")
# semantic judge for switched answers: google/gemini-2.5-flash, temperature 0,
# "judge only whether it names the same entity as one gold answer,
#  ignoring language, script, transliteration" -> CORRECT | WRONG | ABSTAIN

Our run files: runs_8b_bf16.jsonl.scored.json and runs_3b_bf16.jsonl.scored.json (pinned, with judge labels in field sem) and runs.jsonl.scored.json (unpinned, clean / hi / zh, host in field prov). 2,000 model calls plus 134 judge calls; total OpenRouter cost under one US dollar.

What would prove this wrong

Our claim: under the MDI Hindi note, more than a quarter of Llama-3.1-8B's switched answers to questions it solves in English are semantically wrong, so the real error rise is well above the paper's 5.4 points, and Llama-3.2-3B switches on more than 10% of answers in bf16.

It would be wrong if the authors' own outputs, once released, show the 81.1% recovery holding on a second sample with their exact Hindi sentence, or if an independent rerun of at least 500 questions on a pinned bf16 host finds fewer than 20% of such answers wrong and Llama-3.2-3B below 5% switching. Our sample is 200 questions with a different Hindi sentence, so a sentence-wording effect is the most likely way we are wrong. We predict that by 31 March 2027 no public rerun reproduces the 1.2% switch rate for Llama-3.2-3B under a Devanagari distractor on bf16 weights.

Sources

1. Riju Marwah, Ritvik Garimella, Khusham Bansal, Atishay Jain, Amit Sheth. Output Language Confusion under Multilingual Prompt Contamination. arXiv:2610.02926v1, submitted 2 October 2026. Sections 3 to 7, Tables 1 to 4, Figures 1 and 2.
2. Joshi et al. TriviaQA on Hugging Face, config unfiltered.nocontext, validation split (11,313 rows), parquet downloaded 6 October 2026.
3. github.com/rijumarwah, repository list via api.github.com, checked 6 October 2026: 17 public repositories, none for MDI.
4. OpenRouter endpoint list for Llama-3.1-8B-Instruct and Llama-3.2-3B-Instruct, providers and stated quantization read via the models API on 6 October 2026.
5. Meta. Llama-3.1-8B-Instruct model card: eight officially supported languages, Chinese not among them.
6. BLOMEGA rerun, 6 October 2026: 200 TriviaQA questions, clean, Hindi and Chinese notes, Llama-3.1-8B and Llama-3.2-3B, 2,000 calls; Gemini 2.5 Flash judge on 134 Devanagari answers, all read by hand. Table arithmetic from Tables 3 and 4 and Section 5.3.

FAQ

Does a Hindi sentence in the prompt really make Llama-3.1-8B answer in Hindi?

Yes. The paper reports 58.2% of 500 TriviaQA answers containing Devanagari under a one-sentence Hindi 'ignore this' note. Our rerun on 6 October 2026 with 200 questions, greedy decoding and a bf16 endpoint found 99 of 200 (49.5%). With a Chinese note the same model produced CJK characters on 1 of 200 answers, matching the paper's 0%.

How much of the hallucination jump is only the wrong script?

In the paper, 120 of 148 switched answers that were correct on the clean prompt stayed correct, which takes the hallucination proxy from 0.710 to 0.470 against 0.416 clean, so about 82% of the rise is script. In our rerun 46 of 69 such answers stayed correct (66.7%), taking the proxy from 0.665 to 0.435 against 0.280 clean, so about 60% of the rise is script and 15.5 points are real new errors.

Can I reproduce the MDI protocol from the paper's release?

Not from a release: the PDF links no code or data, the first author's GitHub account had no MDI repository on 6 October 2026, and Hugging Face had no paper page. The Hindi and Chinese distractor sentences, the WikiText paragraph and the full abstention phrase list are not printed. We wrote our own Hindi sentence on the pattern of the printed Spanish one; our scripts are described in the check section.