Zero of 65,536 tokens carry both scripts, which is why Arabic facts stay in Arabic

Zero of 65,536 tokens carry both scripts

A paper posted to arXiv on 16 September 2026 pins down why a fact an LLM learns in Arabic stays in Arabic. It pretrains bilingual models from scratch, injects fictive facts at known rates per language, and measures how much a foreign exposure is worth. For English-Arabic at 360M parameters the answer is 0.9%. Then it removes every explanation except one: it trains on two copies of the same English, identical text and identical segmentation, differing only in that the two copies get disjoint token ids. Knowledge still does not cross. CL_eq of -2.6%, and -0.1% even on identical corpora.

We downloaded the tokenizers the paper released and counted. Its joint English-Arabic vocabulary holds 65,536 entries: 31,963 are Arabic script only, 32,198 are Latin only, 1,375 are digits and punctuation, and zero contain characters from both scripts. On 1,012 parallel FLORES-200 sentences, the English and Arabic sides use 6,822 and 8,492 distinct token types and share 279, a Jaccard overlap of 1.86%. Only 101 of those shared types contain a letter, and every one is a Latin stray inside an Arabic sentence.

The barrier is not a modelling abstraction. It is 97.9% of a vocabulary, and you can count it in ten lines.

What was published, and when

arXiv:2609.19291, Why Pretraining Fails to Share Cross-Lingual Knowledge, by Adam Gaber, Uriel Dolev, Elisabeth Fittschen, Bobby Cheng, Yuval Marton and Leshem Choshen. Submitted 16 September 2026. Code at github.com/AdamJaber03/torchtitan-mulitlingual, a fork of TorchTitan, last pushed 18 August 2026.

The method is the contribution. FKD is a set of fictive entity facts, split into 16 groups of 128, injected into pretraining at exposure counts drawn from the cross-product of [0, 20, 100, 1000] per language. Because the facts are invented, all recall is attributable to the injection and the exposure count is known exactly. CL_eq is then the ratio of two OLS regression slopes: how much accuracy in language A rises per exposure in B, divided by how much it rises per exposure in A. A score of 100% means a foreign exposure is worth a native one. The metric is invariant to how good the model is overall, which is what separates transfer from raw competence.

Every CL_eq number in the paper is one full pretraining run.

Our measurement: what a disjoint token space looks like

The repository ships five trained tokenizers under trained_tokenizers/. These are byte-level BPE, so an Arabic character appears in the JSON as escaped bytes; decode them through the GPT-2 byte map first or every Arabic entry looks like Latin.

Byte-level BPE entries decoded through the GPT-2 byte map before classification, so escaped bytes are counted as the characters they encode. BLOMEGA, 22 September 2026, on trained_tokenizers/ in the paper's released repository.

The corpus-level count makes the consequence concrete. Take the 1,012 parallel sentences of FLORES-200 devtest, which are the same meanings in both languages, and tokenize both sides with the joint tokenizer. English emits 27,762 tokens over 6,822 types, of which 88.03% are Latin-lettered and 11.97% neutral. Arabic emits 28,766 tokens over 8,492 types, 91.83% Arabic-lettered, 7.73% neutral, 0.44% Latin. The two sides share 279 types: digits like 1995 and 2000, and stray Latin letters. Not one shared token carries meaning in both languages.

Worth noting because it cuts against intuition: on this corpus Arabic costs only 1.036 times as many tokens as the parallel English. Fertility is not the problem here. This tokenizer treats Arabic fairly by the usual measure, and knowledge still does not cross. That is the paper's point in one number.

The 14x is the baseline, not the fix

The abstract's headline is that mapping languages into a shared token space by word-wise translation recovers up to 12.6% of native learning efficiency, 14 times the baseline. Both halves are true. They point in opposite directions.

CL_eq values as reported in arXiv:2609.19291. The multiples, gains and unrecovered columns are our arithmetic.

The multiple is fragile because the denominator is near zero. Run the same fix at 7B, where standard pretraining already reaches 5.9%, and it is a 2.1x improvement. Run it on Russian, where the baseline is 1.9%, and it is 12.3x. The one quantity that does not move is where the fix lands: 12.6% at 360M and 12.5% at 7B, two orders of magnitude apart in parameters.

So the honest summary of the intervention is: it recovers roughly an eighth of native learning efficiency for Arabic, an eighth again at 7B, and just under a quarter for Russian. That is a real result on a problem where two prior families of fix (code-switching, activation alignment) moved the score by 1.3 and 1.6 points. It is also 87 points short.

The paper's own best number is somewhere else entirely. In the synthetic English1-English2 setting, tying 50% of the embedding pairs takes CL_eq from -0.1% to 82.7%, and 20% tying only reaches 8.6%: a sharp threshold, not a gradient. Merely initialising the pairs identically without tying gives 78.6%. That 82.7% is the paper's cleanest demonstration that the mechanism is right, and it is not transportable, because it needs a perfect one-to-one token correspondence that only exists when one language is a relabelled copy of the other. The distance from 82.7% to 12.6% is the cost of Arabic not being a relabelled copy of English.

One control makes the semantic half of this measurable. Shuffle the word-wise translation dictionary, keeping the shared token inventory but destroying the word correspondence, and CL_eq falls from 12.6% to 3.0%. So of the 11.7 points the fix buys, 2.1 points (18%) come from sharing tokens at all and 9.6 points (82%) come from the tokens meaning the same thing. Shared vocabulary is necessary and nowhere near sufficient.

What is in the release, and what is not

Appendix B says the fictive knowledge dataset and its generation code are publicly available, pointing at the repository. We pulled the full file tree, 549 entries. fictional_entity_data/ contains the generation prompts, three generator scripts, an evaluation generator, two taxonomy JSONs and gemini_seeds.txt with 2,303 seed facts in English, lines of the form "trellix is a company that sells spicy pickles." against the 2,048 facts the paper uses. There are no generated FKD documents and no MCQs, in English or in either of the other two supported languages. The dataset the CL_eq measurements run on is not in the release as of 22 September 2026.

The two word lists that are released, top_arabic_translated.json and top_english_translated.json, are 1,000 entries each. These are the code-switching dictionaries for the Section 4 baseline, not the invertible dictionary word-wise translation needs, which the paper describes in Appendix G and covers 99.7% of word occurrences and about 78% of unique types. The released pair is a useful illustration of why that appendix exists: only 319 of the 1,000 Arabic entries round-trip back to themselves through the English file (31.9%), 1,000 Arabic words collapse onto 819 distinct English targets, and the most common Arabic word in the corpus maps to "Who" in one file while "of" maps to it in the other.

What is fully released, and what this article rests on, is the tokenizers. Five of them, 2.7 to 6.3 MB, enough to reproduce every count above.

What this means for anyone shipping multilingual models

The practical claim in this paper is narrow and unusually well supported: if two languages do not share tokens, pretraining will not build the bridge for you, and no amount of scale, code-switching or representation alignment fixes it afterwards. The 7B run shows scale moving transfer from 0.9% to 5.9%, which is real and nowhere near enough.

• Check your vocabulary's script split before you blame your data mixture. It is a ten-line count on any released tokenizer, as above. If the two halves are disjoint, a fact in one language is structurally invisible to the other, and adding more parallel data will not change that.
• Fertility parity is not transfer parity. The tokenizer we measured costs Arabic only 1.036 times English, which passes every fairness check people normally run, and transfers 0.9% of what a native exposure is worth. These are different problems with different fixes.
• Word-wise translation is cheap and it costs sequence length. The mapping is context-free, applies and inverts on the fly, and keeps the user's native script at the interface. It also needs roughly 30% more tokens than native-script Arabic, which under a fixed token budget meant about 23% fewer Arabic documents in the paper's own runs. The 12.6% is measured with that handicap already applied.
• The soft-mapping result is the one to watch. Reserving a small number of embedding dimensions for language identity gives 9.60% at 90% tying and 10.47% at 99%, which is 76% and 83% of the full-mapping result while restoring a native mechanism for telling the languages apart. That is the version that could extend past a single language pair.

A judgement, marked as one: the finding that survives is the diagnosis, not the treatment. Disjoint token spaces being sufficient to compartmentalise knowledge between two identical copies of English is the kind of result that does not depend on the fix working, and it reframes what a multilingual vocabulary is for.

Check it yourself

Every number in the two measurement sections above comes from four files in the released repository and one public corpus.

pip install tokenizers
curl -sSLO https://raw.githubusercontent.com/AdamJaber03/torchtitan-mulitlingual/main/trained_tokenizers/bpe_65k_en1.0_ar1.0.json
curl -sSL https://dl.fbaipublicfiles.com/nllb/flores200_dataset.tar.gz | tar xz

python3 - <<'PY'
import json, collections
from tokenizers import Tokenizer

def b2u():                       # GPT-2 byte-level map, needed to read the entries
    bs = list(range(33,127)) + list(range(161,173)) + list(range(174,256))
    cs, n = bs[:], 0
    for b in range(256):
        if b not in bs: bs.append(b); cs.append(256+n); n += 1
    return {chr(c): b for b, c in zip(bs, cs)}
U2B = b2u()
dec = lambda t: bytes(U2B[c] for c in t).decode('utf-8', 'replace')

def script(x):
    ar = sum(1 for c in x if 0x0600 <= ord(c) <= 0x06FF or 0xFB50 <= ord(c) <= 0xFEFF)
    la = sum(1 for c in x if c.isalpha() and ord(c) < 0x0250)
    return 'mixed' if ar and la else 'arabic' if ar else 'latin' if la else 'neutral'

tk = Tokenizer.from_file('bpe_65k_en1.0_ar1.0.json')
v = tk.get_vocab()
print(collections.Counter(script(dec(t)) for t in v))   # 0 mixed

D = 'flores200_dataset/devtest/'
types = lambda f: {t for l in open(D+f, encoding='utf-8') if l.strip()
                    for t in tk.encode(l.rstrip()).tokens}
e, a = types('eng_Latn.devtest'), types('arb_Arab.devtest')
print(len(e), len(a), len(e & a), f"{100*len(e&a)/len(e|a):.2f}% Jaccard")
PY

Expected: Counter({'latin': 32198, 'arabic': 31963, 'neutral': 1375}) with no mixed key at all, then 6822 8492 279 1.86% Jaccard.

To check the dictionary round-trip, fetch top_arabic_translated.json and top_english_translated.json from the repository root and count how many Arabic keys come back through the English file: 319 of 1,000.

What would prove this wrong

The paper's central claim is that disjoint token spaces gate cross-lingual knowledge generalization, and its evidence is a synthetic pair of English copies where nothing else differs. The falsification is a third copy that shares tokens but nothing else: train English1 against an English2 whose text is character-permuted but drawn from the same token inventory. If knowledge still fails to cross, the barrier is not the token space.

Our own claim is narrower and easier to test: that the reported 14x is a property of the 0.9% baseline. We predict that by 30 June 2027, any replication of word-wise translation on a new English-to-non-Latin-script pair at 360M will report a post-fix CL_eq between 8% and 30%, and that the reported multiple will vary by more than 5x across those replications while the absolute score stays inside that band. If a replication reports a post-fix CL_eq above 50% on a genuine language pair, the ceiling we describe is not a ceiling and the method generalises better than the paper's own numbers suggest.

A cheaper check, available today: if someone releases the FKD documents and MCQs, recompute CL_eq for the released 7B checkpoint. The paper reports 5.9% native and 12.5% with word-wise translation at that scale. Those two numbers are the load-bearing ones, because they are the only evidence that the effect is not a small-model artifact.

Sources

1. Adam Gaber, Uriel Dolev, Elisabeth Fittschen, Bobby Cheng, Yuval Marton, Leshem Choshen. Why Pretraining Fails to Share Cross-Lingual Knowledge. arXiv:2609.19291, submitted 16 September 2026. Sections 2 to 6, Table 2, Figures 1 and 2, Appendices B and G.
2. github.com/AdamJaber03/torchtitan-mulitlingual, the released code. trained_tokenizers/ (five BPE tokenizers), fictional_entity_data/ (prompts, generators and 2,303 English seeds), top_arabic_translated.json and top_english_translated.json (1,000 pairs each). Full tree of 549 entries retrieved 22 September 2026.
3. NLLB Team. FLORES-200. The 1,012-sentence devtest split used for the parallel token counts, downloaded from dl.fbaipublicfiles.com/nllb/flores200_dataset.tar.gz.
4. Sander Land, Max Bartolo. Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models. EMNLP 2024. Prior evidence that a token existing in a vocabulary says little about whether the model learned anything with it.
5. BLOMEGA measurement, 22 September 2026: script classification of all four released tokenizers after byte-level decoding; parallel token type counts on FLORES-200 devtest; the dictionary round-trip and collision counts; and the multiple-versus-ceiling arithmetic on Table 2.

FAQ

What does CL_eq measure?

How much an exposure to a fact in one language is worth for recalling that fact in another, relative to a native exposure. It is the ratio of two OLS regression slopes fitted over fictive facts injected at known per-language rates. 100% means a foreign exposure is as good as a native one; 0% means complete compartmentalisation.

How bad is cross-lingual knowledge transfer in practice?

For a 360M English-Arabic model trained with standard practice, CL_eq is 0.9%. English-Russian is 1.9%. At 7B parameters English-Arabic reaches 5.9%. Pretraining code-switching gives 2.2% and activation alignment 2.5%, so neither helps.

What does the paper prove about tokenization?

That disjoint token spaces alone are enough to compartmentalise knowledge. Training on two identical copies of English that differ only in token ids gives CL_eq of -2.6%, and -0.1% even when the corpora are identical. Nothing about language difference is involved.

Is the 14x improvement as big as it sounds?

It is a ratio on a 0.9% baseline. Word-wise translation raises English-Arabic CL_eq from 0.9% to 12.6%, leaving 87.4 points unrecovered. The same fix at 7B goes from 5.9% to 12.5%, which is 2.1x. The absolute landing place, about 12.5%, is stable across scale; the multiple is not.

How disjoint is a real joint vocabulary?

We counted the paper's released English-Arabic tokenizer: of 65,536 entries, 31,963 are Arabic script only, 32,198 Latin only, 1,375 script-neutral and zero contain both scripts. On 1,012 parallel FLORES-200 sentences the two sides share 279 token types out of 15,035, and only 101 of those contain a letter.

Is the dataset released?

Not as of 22 September 2026. The repository ships generation prompts, generator scripts and 2,303 English seed facts, but no generated FKD documents and no evaluation MCQs in any of the three supported languages. The five trained tokenizers are released in full.