The "language unalignability" paper's Yami evidence runs on systems near 10 BLEU, and its tables say so

The mechanistic evidence for unalignability is measured on Yami systems that score about 10 BLEU

The paper's mechanistic claim (Section 5) is that models encode a regularity about Yami (its activation profile sits near other Austronesian and low-resource languages) that an intervention on "language-specific neurons" cannot use. The intervention is LAPE-masked LoRA fine-tuning, compared against an unmasked Standard LoRA and against a random mask of matched sparsity, on a 3,566-item Yami and Chinese benchmark. Table 5 reports only paired differences, each the mean of the two translation directions. Figure 5 reports Chinese to Yami BLEU for the five LAPE conditions. Section 5 adds that Yami to Chinese BLEU is 0.000 for LAPE in all nine models in the three cross-pair conditions (Exp 2a, 2b, 3).

Those three facts are enough to solve for the numbers the paper leaves out. Because each Table 5 difference D equals half the Chinese to Yami difference plus half the Yami to Chinese difference, the Yami to Chinese gap is 2D minus the Figure 5 gap. Doing that against the negative control (Exp 0) through Exp 2a, Exp 2b and Exp 3 gives 3.69, 3.70 and 3.69 BLEU for the control's Yami to Chinese score: three routes, one answer, spread 0.01. The same arithmetic gives Exp 1 (Yami-derived mask) 4.12 or 4.14 depending on route, and two rows with zero on both sides (2b vs 3, 2b vs 2a) are predicted at -0.565 and +0.315 against reported -0.57 and +0.31.

With the direction scores fixed, the baseline follows. Exp 1 LAPE averages (9.17 + 4.13) / 2 = 6.65 BLEU, and Table 5 puts it 3.75 below matched Standard LoRA, so Standard LoRA on Yami and Chinese averages 10.40. Going through Exp 0 instead (5.93 plus 4.48) gives 10.41. Our judgement: the reconstruction is sound, and it changes how the headline reads. The "gap" between encoding and use is measured on systems whose best average is about 10 BLEU, and the LAPE versus random contrast that carries the claim is 0.29 BLEU, 2.8% of that level, with an interval of -0.82 to 1.24.

What Hsieh and Lian posted as arXiv:2610.08303 on 6 October 2026

arXiv:2610.08303v1, Language Unalignability: Why Some Concepts Resist Cross-Cultural Benchmark Evaluation, by Shu-Kai Hsieh and Da-Chen Lian of National Taiwan University, was submitted on 6 October 2026 at 13:10 UTC in cs.CL. The arXiv comment calls it a position paper: 32 pages, 10 of main text, 6 figures, 12 tables. Version 1 is the only version as of our check.

The paper names a Translation-Isomorphism Assumption (TIA): that every concept has a translation equivalent whose inverse also preserves meaning, which is what makes a single gold reference possible. It models a concept as a "usage cloud", the set of contextual embeddings of every attested use, and calls a translation pair alpha-unalignable when no map can keep both the cloud centroids within epsilon (lexical faithfulness, M1) and the local neighbour ranks within delta (structural faithfulness, M2). Its running example is Mandarin xīn kǔ le ("you've worked hard"), which a model can render as "good job" across superior-to-subordinate, peer, service and ironic uses and score the same on every single-reference benchmark.

It proposes a four-part profile in place of a scalar score: Cycle-Consistency (CC, one minus the Spearman correlation of pairwise distances before and after a round trip), Pragmatic-Load Disagreement (PLD, model choice against the modal of three native-speaker continuations, adjusted for agreement), Manifold-Curvature Mismatch (MCM, TwoNN intrinsic dimension difference, density difference and Procrustes residual) and the Representation-Intervention Gap (RIG, probe accuracy above chance minus normalised intervention effect). The Yami work comes from the second author's 2026 NTU dissertation.

The paper's evidence, its unprinted numbers and its release, side by side

Paper values from arXiv:2610.08303v1. Reconstructions, ratios and release checks are BLOMEGA computations, 8 October 2026 (00:17 UTC on 9 October).

The reconstruction also explains one figure that otherwise looks odd. Table 5 gives "Yami vs Neg. Ctrl" at +0.72 BLEU, but the Figure 5 bars differ by 1.01 (9.17 against 8.16). The two agree once Yami to Chinese is included: that direction differs by only 0.43 (4.13 against 3.69), and the mean of 1.01 and 0.43 is 0.72. So the small Exp 1 advantage over the unrelated-language control is mostly a Chinese to Yami effect.

We also checked whether the Table 5 intervals match the effect sizes printed next to them. With nine model-level differences, a percentile bootstrap interval should be about 1.96 times the plug-in standard deviation over 3, and d_z gives the standard deviation as delta over d_z. For the nine BLEU rows with |d_z| above 1, the reported half-width over the predicted one runs 0.97 to 1.00. The intervals, effect sizes and means are mutually consistent, which is the result an author would want and a reviewer could not get without doing the sum.

Appendix G gives one more checkable total. NLLB-200 is said to produce 2,928 of the 2,929 empty predictions. Table 12 lists the Yami to Chinese empties by condition; they sum to 2,900, leaving 28 for Chinese to Yami, which fits the stated 0 to 8 per condition. Across the three cross-pair conditions the random mask produced more empties (1,667) than LAPE (1,185) out of 10,698 items each.

How the paper gets from usage clouds to a gap between encoding and use

The argument has two halves that are measured very differently. The probe half (pi-hat in the paper) is a cosine similarity between per-neuron firing probabilities, averaged across layers; it is not a trained probe. The intervention half (iota-hat) is a BLEU difference between fine-tuning regimes. RIG is defined as the first minus the second, but the paper never computes a RIG number: it reports the two halves side by side.

Two features of the design limit what the gap can show, and the authors say both. Every regime trains an unmasked attention LoRA path, so the comparison isolates the added value of choosing MLP neurons, not a neuron-only adapter. And the probe regularity correlates with corpus size (r = 0.879 across 16 languages), with three low-resource African controls (Yoruba, Igbo, Hausa) inside the Austronesian distance range, so what is "encoded" is closer to "low-resource" than to "Austronesian". The Austronesian-specific residual gives Welch's t = -2.01, p = 0.100.

The link back to honorifics is the paper's hypothesis, not its measurement. Yami versus Chinese at about 10 BLEU is a low-resource adaptation problem. None of the nine models is tested on an honorific, an evidential or xīn kǔ le, and no usage cloud is built for any real concept. Our judgement: the framework is a useful vocabulary, and the Yami study is a careful null result about LAPE, but the second does not test the first.

What this means for teams localizing honorifics and register into low-resource languages

• The benchmark critique stands on its own and is cheap to act on. A single reference per source item cannot see register collapse, which is the paper's central point (Section 4.1). For honorific-heavy pairs (Japanese, Korean, Javanese, Thai) or address forms in dubbing, author two to four registered references per item, tagged with speaker relation, and add "no faithful gloss" as an allowed answer, as Section 7 recommends.
• PLD is the one metric you can run without the authors. It needs three native-speaker continuations per probe and an agreement figure. That is a data-collection task, not a modelling one, and it measures what a dubbing director checks: whether the line picks the right social register.
• Do not read the Yami result as "honorifics are unfixable". It is a 10 BLEU low-resource adaptation study about one neuron-selection method. What it does show, at 9 of 9 models, is that restricting the MLP LoRA path to 1% or fewer of its neurons costs about 36% of the baseline score (6.65 against 10.40); the random mask loses 39% (6.36) and the negative-control LAPE run 43% (5.93). For a production fine-tune on a small language pair, our judgement is to start from unmasked LoRA.
• Direction matters more than the headline suggests. Into-Chinese output sat at about 45% of into-Yami BLEU in the two working conditions, and at 0.000 in every cross-pair LAPE run, where models fell back on one stock phrase ("small boat", "conch shell") across thousands of sources (Table 10). Evaluate each direction separately and look at the most frequent output string before trusting an average.
• Pivot data is not a substitute for in-pair data. Standard LoRA trained on Yami-Filipino or Yami-English (pivot-translated) scored about 5.1 and 6.6 on the Yami and Chinese benchmark, against 10.4 for in-pair training. For collection budgets on endangered or minority languages, that is a direct argument for paying for consented, native-speaker parallel data in the target pair.
• Expect no reuse of the Yami corpus. The 31,255 pairs come from third-party providers and are not the authors' to release (Appendix H). Anyone building Yami systems needs their own rights-cleared data.

Check it yourself: release state and the reconstruction

Fetch the PDF and text (pypdf, since the reconstruction reads only the PDF):

curl -sL https://arxiv.org/pdf/2610.08303 -o p.pdf
python3 -c "import pypdf;print('\n'.join(p.extract_text() for p in pypdf.PdfReader('p.pdf').pages))" > p.txt
curl -s "https://export.arxiv.org/api/query?id_list=2610.08303" | grep -E "<published>|arxiv:comment"
# 2026-10-06T13:10:29Z; "Position paper. 32 pages (10 pages main text), 6 figures, 12 tables"

Release state (8 October 2026, 00:17 UTC on 9 October):

curl -s "https://api.github.com/search/repositories?q=language+unalignability" | grep total_count   # 0
curl -s "https://huggingface.co/api/datasets?search=tao_Latn"                                        # []
curl -s https://huggingface.co/facebook/nllb-200-3.3B/raw/main/special_tokens_map.json | grep -c tao_Latn   # 0 (202 tokens, tgl_Latn present)
curl -s https://huggingface.co/api/datasets/openlanguagedata/flores_plus/tree/main/devtest | grep -c tao_   # 0 of 222 files
curl -s https://api.github.com/repos/lopentu/LOBSTER | grep -E '"(created_at|pushed_at|license)"'   # created 2025-10-23, no licence

The reconstruction is pure Python with the paper's numbers typed in (Figure 5 right panel; Table 5 BLEU rows; the Section 5 statement on 0.000 BLEU):

python3 - <<'PY'
c2y={'E0':8.16,'E1':9.17,'E2a':4.19,'E2b':4.82,'E3':5.95}       # Figure 5, Chinese to Yami
t5={('E2a','E0'):-3.83,('E2b','E0'):-3.52,('E3','E0'):-2.95,    # Table 5, BLEU, mean of directions
    ('E1','E0'):0.72,('E2a','E1'):-4.56}
e0=[0-(2*t5[(a,'E0')]-(c2y[a]-c2y['E0'])) for a in ('E2a','E2b','E3')]
E0=sum(e0)/3; E1=E0+2*0.72-(c2y['E1']-c2y['E0'])
print('Exp0 Yami->Chinese',[round(x,2) for x in e0])          # [3.69, 3.7, 3.69]
print('Exp1 Yami->Chinese',round(E1,2), round(-(2*-4.56-(4.19-9.17)),2))   # 4.12, 4.14
print('Standard LoRA',round((9.17+E1)/2+3.75,2), round((8.16+E0)/2+4.48,2))  # 10.4, 10.41
PY

The full script with the interval check, the Holm floors and the Table 12 sum is headline.py in our work folder; it prints the numbers in this note. The four quantities to look for in the PDF: the 0.000 statement in Section 5.2 (page 8), Table 5 (page 22), Figure 5 right panel labels 8.16, 9.17, 4.19, 4.82, 5.95 (page 27), and Appendix H (page 32).

What would prove this reading wrong

Our claim has two parts. First, that the Yami systems behind the RIG evidence average about 10.4 BLEU (Standard LoRA) and 6.65 (Exp 1 LAPE), with Yami to Chinese near 4.1. Second, that the four-metric profile has not been measured on a real honorific or pragmatic marker.

The first part is wrong if the dissertation (Lian 2026) or a v2 of the paper prints per-direction BLEU that differs from our 3.69 and 4.13 by more than 0.1, or a Standard LoRA mean outside 10.3 to 10.5. The only way our arithmetic could fail is if Figure 5 is not the same LAPE runs as Table 5, and three routes agreeing to 0.01 makes that unlikely. The second part is wrong if, by 30 June 2027, the authors publish CC, PLD or MCM values for at least one real concept pair (for example xīn kǔ le and "good job") with the corpus or code to recompute them. We predict that by that date the BETEL evaluation will report PLD on Taiwanese Hokkien or Mandarin items, but that no release will include the Yami corpus.

Sources

1. Shu-Kai Hsieh, Da-Chen Lian (National Taiwan University). Language Unalignability: Why Some Concepts Resist Cross-Cultural Benchmark Evaluation. arXiv:2610.08303v1, 6 October 2026. Sections 1 to 8, Tables 4 to 12, Figures 1 to 5, Appendices C to H.
2. Da-Chen Lian et al. LOBSTER: Linguistics Olympiad Benchmark for Structured Evaluation on Reasoning. ROCLING 2025, November 2025 (ACL Anthology page returned HTTP 200 on 8 October 2026). Code at github.com/lopentu/LOBSTER: created 23 October 2025, 4 commits, no licence file.
3. Tianyi Tang et al. Language-specific neurons: The key to multilingual capabilities in large language models. ACL 2024. The LAPE method tested in Section 5.
4. NLLB Team. facebook/nllb-200-3.3B on Hugging Face; tokenizer special tokens checked 8 October 2026 (202 tokens, no tao_Latn).
5. openlanguagedata/flores_plus on Hugging Face; devtest file list checked 8 October 2026 (222 files, no Yami).
6. BLOMEGA computations, 8 October 2026: direction-level BLEU reconstruction from Figure 5 and Table 5, Standard LoRA levels, Table 5 interval and effect-size consistency, Holm floors, corpus and Table 12 sums, GitHub and Hugging Face searches.

FAQ

Does the unalignability paper measure honorifics or pragmatic markers directly?

No. It defines four metrics (Cycle-Consistency, Pragmatic-Load Disagreement, Manifold-Curvature Mismatch and the Representation-Intervention Gap) but reports no value of the first three for any real concept. Its only Procrustes figure uses synthetic clouds, the BETEL items are illustrations with evaluation "forthcoming", and alpha-unalignability has no numeric epsilon or delta. The measured evidence is a cited FLORES-200 run of ten sentences per language and a Yami fine-tuning study about language-specific neurons.

How good were the Yami translation systems in the Representation-Intervention Gap study?

The paper does not print per-direction scores, but its Figure 5 and Table 5 fix them. By our reconstruction the matched Standard LoRA baseline averages about 10.4 BLEU over the two directions, the Yami-masked LAPE run 6.65 (9.17 Chinese to Yami, about 4.13 Yami to Chinese) and the matched random mask about 6.36. Three independent routes through the tables agree to within 0.01 BLEU.

Can a localization team reproduce or reuse the Yami results?

Not from public material. Appendix H says no code or data is released, because the 31,255-pair Yami-Mandarin corpus comes from third-party providers. On 8 October 2026 (00:17 UTC on 9 October) GitHub search for the paper's terms returned 0 repositories, Hugging Face had no tao_Latn dataset, NLLB-200's tokenizer had no tao_Latn token, and FLORES+ devtest had no Yami file among its 222. The arithmetic check in this note needs only the PDF.