Dialect routing before Whisper pays only above 42% identification accuracy: a check of Itgan's NADI 2026 Arabic ASR paper
On 7 October 2026 independent researcher Ibrahim Almajai posted the Itgan systems for the three Arabic ASR subtasks of NADI 2026, all built from one recipe: Whisper with LoRA adapters on two consumer GPUs. The paper reports that a linear dialect probe recovers 44% of what oracle routing to per-dialect specialists gives. Working from its own cross-dialect table, we find the other side of that number: a wrong route costs 0.71 of what a right one gains, so routing helps only above 41.6% identification accuracy, and a random router would add 0.95 WER points.
A dialect router in front of Whisper specialists breaks even at 41.6% accuracy, and one dialect needs 76%
Table 9 of arXiv:2610.09934 decodes every one of eight per-country specialists on every country of the Subtask 1.1 development set and reports the change in WER against the pooled adapter. The diagonal (each specialist on its own dialect) averages a gain of 1.90 points. The 56 off-diagonal cells average a cost of +1.36 points. The paper states both numbers, rounded to 1.9 and 1.4, and calls the ratio "about seven tenths". We take the next step. If a router is right with probability a and its mistakes are spread evenly, the expected change is -1.90a + 1.36(1 - a), which crosses zero at a = 41.6%. At chance (12.5% for eight classes) the expected change is +0.95 points, and an exact column average of Table 9 gives the same +0.95.
That simple model predicts the paper's measured result well. At the probe's 65.3% accuracy it predicts 40.5% of the oracle prize recovered; the paper's per-utterance routing measured 44%. At the tuned probe's 69.5% it predicts 47.7%; the paper measured 47%. The paper's own shortcut, composing a confusion matrix with Table 9, gave 51%, which the author flags as flattering. Our judgement: for planning, the even-spread model is the better rule of thumb, because it lands within 3.5 points of the measured figure at both accuracies.
The average hides a spread. Applying the same arithmetic one column of Table 9 at a time gives a break-even accuracy per dialect: 12% for Algeria and 16% for Mauritania, whose specialists gain 3.1 and 2.8 points and whose columns are cheap to misroute into, but 61% for Palestine and 76% for Jordan. The paper reports that routing made Jordan worse (48.7 to 49.5 WER) and that Jordan had the lowest router accuracy, 57.5%. Our threshold explains that result: 57.5% is well short of the 76% Jordan needs.
What Itgan submitted to NADI 2026 and posted on 7 October 2026
arXiv:2610.09934 v1, submitted 7 October 2026 (cs.CL), single author Ibrahim Almajai, listed as an independent researcher. It describes entries to the three ASR subtasks of NADI 2026, the second multidialectal Arabic speech shared task (Sullivan et al., ArabicNLP 2026, Budapest): noisy, band-limited country-level ASR across eight national dialects (1.1), mixed-dialect ASR with no dialect labels (1.2), and Tunisian Arabic code-switched with French and English (1.3).
The shared recipe is LoRA at rank 32, alpha 64, on q, k, v, out projections and both feed-forward layers of every block, with the frozen base in bfloat16 and adapters in float32, trained on an RTX 3080 (10 GB) and an RTX 3060 (12 GB). Results on the blind test sets: 57.1% country-average WER on 1.1 (seventh of nine teams, leader 43.2%), 46.7% on 1.2 (third of four), and 14.49% WER with 5.38% CER on 1.3 (second of six submissions, 0.08 points behind first). The 1.3 system's last 0.60 points came with no extra training: a weight-space soup of four LoRA adapters (15.09 to 14.55) and three-way ROVER voting (14.55 to 14.49).
The paper is unusually careful about statistics. Every comparison carries a paired bootstrap with 4,000 resamples, it lists 36 such comparisons without multiplicity correction and says so, and it reports eight failed directions, including speech enhancement (CMGAN, +5.0 WER) and bandwidth extension (VoiceFixer, +18.0 WER) on the noisiest 200 utterances.
What we recomputed and verified, row by row
| Check | Paper says | We found | Source |
|---|---|---|---|
| Break-even router accuracy | not reported | 41.6% overall; per dialect 12% (Algeria) to 76% (Jordan) | Table 9; our arithmetic |
| Random router (12.5%) | not reported | +0.95 WER points, worse than no routing | Table 9; our arithmetic |
| Oracle prize and probe recovery | 1.9 points, 3.4% relative, 0.8 points = 44% | 1.90, 3.4%, 0.83 = 44% (56.56, 55.73, 54.66) | Table 10 |
| Subtask 1.1 country average | 57.1% | 57.125% from the eight rounded country WERs | Table 1 |
| LoRA trainable parameters, six configurations | 6.6M, 27.9M, 15.7M, 28.8M, 57.7M, 115.3M | all six match, computed from the released config.json files | Sections 3 to 6; Hugging Face |
| Table 2 differences vs quoted bootstrap deltas | may differ by about 0.01 | four differ by 0.01, the matched-capacity one by 0.02 (2.63 printed, 2.65 quoted) | Table 2, Section 5 |
| Mel bins above the dead band | 43% of 128 bins peak above 4 kHz | 21.9% (28 bins) peak above 4 kHz; 43.0% (55 bins) peak above 2.1 kHz | Appendix A; Slaney mel, 0 to 8 kHz |
| Runaway hypotheses cut by the length guard (pooled, greedy) | "about 12 of 10,808" in Section 4 | Appendix B gives 14 for the same adapter | Section 4 vs Appendix B |
| Soup transfer, dev to blind | 2.26 points became 0.54 (24%) | 19.46 to 17.19 is 2.27; 15.09 to 14.55 is 0.54; 24% | Table 3, Appendix D |
| Implied insertion share, 1.3 practice test | not reported | sub+del 17.6% vs WER 21.19%, so about 3.6 points are insertions | Tables 3 and 7; our arithmetic |
| Latin-script share of reference words | 33.3% (practice test) | 31.8% practice test, 22.4% validation, raw transcriptions | HF datasets-server; our count |
| Split sizes | 1,600 x 8 train, 10,808 dev, 4,000 test; 3,152; 24,893 / 731 / 842 / 2,123 | all match on Hugging Face; 1.2 blind test has 427 rows (not stated in paper) | HF datasets-server /size |
| turbo encoder vs large-v3 encoder | "also moved its encoder weights" | 100% of sampled elements differ; cosine 0.90 at layer 0 q_proj, 0.43 at layer 31 fc2 | HTTP range reads of model.safetensors |
| Itgan code, adapters, hypotheses | no link given | none found on GitHub or Hugging Face, 8 October 2026 | GitHub and HF search APIs |
Paper values from arXiv:2610.09934v1 (7 October 2026). All "we found" values are BLOMEGA computations or API reads on 8 October 2026; scripts listed in the check section.
Two of these rows are small slips that do not change any conclusion. The mel-bin sentence in Appendix A says 43% of Whisper's 128 bins "peak above that line", where the line is 4 kHz. Computing the 128 Slaney-scale filter centres that Whisper large-v3 uses (0 to 8 kHz at 16 kHz sampling) puts 28 bins, 21.9%, above 4 kHz, and 55 bins, 43.0%, above 2.1 kHz, the effective bandwidth quoted one sentence earlier. Our reading: the 43% is right for the 2.1 kHz edge, so the point stands more strongly than written, since nearly half the input features carry no speech on this data. The second slip is the runaway count: Section 4 says the six-words-per-second guard affects about 12 development utterances, while Appendix B counts 14 on the same pooled greedy adapter.
One row is a definitional question. Appendix F says that after folding hamza, ta marbuta and alif maqsura, "844 word types are still written more than one way, covering 21.9% of all tokens". Running that folding over the 3,152 public Subtask 1.2 dev transcriptions, we find 850 folded keys with more than one surface spelling, covering 22.1% of Arabic tokens: almost the same numbers, but for variation that folding removes, not variation that remains. Spellings that lose a majority vote within those groups cover 5.8% of tokens, close to the paper's "roughly 5 WER points" of unwinnable error. Our judgement: the sentence most likely describes the pre-folding variation, and the 5-point floor holds up under either reading.
How the routed Itgan front end works, and where each WER point comes from
The routing front end is a measurement the paper built after the evaluation, on Subtask 1.1 where the country label exists, to estimate what Subtask 1.2 (no labels, more than fifteen varieties) would gain from dialect identification. The probe is multinomial logistic regression on mean-pooled frozen whisper-large-v3-turbo encoder features, fit on the 12,800 training utterances. Each specialist continues the pooled adapter for 300 optimizer steps (three epochs) or 100 steps (one epoch) on one country's 1,600 utterances. A matched pooled control given the same steps moves country-average WER by under 0.1 points (Table 8), so the specialist gain is specialization, not extra training. A fresh adapter trained from the base on one country does worse than the pooled adapter (Mauritania 80.1 against 76.2).
The free steps are worth as much as the trained ones. On the pooled adapter, the length guard (1.01 points) and beam 5 (1.27 points) together give 2.28 points, larger than the 1.90-point oracle prize from eight specialists. Beam search alone makes the two over-generating dialects worse (Mauritania 73.3 to 77.1, Algeria 66.6 to 68.5) because runaway outputs rise from 8 to 26 across the corpus; the guard removes that interaction. The guard also overlaps with specialization: Mauritania's unguarded specialist gain of 6.3 shrinks to 2.8 once the guard is on.
The base-model result on Subtask 1.2 rests on a comparison the paper is candid about. large-v3 beats turbo by 3.76 points with matched recipes and by 2.65 at matched trainable parameters (28.8M against 27.9M), and the author argues decoder depth (32 layers against 4) is the likely reason, while noting that turbo's post-pruning training also changed its encoder. We measured how much. Reading two tensors per sampled layer from each released model.safetensors with HTTP range requests, every element differs, and cosine similarity drops with depth.
The numbers: q_proj cosine 0.90, 0.84, 0.82, 0.86 and 0.67 at layers 0, 8, 16, 24 and 31; fc2 cosine 0.84, 0.75, 0.73, 0.70 and 0.43. The relative change at layer 31 fc2 is 0.93, about the size of the weight matrix itself. Our judgement: this supports the paper's caution and tightens it. The large-v3 versus turbo comparison is a comparison between two different encoders as well as two decoder depths, and the top encoder layers, which feed the decoder, are the ones that moved most. The routing probe also runs on turbo's encoder, so its 65.3% is a property of that encoder, not of Whisper in general.
What this means for teams collecting dialect speech or building dialect ASR
- Price a dialect router against its break-even before training specialists. With Itgan's interference pattern the overall threshold is 41.6%, but it is per dialect, from 12% to 76%. A dialect whose specialist gains little and whose neighbours cost a lot (Jordan here) is better left on the pooled model unless the identifier is very accurate on it. The calculation needs only a cross-decoding matrix like Table 9, which costs one decode per specialist per dialect.
- Collect dialect labels you can trust, and enough of them per class. The paper's limit is the identifier, not the specialists: the gap to oracle is significant (95% interval -1.2 to -0.9), and tuning the same linear model on its features lifts accuracy only from 65.3% to 69.5%. For data buyers, verified speaker-region metadata at collection time is cheaper than inferring it later with a 65% probe.
- Budget for the orthography floor in your contract metric. On Subtask 1.2 references, about 22% of Arabic tokens belong to words spelled more than one way by our count, and folding both sides lowers zero-shot turbo WER from 62.2% to 57.2% (Appendix F). A transcription spec that fixes hamza, ta marbuta and alif maqsura conventions removes about 5 points of error that no model can win back.
- Check where code-switched speech actually switches. The public Tunisian validation split has 22.4% Latin-script words and the practice test 31.8% by our count, so a model tuned on validation sees less switching than it is scored on. The paper finds French and English words easier (12.5% error) than the Tunisian Arabic around them (20.2%), so effort belongs on the dialect, not on the embedded languages.
- Read the licence before reusing NADI data commercially. None of the NADI 2026 ASR datasets on Hugging Face declares a licence in its card metadata (checked 8 October 2026), while the TEDxTN source corpus is tagged CC BY-NC-ND 4.0. Our judgement: treat the derived splits as research-only until the organisers state terms.
For a related measurement of how far Whisper-class models are from usable on Maghrebi speech, see our note on the best recognizer getting 51% of words wrong on Moroccan YouTube speech. Itgan's 77% blind-test WER on Mauritania and 60% on Morocco (Table 1) sit in the same range.
How to reproduce every number in this note
All scripts live in one working directory and need only Python 3 with numpy. The headline (break-even, random router, per-dialect thresholds, Table 1, 2, 3, 7, 10 arithmetic and LoRA counts) is one command:
python3 -I headline.py
# break-even identification accuracy (routing helps only above): 41.6%
# random router (12.5% accuracy) expected change: +0.95 pts (worse than not routing)
# Jor own gain 0.6, wrong-route cost +1.93, break-even 76%
# LoRA large-v3 wide r32 computed 57.7M vs paper 57.7MThe release and data checks use public APIs, with no login:
curl -s "https://datasets-server.huggingface.co/size?dataset=UBC-NLP/NADI2026_subtask1.1_Robust_ASR"
curl -s "https://datasets-server.huggingface.co/size?dataset=fbougares/NADI_TUN_ASR_2026"
curl -s "https://datasets-server.huggingface.co/size?dataset=UBC-NLP/NADI2026_subtask1.2_MixedASR_test"
curl -s "https://huggingface.co/api/datasets/UBC-NLP/NADI2026_subtask2_MixedASR" # gated: false, no licence field
curl -s "https://api.github.com/search/repositories?q=almajai" # total_count 0 on 8 Oct 2026
python3 -I fetch.py # transcription column only, via /rows, 100 rows per call
python3 -I transcripts.py # Latin share 31.8% / 22.4%; 0 of 2,123 blind refs non-blank; 850 keys, 22.1%
python3 -I melbins.py # 28 of 128 bins above 4 kHz, 55 above 2.1 kHz
python3 -I encoder_sweep.py # range-reads 20 tensors from openai/whisper-large-v3{,-turbo}/model.safetensorsThe encoder check reads the 8-byte safetensors header length, then the JSON header, then the byte range of one tensor (data_offsets), so it fetches about 150 MB instead of 4.6 GB. Tensors checked: model.encoder.layers.{0,8,16,24,31}.self_attn.q_proj.weight and .fc2.weight, both stored as F16. The LoRA count uses r(d_in + d_out) per adapted linear, with eight attention projections per decoder layer (self and cross attention), from d_model 1280, encoder_ffn_dim 5120 and the layer counts in each config.json (32/32 and 32/4). Our transcript tokenizer strips punctuation and digits from the raw transcription field rather than the organisers' clean_transcription normalisation, which is why our Latin share (31.8%) sits 1.5 points under the paper's 33.3%.
What would prove the break-even rule wrong
Our claim: for per-dialect LoRA specialists continued from a pooled Whisper adapter, routing gain is close to linear in identification accuracy, with a break-even near the ratio of wrong-route cost to the sum of right-route gain and wrong-route cost (41.6% on Itgan's numbers), and dialects whose own gain is below their wrong-route cost need well over 60% accuracy.
It would be wrong if a NADI 2026 or later system paper, by 30 June 2027, reports a router below 40% accuracy that lowers country-average WER significantly against the pooled adapter with the same specialists, or reports measured recovery more than 10 points away from the even-spread prediction at the same accuracy. It would also be wrong for Jordan if a router at or below 70% accuracy on Jordan improves Jordan's WER under routing. We also predict that if a NADI 2026 spoken dialect identification entry (Itgan did not enter that track) is plugged into this front end at 80% accuracy, recovery will land between 60% and 72% of the oracle prize (our model says 65.7%).
Sources
- Ibrahim Almajai. Itgan at NADI 2026 shared task: Parameter-Efficient Whisper Adaptation for mixed-dialect and code-switched Arabic ASR (full title on arXiv). arXiv:2610.09934v1, 7 October 2026. Sections 2 to 7, Limitations, Tables 1 to 10, Appendices A to I.
- NADI 2026 data on Hugging Face, read 8 October 2026: UBC-NLP/NADI2026_subtask1.1_Robust_ASR (created 18 June 2026), its test set, UBC-NLP/NADI2026_subtask2_MixedASR (3,152 dev), Subtask 1.2 test (427), fbougares/NADI_TUN_ASR_2026 (24,893 / 731 / 842) and Subtask 1.3 blind test (2,123, blank references).
- OpenAI. whisper-large-v3 (last modified 12 August 2024) and whisper-large-v3-turbo (last modified 4 October 2024): config.json and model.safetensors, read 8 October 2026.
- Fethi Bougares et al. TEDxTN, ArabicNLP 2025; Hugging Face card tagged CC BY-NC-ND 4.0, read 8 October 2026.
- Sullivan et al. NADI-2026: The second multidialectal Arabic speech processing shared task. ArabicNLP 2026, Budapest (cited by the paper; proceedings link not yet found on 8 October 2026).
- GitHub search API, 8 October 2026: no repository matching "almajai" or an Itgan NADI system; a GitHub user named ialmajai (not confirmed as the author) has 5 public repositories, none for NADI 2026.
- BLOMEGA computations, 8 October 2026: break-even and per-dialect thresholds from Table 9, Table 1, 2, 3, 7 and 10 arithmetic, LoRA parameter counts, mel-bin centres, transcript counts, encoder cosine sweep.
FAQ
Should I put a dialect identifier in front of per-dialect Whisper adapters?
Only if the identifier is accurate enough. In Itgan's NADI 2026 Table 9 a specialist gains a mean 1.90 WER points on its own dialect and costs a mean 1.36 points on the seven others, so by our calculation routing breaks even at 41.6% identification accuracy and a random router adds 0.95 points. Their 65.3% linear probe recovers 44% of the oracle gain. The threshold is per dialect: Jordan needs about 76% and its router had 57.5%, and routing did make Jordan worse (48.7 to 49.5 WER).
Is whisper-large-v3-turbo the same encoder as whisper-large-v3?
No. Reading the released safetensors on Hugging Face on 8 October 2026, every element of the encoder tensors we sampled differs, and cosine similarity between the two checkpoints falls from 0.90 (layer 0 q_proj) to 0.43 (layer 31 fc2). Itgan's paper notes this, which is why its 3.76-point large-v3 advantage on mixed-dialect ASR cannot be credited to decoder depth alone.
Can I reproduce Itgan's NADI 2026 results?
Partly. The NADI 2026 data is public on Hugging Face, ungated, and every split count in the paper matches it (for example 10,808 Subtask 1.1 dev and 24,893 Tunisian training utterances). No Itgan code, adapters or hypotheses were public on 8 October 2026, and blind-test references are blank (0 of 2,123 non-empty for Subtask 1.3), so test scores can only be reproduced through the organisers.