A no-model guess scores 36.87 on AlexandriaX's dialectal Arabic error-detection task

A no-model guess scores 36.87 on the error-detection task, 2.03 above the sixth team

Subtask 3 of AlexandriaX asks a system to find the erroneous spans in an English-to-dialect machine translation and label each one with one of six LQM categories. The official ranking metric, the Overall Score, is the mean of a character-level Overlap F1 and an Error Class F1 (paper Section 3.2). The winner, Thakaa, scored 49.82; the NileChat-3B baseline scored 24.29 (Table 5).

We pulled the scoring function from the organisers' Subtask 3 Colab notebook, linked from alexandriax.dlnlp.ai, and ran it on the 138-row dev split released on Hugging Face. A prediction that marks each full MT output as a single span and labels it sociolinguistics scores 36.87 Overall (34.37 Overlap F1, 39.37 Error Class F1). On the 1,125-row training split the same guess scores 36.44, so the number does not hinge on the small dev set.

Dev and the blind test are different samples, so this is not a leaderboard entry. They are the same size and shape, though: 138 against 145 rows, split across the same five directions in the same proportions. Our judgement is that the Overall Score gives roughly 37 points for free, and the discriminating range on the leaderboard is 37 to 50, not 24 to 50.

What was released on 19 September 2026, and what came before it

arXiv:2609.22796, AlexandriaX 2026: The First Shared Task on Dialectal Arabic Machine Translation, by Abdellah El Mekki, AbdelRahim A. Elmadany, Samar M. Magdy, Saad Ezzini, Mo El-Haj, Mustafa Jarrar, Zaid Alyafeai, Bernard Ghanem and Muhammad Abdul-Mageed (UBC, KFUPM, Lancaster, HBKU, KAUST), submitted 19 September 2026. It is the overview paper for the shared task at ArabicNLP 2026, which the project page lists for presentation between 24 and 29 October 2026.

The task ran on a public timeline: training and dev data on 1 June 2026, blind test inputs on 20 July, final predictions due 25 or 30 July, official results on 30 July or 4 August (the page shows both dates for each of these two milestones). Registrations were 38, 33 and 35 for the three subtasks; 7, 4 and 6 teams submitted test predictions, 13 teams in all, 12 with system papers (Section 4.1).

• Subtask 1: context-aware English-to-dialect dialogue translation into 13 country varieties, built on the Alexandria corpus. Best spBLEU 30.42 constrained (at most 5B parameters, Alexandria data only) and 33.49 unconstrained, against a NileChat-3B QLoRA baseline of 21.17 (Tables 2 and 3).
• Subtask 2: direct cross-dialect translation of banking queries, 3,574 blind instances over 30 directed pairs, with no parallel training data. Thakaa won with 28.40 spBLEU using retrieval over the released test inputs; a zero-shot Gemma-4-31B baseline scored 24.69 and copying the source scored 12.81 (Table 4).
• Subtask 3: span-level error detection and classification in five directions (Egyptian, Emirati, Mauritanian, Moroccan, Palestinian), 1,125 train, 138 dev and 145 test rows.

How far above a no-model guess does each Subtask 3 team sit?

Official rows are the paper's Table 5 on the 145-row blind test. Our rows use the organisers' scorer, unchanged, on the released splits. All values are 0 to 100.

Official scores: arXiv:2609.22796 Table 5, and the leaderboard at alexandriax.dlnlp.ai/leaderboard.html, which gives six decimals (Thakaa 0.498212 = (0.542642 + 0.453782) / 2). Our rows: BLOMEGA measurement, 24 September 2026.

Five of the six teams clear the whole-output guess by between 2.39 (Axiom) and 12.95 (Thakaa) points. ArabicMTDiagnostics sits 2.03 below it and the official baseline 12.58 below it. The every-word guess, which floods the output with one span per token, scores 23.47: a lot of spans hurts, one big span helps.

Why one span and one label beat a fine-tuned 3B model

Three properties of the scorer combine. We read them from the code, not the paper.

1. Overlap F1 counts characters, not spans. _char_f1 builds a character mask for gold and for prediction per row. A span that covers the whole output recalls every gold character, and precision equals the fraction of the output that is erroneous. Dev outputs average 86.1 characters and carry 1.76 gold spans, which is where 34.37 comes from.
2. "Error Class Macro-F1" is computed on pooled counts. evaluate_jsonl_files adds up class-correct matches across all categories and computes one F1 from the totals, which is a micro score. Each prediction is paired with at most one gold span, the one it overlaps most, and sociolinguistics is 141 of the 243 dev spans (58.0%). Recomputed as a true per-class macro F1, the same guess scores 10.75, not 39.37. The paper and the scorer both call it Macro-F1.
3. Exact match is reported but not ranked. Our guess scores 0.52 exact-match F1 and it does not matter, because the Overall Score leaves exact match out.

The boundary problem is separate and affects everyone who trains on the data. In 494 of 1,997 training spans (24.7%) and 50 of 243 dev spans (20.6%), model_prediction[start:end] is not equal to the span's own text field: in 489 of the 494 training cases the slice is one character longer, usually a trailing space, and in the other 5 it is two characters longer. A system that finds exactly the gold words and marks them tightly scores 76.54 exact-match F1 against the dev gold, not 100. If the blind test gold has the same slack, part of every team's low exact-match score (the best was 23.19) is the annotation file rather than the model. REGLAT's 0.41 exact match with 42.72 Overlap F1, which its authors called a "dilation trap", shows how far apart the two metrics can drift.

The documentation disagrees with itself too. The project page defines the Overall Score as the "average of Exact Match F1 and Error Class Macro-F1". The paper and the leaderboard decimals use Overlap F1. Under the page's formula REGLAT would score 19.05 and drop from fourth to sixth, behind Axiom (23.52) and ArabicMTDiagnostics (21.87). The leaderboard is internally consistent; the page text is wrong.

Which dialects did no team fix? Mauritanian, in every row of both tables

We recomputed Subtask 1 from the per-dialect Tables 6 and 7. Every published average reproduces to two decimals. Beyond the averages:

• Mauritanian is the minimum for all 7 rows in both tracks, baseline included: 9.83 for the baseline, 20.18 for the best unconstrained system. The paper says it was "frequently among the weakest"; in the published numbers it is always the weakest. Syrian is the maximum for 5 of 7 unconstrained rows, Jordanian for the other 2.
• Every dialect was beaten by someone, by 10.35 to 17.77 spBLEU in the unconstrained track. The largest gain is Omani (+17.77, Thakaa); relative to the baseline, Mauritanian more than doubled (+105.3%) and still finished last.
• The overall winner does not win everywhere. CUNI, second overall at 32.50, beats Thakaa on Egyptian, Jordanian, Libyan, Moroccan and Sudanese. In the constrained track LahjaMT, second overall, wins Moroccan, Mauritanian and Tunisian.
• Alkhder, the only team below the baseline on average (20.36 against 21.17), beats it on exactly three dialects: Moroccan, Mauritanian and Tunisian, all three of them Maghrebi.
• Training volume barely predicts difficulty. Across the 11 dialects with training data, the Spearman correlation between training turns (2,034 Tunisian up to 14,933 Palestinian) and baseline spBLEU is 0.345. Baseline and winner rank the dialects almost identically (Spearman 0.94 over all 13), so whatever makes a dialect hard survives the best systems.

The two "unseen" dialects deserve a caveat the paper only half states. Libyan and Sudanese had no train or dev split, but the public test split on Hugging Face, released with the training data, contains 1,109 Libyan and 1,106 Sudanese turns with gold Arabic. Thakaa trained a LoRA adapter on those turns and gained 0.83 spBLEU in the constrained track, 29.59 to 30.42 (Section 4.3), which the rules allowed. The unseen-dialect result is therefore a measure of what 1,100 in-dialect examples buy, not of zero-shot generalisation. Teams without that route gained less there: FCDS gained 5.93 spBLEU over the baseline on the 11 seen dialects but 1.93 on the two unseen ones (our arithmetic from Table 7), and NAMAA's constrained system fell below the baseline on exactly those two.

Does the released data match the paper? Mostly, to the row

Our counts: BLOMEGA, 24 September 2026, from the Hugging Face files listed. "Not reported" means the document gives no figure.

The Subtask 1 release is clean: split sizes match the project page for all 13 dialects and three splits, and at most 5 of a dialect's roughly 1,100 public-test English turns also appear in its training split. The Subtask 3 test file shares an id, English source and direction with a training row in 78 of 145 rows. These are the same source sentences translated by a different MT system (only 2 MT outputs repeat), with gold spans visible for the training version. We tested whether that leaks: copying the sibling's spans wherever their text reappears scores only 11.24 Overall on dev, so it is a weak shortcut, not an answer key.

Subtask 2 has two oddities in its tables. NAMAA Community ranks third on 24.53 spBLEU, below the zero-shot Gemma-4-31B baseline at 24.69, so only two of four teams beat the model baseline on the primary metric; on chrF++ NAMAA (39.04) edges ALEXIS (39.02), reversing their spBLEU order. And ALEXIS, second overall, scores below the copy-the-source baseline on Egyptian (13.53 against 13.96) and Lebanese (14.68 against 17.32) targets while winning Moroccan, Palestinian, Saudi and Tunisian (Table 8). A single averaged score hides a system that is two different systems.

What this means if you evaluate or buy dialectal Arabic MT

• Put a no-model floor next to any span-detection score. For AlexandriaX Subtask 3 it is about 37 Overall. A vendor claiming 40 on this metric is 3 points above a constant; claim credit for the gap, not the level.
• Report per-class macro F1 when classes are this skewed. With sociolinguistics at 58% and graphetics at 2 spans in 1,997, a pooled score mostly measures whether you say "sociolinguistics". Thakaa's reported 75.35% category accuracy on matched spans is more informative than any of the pooled numbers.
• Normalise offsets before training. Re-anchor every gold span to its text field and strip whitespace. It is ten lines of code and it removes a fifth of the boundary noise from the training signal.
• Budget Maghrebi and Sahelian data first. Mauritanian finished last for every system, and the Maghrebi varieties are where the baseline is weakest. More Palestinian data (already 14,933 turns) is the least useful purchase in this corpus, in our judgement.
• Treat single-reference spBLEU as a floor. Our count of public-test reference turns containing a Latin-script word of three or more letters runs from 0.4% of Syrian turns to 45.8% of Tunisian ones (29.1% Moroccan, 19.7% Lebanese): code-switching the metric cannot credit when the system writes the same word in Arabic script.

For how reference-free QE handles post-edited localisation data, see our note on CometKiwi and LocQE post-edits; for another metric that rewards a shortcut, the WMT26 masculine-preference gambit.

Check it yourself

You need the dev file and the scorer from the organisers' Subtask 3 Colab (drive id 1tZNwN_MRGU41zkwVqPgtCLggb5WqWcd3, linked from the project page). Save its "Scorer" cell as scorer.py.

curl -sL -o dev.jsonl https://huggingface.co/datasets/UBC-NLP/AlexandriaX_Subtask_3/resolve/main/dev.jsonl
curl -sL -o nb.ipynb "https://drive.google.com/uc?export=download&id=1tZNwN_MRGU41zkwVqPgtCLggb5WqWcd3"
python3 - <<'PY'
import json
nb = json.load(open('nb.ipynb'))
exec(''.join(nb['cells'][18]['source']))          # the organisers' scorer cell
G = [json.loads(l) for l in open('dev.jsonl')]
with open('whole.jsonl', 'w') as f:
    for r in G:
        n = len(r['model_prediction'])
        f.write(json.dumps({'id': r['id'], 'direction': r['direction'],
            'LQM_prediction_tagged_errors': [{'text': r['model_prediction'], 'start': 0,
            'end': n, 'category': 'sociolinguistics'}]}, ensure_ascii=False) + '\n')
s = evaluate_jsonl_files('dev.jsonl', 'whole.jsonl')
print(round(100*s['overall_score'], 2), round(100*s['overlap_span_f1'], 2), round(100*s['class_macro_f1'], 2))
bad = sum(r['model_prediction'][e['start']:e['end']] != e['text']
          for r in G for e in r['LQM_prediction_tagged_errors'])
print('offset mismatches:', bad, 'of', sum(len(r['LQM_prediction_tagged_errors']) for r in G))
PY

Expected output: 36.87 34.37 39.37, then offset mismatches: 50 of 243. Swap dev for train to get 36.44 and 494 of 1,997. The cell index is 18 in the notebook as fetched on 24 September 2026; if it moves, search for def evaluate_jsonl_files.

For the Subtask 1 counts, the per-dialect Parquet files are at https://huggingface.co/datasets/UBC-NLP/alexandria/resolve/main/<CC>/<split>-00000-of-00001.parquet; count turns as the length of english_conversation per row. LY/test and SD/test carry the gold Arabic in dialectal_conversation.

What would prove this wrong

Our claim is that the whole-output guess would land in the same place on the blind test as on dev and train. We predict that if the organisers release the Subtask 3 test gold, or accept a post-evaluation submission on CodaBench, by 31 December 2026, the whole-output sociolinguistics prediction will score between 33 and 40 Overall on the 145 test rows. A score under 30 would mean dev and train are unrepresentative of the test, and the free floor we describe is smaller than we say.

A second, longer test: if a 2027 edition keeps the same Overall Score, we predict that at least one submitted system will score below the whole-output guess on its own test set. If the organisers switch to per-class macro F1 and tight offsets, we predict the gap between the top team and a no-model guess widens by more than 10 points, because the guess falls to about 11 on the class half.

Sources

1. Abdellah El Mekki, AbdelRahim A. Elmadany, Samar M. Magdy, Saad Ezzini, Mo El-Haj, Mustafa Jarrar, Zaid Alyafeai, Bernard Ghanem, Muhammad Abdul-Mageed. AlexandriaX 2026: The First Shared Task on Dialectal Arabic Machine Translation. arXiv:2609.22796, submitted 19 September 2026. Sections 2 to 4, Tables 2 to 10, Appendix B.
2. AlexandriaX 2026 project page (split sizes, timeline, Overall Score definition) and official leaderboard, both fetched 24 September 2026.
3. UBC-NLP/alexandria on Hugging Face, last modified 24 June 2026; UBC-NLP/AlexandriaX_Subtask_3, last modified 20 May 2026; UBC-NLP/AlexandriaX_Subtask_3_Test, last modified 16 July 2026.
4. AlexandriaX Subtask 3 Colab baseline, including the scorer evaluate_jsonl_files, fetched 24 September 2026.
5. BLOMEGA measurement, 24 September 2026: no-model baselines and a tight-boundary oracle scored with the organisers' scorer on the Subtask 3 train and dev splits; offset, overlap and category recounts; split-size, duplicate and train-test overlap audit of the Subtask 1 release; recomputation of Tables 2 to 10.

FAQ

What is the AlexandriaX 2026 shared task?

A shared task at ArabicNLP 2026 on dialectal Arabic machine translation, with three subtasks: context-aware English-to-dialect dialogue translation into 13 varieties, cross-dialect translation of banking queries across six dialects, and span-level MT error detection and classification in five directions. Thirteen teams submitted test predictions; the overview paper is arXiv:2609.22796, submitted 19 September 2026.

Who won, and by how much over the baselines?

Thakaa won all four rankings: 30.42 spBLEU in the constrained and 33.49 in the unconstrained Subtask 1 track against a 21.17 baseline, 28.40 spBLEU on Subtask 2 against a 24.69 zero-shot baseline, and 49.82 Overall on Subtask 3 against a 24.29 baseline.

How good is 49.82 on Subtask 3?

Better than it looks against the official baseline and worse than it looks in absolute terms. With the organisers' scorer, labelling each whole translation as one sociolinguistics error scores 36.87 on the released dev set and 36.44 on train, with no model. We read the leaderboard's useful range as roughly 37 to 50.