SHAMS, the Levantine Arabic pronunciation benchmark, checks out on paper but is not downloadable, and none of it is licensed for commercial use

SHAMS's numbers hold up, but on 5 October 2026 nobody outside the authors can run it, and a commercial team never can

The paper's arithmetic is clean. We recombined the per-source WER tables for Jordanian (Table 12) and Palestinian Urban (Table 13), weighting by the word counts in Table 1, and got the per-variety WER in Table 8 for all 18 system and variety cells to within 0.1 points. The release is the problem. Four days after submission, shams-nlp/shams on Hugging Face answers 401 to an anonymous request, the shams-nlp organisation lists 0 public datasets, and github.com/bensapirstein/shams does not exist publicly. The project page at shams-nlp.github.io is one logo image with a noindex tag.

The licence problem outlasts any release date. SHAMS draws 393 utterances from Casablanca, which Hugging Face lists as CC BY-NC-ND 4.0, and 907 from the LangMedia archive, whose content licence page reads "Creative Commons Attribution-NonCommercial-ShareAlike 4.0". Section 2.5 of the paper calls LangMedia "CC BY-SA". Our reading (not legal advice): every one of the 1,300 utterances comes from a NonCommercial source, so a vendor scoring a paid Levantine ASR, TTS or dubbing product on SHAMS has no clean licence to do so.

We also pulled Casablanca's Levantine splits, which SHAMS samples from, and found one figure that does not fit its pool. SHAMS reports 31.0% of Palestinian Rural words as spoken by women. Casablanca's own gender labels put women at 5.9% of words in the Palestine test split, the split all five Casablanca IDs printed in the paper come from.

What the SHAMS authors published on 1 October 2026

arXiv:2610.01427 v1, SHAMS: An Audio-Grounded Pronunciation Benchmark for Levantine Arabic, was submitted on 1 October 2026 at 10:26 UTC by Ben Sapirstein (Reichman University) with Roy Mattar (independent), Guy Mor-Lan (Hebrew University of Jerusalem), Ahlam Mohamed and Letizia Cerqueglini (Tel Aviv University) and Morris Alper (University of Miami). The arXiv comment says it is accepted to ArabicNLP 2026. There is one version.

Each utterance has four aligned tiers: audio, unvocalized Arabic, fully diacritized Arabic, and a phonetic transcription in the Semitic notation used by the Encyclopedia of Arabic Language and Linguistics. The five varieties are Palestinian Urban (230 utterances), Palestinian Rural (234), Jordanian (305), Lebanese (316) and Syrian (215), roughly 3,100 words each, 15,506 words and 128.2 minutes in total (Table 1). For 520 utterances the phonetic tier was written by hand; for the other 780, the Levanti model produced a first pass that annotators corrected against the audio, and 161 utterances (12.4%) kept diacritics identical to Levanti's output.

Four tasks are scored: diacritization (DER), grapheme-to-phoneme (CER), ASR (WER and CER) and audio-to-phoneme (CER), each reported per variety and at a "strict" and a "neutral" normalization tier. Inter-annotator agreement on diacritics, measured on 150 utterances and 7,290 letter slots, is 93.2% with Cohen's kappa 0.898 (Table 3). The phonetic and orthographic tiers were each produced by a single expert (Limitations).

The paper's claims against its release, its sources and its own tables

Paper values from arXiv:2610.01427v1. Sums, recombinations, word counts and release checks are BLOMEGA computations, 5 October 2026.

The recombination is the strongest evidence that the paper's results tables were produced by one consistent pipeline. WER in Table 8 behaves as a corpus-level rate: weighting the Casablanca and LangMedia halves of Jordanian by their word counts (1,451 and 1,722) reproduces Whisper v3's 39.5 exactly from 53.8 and 27.4. Weighting by utterances instead gives 39.3, so the tables also tell you which averaging the authors used. Table 3's per-variety slots sum to 7,290, the Palestinian and non-Palestinian rows match their components, and the Table 14 "All" column equals the utterance-weighted mean of the variety columns within 0.06 points for all five rows. The two discrepancies we found are a 0.1-minute rounding gap in Table 1 and a 4.5 in Appendix B.3 that the table puts at 4.4. Neither changes a conclusion.

The speaker-gender figure is harder to square. Casablanca labels each segment's speaker, and in the Palestine test split 625 of 667 segments are male and 41 female. All five Casablanca IDs quoted in the paper (6539_1, 46385_1, 46392_1, 46465_1, 46857_1) sit in Casablanca's test split, and 46385_1 and 46465_1 are labelled Palestinian Rural and Urban in SHAMS's own example tables. If every Palestinian Rural utterance came from that split, Casablanca's labels allow at most 518 female words, 16.4% of 3,153, not 31.0%. Adding the validation split takes the pool to 1,126 female words, close to the roughly 1,111 that SHAMS's Palestinian Rural and Urban Casablanca rows imply together. So either SHAMS assigned speaker gender itself, or it drew almost every female Palestinian segment Casablanca has from both splits. Our judgement: both are defensible, but the paper says neither, and the Limitations section already notes that source and gender are entangled.

Why audio-to-phoneme error is mostly the phonetic conversion, not the listening

The A2P experiment in Section 3.4 is the paper's most useful design for anyone building pronunciation data. Feeding the gold orthographic transcript through the Gemini 3.5 Flash G2P stage gives 13.0% CER. Feeding Gemini's own ASR output through the same stage gives 18.9%. Asking Gemini to go straight from audio to phonetic transcription gives 20.5%, worse than the cascade. GPT-Audio shows the same order: 23.3% cascaded, 24.9% direct. The authors read this as roughly 13 points intrinsic to phonetic conversion and about 6 added by ASR (18.9 minus 13.0 is 5.9, our arithmetic).

The 13-point floor is where dialect lives. Table 7 shows that no system recovers the rural Palestinian [k] reflex of qaf (ilḥak ya sīdi comes back as MSA [q] or urban glottal stop), and Appendix A lists one spelling, قاعد, realised as qāʿid, gāʿid, kāʿid and ʾāʿed across the corpus. Orthography does not encode that choice, and a text-only G2P stage cannot guess it. In the Casablanca Palestine test split we count 610 of 8,833 whitespace tokens (6.9%) containing ق; in Jordan test, 572 of 8,804 (6.5%). That is the share of words where a qaf reflex decision has to be made in this kind of material, before vowels and imāla are considered.

The variety breakdown points the same way. For Gemini 3.5 Flash, the gap between the best and worst variety is 1.3 DER points on diacritization and 1.8 CER points on G2P, but 3.8 CER points on ASR, 4.4 on A2P and 11.0 WER points on ASR. Whisper large v3's ASR CER runs from 10.9% (Palestinian Urban) to 32.0% (Palestinian Rural), a 21.1-point spread. Text tasks barely care which Levantine variety they get; audio tasks care a great deal.

What this means for teams building Levantine speech, dubbing or pronunciation data

• Do not plan a commercial evaluation on SHAMS. Casablanca is CC BY-NC-ND 4.0 and LangMedia's site says CC BY-NC-SA 4.0. Palestinian Rural is 234 of 234 Casablanca utterances, so the only rural variety in the benchmark is also the one whose audio SHAMS cannot ship. A team that needs rural Palestinian or any Levantine audio for a paid product needs its own consented recordings.
• Use the published numbers as a model-selection prior now. They are internally consistent. For Levantine diacritization a few-shot Gemini 3.5 Flash (11.6% DER) beats GPT 5.5 (15.0%) and Claude Opus 4.8 (18.2%), and all three beat the MSA tools at 24.4% to 27.4%. For G2P the three frontier LLMs sit together at 11.9% to 12.2% CER, ahead of Levanti at 18.3%.
• Budget human phonetic review per variety, not per language. Variety spread is small on text and large on audio (chart below). A dubbing QA plan that samples "Levantine" as one pool will hide a 21-point CER gap like Whisper v3's.
• Cascade, then review the phonetic step. Direct A2P lost to ASR plus G2P for both audio models. Our judgement: for pronunciation lexicons or TTS front ends, transcribe with the strongest dialect ASR, convert, and spend native-speaker time on qaf, interdentals and imāla, which the paper shows stay wrong after normalization.
• Record speaker metadata yourself. SHAMS's gender split cannot be checked against its source until the release is public. Our judgement: any benchmark that reports per-gender or per-speaker balance should ship the speaker labels it used, with their provenance.

A related BLOMEGA note on AlexandriaX dialectal Arabic error detection found a no-model guess scoring 36.87 there, above one of six teams. The common lesson for dialectal Arabic evaluation: check what a trivial or in-distribution system scores before reading a leaderboard, which is why SHAMS's own flag on Levanti (7.8% DER, partly its own silver labels) matters.

Check it yourself

Release state, each one command (run on 5 October 2026, 19:08 UTC):

curl -s -o /dev/null -w "%{http_code}\n" https://huggingface.co/api/datasets/shams-nlp/shams      # 401
curl -s https://huggingface.co/api/organizations/shams-nlp/overview | grep -o '"numDatasets":[0-9]*'   # 0
curl -s -o /dev/null -w "%{http_code}\n" https://api.github.com/repos/bensapirstein/shams          # 404
curl -s https://api.github.com/repos/shams-nlp/shams-nlp.github.io/commits | grep '"message"'     # 1 commit
curl -s https://huggingface.co/api/datasets/UBC-NLP/Casablanca | grep -o '"license:[^"]*"'         # cc-by-nc-nd-4.0
curl -sL https://www.langmedia.fivecolleges.edu/content-license | grep -o 'Attribution-NonCommercial-ShareAlike 4.0'

Casablanca pool counts and gender shares, from the datasets-server rows API (fields seg_id, transcription, gender, duration; audio not downloaded):

python3 - <<'PY'
import json, urllib.request
for cfg, n in [('Jordan', 848), ('Palestine', 667)]:
    for sp in ('test', 'validation'):
        rows = []
        for off in range(0, n, 100):
            u = f"https://datasets-server.huggingface.co/rows?dataset=UBC-NLP/Casablanca&config={cfg}&split={sp}&offset={off}&length=100"
            rows += [r['row'] for r in json.load(urllib.request.urlopen(u))['rows']]
        w = lambda g: sum(len(r['transcription'].split()) for r in rows if r['gender'] == g)
        q = sum('ق' in t for r in rows for t in r['transcription'].split())
        print(cfg, sp, len(rows), 'F words', w('F'), 'F share', round(100*w('F')/(w('F')+w('M')), 1), 'qaf tokens', q,
              [r['seg_id'] for r in rows if r['seg_id'] in {'6539_1','46385_1','46392_1','46465_1','46857_1'}])
PY

Expected: Palestine test 667 rows, 518 female words, 5.9%, 610 qaf tokens, IDs 46385_1, 46392_1, 46465_1, 46857_1; Jordan test 848 rows, 39.6%, ID 6539_1; Palestine validation 6.8%. Table recombination (word-weighted, Jordanian words 1,451 Casablanca and 1,722 LangMedia; Palestinian Urban 267 and 2,908):

python3 -c "
jor=[(34.5,34.9,34.7),(29.0,34.1,31.8),(53.8,27.4,39.5),(24.8,34.8,30.3),(76.1,61.6,68.2),(30.3,37.8,34.4),(22.9,30.5,27.1),(19.4,22.2,21.0),(29.4,28.1,28.7)]
pu=[(39.8,31.2,31.9),(36.5,28.4,29.1),(29.3,29.6,29.5),(32.0,35.6,35.3),(64.7,67.8,67.5),(28.9,40.1,39.2),(31.6,36.2,35.8),(29.3,26.3,26.5),(39.5,35.6,35.9)]
for rows,a,b in [(jor,1451,1722),(pu,267,2908)]:
    print(max(abs((c*a+l*b)/(a+b)-p) for c,l,p in rows))"

Expected maximum deviation: below 0.1 for both varieties. The LangMedia licence sentence in the paper is in Section 2.5, page 4; the Palestinian Rural female share is the M/F column of Table 1, page 3.

What would prove this wrong

Three claims are falsifiable. First, release: we say SHAMS was not publicly downloadable on 5 October 2026. A public commit or dataset revision on shams-nlp/shams or bensapirstein/shams dated on or before that day would refute it.

Second, licence: we predict that when the dataset card appears (we expect one by 31 December 2026), the LangMedia-derived rows will be listed under a NonCommercial licence, or the card will cite a separate written grant from the Five College Center for World Languages. A card that lists plain CC BY-SA for LangMedia rows with no such grant, while LangMedia's page still says CC BY-NC-SA 4.0, would mean the paper's statement stands and our reading of the source was wrong or outdated.

Third, gender: once the Casablanca IDs are public, joining them to UBC-NLP/Casablanca by seg_id will show either that Palestinian Rural draws on the validation split, or that SHAMS's gender labels differ from Casablanca's on dozens of segments. If the join shows only test-split IDs with Casablanca's labels yielding 31% female words, our count is wrong, and we will correct this note.

Sources

1. Ben Sapirstein, Roy Mattar, Guy Mor-Lan, Ahlam Mohamed, Letizia Cerqueglini, Morris Alper. SHAMS: An Audio-Grounded Pronunciation Benchmark for Levantine Arabic. arXiv:2610.01427v1, 1 October 2026, accepted to ArabicNLP 2026. Sections 2 to 5, Limitations, Tables 1, 3, 4, 6, 7, 8, 10, 12, 13, 14; Appendices A and B.
2. huggingface.co/datasets/shams-nlp/shams, HTTP 401 on 5 October 2026; shams-nlp organisation, 0 public datasets.
3. github.com/bensapirstein/shams, HTTP 404 on 5 October 2026; shams-nlp/shams-nlp.github.io, created 16 September 2026, 1 commit.
4. Talafha et al. Casablanca: Data and Models for Multidialectal Arabic Speech Recognition, EMNLP 2024; Hugging Face dataset UBC-NLP/Casablanca, CC BY-NC-ND 4.0, last modified 14 November 2024, read 5 October 2026.
5. Five College Center for World Languages. LangMedia content licence (CC BY-NC-SA 4.0) and LangMedia: Arabic, read 5 October 2026.
6. Guy Mor-Lan. Levanti: A Levantine Arabic dataset, Hugging Face, CC BY-NC 4.0; the model behind the silver first-pass labels.
7. BLOMEGA computations, 5 October 2026: Table 1 sums and source shares, Table 8 recombination from Tables 12 and 13, Table 3 and Table 14 checks, Casablanca Levantine row counts, gender word shares and qaf token counts.

FAQ

Can I download SHAMS today?

Not as of 5 October 2026. huggingface.co/datasets/shams-nlp/shams returns HTTP 401 (the shams-nlp organisation lists 0 public datasets), github.com/bensapirstein/shams returns 404, and shams-nlp.github.io is a logo-only page marked noindex, committed on 16 September 2026 as a "temporary landing page". The paper (v1, 1 October 2026) says the data, evaluation code and model outputs are released at those URLs.

Can SHAMS be used to evaluate a commercial Arabic ASR or TTS product?

Not on the licences as we read them, and this is our reading, not legal advice. The 393 Casablanca utterances (34.2 minutes) are CC BY-NC-ND 4.0 on Hugging Face and SHAMS ships only their IDs, not the audio. The 907 LangMedia utterances (93.9 minutes) come from a site whose content licence page says CC BY-NC-SA 4.0, while the paper describes it as CC BY-SA. So all 1,300 utterances trace back to a NonCommercial source. Palestinian Rural, the only rural variety, is 100% Casablanca.

Which model is best on Levantine Arabic in SHAMS?

It depends on the task. Diacritization: Levanti 7.8% DER, but it produced silver labels for 780 utterances, so Gemini 3.5 Flash at 11.6% is the best independent score. G2P: Gemini 3.5 Flash, Claude Opus 4.8 and GPT 5.5 at 11.9% to 12.2% CER. ASR: Gemini 3.5 Flash at 27.6% WER and 9.6% CER. Audio to phonemes: Gemini 3.5 Flash 20.5% CER direct, 18.9% through an ASR then G2P cascade, against a 13.0% gold-text floor.