DialectSentEval 2026: copying the input beats 7 of 15 Arabic sentiment-swap entries on BLEU, and 24% of test sentences are in the training file

Copying the input sentence scores BLEU 45.13 on the DialectSentEval swap dev set, above 7 of the 15 ranked test entries

We downloaded the released Subtask 2 files of DialectSentEval 2026 on 7 October 2026 and scored trivial baselines with the task's own classifier. Returning the source sentence unchanged gets BLEU 45.13 and chrF 65.03 against the human references on the 964-pair development set. In the paper's Table 2, 7 of 15 ranked test rows score below 45.13 BLEU and 5 of 15 below 65.03 chrF. A lookup that returns the training rewrite whenever a source sentence already appears in training, and copies otherwise, reaches BLEU 56.39 and chrF 71.77 on dev, next to the winner's test 56.91 and 72.18. These are dev numbers against test numbers, because test references are not released, so read them as a scale marker, not a re-ranking.

Two more measurements explain why. 461 of the 1,931 test sources (23.9%) occur verbatim in the training or dev file. And the official style classifier, CAMeL-Lab's bert-base-arabic-camelbert-da-sentiment, labels 6,751 of 6,752 training references and all 964 dev references with the target polarity, while it agrees with the given source polarity on only 68.4% of training rows. Our judgement: the references and the scorer move together, so the official rank (style accuracy only) measures how well a system pleases one classifier, which several teams reranked against directly.

What the DialectSentEval 2026 overview paper reports

arXiv:2610.06298v1, DialectSentEval 2026: Arabic Dialect Sentiment Analysis and Swapping Shared Task, was submitted on 5 October 2026 by Saad Ezzini (corresponding, King Fahd University of Petroleum and Minerals) and eight co-authors from KFUPM, Onaizah Colleges, Lancaster University, UM6P, VinUniversity, El Technology (Qatar) and the University of Luxembourg. It is accepted at ArabicNLP 2026, co-located with EMNLP 2026 in Budapest on 24 to 29 October. Fifteen teams took part and 16 system papers were submitted.

Subtask 1 is three-way sentiment (positive, negative, neutral) on MDS-3, described as 3,000 hotel-review sentences translated by native speakers into Moroccan, Egyptian, Lebanese, Jordanian and Saudi, "balanced across five major dialects", split 80/20 by sentence ID, scored by macro F1. ALEXIS won at 0.8667; rank 12 scored 0.7722. Subtask 2 rewrites a dialect sentence to the opposite polarity. The paper describes 9,647 pairs extended from MA'AKS, 5,331 negative and 4,316 positive, in Gulf (5,893), Levantine (1,655), Egyptian (1,561) and Maghrebi (538), split 6,752 / 964 / 1,931. Input is "an Arabic sentence along with its original sentiment polarity". Scoring is style accuracy by the CAMeLBERT-DA classifier (the ranking metric), plus BLEU and chrF "between the generated sentence and the reference sentence". ALEXIS won at 0.9581.

The paper's arithmetic is internally consistent: the dialect counts and the polarity counts each sum to 9,647, and the percentages round correctly (61.1, 17.2, 16.2, 5.6). The problems sit between the paper, the competition pages and the files.

Where the paper, Codabench and the released files disagree

Paper values from arXiv:2610.06298v1. Codabench values from the public API on 7 October 2026. Lahja values from Shaden-Saad/DialectSentEval2026 main.tex. Counts marked "our" are BLOMEGA computations, 7 October 2026.

The Subtask 1 numbers come from participants who held the files; we could not download Subtask 1 data ourselves. The Lahja team (Alzeer and Akram, University of Aberdeen) report the split, the 58 verbatim Egyptian test sentences and the 201 relabelled review IDs in their system paper. The ALEXIS Subtask 1 README describes an "exact-match lookup override": test sentences byte-identical to a training sentence receive the training label. That is a legitimate use of the released data, and it won.

The organisers' earlier AraSentEval 2026 site (OSACT7, LREC 2026) described the same 3,000-sentence MDS-3 as "balanced across four major dialects: Moroccan, Egyptian, Jordanian, and Saudi" and its swap set as 5,000 MSA pairs. Our reading: DialectSentEval added Lebanese as a test-only dialect and kept the old size figure. The 5,000-pair MSA MA'AKS file is public on Hugging Face as sabudalfa/MAAKS (no licence tag); the 9,647-pair dialect extension is not.

How one classifier sits on both sides of the swap benchmark

Style accuracy is the share of outputs the CAMeLBERT-DA sentiment model labels with the target polarity. We ran that model (Hugging Face revision b0df630, Apache-2.0) on the released files. It returns positive or negative for every human reference, matching the target on 6,751 of 6,752 training rows and 964 of 964 dev rows. On the source side it predicts neutral for 641 training sentences and agrees with the given polarity on 68.4% of training rows and 68.8% of dev rows.

Two readings fit those numbers. The references may have been filtered or revised until the classifier accepted them, or annotators may have written rewrites so overtly polar (emoji, strong adjectives) that any classifier agrees. Either way the consequence is the same: matching the reference style and pleasing the scorer are the same objective. AKARROUCH and Sabaa both rerank candidates by CAMeLBERT-DA target-sentiment probability (their READMEs), and NAMAA reports that reranking alone lifted dev style accuracy from 79.56 to 93.57.

The missing test polarity then moves the ranking. Teams first had to guess which way to flip. Sabaa reports reaching the polarity it aimed for 90.37% of the time but aiming in the right direction only about 80% of the time; NAMAA inferred direction with the scoring classifier, which by our measurement agrees with gold source labels 68.8% of the time on dev. Dev-to-test drops line up with that: Sabaa 0.9564 to 0.7447, NU-SentEval (areejkhaled) 0.7790 to 0.6898, NAMAA's Hassan account 0.8828 on the dev board to 0.7343 on test. A character n-gram logistic regression trained on the training sources predicts dev source polarity at 92.6% (90.7% on the 766 dev rows whose source is not in training), so the direction step is learnable, but every point lost there is a point of style accuracy lost.

The content metrics add little. Against the reference, copying already reaches 45.13 BLEU because human rewrites keep most words: median length is 10 words for both source and reference. Against the input, as the Codabench page words it, copying scores 100 and the human reference itself scores 45.13 BLEU and 65.16 chrF. Row 15 of Table 2 (style 0.2807, BLEU 43.06, chrF 64.61) sits close to our copy profile (0.222, 45.13, 65.03). That is a resemblance, not proof of what that team submitted.

Subtask 1 is less fragile. The first-to-second gap is 0.0457 macro F1 (5.57% relative, so the paper's "+4.57%" is percentage points), almost the whole 0.0488 spread from rank 2 to rank 12. A word and character TF-IDF SVM scored 0.709 on the same test set, 0.0632 below the last ranked system. Always predicting "positive" (38% of training labels) would give about 0.18 macro F1 if the test shares match training; that is our estimate, the shares of test labels are not reported. Codabench also ranks accuracy differently: LahjaSensei is second on accuracy (0.8476) while sixth on F1.

What this means if you localize into Arabic dialects or buy dialect sentiment data

• Do not use Subtask 2 style accuracy as a quality score for dialect rewriting. It is agreement with one BERT classifier that also shaped, or at least agrees perfectly with, the references. If you evaluate tone-shifting for marketing copy or dubbing scripts, use native-speaker judgement on polarity and meaning, and keep the classifier as a cheap pre-filter.
• Report a copy baseline next to any BLEU or chrF for minimal-edit tasks. On this data the floor is 45.13 BLEU and 65.03 chrF before a single word changes. Any vendor claim of "50 BLEU" on sentiment or formality transfer means little without it.
• Deduplicate before you split. 23.9% of swap test sources and 55% of the Egyptian Subtask 1 test sentences were already in training. When you commission dialect data, require that source texts are unique per split and that parallel renderings of one review stay on one side, and check it with a hash join on delivery.
• Ask for the dialect label per row. The paper reports four dialect families for 9,647 pairs, but the released files carry no dialect field, so nobody outside the organisers can compute per-dialect scores. For buyers, a dialect corpus without a per-row dialect tag cannot be audited for coverage, and Maghrebi is 5.6% of this one.
• Plan for disagreement across dialect renderings. 53.5% of Subtask 1 review IDs changed gold label between dialect versions. Whether that is real pragmatic shift or annotator noise, a translated sentiment set needs per-dialect adjudication, which is where native-speaker review budget belongs. Our earlier note on a no-model guess scoring 36.87 on dialectal Arabic error detection found the same pattern of trivial baselines sitting close to ranked systems.
• Check the licence before training. The organiser repo has no licence, Codabench terms say "research purposes only", and the public copy lives in a participant repo without a licence. Commercial use is not cleared by anything we found. See also our note on SHAMS, a Levantine benchmark with no commercial licence.

How to reproduce the copy baseline and the overlap count

Everything runs on a laptop CPU. Keep downloaded files in their own directory and run Python with -I.

mkdir -p dl/lahja scripts
R=ahmadalmoustafa/LahjaSensei-DialectSentEval-2026
for f in Train Val Test; do
  curl -sL -o dl/lahja/SentimentSwapSharedTask$f.xlsx \
    https://raw.githubusercontent.com/$R/main/data/SentimentSwapSharedTask$f.xlsx; done
shasum -a 256 dl/lahja/*.xlsx
# 7 Oct 2026: Test fd7c000f..539734, Train cc9617b7..6fe, Val ae2ce20e..637ff6
python3 -m venv venv && venv/bin/pip install openpyxl sacrebleu scikit-learn torch transformers
venv/bin/python -I scripts/s2_stats.py dl/lahja       # rows, polarity, overlap, copy BLEU/chrF
HF_HOME=$PWD/dl/hf venv/bin/python -I scripts/s2_camel.py dl/lahja        # official classifier, dev
HF_HOME=$PWD/dl/hf venv/bin/python -I scripts/s2_camel_train.py dl/lahja  # official classifier, train

Expected fields: val_copy_BLEU 45.13, val_copy_chrF 65.03, test_src_exact_in_train_or_val_src 461, Test_columns ["id","source"], train_reference_style_acc 0.9999 with 1 mismatch, train_source_label_agreement 0.6838. Sources are whitespace-normalised before matching; raw byte matches give 440 test and 197 dev overlaps. BLEU is sacrebleu.corpus_bleu(sources, [references]) with default tokenisation, chrF is corpus_chrf with word order 0.

The leaderboards and competition pages are public JSON:

curl -s "https://www.codabench.org/api/competitions/16337/"   # pages[].content: Evaluation tab text
curl -s "https://www.codabench.org/api/phases/26843/get_leaderboard/"   # Subtask 2 final board
curl -s "https://www.codabench.org/api/phases/26841/get_leaderboard/"   # Subtask 1 final board
curl -s -o /dev/null -w "%{http_code}\n" https://ezzini.github.io/DialectSentEval/   # 404 on 7 Oct 2026
curl -s https://api.github.com/repos/ezzini/DialectSentEval2026 | grep -E '"(license|pushed_at)"'

The Subtask 1 facts are in main.tex of github.com/Shaden-Saad/DialectSentEval2026, Section "Data" (lines 97 to 110 on 7 October 2026). The scoring programs on Codabench are marked private, so the exact BLEU reference side cannot be confirmed from outside.

What would prove this wrong

Our claim is that the Subtask 2 ranking rewards agreement with the CAMeLBERT-DA classifier and that its content metrics do not separate rewriting from copying. Three things would show we are wrong.

First, if the organisers release the 1,931 test references or the scoring program and a copy-the-input submission scores below 40 BLEU against them, our dev-set floor does not transfer to test. We predict it will land between 40 and 50. Second, if the official classifier labels fewer than 95% of the test references with the target polarity, the dev and train result (100% and 99.99%) is a coincidence of those splits. Third, if a camera-ready version or the ACL Anthology record by 31 March 2027 reports per-dialect Subtask 2 scores computed from a released dialect field, our point about the missing column is moot. We expect none of the three by that date.

Sources

1. Saad Ezzini, Shadi Abudalfa, Maram Alharbi, Salmane Chafik, Hind Alatawi, Mo El-Haj, Ahmed Abdelali, Osamah Alnahari, Salima Lamsiyah. DialectSentEval 2026: Arabic Dialect Sentiment Analysis and Swapping Shared Task. arXiv:2610.06298v1, 5 October 2026, accepted at ArabicNLP 2026. Sections 3 to 6, Tables 1 and 2.
2. ezzini/DialectSentEval2026 and its site ezzini.github.io/DialectSentEval2026, created 15 May 2026, 3 commits, no licence; checked 7 October 2026.
3. Codabench competitions 16336 (Subtask 1) and 16337 (Subtask 2), overview, dataset, evaluation and terms pages plus final leaderboards via the public API, read 7 October 2026.
4. ahmadalmoustafa/LahjaSensei-DialectSentEval-2026, data folder committed 15 August 2026: the three Subtask 2 files we measured.
5. Shaden Alzeer, Raja Naeem Akram. Lahja at DialectSentEval Subtask 1, paper source at Shaden-Saad/DialectSentEval2026: Subtask 1 split, label shares, overlap and TF-IDF baseline.
6. System repos and READMEs: ALEXIS Subtask 1, NAMAA Subtask 2, Sabaa Subtask 2, AKARROUCH; read 7 October 2026.
7. CAMeL-Lab/bert-base-arabic-camelbert-da-sentiment, Apache-2.0, revision b0df630; the official style classifier.
8. sabudalfa/MAAKS on Hugging Face (5,000 MSA pairs, last modified 31 January 2026) and Mughaus et al., MA'AKS, Language Resources and Evaluation 60(1), 2026.
9. ezzini/AraSentEval site text (OSACT7 at LREC 2026): earlier four-dialect description of MDS-3.
10. BLOMEGA computations, 7 October 2026: copy and lookup baselines, overlap counts, classifier agreement, polarity inference baseline, table arithmetic (scripts s2_stats.py, s2_camel.py, s2_camel_train.py, s2_extra.py, tables.py).

FAQ

Is the DialectSentEval 2026 data publicly downloadable?

Not from the organisers in an open form. The paper's footnote URL ezzini.github.io/DialectSentEval/ returned HTTP 404 on 7 October 2026; the real site is ezzini.github.io/DialectSentEval2026/, whose repository has no licence file. The data sits in the Files tab of Codabench competitions 16336 and 16337 under 'research purposes only' terms. A copy of the three Subtask 2 files (6,752 train, 964 dev, 1,931 test rows) is redistributed in the participant repo ahmadalmoustafa/LahjaSensei-DialectSentEval-2026, also with no licence. We found no Subtask 1 data on GitHub or Hugging Face.

How strong is a copy-the-input baseline on the Arabic sentiment swap task?

On the 964-pair development set, returning the source unchanged scores BLEU 45.13 and chrF 65.03 against the human references, with style accuracy 0.222 by the official CAMeLBERT-DA classifier. Seven of the 15 ranked test rows in the paper's Table 2 score below 45.13 BLEU. If BLEU and chrF are computed against the input, as the Codabench evaluation page says, copying scores 100 on both. The ranking uses style accuracy alone, so copying cannot win, but the content metrics do not separate rewriting from copying.

Which result from the shared task should a dialect data buyer trust?

Treat the Subtask 1 macro F1 range (0.7722 to 0.8667 on 525 test sentences, Lebanese unseen in training) as a rough ceiling for hotel-review sentiment, and note the winning system used an exact-match lookup against training text. Treat Subtask 2 style accuracy as agreement with one classifier: that classifier labels 6,751 of 6,752 training references as the target polarity but agrees with only 68.4% of training source labels, and the test file withheld the source polarity the paper says was given.