The mBERT quality gate in Surrey's edge speech translator fires on 86% of sentences, and switching it off scores no worse

At tau 0.90 the QE gate behaves like a random sample of 872 sentences

The paper (arXiv:2610.07545) uses mean-pooled mBERT cosine between the English source and the Opus-MT Spanish output as a quality estimate. When it falls below a threshold tau, the system generates an N-best list by beam search and picks one candidate, either by the same mBERT score (M1), by minimum Bayes-risk consensus over chrF (M2), or by anchor-boosted beam search (M3). Its best result, M2 at tau 0.90 and N=10, is +0.67 BLEU and +0.51 chrF over greedy decoding on the 1,012-sentence FLORES-200 devtest, p below 0.001.

Our rerun of that configuration gives +0.66 BLEU with a paired-bootstrap 95% interval of +0.31 to +1.03 (the repository README reports +0.349 to +1.025). The gate flags 872 sentences. Correcting all 1,012 instead gives +0.80 BLEU and +0.54 chrF. Over 20 random draws of 872 sentences, MBR averages +0.68 BLEU (range +0.57 to +0.86). The difference between ungated and gated MBR is +0.14 BLEU, 95% interval -0.03 to +0.36, so the honest reading is that the gate neither helps nor measurably hurts quality at this threshold. What it buys is compute: 140 sentences (13.8%) skip the second pass.

Our judgement: the paper's quality numbers are real and reproducible, but the attribution is not. The improvement is a property of MBR over a beam N-best list. The QE model's contribution at tau 0.90 is to send almost everything to MBR.

What arXiv:2610.07545 published on 6 October 2026, and what the repository holds

"Quality-Aware Self-Correcting Speech Translation on an Edge Device" by Zubair Ajmal Farooq (School of Computer Science and Electronic Engineering) and Diptesh Kanojia (Institute for People-Centred AI), both at the University of Surrey, was submitted to arXiv on 6 October 2026 as v1, cs.CL, 7 pages. The arXiv comment says it is a full paper submitted to the Convergence 2026 proceedings, with a poster at Convergence 2026. The cascade is Whisper-tiny (about 39M parameters) for ASR, Helsinki-NLP Opus-MT en-es (about 78M) for MT, bert-base-multilingual-cased for QE (about 680 MB resident) and eSpeak-ng for TTS. All four fit in 3.6 GB on the Nano; per-utterance latency is about 82 s with sequential model loading and about 41 s with all models resident.

The text-side evaluation runs on FLORES-200 devtest English to Spanish, which means the 0.67 BLEU result never touches ASR output. A 4 by 3 grid sweeps tau over 0.75, 0.80, 0.85, 0.90 and N over 3, 5, 10 for all three methods (Appendix A, Table 4). COMET-22 was computed on Google Colab because it does not fit on the Nano.

The arXiv comment links github.com/juebae/speech-translation_edge_device, which redirects to surrey-nlp/speech-translation_edge_device. On 7 October 2026 the GitHub API showed it created 5 February 2026, last pushed 5 October 2026, 26 commits (first commit dated 29 April 2026), 22 files, no licence file (the README says "Academic research project. Please contact the author before reusing it."). The README states that datasets/, results/ and all JSON files are git-ignored, and that the COMET, GER and bootstrap notebooks "are not in this repo". So the scripts are released; the hypotheses behind the tables are not.

Our rerun matches the paper's grid within 0.02 BLEU, and the controls it never ran change the reading

We downloaded the public FLORES-200 tarball, translated eng_Latn.devtest with Opus-MT greedy (max length 128), scored each source and output pair with mBERT exactly as phase3a_qe_module.py does, generated beam N-best lists for N of 3, 5 and 10, and applied M1 and M2 as in phase4_selfcorrect.py. We skipped M3 and COMET. Every paper value we could recompute came back within 0.02 BLEU or chrF.

Two controls stand out. First, at N=3 the plain beam-search top hypothesis on gated sentences gives +0.60 BLEU, slightly above MBR's +0.57: at that pool size the "self-correction" gain is beam search beating greedy decoding. MBR only pulls ahead at N=10 (+0.66 against +0.46). Second, the ranker finding holds. Inside an N-best list the mBERT score has essentially zero rank correlation with chrF against the reference (mean Spearman 0.007, -0.006 and 0.02), and picking a random candidate does better than M1 at N=10 (+0.11 against -0.14).

As a sentence-level detector the score is weak but not empty. Mean-pooled cosine correlates with sentence chrF at Spearman 0.17 (p of 5e-8), above a length-ratio heuristic at 0.11 and a random score at 0.04. Sentences the gate flags at tau 0.90 average sentence chrF 54.17 against 56.46 for those it passes, and MBR's mean per-sentence chrF gain is 0.56 inside the gate against 0.32 outside it at N=10. The problem is the threshold: a signal this soft cannot separate much when it fires on 86% of the data.

Why tau 0.90 sends almost every sentence to correction

The paper describes the QE score as cosine similarity "s in [-1, 1]". The released code does something else: it computes the cosine of mean-pooled final hidden states and then rescales it as (sim + 1) / 2, clipped to 0 to 1. A threshold of 0.90 on that scale is a raw cosine of 0.80. On our 1,012 greedy outputs the raw mean-pooled cosine has median 0.742 (10th to 90th percentile 0.668 to 0.810), so the rescaled score sits between 0.834 and 0.905 for 80% of sentences. Every tau in the grid lands inside or below that narrow band, which is why the trigger rate jumps from 1.3% to 22.7% to 86.2% across three steps of 0.05.

Pooling matters as much as scale. The paper does not say which pooling it uses; the code mean-pools over all tokens including [CLS] and [SEP]. With [CLS] pooling the same tau 0.90 would fire on 50.1% of sentences, and the [CLS] cosine has no correlation with sentence chrF at all (Spearman -0.026, p of 0.41). Anyone porting the gate to a different encoder or pooling will get a different trigger rate from the same tau, so the threshold should be set from a target trigger rate, not copied.

What this means if you run MT or dubbing pipelines with a QE trigger

For an on-device or cost-capped pipeline, the practical rule from this rerun is to measure the gate against two controls before trusting it: correct everything, and correct a random sample of the same size. If neither loses to the gated system, the gate is a compute budget, not a quality signal, and you can set its threshold purely on latency. On this setup a gate that skips 13.8% of sentences costs about 0.14 BLEU (not significant) and saves 13.8% of second-pass decoding.

The paper's memory argument survives in a modified form. Its point that the QE model can be unloaded during selection (freeing about 680 MB) is correct, because MBR never consults it. Our data suggest a stronger version: at tau 0.90 the QE model could be dropped from the critical path entirely and replaced by "always run MBR", or by a length-ratio check that ranks sentence chrF at Spearman 0.11 against mBERT's 0.17 at no memory cost. That is our inference from one language pair, one domain and text input.

For localization QA routing, where a QE score decides which segments a human post-editor sees, the result is a warning about sentence-embedding similarity as QE. A within-list Spearman near zero means the score cannot tell close paraphrases apart, and the whole-sentence AUC of 0.61 for flagging the worst quarter is close to the 0.57 of a length heuristic. Teams routing human review should calibrate any QE model on their own human-scored segments; the repository README's Pearson 0.41 between this score and COMET is a correlation with another metric, not with a human judgement.

Three internal inconsistencies are worth knowing before citing the paper. Table 3 lists 485, 542 and 580 triggered sentences for N of 3, 5 and 10 at tau 0.90 (47.9% to 57.3%), while Table 4 and the code give 86.2% (872 sentences) and the gate is computed before N is chosen, so the trigger count cannot vary with N. Table 1 says it reports "the strongest configuration for each correction method" but shows M1 and M3 at N=10, their weakest settings; M1's best is N=3 at +0.36 BLEU, and at N=3 the M1 versus M2 gap is 0.23 BLEU, against the 0.81 that Table 1 displays at N=10. And the README says median GER is highest at N=5 (+0.0081) while the paper's Table 3 puts N=5 lowest (+0.0023) and N=3 highest. We could not check COMET or GER ourselves because the hypotheses are not released.

Check it yourself

Release state, one command each (results as of 7 October 2026: 26 commits, no license, no data files):

curl -sL https://api.github.com/repos/surrey-nlp/speech-translation_edge_device | grep -E '"(created_at|pushed_at|license)"'
curl -sIL "https://api.github.com/repos/surrey-nlp/speech-translation_edge_device/commits?per_page=1" | grep -i '^link'
curl -sL "https://api.github.com/repos/surrey-nlp/speech-translation_edge_device/git/trees/main?recursive=1" | grep '"path"'

Full rerun on CPU (about 11 minutes on an Apple-silicon laptop; Python 3.14, torch 2.14.1, transformers 5.19.0, sacrebleu 2.6.0). Our two scripts follow phase3a_qe_module.py and phase4_selfcorrect.py; the only intentional change is batching of 8 sentences in generation.

python3 -m venv venv && ./venv/bin/pip install torch transformers sentencepiece sacremoses sacrebleu scipy
mkdir -p dl/flores && curl -sL https://dl.fbaipublicfiles.com/nllb/flores200_dataset.tar.gz | tar -xz -C dl/flores \
  flores200_dataset/devtest/eng_Latn.devtest flores200_dataset/devtest/spa_Latn.devtest
./venv/bin/python -I run_check.py dl/flores/flores200_dataset/devtest results   # writes results/raw.json
./venv/bin/python -I analyze.py results/raw.json results/summary.json           # grid, controls, correlations
./venv/bin/python -I gate_test.py results/raw.json results/gate_test.json       # paired bootstrap, seed 42

Fields to read: summary.json keys baseline (26.09, 54.85), trigger["0.9"].ours_scaled_pct (86.2), grid rows with tau 0.9, controls_tau0.90["10"].M2_ungated_all_sentences_dBLEU (0.8) and M2_random_gate_same_count_dBLEU_mean (0.68), within_nbest_mean_spearman_qe_vs_chrf; gate_test.json key "10".dBLEU_ungated_minus_gated_ci95 (-0.03 to 0.36). The paper's Table 3 versus Table 4 trigger mismatch is visible on pages 4 and 6 of the PDF.

What would prove this wrong

Our claim is that on FLORES-200 English to Spanish the mBERT gate at tau 0.90 adds no measurable quality over correcting every sentence or a random 86% of them, and that the paper's gains come from beam search plus MBR.

It would be wrong if the authors' own hypothesis files, once released, show gated M2 beating ungated M2 by a significant margin on COMET-22 (one-tailed p below 0.05), which we could not test. It would also be wrong if, on noisy ASR transcripts from FLEURS rather than clean text, a gate tuned to fire on 50% or fewer of sentences beat a random gate of the same size by 0.3 BLEU or more. We predict that by 30 June 2027 no published follow-up of this system will report such a gate-versus-random or gate-versus-all comparison favouring the mBERT gate, and that any reported comparison with COMET-Kiwi as the gate will show a trigger rate under 50% doing at least as well as tau 0.90 with mBERT.

Sources

1. Zubair Ajmal Farooq, Diptesh Kanojia (University of Surrey). Quality-Aware Self-Correcting Speech Translation on an Edge Device. arXiv:2610.07545v1, submitted 6 October 2026. Abstract, Sections 3 to 8, Tables 1 to 4, Appendix A.
2. surrey-nlp/speech-translation_edge_device (redirect from juebae/speech-translation_edge_device), checked via api.github.com on 7 October 2026: created 5 February 2026, last push 5 October 2026, 26 commits, no licence file. Files read: README.md, phase3a_qe_module.py, phase3a_mt_module.py, phase4_selfcorrect.py, requirements.txt.
3. NLLB Team / Meta. FLORES-200 dataset tarball, devtest eng_Latn and spa_Latn, 1,012 lines each, downloaded 7 October 2026 (HTTP 200).
4. Helsinki-NLP. opus-mt-en-es on Hugging Face; Google. bert-base-multilingual-cased on Hugging Face. Both downloaded 7 October 2026.
5. Fernandes et al. Quality-Aware Decoding for Neural Machine Translation. NAACL 2022. Cited by the paper for QE reranking.
6. Freitag et al. High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics. TACL 2022. Cited by the paper for MBR.
7. BLOMEGA computations, 7 October 2026: CPU rerun of greedy, QE, beam N-best, M1 and M2 over the full tau by N grid; ungated, random-gate, beam top-1 and random-candidate controls; Spearman and AUC against sentence chrF; paired bootstrap (1,000 resamples, seed 42); cross-check of Tables 1, 3 and 4 and the README.

FAQ

Does an mBERT cosine similarity gate pick out the translations that need fixing?

Weakly. On 1,012 FLORES-200 English to Spanish sentences translated greedily by Opus-MT, mean-pooled mBERT cosine correlates with sentence chrF at Spearman 0.17, against 0.11 for a plain length-ratio heuristic, and flags the worst quarter of sentences with an AUC of 0.61 against 0.57 for length ratio. At the paper's threshold of 0.90 it flags 872 of 1,012 sentences (86.2%), and MBR applied to all 1,012 scores +0.80 BLEU versus +0.66 when gated.

Is the mBERT QE score useful for reranking N-best translations?

No, and our rerun agrees with the paper on this half of its claim. Inside each beam's N-best list the mBERT score correlates with sentence chrF at a mean Spearman of 0.007 (N=3), -0.006 (N=5) and 0.02 (N=10). QE reranking at N=10 changes BLEU by -0.14, below simply taking the beam's top hypothesis (+0.46) and below MBR (+0.66).

Can I reproduce the arXiv 2610.07545 results without a Jetson Nano?

The BLEU and chrF part, yes. The released repository github.com/surrey-nlp/speech-translation_edge_device (26 commits, last push 5 October 2026, no licence file) holds the scripts but no data, hypotheses or COMET notebooks. Following its code on a laptop CPU with the public FLORES-200 devtest, we got baseline BLEU 26.09 and chrF 54.85, trigger rates 0.1%, 1.3%, 22.8% and 86.2%, and MBR N=10 at +0.66 BLEU, against 26.09, 54.85, 22.7% and +0.67 in the paper. The full run took about 11 minutes.