Japanese subtitles scored 12.19 chrF at IWSLT 2026. Re-scored without the word segmenter, 28.18

The 2026 subtitling track added Japanese, Chinese and a YouTube domain

July 2026, San Diego. The IWSLT automatic subtitling track, running since 2023, expanded to five target languages (Arabic, Chinese, German, Japanese, Spanish) from English audio-visual content across three domains: ITV entertainment series, Asharq Business with Bloomberg economic news, and audio from the YODAS YouTube dataset, which is new this year. Three teams entered: AppTek, the MT unit of Fondazione Bruno Kessler (FBK), and Huawei Translation Service Center (HW-TSC). Results are in Speech Translation and Metrics in 2026, Track IV, Tables 31 to 33.

The constraints came from published industry style guides, not from the organisers' preference. Maximum reading speed of 21 characters per second for Arabic, German and Spanish from TED's subtitling tips, 4 for Japanese and 9 for Chinese from Netflix's Japanese and Chinese Simplified timed text style guides. Line length 42, 13 and 16 characters respectively, whitespace included. Two lines per subtitle, everywhere. Half-width characters count as 0.5 in Japanese.

Scoring is three-sided. Subtitle quality is SubER, the primary ranking metric. Translation quality is BLEU, chrF and BLEURT. Compliance is measured as the rate of subtitles inside the CPS limit, the rate of lines inside the CPL limit, and the rate of subtitles at or under 2 lines. Before any metric is computed, automatic subtitles are realigned to the reference with mweralign, a variant of the AS-WER algorithm. That realignment step is where the Japanese problem starts.

The published Japanese numbers and the re-scored ones

Table 1 is the 2026 test set for the ITV entertainment domain, primary systems only, one row per language, plus the organisers' own re-scoring of the two CJK rows.

Read the last row against the first. The human reference Japanese subtitles comply with the 4 characters-per-second limit 44.89% of the time. The machine complies 91.88% of the time. The benchmark's own ground truth violates the benchmark's own rule more than half the time, and a system is penalised on SubER and BLEURT for matching the rule rather than the reference.

The organisers did the arithmetic on the relaxation too. Moving the Japanese threshold from 4 characters per second to 6 raises the reference compliance rate from 44.89% to 95.74%. Two characters per second is the difference between a rule almost nobody follows and a rule almost everybody follows.

The Japanese penalty is applied twice, and only once on purpose

Automatic subtitles do not line up with reference subtitles block for block, so the scorer realigns them first. For languages with spaces, realignment operates on tokens the text already provides. For Japanese it has to invent them.

The organisers are explicit about the logic: chrF is sensitive to word-level segmentation during hypothesis re-segmentation but not during evaluation itself, so comparing the two scoring paths isolates the segmentation effect. Chinese still gained 7.38 points, because realignment still ran, but Chinese evaluation is character-level and therefore largely insulated. Japanese gained 15.99. The paper's conclusion, in its own words: "the underlying translation quality is not as divergent as the official scores would indicate."

This does not make the Japanese output good. BLEURT for AppTek Japanese is .2521 against .5681 for Chinese and .5735 for Spanish, and BLEURT does not run through mweralign word tokens the same way. The honest reading is that roughly half of the visible Japanese deficit is instrumentation, and the rest is real.

On a frozen test set, translation quality really did move

The track keeps re-scoring old submissions on old test sets, which makes a clean four-year series possible. ITV tst23, German and Spanish, primary runs only.

The compliance columns tell the opposite story. On German tst26, AppTek's contrastive run hit 99.26% reading-speed compliance and the primary run hit 90.48%, and the primary run scored better BLEURT. On the Asharq-Bloomberg German section, the organisers note AppTek's 2026 system improved BLEURT from .6020 to .6234 while reading-speed compliance fell from 92.44 to 73.50. Their summary: the optimal tradeoff between translation quality and subtitling spatiotemporal constraints "has yet to be conclusively resolved."

The 2026 systems reveal how that tradeoff is now negotiated. AppTek runs a hybrid production ASR plus Transformer Big MT with genre, speaker gender and length-class metadata, uses its Intelligent Line Segmentation to place breaks, iteratively shortens translations with the length parameter until CPS, CPL and LPB are all satisfied, and post-edits with gpt-4o in roughly 20-sentence chunks. For Japanese and some German runs it disables strict CPS enforcement, explicitly because it avoids a large drop in MT metrics and matches the low compliance of the human references. FBK runs an entirely open stack: SpeechBrain VAD, Whisper large-v3, MADLAD-400-10B-MT, with a second pass that re-transcribes aggregated segments using Voxtral's 32k-token context and realigns via mweralign. HW-TSC compresses non-compliant blocks with Qwen3-32B in two passes, greedy at temperature 0 removing auxiliaries and conjunctions, then temperature 0.3 for deeper compression, always retaining proper nouns.

What to do with this if you buy subtitling

Never accept a cross-language metric comparison in CJK without asking how it was segmented. A vendor reporting "chrF 12 in Japanese versus 28 in Chinese" may be reporting the same translation quality through two different scorers. Ask for the tokenisation, and ask for a character-level re-score alongside.

Write the reading-speed limit you will actually enforce into the brief. If your reference material is 44.89% compliant at 4 cps and 95.74% at 6, then 4 is an aspiration and 6 is your operating rule. Say which one the vendor is being scored against, because at 4 cps a system that complies is punished by every reference-based metric you are also using.

Score compliance and quality separately and weight them yourself. Every 2026 result shows them trading against each other: AppTek turned CPS enforcement off in Japanese to protect its MT scores, and lost 18.94 points of Asharq German compliance to gain 0.0214 BLEURT. That is your decision to make, not your vendor's.

An open stack is now within a few points of a production stack. On German tst26 ITV, FBK's fully open pipeline scored SubER 75.51 and BLEURT .5252 against AppTek's 74.01 and .5408, and FBK beat AppTek outright on German in the Asharq-Bloomberg domain. If your constraint is that no audio leaves your infrastructure, the penalty for that in 2026 is small and measurable.

Budget separately for Japanese. Even after correcting for the segmenter, AppTek's Japanese BLEURT of .2521 sits less than half of its Spanish .5735. Japanese needs the reviewer hours, and the 13-character line limit with half-width counting is a real production constraint that no length-control parameter fully absorbs.

Check it yourself

# 1. the tables and the re-scoring paragraph (Track IV)
curl -sL -o iwslt2026.pdf https://aclanthology.org/2026.iwslt-1.39.pdf
python3 -c "
import pypdf
t='\n'.join(p.extract_text() or '' for p in pypdf.PdfReader('iwslt2026.pdf').pages)
i=t.find('Track IV Subtitling'); print(t[i:i+9000])"     # task, thresholds, systems
python3 -c "
import pypdf
t='\n'.join(p.extract_text() or '' for p in pypdf.PdfReader('iwslt2026.pdf').pages)
i=t.find('Table 33:'); print(t[i-4200:i+400])"           # ITV results, all languages

# 2. the metrics, as code
pip install subER sacrebleu
#    SubER:      https://github.com/apptek/SubER
#    compliance: https://github.com/hlt-mt/FBK-fairseq/blob/master/examples/
#                speech_to_text/scripts/subtitle_compliance.py
#    realigner:  mweralign, a variant of AS-WER (Matusov et al., 2005)

# 3. reproduce the segmentation effect on your own Japanese SRT pair
#    score once with tok=ja-mecab, once with the whole file as a single
#    character string, and compare chrF. The organisers' delta was +15.99.
sacrebleu ref.ja.txt -i hyp.ja.txt -m chrf --tokenize ja-mecab
python3 -c "
import sacrebleu
r=open('ref.ja.txt').read().replace('\n','')
h=open('hyp.ja.txt').read().replace('\n','')
print(sacrebleu.sentence_chrf(h,[r]).score)"

# 4. the reading-speed arithmetic in this note
python3 - <<'PY'
ref = {"4 cps": 44.89, "6 cps": 95.74}
sys = {"AppTek prmry": 91.88, "AppTek cntrs1": 75.78,
       "FBK prmry": 61.60, "FBK cntrs1": 56.63, "FBK cntrs2": 23.16}
print("relaxing 4 cps to 6 cps moves the human reference by",
      round(ref["6 cps"]-ref["4 cps"],2), "points")
beat = [k for k,v in sys.items() if v > ref["4 cps"]]
print(len(beat), "of", len(sys), "system runs are more compliant than the reference:", beat)
PY
# relaxing 4 cps to 6 cps moves the human reference by 50.85 points
# 4 of 5 system runs are more compliant than the reference: ['AppTek prmry', 'AppTek cntrs1', 'FBK prmry', 'FBK cntrs1']

The style guides the thresholds come from are public: TED subtitling tips, Netflix Japanese and Chinese Simplified timed text guides. Check yours against them before you set an acceptance threshold.

What would prove this wrong

The claim under test is that a large share of the Japanese subtitling deficit on reference-based metrics is produced by word-level re-segmentation rather than by translation. It is wrong if a controlled re-scoring of the IWSLT 2026 Japanese submissions, published by 31 December 2027, shows a character-level chrF within 2 points of the segmenter-based chrF, that is a delta under 2 rather than the +15.99 the organisers measured. It is also wrong if BLEURT, which does not depend on the same tokenisation, moves by a comparable amount under a character-level realignment; the organisers reported the chrF delta only, so this half is untested.

A second prediction, marked as judgement: the 2027 subtitling track will either relax the Japanese reading-speed threshold above 4 characters per second or report reference compliance alongside system compliance, because keeping a rule that the ground truth breaks 55% of the time makes the compliance column unusable. If the 2027 task page still specifies 4 cps with no reference baseline published, this reading was wrong.

FAQ

Why do Japanese subtitles score so badly on automatic metrics?

Mostly word segmentation. Japanese has no whitespace, so the scorer must segment before aligning. Re-scoring AppTek's ITV Japanese as character strings moved chrF from 12.19 to 28.18. Chinese, evaluated at character level, moved from 28.14 to 35.52. The gap fell from 16 points to 7.

What is the maximum subtitle reading speed for Japanese?

IWSLT 2026 used 4 characters per second, half-width counted as 0.5, from Netflix's Japanese timed text style guide, against 21 for Arabic, German and Spanish and 9 for Chinese. Line length was 13 characters for Japanese, 16 for Chinese, 42 for the rest, and 2 lines maximum everywhere.

Do human subtitlers meet the 4 cps limit?

In this data, 44.89% of the time on the ITV development references, rising to 95.74% at a 6 cps threshold. AppTek's primary Japanese system complied 91.88% of the time.

Has automatic subtitling improved since 2023?

On translation quality, yes, measured on a frozen test set: best primary BLEURT on ITV tst23 rose from .4438 to .5520 for German and .4530 to .5514 for Spanish. SubER and compliance were volatile and peak values were not always the most recent.

Sources

1. IWSLT 2026 organisers (60 authors), Speech Translation and Metrics in 2026: Findings of the IWSLT Campaign, Proceedings of the 23rd International Conference on Spoken Language Translation, San Diego, July 2026. Track IV (Subtitling): task description and thresholds, Table 8 (set statistics), metrics, system descriptions, Results (Japanese re-scoring, reference CPS, year-over-year trend), Tables 31 to 33. PDF.
2. Mauro Cettolo, Roldano Cattoni, Matteo Negri, Luisa Bentivogli, The FBK Sentence-Aware Subtitling System at the IWSLT 2026 Subtitling Track, IWSLT 2026, pages 68 to 77. The two-stage open-source pipeline described above.
3. HW-TSC, HW-TSC's Submission to the IWSLT 2026 Subtitling Track, IWSLT 2026. Qwen3 streaming ASR plus Qwen3-32B two-pass subtitle compression.
4. Style guides the thresholds are taken from: TED subtitling tips; Netflix Japanese Timed Text Style Guide; Netflix Chinese Simplified Timed Text Style Guide.
5. Metric implementations: SubER (Wilken et al., 2022); the FBK-fairseq subtitle compliance script; sacreBLEU (Post, 2018) for BLEU and chrF; BLEURT (Sellam et al., 2020).
6. Compliance arithmetic in this note computed by BLOMEGA on 16 September 2026 from the Track IV figures. Code in "Check it yourself".

Related BLOMEGA guides: Your multilingual LLM judge prefers the machine translation · The cost of a dubbed minute in 2026 · Localization is the new default.