A trim-and-pad script beats Index-Homura's 81.92% syllable hit rate on SandGlass, so dubbing teams must read the quality column

A trim-and-pad script beats Index-Homura's syllable hit rate, so the quality column carries the result

SandGlass (released as IndexTeam/Sandglass-Bench, MIT licence) has 3,600 cases: 300 Chinese subtitle lines, 4 target languages and 3 length targets each. Only the 900 English rows carry a reference translation. On those 900 we ran syllable_metrics() from the release's own evaluate.py, unchanged, on two inputs. First, the reference unchanged for all three targets. It gets 18.67% within 10%, 20.22% within one syllable, and a slope of 0 (it can't respond to the target). Second, the reference with trailing words cut until it fits, padded with "yeah" when it falls more than one syllable short. That gets 100% within one syllable, 93.33% within 10%, mean relative deviation 0.049 and a slope of 0.962.

Compare that with paper Table 9. Index-Homura-9B (RL) gets 74.42% within one syllable, 81.92% within 10%, deviation 0.0693 and slope 0.968. Table 10 puts its English-only rate at 78.67%. Our script's outputs read like "If we go by the yeah yeah yeah" for a 9-syllable target, so they would fail any quality check. That's the point. In our judgement, syllable adherence on SandGlass is easy to game and close to meaningless on its own. Homura's real claim is the pair: 81.92% adherence at 0.7863 judged quality. Table 9 also shows that pair is worse on quality than the Homura-9B SFT checkpoint (0.8581), GPT-5.6-Sol (0.8715) and DeepSeek-V4-Flash (0.8742).

One caveat: our baselines cover English only, because only English has a reference to trim, and they were not run through the Gemini-2.5-Flash quality judge. The comparison holds the syllable metric fixed. It does not compare full benchmark scores.

What Bilibili's Index team released between 28 September and 3 October 2026

The paper is arXiv 2609.40181v1, Index-Translate: A Multilingual Translation Model Family: Text, Speech, Controlled Dubbing, and Long-Document Translation. It was submitted 30 September 2026 by Tianjiao Li (project lead) and ten co-authors of the Index LLM Team, copyright Bilibili Index Team. All models start from Qwen3.5 base models. They get 167.77B tokens of multilingual mid-training (Table 2), then General, Instruction and Meme specialists merged at 0.8/0.1/0.1 and refined with multi-teacher on-policy distillation (MOPD). Three specialists sit on top: Index-Echo (speech-to-text and speech-to-speech), Index-Homura (syllable-controlled translation) and Index-NativeLong (whole-book translation in one generation).

The weights came out before the paper. We checked the Hugging Face API on 4 October 2026. The 2B and 9B checkpoints of Index-Translate, Homura, NativeLong and both Echo variants all have createdAt 28 September 2026. Index-Translate-35B-A3B-preview followed on 2 October, and GGUF, FP8 and FP4 variants arrived on 2 and 3 October. Every IndexTeam model repo is tagged Apache-2.0 and none is gated. The four benchmark repos (SandGlass, InstTrans, Meme, NAtIveLong) were created on 3 October 2026. The GitHub repo bilibili/Index-Translate (Apache-2.0, created 15 September 2026) showed 40 commits and 584 stars when we checked.

Parameter counts are the safetensors.total field of the HF model API; GB is the sum of file sizes. Checked 4 October 2026.

Which Index-Translate numbers hold up against the tables and the release

We rechecked each headline claim against the paper's own rows and the released files. Most of the arithmetic holds. The weak points are scope and one copied cell.

Paper values from arXiv 2609.40181v1. Counts, recomputations and dataset checks are BLOMEGA's, 4 October 2026.

The "100B-scale" claim holds against the one 100B-plus model in the table, and decisively. The "frontier" claim depends on which column you read. On the reference-based metrics (FLORES and WMT24++ COMET-22) the 9B model and the 35B-A3B preview edge GPT-5.6-Sol: 0.8789 and 0.8794 against 0.8650 on FLORES. On the one reference-free, LLM-judged score (WMT26), every frontier row leads: GPT-5.6-Sol 89.10, DeepSeek-V4.1-Flash 83.55, Gemini 3.5 Flash Lite 79.52, Index-Translate-35B-A3B 76.76. In our judgement a localization buyer should weight the WMT26 gap more than the 0.014 COMET margin.

How SandGlass turns a subtitle's duration into a syllable target and a score

The target comes from the clock, not the source text. Each Chinese line has a timed duration (0.84 to 7.0 seconds, median 2.3). The anchor is duration times a fixed speaking rate per language: 6.19 syllables per second for English, 7.84 for Japanese, 7.82 for Spanish, 8.4 for Arabic. The short and long targets are 0.75 and 1.25 times the anchor. We recomputed all 3,600 targets from the metadata, and none deviate from that rule. Median natural targets come out at 14 syllables (English), 18 (Japanese and Spanish) and 19 (Arabic). The rates are calibrated to roughly natural length. The English reference translations have a median of 13.5 syllables against a median natural target of 14.

Two scorers then run independently. A rule-based counter (syllable_calculation.py, "V3", 22 languages) counts the output. Its own docstring rates zh, en, ja and es as high reliability and Arabic as medium. Homura-9B's Arabic within-10% rate of 79.89% (Table 10) is therefore measured with the counter's weaker setting. A Gemini-2.5-Flash judge then grades quality on 0, 0.5 or 1 with no reference, and it is told the length limit. The README adds a composite: score = (syllable_reward + quality) / 2. That gives a perfectly fitted output with zero quality 0.5 points. A faithful, natural line that misses the count by 50% or more gets only half its quality score.

Homura is trained against the same kind of target. GRPO jointly optimises a quality reward and a length reward on the requested count (Section 3.2, following HOMURA, arXiv 2601.10187). The paper is candid about the result: "precise syllable control transfers poorly" through merging and distillation, and MOPD "performs poorly at following length-related constraints". Both statements point to the same thing as our trim-and-pad run. Hitting a count and translating well are separate skills, and when they compete in training the count wins.

What this changes for dubbing and localization teams

• Never accept a syllable or length hit rate without the paired quality score from the same run. Our 93.33% trim-and-pad result shows that adherence on its own can be bought at zero cost. When a vendor or model card quotes "within 10%", ask for quality on the same cases and the same judge.
• Use Homura to draft lines for lip-sync and timing, then review for meaning. The Homura-9B RL checkpoint gives up 0.0718 quality (0.8581 to 0.7863) to go from 45.75% to 81.92% adherence. That is a draft you edit, not a line you ship. Our judgement: route its output to a human adapter. The paper does not report quality separately for the short (0.75x) targets, where compression is hardest, so test those yourself.
• Pin the decoding settings. The released client runs at temperature 0.3 with a bare prompt. Table 9 used greedy decoding and a prompt with the video summary and the lines before and after. In-house results with the stock client will not match the paper, and the model card says so.
• Re-derive speaking rates for your own content. SandGlass uses one rate per language for every genre, with Japanese at 7.84 and Spanish at 7.82 syllables per second. Anime, documentary and game dialogue are not delivered at one rate. If your timing comes from your own voice talent, rebuild the targets from measured rates before comparing systems.
• Treat Arabic, and anything outside en/ja/es/ar, as untested for count control. The counter rates Arabic medium, Homura is measured on 4 targets, and the model card says those results "do not establish equal control precision" across the 150-language inventory.
• Licences differ across the release. Weights are Apache-2.0 and SandGlass is MIT. InstTrans and Meme are CC BY-NC 4.0, so they are fine for internal evaluation but not for a commercial training set. NativeLong ships no text at all, and GuoFeng and BWB have their own upstream terms.

For a related check on how a judge model's verdicts can drive a localization metric, see our note on TermJudge's terminology lead and its empty code repository.

How to reproduce every number here in 10 minutes

Release state, straight from the Hugging Face and GitHub APIs:

curl -s "https://huggingface.co/api/models?author=IndexTeam&limit=200&full=true" \
  | python3 -c "import json,sys;[print(m['id'],m['createdAt'][:10],[t for t in m['tags'] if t.startswith('license')]) for m in json.load(sys.stdin) if m['createdAt']>'2026-09']"
curl -s "https://huggingface.co/api/datasets?author=IndexTeam" | python3 -m json.tool | grep -E '"id"|createdAt'
curl -s "https://huggingface.co/api/models/IndexTeam/Index-Homura-9B" | grep -o '"total":[0-9]*'
curl -s https://api.github.com/repos/bilibili/Index-Translate | grep -E '"(created_at|stargazers_count)"'

The SandGlass baselines use the release's own scorer, unmodified:

hf download IndexTeam/Sandglass-Bench --repo-type dataset --local-dir sg && cd sg
python3 -m venv v && v/bin/pip install pyphen num2words fugashi unidic-lite
v/bin/python - <<'PY'
import json,statistics,sys; sys.path.insert(0,'.')
from evaluate import syllable_metrics, controllability
import syllable_calculation as sc
R=[json.loads(l) for l in open('sandglass_bench.jsonl')]
EN=[r for r in R if r['metadata']['target_language']=='en']
cnt=lambda s: sc.cal_syllable_count(s,lang='en')
def fit(ref,t):
    w=ref.split()
    while len(w)>1 and cnt(' '.join(w))>t: w=w[:-1]
    s=' '.join(w)
    while cnt(s)<t-1: s+=' yeah'
    return s
for name,f in [('reference',lambda r:r['label']),('trim_pad',lambda r:fit(r['label'],r['metadata']['target_syllables']))]:
    rows=[dict(uuid=r['metadata']['uuid'],target_kind=r['metadata']['target_kind'],target_language='en',
               target_syllables=r['metadata']['target_syllables'],quality=None,**syllable_metrics(f(r),r['metadata'])) for r in EN]
    print(name,{k:round(statistics.mean(x[k] for x in rows),4) for k in ['dev','hit_pm1','hit_10']},
          'slope',round(controllability(rows)['slope'],3))
PY

Expected output: reference {'dev': 0.3649, 'hit_pm1': 0.2022, 'hit_10': 0.1867} slope 0.0 and trim_pad {'dev': 0.049, 'hit_pm1': 1, 'hit_10': 0.9333} slope 0.962. To check the target rule, compare metadata.target_syllables with round(round(duration*speech_rate)*target_factor); all 3,600 rows match. The data file's SHA-256 should be 88cb4506ff5dd95ba171ab746bbf7fcf3d4e12df27b51d6554673c3f4bc4b6f1, as recorded in manifest.json. The duplicated DeepSeek and Gemini instTrans cells are on page 7 of the PDF, Table 4(a).

What would prove this wrong

Our claim: on SandGlass, syllable adherence alone does not separate a dubbing model from a trivial length fitter, and Index-Homura's gains in adherence were paid for in judged quality.

It would be wrong if, by 31 March 2027, the Index team or anyone else publishes per-case SandGlass outputs showing an Index-Homura checkpoint at 80% or more within 10% and Gemini-2.5-Flash quality of 0.85 or more on the same 3,600 cases, matching the SFT checkpoint's 0.8581 without losing control. It would also be wrong if human raters on a sample of Homura-9B RL outputs prefer them over the 9B SFT outputs for dubbing use, which would mean the judge's 0.0718 quality drop does not reflect what adapters see. We predict neither appears by that date. We also predict the next Homura release will report a composite or a paired metric rather than adherence alone.

Sources

1. Tianjiao Li, Mengran Yu, Chenyu Shi, Lusheng Zhang, Qisi Chen, Yanshan Zhou, Ji Qi, Jingying Liu, Yuang Feng, Ziang Cui, Tianxing Yan (Index LLM Team, Bilibili). Index-Translate: A Multilingual Translation Model Family. arXiv 2609.40181v1, submitted 30 September 2026. Tables 1 to 20, Sections 3.2, 4.5, 4.6, 5.2, 5.3, Appendix C.
2. IndexTeam. Sandglass-Bench on Hugging Face, public-v1, created 3 October 2026, MIT. Downloaded and scored by BLOMEGA on 4 October 2026.
3. IndexTeam. Index-Homura-9B model card, created 28 September 2026, Apache-2.0; client defaults and per-language table.
4. IndexTeam. Index-Translate-35B-A3B-preview, created 2 October 2026, and Index-Nailong-9B (Index-NativeLong-9B), created 28 September 2026.
5. IndexTeam. NAtIveLong dataset card, created 3 October 2026: 84 BWB and 84 GuoFeng cases, metadata and reconstruction only.
6. bilibili. Index-Translate on GitHub, Apache-2.0, created 15 September 2026, 40 commits on 4 October 2026.
7. Ziang Cui et al. HOMURA: Taming the Sand-Glass for Time-Constrained LLM Translation via Reinforcement Learning. arXiv 2601.10187, 2026. Origin of SandGlass and the length reward.
8. BLOMEGA computations, 4 October 2026: SandGlass target-rule check, English reference and trim-and-pad baselines with the release scorer, Table 4/5 recomputation and win counts, HF API release dates, sizes and licences.

FAQ

Does Index-Homura actually hit the requested syllable count for dubbing?

On SandGlass, Index-Homura-9B (RL) lands within 10% of the requested count on 81.92% of 3,600 cases and within one syllable on 74.42% (paper Table 9), against 47.64% within 10% for GPT-5.6-Sol. Judged quality is 0.7863, below its own SFT checkpoint (0.8581) and GPT-5.6-Sol (0.8715). Using the benchmark's own scorer, we found that a script trimming or padding the English reference hits 93.33%, so the count is easy to hit and the quality score decides whether the line is usable.

Is Index-Translate open source and under what licence?

Yes for weights. On 4 October 2026 every IndexTeam model repo for the family (2B, 9B, 35B-A3B-preview, Homura, NativeLong published as Index-Nailong, Echo S2TT and S2ST) was tagged Apache-2.0 and ungated. The 2B and 9B checkpoints were created on 28 September 2026 and the 35B-A3B preview on 2 October. SandGlass is MIT. InstTrans and Meme benchmarks are CC BY-NC 4.0. NativeLong ships case metadata only, under upstream corpus terms.

Does Index-Translate really cover 150 languages?

150 is the supported inventory in Appendix C (22 core plus 128 additional entries). Reported text scores cover at most 83 languages (21 in FLORES and 62 low-resource, if disjoint), and the expansion study uses 81 targets. Syllable control is measured on English, Japanese, Spanish and Arabic only, speech-to-speech on 6 directions, and the released NativeLong client on 4 directions.