The same Gemini interpreter scored 40.3% or 75.2% on hard conversations, depending on the brief
On 1,090 hard conversational scenarios, Gemini 3.1 Pro acting as an interpreter passed 40.3% of communicative-goal checks with a plain "translate this" instruction and 75.2% with a brief describing the setting, the speakers and the cultural norms of the language pair. Same model, same scenarios, same judge: a 34.9-point swing. The benchmark, from KAIST and published 17 September 2026, also finds every one of 10 interpreter setups losing ground from meaning to social fit, and standard MT metrics blind to most of the loss.
KAIST released a benchmark for interpreter agents, not sentences, on 17 September 2026
17 September 2026. Faiz Ghifari Haznitrama and Alice Oh of the KAIST School of Computing posted Evaluating Communicative Success in Machine-Translated Conversation, arXiv:2609.19885. It targets the product category shipping fastest in localisation right now: an agent that sits between two people who share no language and relays a live conversation.
The setup: 2,812 OpenSubtitles dialogue segments chosen for difficulty across six language pairs among Arabic, Bengali, Indonesian and Korean, each used in both directions for 5,624 scenarios and 12 translation directions, with no English in any pair. Each scenario gets a checklist of yes/no criteria in three layers (semantic, pragmatic, cultural-social), written by Gemini 3.1 Pro with the conversation context in view. A simulated native-language recipient replies to each translation, and a judge scores translation and reply against the checklist. Ten interpreter setups were run, giving 56,240 scored outputs.
Ten interpreters, and all ten lose points from meaning to social fit
| Interpreter | Type | Pass rate | Strict | L1 meaning | L2 intent, tone | L3 social fit | L1 minus L3 | Source |
|---|---|---|---|---|---|---|---|---|
| Gemini 3.1 Pro | frontier LLM | 91.2 | 89 | 98 | 93 | 87 | 11 | Fig. 3 |
| DeepSeek V4 Pro | frontier LLM | 82.9 | 81 | 93 | 86 | 76 | 17 | Fig. 3 |
| Qwen3.5 Flash | efficient LLM | 78.5 | 77 | 89 | 81 | 72 | 17 | Fig. 3 |
| Gemini 3.1 Flash Lite | efficient LLM | 78.2 | 77 | 91 | 79 | 72 | 19 | Fig. 3 |
| GPT-5.4 Mini | efficient LLM | 77.2 | 76 | 91 | 80 | 69 | 22 | Fig. 3 |
| DeepSeek V4 Flash | efficient LLM | 75.8 | 74 | 91 | 79 | 67 | 24 | Fig. 3 |
| Google Translate | MT system | 54.3 | 53 | 79 | 56 | 43 | 36 | Fig. 3 |
| NLLB-200 3.3B | MT system | 52.1 | 51 | 70 | 53 | 44 | 26 | Fig. 3 |
| Tiny Aya | small LLM | 46 | 44 | 62 | 46 | 40 | 22 | Fig. 3 |
| SeamlessM4T v2 | MT system | 24.2 | 24 | 43 | 23 | 17 | 26 | Fig. 3 |
The gap between layers is the paper's central result, and its size tracks capability. Gemini 3.1 Pro loses 11 points from L1 to L3. The four efficient LLMs lose 17 to 24. Google Translate keeps 79% of meaning criteria and 43% of social-fit criteria, a 36-point fall, the steepest in the table. Two systems that finish close overall can reach that total with very different layer profiles: Gemini 3.1 Flash Lite and Qwen3.5 Flash are 0.3 points apart overall and 2 points apart on L1.
Language direction matters in both roles (p < 0.001 for source and for target). Averaged across the ten systems, Bengali is 10 points harder than a system's own mean as a source language and 6 points easier as a target; Arabic is the reverse, 10 points harder as a target. The authors read Bengali as a comprehension problem and Arabic as an adaptation problem.
Conversation does not rescue weak systems. In six-turn scripted conversations the three LLM configurations score 92% to 94% per turn and NLLB and Tiny Aya 45% to 47%, and every system scores lower on conversation-level criteria (cross-turn consistency, goal completion) than on its own turns: 40.7% against 45.0% for NLLB. Giving Gemini 3.1 Flash Lite the translated history improved all six pairs, significantly in three, by 1.3 to 2.4 points.
The checklist catches register and honorific failures that CometKiwi and MQM judges score as perfect
The paper quantifies the disagreement on Gemini 3.1 Pro's 5,624 outputs (Table 10). Of the outputs each metric places in its top quartile, 10.3% (RATE) to 12.2% (MQM-APE) score 0.6 or lower on the checklist; CometKiwi's figure is 10.7%. Of the outputs each metric places in its bottom quartile, 62.0% (CometKiwi) to 69.6% (MQM-APE) still pass 90% or more of their criteria. Within a strong system, the metrics are ranking by something other than whether the conversation worked.
Pooling all ten systems hides this. Across about 56,000 outputs CometKiwi correlates with the checklist at Spearman ρ = 0.321, GEMBA-MQM 0.436, MQM-APE 0.301 and RATE 0.529. The authors show every one of those pooled values exceeds every within-system value for the two strongest interpreters, with bootstrap intervals no wider than ±0.03: the pooled correlation is the metric correctly ranking SeamlessM4T below Gemini, not the metric seeing tone.
Pooled over seven setups, scenario context adds about 4 points (36.4 to 40.3), a structured specification-aware brief about 9 more (49.0), and the pair-specific cultural context about 2 more (51.3). The briefs also get longer as they add content, and the authors note the gains are not length-matched against a same-length control.
If you ship an interpreter feature, the brief is part of the product and fidelity scores are not acceptance tests
Write the brief like a localisation kit. The strongest lever in the paper is not the model; it is telling the model who is speaking to whom, in what setting, and what the target culture expects of that relationship. For Gemini 3.1 Pro that was worth 34.9 points on hard cases. For Tiny Aya it was worth 4.3. A small model cannot use a brief a large model can.
Test L3 separately, in every target language. A single pass rate hides where the loss sits. Korean speech levels, Bengali honorific pronouns, Indonesian face-preserving formality and Arabic register with gender agreement are the four mechanisms the paper's failure cases span. Each needs criteria written by someone who knows that language's rules.
Do not accept an interpreter on CometKiwi or an MQM judge alone. Among a strong system's outputs, roughly one in ten of those the metrics rank highest fail the conversation, and roughly two in three of those they rank lowest are fine. The first is deployment risk. The second is a false rejection of a translation that correctly adapted to context.
Dedicated MT systems were tested with a handicap, so do not over-read their rows. Google Translate, NLLB-200 and SeamlessM4T v2 received raw source text only, while LLM interpreters received the full brief. L2 and L3 criteria test tone and honorifics that need exactly that context. The authors flag this confound themselves and did not run a matched condition.
A judgement, marked as one: Gemini 3.1 Pro wrote the checklists, served as the primary judge, and is the top-scoring interpreter. The authors tested for this. Human-judge agreement was 75.4% (κ = 0.403) for Gemini against 69.2% (κ = 0.328) for GPT-5.4 on the same items, and DeepSeek V4 Pro was rejected as a judge after its accuracy fell from 96.6% to 38.6% under polarity inversion. That is good practice and still leaves the top row graded by its own model family. Read Gemini's 91.2 with that in mind; the layer ordering, which holds for every system and every judge, is the sturdier finding.
Check it yourself
# The paper: Figure 3 (page 5), Tables 1-2 (page 7), Tables 9-11 (pages 26-31)
curl -L -o comm.pdf https://arxiv.org/pdf/2609.19885
python3 -c "import pypdf; r=pypdf.PdfReader('comm.pdf'); print(r.pages[25].extract_text()[:2500])"
# Source dialogue: OpenSubtitles (ODC-By 1.0), e.g. the Bengali-Korean slice
# https://huggingface.co/datasets/Helsinki-NLP/open_subtitles
# The scoring rule, so you can run the same metric on your own interpreter logs:
python3 - <<'EOF'
# p_s = share of a scenario's criteria met; benchmark score = unweighted mean of p_s
scenarios = [
{"L1": [1,1,1,1], "L2": [1,1,0], "L3": [0,0,1]}, # meaning kept, register lost
{"L1": [1,1,1], "L2": [1,1], "L3": [1,1]},
]
p = [sum(sum(v) for v in s.values()) / sum(len(v) for v in s.values()) for s in scenarios]
print(round(100*sum(p)/len(p), 1)) # 85.0
for layer in ("L1","L2","L3"):
print(layer, round(100*sum(sum(s[layer])/len(s[layer]) for s in scenarios)/len(scenarios), 1))
# L1 100.0 / L2 83.3 / L3 66.7
EOFThe paper says its scoring code, derived scenarios and system outputs are released, but gives no repository URL in the arXiv version, and we could not locate one on 19 September 2026. The numbers above can be checked against the PDF. To reproduce the method on your own product, the judge and checklist-generation prompts are printed in Appendix K.
What would prove this wrong
The central claim is that interpreter quality degrades from semantic to cultural-social success for every system, and that the translation brief is a first-order variable. The layer claim is wrong if a replication on at least 1,000 scenarios in at least three non-English pairs, using a judge from a different model family than the top interpreter, finds any of the top three systems scoring higher on cultural-social criteria than on semantic ones. The brief claim is weakened if the same replication, with a non-Gemini judge, finds Gemini 3.1 Pro's gain from direct instruction to cultural-context brief below 15 points, against the 34.9 reported. We will look for either by 30 June 2027.
Sources
- Faiz Ghifari Haznitrama and Alice Oh (KAIST), Evaluating Communicative Success in Machine-Translated Conversation, arXiv:2609.19885v1, 17 September 2026. Figure 3 (pass rates by system, language and layer), Figure 4 (multi-turn), Table 1 and Table 9 (brief ablation), Table 2 (judge tests), Section 7.3 (human agreement), Table 10 (metric quartile disagreement and pooled correlations), Table 11 (qualitative cases), Appendix A (context asymmetry).
- Pierre Lison and Jörg Tiedemann, OpenSubtitles2016, LREC 2016, aclanthology.org/L16-1147. Dataset mirror: Helsinki-NLP/open_subtitles.
- Ricardo Rei et al., CometKiwi, WMT 2022; Tom Kocmi and Christian Federmann, GEMBA-MQM, WMT 2023. The two families of fidelity metric the benchmark is compared against.
- Yuto Kayano and Saku Sugawara, Specification-aware machine translation and evaluation for purpose alignment, WMT 2025. The brief the paper's cultural-context brief extends.
Related BLOMEGA guides: When human post-editing wins, by discourse dependency · Live dubbing latency at IWSLT 2026 · Multilingual LLM judges and translationese bias
FAQ
How do you evaluate an AI interpreter or live translation agent?
Score each translated turn against a scenario-specific checklist of yes/no criteria at three layers: semantic (meaning, entities, quantities), pragmatic (speech act, intent, tone) and cultural-social (register, honorifics, face). KAIST's benchmark of 17 September 2026 does this over 5,624 OpenSubtitles scenarios in Arabic, Bengali, Indonesian and Korean, with a judge that also sees the recipient's reply.
Is Google Translate good enough for live conversation?
On this benchmark it passed 54.3% of communicative-goal criteria against 91.2% for Gemini 3.1 Pro, and fell from 79% on meaning criteria to 43% on cultural-social criteria. It received raw source text only, while LLM interpreters received a context brief, so part of that gap is the missing context.
How much does the translation prompt matter for an LLM interpreter?
A lot for strong models. On about 1,090 hard scenarios Gemini 3.1 Pro scored 40.3% with a direct instruction and 75.2% with a brief describing the setting, speakers and the cultural norms of the language pair. Pooled over seven models the gain was 36.4% to 51.3%. Tiny Aya gained only 4.3 points.
Do CometKiwi and MQM judges catch cultural errors in translation?
Mostly not. Among Gemini 3.1 Pro's outputs, 10.3% to 12.2% of those each metric ranked in its top quartile failed the checklist at 0.6 or below, and 62.0% to 69.6% of those ranked in the bottom quartile passed at 0.9 or above. In one Bengali to Korean case CometKiwi gave 0.89 and both MQM judges found zero errors while the checklist passed 4 of 26 criteria.