The "improved" AI earthquake translation into Chinese deleted the outdoor instructions, and nobody re-tested it

The discourse-revised Chinese text keeps fewer earthquake instructions intact than the plain prompt did

The paper prints all its material: the 442-word English source (Appendix A), the plain-prompt ChatGPT-4o translations (Appendix B Chinese, Appendix C Spanish) and the discourse-prompt revisions (Appendices D and E). We split the English into 27 actionable instructions across six situations (indoors, disability, outdoors, vehicle, trapped, after) and marked each one in each translation as intact, altered (a reader could act differently) or missing, with a verbatim quote for every intact or altered call. Where the paper's own reader test exists, we used its result.

Appendix D, the revised Chinese, contains no "If you are outdoors" section at all: no 路灯 (streetlights), no 空旷 (open area), no instruction to stay outside. It also turns "Stop as quickly as safety permits" into 立即停车 (stop immediately), drops "under a sturdy table or other piece of furniture" from the cover instruction, and drops "and it is close to you" from the doorway rule. Appendix E, the revised Spanish, fixes the door/doorway error but moves "stay there" above the outdoors heading into the disability section, so its count stays at 25 of 27, level with the plain Spanish.

The paper does not mention any of these changes. It could not have caught them through its readers either: the 117 students answered comprehension questions on the plain translations only. The revisions were shown to them for comparison and comment, and the "considerable improvement" in the abstract is the authors' reading of those revisions, not a re-run of the test.

What arXiv:2610.08601 tested, and on which model

arXiv:2610.08601, Generative AI translations in high-stakes emergency messaging, by Nune Ayvazyan (Universitat Rovira i Virgili), Yu Hao (University of Melbourne) and Anthony Pym (Universitat Rovira i Virgili), was submitted on 6 October 2026 as v1; the PDF itself is labelled "Pre-print version 5.2 (August 2026)". No v2 existed when we checked on 7 October 2026 (the v2 URL returned HTTP 404).

The source is the Victoria State Emergency Service earthquake page, downloaded in October 2024 and cut from 892 to 442 words so that suspect passages were concentrated. The authors ran it through ChatGPT-4o with "Translate this text into Chinese" and "Translate this text into Spanish", then showed the outputs without the English to 62 Chinese-L1 master's students in Melbourne and 55 Spanish-L1 undergraduates in Tarragona, who answered six multiple-choice actionability questions. Next they had ChatGPT list "the discourse and stylistic features of the best warning texts in simplified Chinese for PR China / Spanish for Spain" and "use those features to revise the previous translation". Students compared those revisions with their own manual revisions and rated them.

The context is policy, not lab curiosity. The paper recounts that the US National Weather Service used Lilt's AI system to publish Spanish and Chinese warnings (NOAA announced it on 26 October 2023) and that translation stopped in April 2025 (the paper gives 9 April; CNN's report is dated 8 April 2025). The authors' question is whether a cheap prompt can stand in for a two-year, corpus-built system. Their conclusion is a qualified yes, with human revision and "low, vigilant trust".

The VICSES page still carries the same English wording on 7 October 2026 (we fetched it: HTTP 200, "Do not use a doorway except if you know it is a strongly supported, load-bearing doorway and it is close to you" unchanged) and still offers no translation of it, only a link to general information in other languages.

The reader test, recomputed, and what the revisions changed

Table 1 is the paper's only quantitative result on actionability. We retyped its counts and recomputed every percentage. Five of the eight rows are consistent. Three are not, and the participant count differs between caption and method.

Paper values from arXiv:2610.08601v1. Recomputed percentages and the 27-instruction audit are BLOMEGA computations, 7 October 2026.

None of the arithmetic slips changes a conclusion: Chinese "drop" is still about half wrong, Spanish "doorway" still under half right. They matter for anyone who wants to reuse Table 1 as a baseline, because the denominators are not stated and have to be inferred from the counts (Spanish rows range from 47 to 77 responses for 55 people, since "drop" allowed several answers).

What did the revision do to those eight trouble spots? From the paper's own Appendices and its page 8 commentary: two were fixed (Spanish puerta became marco de una puerta; Chinese 待办事项清单, to-do list, became 安全步骤清单, safety-step checklist), one was partly changed (Chinese 使用门口 became 站在门口, "stand in the doorway"), and five kept the tested wording (Chinese 趴下 and 掩护, Spanish agáchate, sujétate and continúa con precaución). One detail the paper describes slightly differently: it says the Chinese for "cover" "remained unchanged", which is true of the main bullet (找掩护), but the slogan lines in Appendix D now read 趴下、遮挡、抓紧, swapping in 遮挡 (block, screen).

Where a style prompt can remove content without anyone noticing

The second prompt asks for style: bullet points, block language, action-first sentences. It does not ask to keep every instruction, and the workflow has no step that counts them. The paper's own description of what changed (more bullets, questions as Spanish headers, bold action verbs in Chinese) matches what we see; the deletions came along with the restructuring. Our judgement: a restyling pass is a rewrite, and a rewrite needs the same completeness check as a first translation.

That bottom panel is the second result. We back-translated each section of the paper's four printed translations through the BLOMEGA hub and asked an LLM judge to mark each instruction. It found the Appendix D deletions, the "stop immediately" change and both misplaced "stay there" lines (Chinese plain, Spanish revised), 6 of the 15 problems in our audit. It missed two other dropped conditions in Appendix D (the table, "close to you") and passed all five tested wordings that a third or more of readers got wrong (Chinese drop 51.7%, doorway 33.8%, list 53.8%; Spanish doorway 44.7%, proceed 72.9% wrong), because 趴下 back-translates to "get down", 待办事项清单 to "a simple to-do list" and puerta to "door", and a judge reading English sees nothing alarming. Back-translation finds missing content. It does not find misleading content that is still there.

Two prompts on a 2026 model: the doorway is fixed, the posture verb is not

The paper notes its findings are time-bound: some GPT errors from October 2024 were gone by April 2025. We reran both conditions on 7 October 2026 through the BLOMEGA hub's llm-gateway function, which resolved 264 of the 265 logged rerun calls to claude-sonnet-4-5 (one judge call fell back to grok-4). Three independent runs, temperature fixed at 0.4 by the gateway. Two compromises, stated plainly: the gateway caps prompts at 2,000 characters, so we translated section by section in chunks of about 450 characters rather than as one document, and the second step had to restate the English with the earlier translation instead of relying on chat history.

Paper columns from Appendices B to E (ChatGPT-4o, October 2024). Rerun columns: BLOMEGA, 3 runs per condition, claude-sonnet-4-5 via llm-gateway, 7 October 2026.

Four consequences for teams translating safety content or recording it for dubbing and voice prompts:

The posture verb is the stubborn error. Six of six Chinese outputs in 2026, plain and restyled, still render DROP as 趴下, the word with 75 of 145 wrong responses in the paper, many of them accepting a face-down position. Neither model generation nor the style prompt removed it; one of three restyled runs glossed it as 迅速蹲下或趴在地面 (quickly crouch or lie on the ground), which adds the crouch and keeps the lying down. The authors' own proposed fix is source-side: write "sit" or "crawl" after the slogan. That is a writing-guide change, and it is cheaper than any prompt.

Plain prompts on a current model trade one doorway problem for another. The Chinese now forbids the frame, as the authors saw by April 2025. The plain Spanish switched from puerta (door) to entrada (entrance) in all three runs; the readers never saw entrada, so whether it misleads is untested, but it is still not the frame. Only the restyled Spanish named the marco or vano.

Restyling adds as well as removes. In one 2026 run the Chinese revision inserted 根据应急安全规定，违反上述要求将造成人身伤害 ("according to emergency safety regulations, violating the above requirements will cause personal injury"), and in another it told readers to "strictly observe the above regulations". The feature list the model wrote for itself asked for "authority references (根据规定)". No regulation is cited in the source. In the same run, the outdoor section changed from "stay there" to "move quickly to an open area" followed by "do not move until the shaking stops". In one Spanish run, "proceed cautiously once the earthquake has stopped" became "No circule hasta que cese el seísmo" (do not drive until the quake stops), which drops the caution.

Measure the revised text, not the prompt. Our judgement: the paper's claim that discourse prompts "considerably improve" actionability may well be true for Spanish, where the one high-risk fix landed and nothing was lost. For Chinese the printed evidence points the other way on completeness, and the comprehension effect is unmeasured. A team adopting this two-step workflow should run an instruction inventory against every rewritten version and put the rewritten version, not the first draft, in front of native readers.

Check it yourself

Everything in the paper-side audit comes from the PDF. The outdoor deletion takes one pipeline (pdftotext puts layout spaces inside Chinese lines, so strip whitespace first):

curl -sL https://arxiv.org/pdf/2610.08601 -o 2610.08601.pdf
pdftotext -layout 2610.08601.pdf p.txt
# Appendix B (plain) vs Appendix D (discourse): count streetlights / open area
sed -n '/^Appendix B/,/^Appendix C/p' p.txt | tr -d ' \n' | grep -o '路灯\|空旷' | wc -l   # 2
sed -n '/^Appendix D/,/^Appendix E/p' p.txt | tr -d ' \n' | grep -o '路灯\|空旷' | wc -l   # 0
sed -n '/^Appendix D/,/^Appendix E/p' p.txt | tr -d ' \n' | grep -o '立即停车'           # stop immediately

Table 1, recomputed from its own counts (correct, not sure, wrong):

python3 - <<'PY'
rows={'zh drop':(69,1,75),'zh door':(43,0,22),'zh cover':(61,0,4),'zh list':(30,0,35),
      'es drop':(69,4,4),'es door':(21,5,21),'es hold':(38,4,6),'es proceed':(12,1,35)}
for k,v in rows.items(): n=sum(v); print(k,n,[round(100*x/n,1) for x in v])
PY
# zh drop 145 [47.6, 0.7, 51.7]  (prose: 144)   es drop 77 [89.6, 5.2, 5.2]  (printed 84/8/8)
# es door 47 [44.7, 10.6, 44.7]  (printed wrong 46)

Our working files: inventory.json (the 27 instructions), audit_paper.json (status plus verbatim quote per instruction per appendix), scripts/audit_paper.py (verifies all quotes exist in the PDF text and tallies), scripts/rerun.py RUN_ID (both prompts, back-translation, judge, via POST /functions/v1/llm-gateway with {"action":"test","prompt":...}), scripts/judge_paper.py (judge calibration), scripts/lexcheck.py (the string checks in the table above). The rerun outputs are in runs/r1 to runs/r3, with every prompt and response logged in calls.json. The tallies in the first chart reproduce with sh scripts/headline.sh.

What we could not do: run the paper's multiple-choice questions on human readers for the revised texts, or call ChatGPT-4o itself (the hub's OpenAI balance is empty, so the gateway routed to Anthropic). The rerun is therefore a different model under a chunked procedure, a check on whether the failure types persist, not a replication.

What would prove this wrong

Our claim is that the discourse-specific prompt, as printed, did not make the Chinese earthquake text more complete or more actionable, and that the stubborn error is the posture verb, not register or layout.

It would be wrong if a reader test of Appendix D, run with the paper's six questions on a comparable Chinese-L1 group, shows the "drop" item at 70% correct or better (it was 47.6% on the plain text with the same verb), or if the authors show the outdoor section was present in their actual output and lost only in typesetting. We predict that by 30 June 2027 no version of arXiv:2610.08601 or follow-up from the same group reports a comprehension test on the revised translations showing the Chinese "drop" item above 60% correct while 趴下 remains the only verb used.

Sources

1. Nune Ayvazyan, Yu Hao, Anthony Pym. Generative AI translations in high-stakes emergency messaging. arXiv:2610.08601v1, submitted 6 October 2026 (PDF labelled pre-print version 5.2, August 2026). Methodology, Results, Table 1, Appendices A to E.
2. Victoria State Emergency Service. Earthquake. Source text of the experiment; fetched by BLOMEGA on 7 October 2026, HTTP 200, wording unchanged.
3. Susan Buchanan, NOAA. NOAA uses artificial intelligence to translate forecasts, warnings into Spanish and Chinese, 26 October 2023 (cited in the paper).
4. CNN. National Weather Service no longer translating products for non-English speakers, 8 April 2025 (cited in the paper).
5. Allison Yarborough, LILT. LILT supports AI-powered translated forecasts offered by NOAA's National Weather Service, 2023 (cited in the paper).
6. BLOMEGA computations, 7 October 2026: 27-instruction audit of Appendices B to E, Table 1 recomputation, back-translation judge calibration (98 of 108 agreement), three reruns of both prompts on claude-sonnet-4-5 via the BLOMEGA hub llm-gateway (265 calls).

FAQ

Did the discourse-specific prompt improve the AI earthquake translations in arXiv:2610.08601?

Partly, and it was not measured. The revised texts (Appendices D and E) were never given to readers; the "marked improvement" is the authors' reading. In Spanish the revision fixed the door/doorway error and kept 25 of 27 instructions intact, the same count as the plain prompt. In Chinese it fixed the to-do-list error but kept only 20 of 27 intact against 23 of 27 for the plain prompt: it deleted the three-instruction outdoor section, dropped "under a sturdy table", dropped "close to you" from the doorway rule and turned "stop as quickly as safety permits" into "stop immediately" (BLOMEGA audit, 7 October 2026).

Are the reader-test numbers in Table 1 of arXiv:2610.08601 internally consistent?

Five of eight rows are. Three are not: the Chinese "drop" row sums to 145 responses where the prose says 144 and its not-sure percentage is blank; the Spanish "drop" row sums to 77 where the prose says 76 and its printed 84/8/8 percent should be 89.6/5.2/5.2 for those counts; the Spanish "doorway" row prints 46% wrong for 21 of 47 responses, which is 44.7%. The caption says 65 Chinese participants, the methodology says 62. None of these changes the paper's direction.

Can back-translation catch these emergency translation errors?

Not the ones readers actually tripped on. We back-translated the paper's four printed translations and had an LLM judge each of 27 instructions. It agreed with our manual audit on 98 of 108 calls and flagged all 3 deleted instructions and both misplaced lines, but it marked as intact all 5 tested wordings that a third or more of the paper's 117 readers got wrong (Chinese "drop", "doorway" and "to-do list", Spanish "puerta" and "continúa con precaución").