STI-Bench audit: 282 of 2,064 questions excluded, and the repaired set still leaks on one task

What did the ReSTI audit find, and when?

Pengzhan Sun, Ramanathan Rajaraman, Shiu-Hong Kao, Junbin Xiao and Angela Yao (National University of Singapore; University of Science and Technology of China) posted ReSTI: A Source-Grounded Audit and Repair of STI-Bench as arXiv:2609.24727v1 on 21 September 2026. The annotation release itself, ReSTI v5, is dated 15 August 2026 in the repo's release.json; the files were repackaged into github.com/pengzhansun/ReSTI on 21 September with byte-identical contents.

STI-Bench (Li et al., arXiv:2503.23765, first posted 31 March 2025, ICCV 2025) asks 2,064 five-way questions over roughly 300 videos across eight tasks, from 3D Video Grounding to Trajectory Description. The Hugging Face copy, MINT-SJTU/STI-Bench, holds a 377,484-byte qa.parquet with 2,064 rows and a 1,003,401,772-byte video.zip, Apache-2.0, last modified 12 January 2026.

The audit writes every question as a measurement contract (source, entity, time, coordinate frames, operator, unit, formatting rule, distractor policy, Eq. 1) and gives each row one of four verdicts: retained, corrected, re-contracted, or excluded. The paper's split is 652 retained, 547 corrected, 583 re-contracted and 282 excluded (Section 1). It explicitly reports no model scores: "they do not measure changes in model accuracy or rankings" (Section 8). So there is no before-and-after leaderboard to check, and we could not compute ranking shifts.

Do the paper's counts survive a check against the released files?

Mostly yes. We downloaded annotations/resti.jsonl and annotations/excluded.jsonl (SHA-256 matches checksums.sha256) and the legacy qa.parquet, then joined on the Legacy Row Key field. Every row in the right-hand column is our computation.

Paper values from arXiv:2609.24727v1. Checks run 23 September 2026 against ReSTI commit on main (pushed 21 September 2026) and MINT-SJTU/STI-Bench on Hugging Face.

Two details are worth pulling out. First, the release notes warn that the legacy combination of video, legacy ID and timestamps collides; we found the cause: Omni6DPose video 000070.mp4 has two legacy rows with ID 2, one asking speed and one acceleration. An evaluator keyed on the legacy ID silently scores 1,781 records. Second, 211 of the 282 exclusion records carry a historical reason code saying the source reconstruction was "not implemented" or "not yet reconstructed" (77 Waymo object tracks, 73 ScanNet, 44 Waymo dimensions, 17 Waymo relations). All 211 also carry a later verification verdict naming a question-level defect, so we read them as verified exclusions. Our judgement: exclusion density still partly tracks where the team built reconstruction tooling, which is why Waymo loses 20.3 percent of its rows (161 of 793) against 9.5 percent for ScanNet (82 of 865) and 9.6 percent for Omni6DPose (39 of 406).

How did a frame, a plane and a timestamp corrupt the labels?

The Grounding error is a missing matrix. ScanNet ships object annotations in an axis-aligned world (transform A takes raw scan coordinates into it) and camera poses in the raw scan world. The correct camera-frame center is P(t)^-1A^-1c_ann (Eq. 2). The released centers match P(t)^-1c_ann, skipping A^-1. In the paper's worked example, a white table in scene0041_00 at 4.0 s, the released center lands 9.15 m from the table and 0.004 m from where the faulty formula puts it (Figure 4). Across rows, corrected centers agree with an independently built source trace to a median 0.006 m, released ones sit a median 2.61 m from the nearest physical object (Section 5.1).

The Orientation error is a wrong plane. The legacy value is recovered by projecting one camera axis onto raw-world X-Z; ScanNet's Z is vertical, so the angle mixes heading with tilt. Scene0012_00 gives -162.965° on X-Z, matching the released -162.97°, and +146.807° on the horizontal X-Y plane (Figure 5). The best of five standard heading conventions reproduces only 14 of 104 legacy ScanNet targets, and the replacement operator reproduces all 102 corrected targets in the checked subset (Section 6.1).

The Waymo pose error is a timestamp. Questions like "Given the initial pose, estimate the camera pose" never named the target time, and 90 of 121 retained legacy matrices were taken at the wrong source instant; replacing them moves translation by a median 0.77 m and at most 31.6 m (Sections 4 and 7). ScanNet pose labels, by contrast, were right all along: 144 of 144 values stay, but their distractors reused identical rotations.

None of this is annotator noise in the crowdsourcing sense. No human mislabelled anything; a conversion script applied one wrong operator to every row of a task. That is why the error rates are near 100 percent within affected tasks instead of scattered, and why noise-tolerant training or majority-vote relabelling would not have caught it.

How much of the benchmark actually changed, task by task?

The paper gives per-task accepted and excluded counts (Figure 3a). We added the per-task share of kept answers whose value differs from the legacy answer detail, using exact numeric comparison of the parsed values.

Read the chart with its caveats. The Grounding 100 percent is partly a format change: every answer now serializes a box-to-camera rotation. The 37 ScanNet speed changes are all 0.01 m/s, a last-digit shift, and 24 changed single-value ScanNet displacement answers move by a median 0.01 m. A value change is not always a correction of substance. The Dimensional Measurement result is the substantive one: 152 of 289 questions (52.6 percent) excluded, and 136 of the 137 survivors (99.3 percent) carry a new value.

For anyone who reported STI-Bench numbers, three decisions follow. Scores on the legacy set are not comparable with ReSTI scores: all 1,782 option sets are new, the keyed letter stayed the same on only 347 rows (19.5 percent, what independent reassignment gives), and 1,684 question texts were rewritten; only 98 Waymo Spatial Relation questions are verbatim. Per-task legacy scores on Orientation and Grounding measured agreement with a wrong operator, so a model that got them "right" learned the bug or guessed; Waymo Orientation could be solved at 100 percent by always picking the smallest option. And if you build a similar benchmark from 3D sources, the cheapest check is the one ReSTI ran first: back-project each answer into the source scene and measure distance to the object it names.

Our finding: the repaired Speed and Acceleration options still point at the answer

ReSTI tested option-only attacks on Grounding and Orientation and reports 22.61 percent and 20.0 percent after repair (Sections 5.2 and 6.3). It does not report them for the other scalar tasks, and its release notes state the checks "do not certify that every possible answer-only strategy is at chance". So we ran the same family of attacks, sorting the five numeric options and always picking one rank, on all 916 kept questions whose options parse as single numbers.

Pooled, the repair works. Median-pick falls from 46.9 percent on legacy scalar rows to 20.1 percent (184 of 916) on ReSTI, and the answer letters are balanced at 357, 357, 357, 357 and 354. Cell by cell, one leak stands out. In Waymo Speed and Acceleration the keyed answer is the smallest option in 43 of 92 questions, 46.7 percent against 20 percent chance, binomial p = 7.3 × 10^-9. That survives a Bonferroni correction over all 66 cell-attack tests we ran (adjusted p = 4.8 × 10^-7). Split by wording, the smallest option is correct on 32 of 77 speed questions (41.6 percent) and 11 of 15 acceleration questions (73.3 percent). ScanNet Speed and Acceleration shows the same direction at 36 of 115 (31.3 percent, uncorrected p = 0.0027).

The other elevated cells, 35.8 percent on ScanNet Dimensional and 35.7 percent on Waymo Displacement, do not survive correction on samples of 53 and 56, so we treat them as unconfirmed. To keep the in-sample choice honest, we ran a rank attacker that learns the best rank per cell on half the rows and scores on the other half: across 200 random splits it averages 24.4 percent on all 916 scalar questions. That is 4.4 points of free accuracy for a video model with any rank prior on these tasks, concentrated in two cells. Our reading of the likely cause, a judgement: speeds are non-negative and distractors are drawn with a separation floor, so a slow or near-zero true value pushes most distractors above it, the same asymmetry that made Waymo Orientation 100 percent smallest-pick in the legacy set.

Check it yourself

Everything below runs on the Python standard library for the ReSTI part; the legacy comparison needs pandas and pyarrow. The output shown is from our run on 23 September 2026.

curl -sLO https://raw.githubusercontent.com/pengzhansun/ReSTI/main/annotations/resti.jsonl
curl -sLO https://raw.githubusercontent.com/pengzhansun/ReSTI/main/annotations/excluded.jsonl
shasum -a 256 resti.jsonl excluded.jsonl
# 2e31f88b67a9d08b...  resti.jsonl     (matches checksums.sha256 in the repo)
# 5066d38a43335346...  excluded.jsonl

python3 - <<'PY'
import json, re, collections
R = [json.loads(l) for l in open("resti.jsonl")]
X = [json.loads(l) for l in open("excluded.jsonl")]
print(len(R), "accepted,", len(X), "excluded")
num = re.compile(r"^\s*(-?\d+(?:\.\d+)?)")
cells = collections.defaultdict(lambda: [0]*5)
for q in R:
    try: v = {k: float(num.match(s).group(1)) for k, s in q["Candidates"].items()}
    except (AttributeError, TypeError): continue
    order = sorted(v, key=v.get)
    cells[(q["Source"], q["Task"])][order.index(q["Answer"])] += 1
for c, ranks in sorted(cells.items()):
    n = sum(ranks); print(c, n, ranks, "smallest=%.1f%%" % (100*ranks[0]/n))
PY
# 1782 accepted, 282 excluded
# ('ScanNet', 'Speed & Acceleration') 115 [36, 31, 15, 26, 7] smallest=31.3%
# ('Waymo', 'Speed & Acceleration') 92 [43, 17, 14, 12, 6] smallest=46.7%
# ('Waymo', 'Ego-Centric Orientation') 81 [17, 16, 16, 16, 16] smallest=21.0%

# legacy side: 2,064 rows, 1,014 key-detail mismatches
curl -sL -o qa.parquet https://huggingface.co/datasets/MINT-SJTU/STI-Bench/resolve/main/qa.parquet
python3 -c "import pandas as pd; q=pd.read_parquet('qa.parquet'); \
print(len(q), sum(dict(c).get(a)!=d for c,a,d in zip(q.Candidates,q.Answer,q['Answer Detail'])))"
# 2064 1014

To join the two, strip the |uid= suffix from Legacy Row Key and match on Source|Task|Video|ID; one legacy key (Omni6DPose, 000070.mp4, ID 2) is shared by a speed row and an acceleration row, so disambiguate that one by question wording. The verdict split (retained, corrected, re-contracted) is not a field in the release, so the paper's 652 / 547 / 583 cannot be reproduced row by row; our 656 identical values is a proxy.

What would prove this wrong

For the paper's central claim: an independent re-derivation of ScanNet Grounding from the official .sens poses and axisAlignment matrices. If ReSTI's corrected centers sit more than 0.05 m (median) from the source object centers, rather than the reported 0.006 m, the repair is itself wrong. If the legacy centers turn out to match the objects once some other documented convention is applied, the "missing alignment" diagnosis fails.

For our finding, a dated prediction: by 31 March 2027, either a ReSTI release after v5 regenerates the Speed and Acceleration distractors, or a published blind baseline (no video, options only) on ReSTI v5 scores above 30 percent on Waymo Speed and Acceleration. If a text-only frontier model given the ReSTI v5 options scores at or below 22 percent on that cell, the leak exists in the file but is not exploited in practice, and our 46.7 percent matters less than we think.

A caveat on our own numbers: the rank attack treats every option as a bare number and drops 866 rows with structured, textual or multi-value answers, so it says nothing about Grounding, Pose, Spatial Relation or Trajectory, where the paper's own attacks apply.

Sources

1. Sun, P., Rajaraman, R., Kao, S.-H., Xiao, J., Yao, A. ReSTI: A Source-Grounded Audit and Repair of STI-Bench. arXiv:2609.24727v1, 21 September 2026. Sections 1, 4 to 8; Figures 1 to 5.
2. ReSTI v5 annotation release, github.com/pengzhansun/ReSTI: annotations/resti.jsonl, annotations/excluded.jsonl, release.json, docs/release.md, validation.json. Release dated 15 August 2026, repackaged 21 September 2026; accessed 23 September 2026.
3. Li, Y., Zhang, Y., Lin, T., Liu, X., Cai, W., Liu, Z., Zhao, B. STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding? arXiv:2503.23765, 31 March 2025; ICCV 2025.
4. MINT-SJTU. STI-Bench on Hugging Face: qa.parquet (2,064 rows, 377,484 bytes), video.zip (1,003,401,772 bytes), Apache-2.0, last modified 12 January 2026; accessed 23 September 2026.
5. MINT-SJTU. STI-Bench evaluation code, GitHub.
6. Northcutt, C. G., Athalye, A., Mueller, J. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. NeurIPS Datasets and Benchmarks, 2021.
7. BLOMEGA. Five LLM runs agree at 0.995, and the measure correlates 0.029 with what it was built to measure, 22 September 2026.