Four VLMs graded Seoul sidewalks against a laser meter, and none beat always answering "Grade A"

What changed, and when

On 15 September 2026 Seung Jae Lieu, Diego Morra, Chiara Cadoni, Wonseop Song, Martina Mazzarello and Carlo Ratti (MIT Senseable City Lab) posted Can VLMs Reliably Assess Sidewalk Accessibility Attributes from Pedestrian-Level Imagery? (arXiv:2609.17882v1). It is a rare VLM-as-annotator study where the reference is not another annotator but a physical measurement.

The team photographed sidewalks near four sites serving older adults, children, hospital patients and blind people in Eunpyeong-gu, Seodaemun-gu and Mapo-gu between 7 and 11 July 2026, with iPhones held 1.0 m above the path. Each segment was then measured: effective width with a Sincon SD-70 laser distance meter, longitudinal and cross slope on the surface, and pavement condition graded A to E by one researcher against the Ministry of Land, Infrastructure and Transport rubric. About 11 hours of fieldwork produced 514 labelled images.

GPT-5.2, Gemini 3 Flash (preview), Qwen3-VL-8B-Instruct and InternVL3.5-8B were each queried 30 times per image per attribute at temperature 1.0: 246,720 queries. A fifth model, Claude Opus 4.6, was dropped after a 10-image pilot because it returned the identical width on all 30 samples for every image, so its answers carried no spread to calibrate. The authors note its estimates were as close to the field values as the others'; the exclusion is about the method, not accuracy.

Calibration always hit 90%, which is exactly why it is not the result

Split conformal prediction guarantees coverage for any model under exchangeable data, so every one of the 12 model-attribute cells lands at 90.2% to 91.5%. What separates the models is how wide the guaranteed interval has to be. That is the number to read.

Against Korean thresholds the calibrated intervals decide almost nothing. Width is the best case: GPT-5.2's asymmetric interval is 2.01 m wide (about ±1.0 m) against a 2.0 m standard and a 1.5 m constrained minimum, and 34.4% of the surveyed segments sit below 2.0 m. Longitudinal slope needs 7.11° to 9.04° to reach 90%, against limits of 3.18° and 4.76°. Cross slope needs 8.21° to 9.97° against 1.15° and 2.29°, so no segment can be classified either way. Gemini-3-Flash's raw cross-slope answers cluster so tightly around zero (mean base width 0.71°) that they miss the measurement on 83% of images.

The most useful single finding for anyone running LLM or VLM labelling at scale: agreement between samples is not evidence of correctness. On the quarter of images where each model's 30 answers agreed most closely, the measured width fell outside the base interval on 77% to 90% of images, longitudinal slope on 53% to 85%, and cross slope on up to 96% (GPT-5.2).

A consistent bias shows up as confidence

The width errors all point one way. The authors' reading, marked by them as plausible rather than shown: the models estimate the paved span from curb to building line, while the regulation counts only the unobstructed passage after street furniture and fixed obstacles. A model that answers the wrong question consistently will agree with itself, which is why self-consistency failed as a confidence signal.

Pavement grade fails for the same reason, harder. Three of the four models read the texture of new pavement as a defect and call it B. Their sampling is near-deterministic (mean modal mass 0.72 to 0.97), so the true grade often never appears in 30 samples: for 70% (GPT-5.2), 83% (Qwen3-VL-8B) and 81% (InternVL3.5-8B) of calibration images, rho in Table 1. The frequency-based conformal score can only include grades the model actually says. Once rho exceeds the risk level alpha, the calibrated threshold is forced to 1 and the prediction set becomes all five grades on every image. With 257 calibration images and alpha = 0.10 the conformal quantile is the 233th smallest score; GPT-5.2 has only about 78 images with any nonzero sampled frequency on the true grade, so the set is A to E on all 200 splits.

Gemini-3-Flash sits on the knife edge: rho = 0.095 leaves about 232.6 usable images against the 233 needed, which is why 52% of its splits degenerate at alpha = 0.10 and the averaged "coverage 0.950, set size 3.50" describes a mixture that happens on no single split. The authors flag this and recommend reporting the degenerate-split fraction next to any averaged conformal result.

The constant baseline is ours and it is uncomfortable. A set containing only "A" covers 80.2% of images with one grade; "A or B" covers 96.1% with two. Gemini's best informative sets are 1.51 grades at 81.2% and 1.76 grades at 87.9%. On this sample, a label prior that never looks at the photo is about as sharp as the best calibrated VLM. The caveat cuts both ways, and the authors state it: only 6 of 514 images are grade D, none are E, so neither the prior nor any model is shown to find the segments that need repair.

What it means if you use a VLM to pre-label or triage

• Buy a small measured calibration set before you trust any model output. Here, 11 hours of field time for 514 images was enough to show, before deployment, that three of four models carry zero information on pavement grade and all four are unusable on cross slope. That is cheap relative to discovering it from complaints.
• Never route on self-agreement. A "send to human review only when the model's samples disagree" rule would have sent the confidently wrong images straight through: up to 96% of the most self-consistent cross-slope answers were outside their own interval.
• Report the majority-class baseline for every categorical attribute. With 80.2% of labels in one class, top-1 accuracy of 62.3% is below the floor. The paper reports top-1 but not the prior; add it to your own evaluation template.
• Calibrate tails separately when the error is one-sided. For width, asymmetric calibration shortened intervals by 5.7% to 33.2% at unchanged coverage. Where errors are roughly symmetric it does nothing useful: longitudinal slope widths moved -1.2% to +4.0% (longer for three of four models), cross slope -1.8% to +1.7%.
• Check that your model varies at all. Claude Opus 4.6 gave one value 30 times at temperature 1.0. Sampling-based uncertainty needs spread; test for it on ten items before paying for 246,720 calls.
• Budget. 30 width samples cost about $0.03 per image on GPT-5.2 and $0.12 on Gemini-3-Flash ($15 and $62 for all 514). The open 8B models cost nothing per query and produced the widest intervals.

Judgement: the paper is careful not to oversell, and the transferable result is methodological. Conformal calibration against physical ground truth is the honest way to price a VLM label, and on this task the price came back as "not yet" for everything except far-from-threshold width.

Check it yourself

The paper says the annotated images and field values are released, but v1 carries no download link; we found none on 20 September 2026. What you can reproduce from the printed tables:

python3 - <<'EOF'
import math
grades = {"A": 412, "B": 82, "C": 14, "D": 6, "E": 0}          # Table 5
n = sum(grades.values())
print("constant A:", round(grades["A"] / n, 3), " A or B:", round((grades["A"] + grades["B"]) / n, 3))
top1 = {"GPT-5.2": 0.206, "Gemini-3-Flash": 0.623, "Qwen3-VL-8B": 0.109, "InternVL3.5-8B": 0.101}  # Table 9
print("models beating constant A:", [m for m, v in top1.items() if v > grades["A"] / n])
k = math.ceil((257 + 1) * (1 - 0.10))                        # conformal index, n_cal = 257
for m, rho in {"GPT-5.2": 0.696, "Gemini-3-Flash": 0.095}.items():
    print(m, "needs", k, "has", round((1 - rho) * 257, 1), "degenerate:", (1 - rho) * 257 < k)
sym, asy = [2.13, 2.87, 3.71, 4.59], [2.01, 2.27, 2.75, 3.07]  # Table 6
print("asymmetric shortening %:", [round((s - a) / s * 100, 1) for s, a in zip(sym, asy)])
EOF
# expected: constant A: 0.802  A or B: 0.961 / [] / GPT-5.2 needs 233 has 78.1 True / Gemini-3-Flash needs 233 has 232.6 True / [5.6, 20.9, 25.9, 33.1]

The last line gives 5.6% to 33.1% rather than the printed 5.7% to 33.2% because it divides the split-averaged widths in Table 6; the paper averages the per-split ratios. The difference is rounding, not a discrepancy.

• Prompts are verbatim in Appendix B, one per attribute; the longer width prompt the pilot rejected is in Appendix A. The authors note that prompt wording shifted the location of answers, so the prompt is part of the instrument.
• Model identifiers used (Table 4): gpt-5.2, gemini-3-flash-preview, claude-opus-4-6 (pilot only), Qwen/Qwen3-VL-8B-Instruct, OpenGVLab/InternVL3_5-8B-HF.
• Method references: split conformal prediction as in Angelopoulos and Bates; the frequency score for API models as in Su et al., API Is Enough; the per-tail construction as in Romano et al., Conformalized Quantile Regression.

What would prove this wrong

Prediction, dated: by 30 June 2027, no VLM evaluated on this dataset or an equivalent field-measured set of at least 500 forward-facing pedestrian images will produce a calibrated 90% cross-slope interval with a half-width below 2.29°, the relaxed Korean limit. The reason is geometric, not a matter of model scale: tilt perpendicular to the walking direction barely shows in a centre-line photo. A result under 2.29° from standard phone imagery would mean the limit is the models, not the camera angle, and would reopen image-only compliance screening.

Second, cheaper test: on any pavement set where one grade holds 80% or more of images, report the constant-answer baseline beside VLM top-1. We expect it to beat at least half of the models evaluated.

Sources

1. Seung Jae Lieu, Diego Morra, Chiara Cadoni, Wonseop Song, Martina Mazzarello and Carlo Ratti, Can VLMs Reliably Assess Sidewalk Accessibility Attributes from Pedestrian-Level Imagery?, arXiv:2609.17882v1, 15 September 2026. Sections 3 to 6; Tables 1, 2, 4 to 9; Appendix B.
2. Anastasios N. Angelopoulos and Stephen Bates, A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification, arXiv:2107.07511, first posted 15 July 2021.
3. Jiayuan Su et al., API Is Enough: Conformal Prediction for Large Language Models Without Logit-Access, arXiv:2403.01216, 2 March 2024.
4. Yaniv Romano, Evan Patterson and Emmanuel Candès, Conformalized Quantile Regression, arXiv:1905.03222, 8 May 2019.

Related BLOMEGA research: LLM annotators at kappa near zero · Rare-event labelling and the prevalence effect · Data annotation: latest research

FAQ

Can vision-language models measure sidewalk accessibility from photos?

Not precisely enough for compliance in the MIT test (arXiv:2609.17882, 15 September 2026). On 514 Seoul images with laser-measured ground truth, calibrated 90% intervals were about plus or minus 1.0 m for width at best (GPT-5.2), 7.11 to 9.04 degrees wide for longitudinal slope and 8.21 to 9.97 degrees for cross slope, against Korean limits of 2.0 m, 3.18 degrees and 1.15 degrees.

Is agreement between repeated LLM or VLM samples a sign the answer is right?

No. On the quarter of images where each model's 30 samples agreed most, the measured sidewalk width fell outside the sample interval on 77% to 90% of images, and cross slope on up to 96% for GPT-5.2. Consistent answers can be consistently wrong.

How well did VLMs grade pavement condition?

Modal-grade accuracy was 62.3% for Gemini-3-Flash, 20.6% for GPT-5.2, 10.9% for Qwen3-VL-8B and 10.1% for InternVL3.5-8B. Because 412 of 514 images were grade A, always answering A would score 80.2%. Three models called new pavement grade B.

Why was Claude Opus 4.6 excluded?

In a 10-image pilot at temperature 1.0 it returned the same width on all 30 samples for every image. The sampling-based conformal method needs variation between samples, so the model could not be calibrated. The authors state its point estimates were as close to the field values as the other models'.