VisionQ's gold answer is the authors' own method on all 309 benchmark questions, and 3,762 of its 3,911 records say "no human review yet"

What the VisionQ authors posted on 30 September 2026

Vu Dinh Xuan, Duc-Hai Nguyen and seven co-authors (University of Information Technology VNU-HCM, University College Cork, Hanoi University of Science and Technology, Trinity College Dublin; corresponding author Hoang D. Nguyen, UCC) posted VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision as arXiv:2610.00666v1 (cs.CV), 29 pages. The work is funded by Research Ireland grant 23/RC/13506.

It has four parts. A corpus of 1,409 CVPR 2023, ICCV 2023 and CVPR 2024 papers yields 3,651 candidate figures, 1,800+ validated comparison figures and 3,911 data points described as hand-annotated (Table 1). A six-axis, 51-leaf taxonomy of visual criteria (Image Appearance, Object Form, Scene Layout, Relation, Reference Fidelity, Prompt Match) was induced from 9,228 claims. VisionQ-Bench is 332 multiple-choice questions, 309 after a validity filter, each showing lettered crops of 2 to 4 methods with names and captions hidden and asking which best satisfies one named criterion. VisionQ-Judge is a DPO-tuned Gemma-4-E4B trained on a separate 4,524-question set, VisionQ-MCQ.

The headline: GPT-5.5 and GPT-5.3-codex tie at 0.631 against a chance level of 0.322, Grok-4.20 trails at 0.398, and VisionQ-Judge moves from 0.450 to 0.482 on the benchmark and from 49.9% to 52.4% on its own 717-question test split, with the last-option share falling from 54.1% to 47.1%. The paper's limitations section says ground truth comes from author claims that "may be aspirational, imprecise, or biased toward favorable comparisons", and that no human baseline was measured. It does not quantify either.

Does the release match the paper? The benchmark does, the annotation claim does not

Both links in the abstract resolve. On 4 October 2026 at 21:03 UTC, github.com/ReML-AI/visionq returned HTTP 200 (one commit, 29 September 2026, MIT code and CC BY 4.0 annotations) and huggingface.co/datasets/visionq-anon-2026/VisionQ-1k returned HTTP 200 (four commits, created 7 May 2026, last modified 2 October 2026, 17,835 files, about 10.0 GB). The repo ships raw per-question outputs for all 21 judge rows, which is rarer than it should be and is what made this audit possible.

Paper numbers from arXiv:2610.00666v1. Release rows checked by BLOMEGA on 4 October 2026 against GitHub ReML-AI/visionq at commit 0620e29b and HF visionq-anon-2026/VisionQ-1k at sha b440eef9.

The benchmark side is clean: running the repo's own analysis/benchmark_stats.py reproduces every overall and per-axis cell of Table 5, including the year check (2023 minus 2024 accuracy, mean -0.7 pp, higher on 2023 for 6 of 21 models). The annotation side is where the paper and the data diverge. The release's docs/status-semantics.md defines proposed as "No human review yet" and sets an MCQ eligibility rule of overall_status == "validated". By that rule, 5 of 3,911 records qualify. The paper's Appendix D.1 says MCQ items were drawn from records with status "proposed" or "validated". Our reading, marked as a judgement: humans may well have reviewed the crops through another channel, but the shipped status field does not record it, and "hand-annotated" overstates what the data documents.

How does a VisionQ question get its gold answer?

The paper is explicit about the rule: "Every question has exactly one correct answer, the author-claimed winner" (Sec. 3.3), and for the training set, "The winner_method field contains the paper's method name" (App. D.1). Section 3.2 adds that each MCQ question contains exactly one panel from the paper's proposed method. What it does not state is the consequence for the benchmark: the task reduces to "find the authors' panel", and the gold never disagrees with the authors.

We checked that on the 309 eligible questions. 235 gold answers carry an "Ours", "our" or "proposed" label in the hidden method field (for example DINAR (Ours), IDM (ours)). Another 43 match a name in the paper title (ALTO, SCADE, PRDP, UniDepth, LeftRefill, Retro-FPN and others). 22 are the subject of the claim text (ESD-x, ScatterNSG, VLC, COSeg, PYoCo, MCC). The last 9 we read by hand: w/ L_curv and w/ topology warmup in Neuralangelo, the full Pli+Sep+Sa model, F2-NeRF's perspective warp ("Our perspective warping renders more clear images"), and similar own variants. Not one gold answer is a baseline.

Letter position is not the problem. Gold letters are close to uniform: on 4-choice items 41, 40, 40 and 42 across A to D. Always answering A scores 33.3%, the best fixed letter per question size 35.3%, against a chance level of 32.2%. On the 717-question judge test split, always-A scores 41.3% and the best fixed letter per size 42.3% against chance of 39.9%, so the base Gemma's 49.9% sits 7.6 points above the strongest positional shortcut.

Is 63.1% low because judges are weak, or because they disagree with authors?

If the 20 judges were noisy readers of a clear signal, pooling them should beat any one of them. It does not. A plurality vote of the 20 rated judges (19 external VLMs plus VisionQ-Judge) gets 189 of 309 right when ties count as misses (61.2%) and 195 when 8 ties are resolved toward the gold (63.1%), level with the best single judge. Errors are correlated, which is what you would expect if many questions have a defensible answer other than the authors' panel.

The per-question spread says the same. On 131 of 309 questions (42.4%), fewer than half the judges match the gold. On 41 (13.3%), at least 14 of 20 judges converge on the same non-gold panel. On 12, none of the 20 match the gold. Those 41 include InvRender over the authors' method on a relighting figure (19 of 20), Point Transformer over Retro-FPN on a segmentation crop (20 of 20), and a baseline over the authors' method on two segmentation fidelity questions (20 and 19 of 20). We did not look at the crops, so we cannot say who is right in each case. What the data shows is that "judges are wrong" and "authors' claims are generous" are not separable in this design, and the paper has no human baseline to split them.

Why do 44 eligible questions still offer the input or ground truth as an answer?

The paper's validity filter removed 9 questions for offering a context panel. Matching the hidden method labels against input and reference names (Ground-Truth, (b) Label, (a) Source, (c) Visual Ref., Low-Resolution, Style Image, Partial, Condition, RGB, Image and similar), we find 44 more among the 309 eligible (14.2%). Across those 44, the 20 judges put 201 of their 880 answers (22.8%) on the context panel. GPT-5.5 scores 21 of 44 there and 174 of 265 elsewhere.

Some of these are not judge errors at all. Three questions from the IDM super-resolution paper ask which panel shows "more credible identities close to the ground-truth" and include the ground truth as an option; 16, 15 and 13 of 20 judges pick it, and the gold says IDM. A judge that is right about the image is scored wrong. The training set's distractor rule (App. D.1) excludes ground-truth and reference crops by regex; the benchmark does not apply the same rule.

What should you do with VisionQ if you build or buy judge evaluations?

Read the score as "agrees with authors", and say so. A 63.1% judge picked the authors' panel 63.1% of the time with names hidden. That is a useful measure if your use case is "would this judge back the paper", and a weaker one if you need "is this output better". When we run blind preference studies, the reference set always contains items where the requester's own system loses, because a gold set where one side always wins cannot detect a judge that has learned "pick the newest-looking output".

Re-score on the 265 clean questions. Dropping the 44 items that offer an input or ground-truth panel lifts GPT-5.5 from 63.1% to 65.7% and the mean of the 20 judges from 54.8% to 56.2%. Rankings barely move (Spearman 0.98 between the two orderings, no judge shifts more than two places), so the filter matters for absolute claims about how far judges sit above chance more than for leaderboard order.

Add a human baseline before quoting per-axis results. The paper reports none. On the 41 questions where 14 or more of 20 judges agree against the gold, two or three blind human raters per item would show whether the label or the judges are off. That is roughly 120 judgements, a day of work, and it would turn the benchmark's most informative slice into a label-audit set.

Treat the VisionQ-MCQ training results as not yet reproducible from public files. The training script reads mcq/dpo_v5_1.jsonl, which is not on Hugging Face; 308 of the 717 test questions reference crops that are not in the dataset. The benchmark itself is reproducible: all 1,105 bench crops are present, at the repo root rather than under bench/.

Ask for the review status, not the adjective. "Hand-annotated" and "human-verified" appear in the paper and on the dataset card; the per-record field says proposed for 96.2% of records. If you license or build on annotated data, the status field, the annotator count per item and an agreement statistic are the three things to check, in that order.

Check it yourself

Everything runs from the GitHub repo (about 12 MB, no images needed). This script prints the counts quoted above.

# release checks, 2026-10-04 21:03 UTC
curl -s -o /dev/null -w '%{http_code}\n' https://github.com/ReML-AI/visionq                          # 200
curl -s -o /dev/null -w '%{http_code}\n' https://huggingface.co/api/datasets/visionq-anon-2026/VisionQ-1k   # 200
git clone --depth 1 https://github.com/ReML-AI/visionq gh      # commit 0620e29b, 2026-09-29
python3 gh/analysis/benchmark_stats.py                          # reproduces Table 5

# files used: gh/data/benchmark/test_questions.jsonl (fields: id, choices, answer, methods)
#             gh/results/benchmark/predictions/*.jsonl (fields: id, pred_letter, correct, quality_flags)
#             gh/data/benchmark/data_points.jsonl (field: validation.overall_status)
python3 - <<'PY'
import json,glob,re,collections
G='gh/'
Q={(r:=json.loads(l))['id']:r for l in open(G+'data/benchmark/test_questions.jsonl')}
P={f.split('test_questions_')[1][:-9]:{(r:=json.loads(l))['id']:r for l in open(f)}
   for f in glob.glob(G+'results/benchmark/predictions/*.jsonl')}
E=[i for i,r in P['haiku'].items() if r['quality_flags']['benchmark_eligible']]
J=[m for m in P if m!='or_gemma4-4b-base']
ours=re.compile(r'\bours?\b|\bour\b|proposed',re.I)
win=lambda i:Q[i]['methods'][Q[i]['choices'].index(Q[i]['answer'])]
g=lambda i:Q[i]['answer'][1]
print(len(E),sum(bool(ours.search(win(i))) for i in E))          # 309 235
print(sum(g(i)=='A' for i in E)/len(E))                          # 0.333
V=[collections.Counter(P[m][i]['pred_letter'] for m in J if P[m][i]['pred_letter']) for i in E]
lead=[[k for k,v in c.items() if v==max(c.values())] for c in V]
print(sum(l==[g(i)] for l,i in zip(lead,E)),sum(g(i) in l for l,i in zip(lead,E)))   # 189 195
k=[sum(P[m][i]['correct'] for m in J) for i in E]
print(k.count(0),sum(x<10 for x in k))                           # 12 131
ctx=re.compile(r'^\(?[a-z]?\)?\s*(ground[- ]?truth|gt|label|input|rgb( input| left)?|original( design)?|condition|events|source|reference|visual ref\.?|style image|partial|forged|seen|query|support( mask)?|low[- ]resolution|right image|image|scene|main image|overlapped frames|building point clouds|multi[- ]view)$',re.I)
H={i for i in E if any(ctx.match(m.strip()) for m in Q[i]['methods'])}
print(len(H),sum(P['or_gpt-5.5'][i]['correct'] for i in H),
      sum(P['or_gpt-5.5'][i]['correct'] for i in E if i not in H))  # 44 21 174
D=[json.loads(l) for l in open(G+'data/benchmark/data_points.jsonl')]
print(collections.Counter(r['validation']['overall_status'] for r in D))
# proposed 3762, excluded 138, skipped 6, validated 5
PY

The non-"Ours" winners (43 title matches, 22 claim subjects, 9 by reading) were classified by matching the hidden winner label against data/paper_list.csv titles and the v5_claim_raw field, then reading the remainder; the list is short enough to verify by eye. The Hugging Face file list comes from https://huggingface.co/api/datasets/visionq-anon-2026/VisionQ-1k (field siblings); the status definitions are in docs/status-semantics.md and the VLM box-drawing step in README Section 6 of the same dataset. The judge-split baselines use gh/data/judge/test_questions.jsonl.

What would prove this wrong

Our claim is that VisionQ-Bench's ceiling and its per-axis gaps are partly a property of its gold labels, not only of the judges. Two things would break that.

First, a human baseline. If three blind raters per item, shown exactly the composites the judges saw, agree with the authors' panel on more than 85% of the 41 questions where 14 or more of 20 judges disagree with the gold, the judges are wrong and our reading is not. If they side with the judge consensus on more than a third of those 41, a sizeable share of the benchmark's "errors" are label errors.

Second, a dated prediction: by 30 June 2027, if the authors or anyone else releases a VisionQ version whose gold is set by blind human raters rather than by author claims, the best judge on that version will score at least 8 points higher than its score on the author-claim gold for the same items. If the gap is under 3 points, author-claim bias is not what holds scores down.

Sources

1. Xuan, V. D., Nguyen, D.-H., Dao, M.-D., Giao, V. Q., Nguyen, Q. H., Hua, B.-S., O'Sullivan, B., Murphy, D., Nguyen, H. D. VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision. arXiv:2610.00666v1, 30 September 2026. Sections 2.4, 3.2, 3.3, 4, 5.3, 6, Tables 1 to 3 and 5, Appendix D.1.
2. VisionQ authors. ReML-AI/visionq (MIT code, CC BY 4.0 annotations), commit 0620e29b, 29 September 2026. Accessed 4 October 2026.
3. VisionQ authors. visionq-anon-2026/VisionQ-1k v4.4 (CC BY 4.0 metadata, research-only crops), created 7 May 2026, last modified 2 October 2026, sha b440eef9. README, docs/status-semantics.md, eval/README.md. Accessed 4 October 2026.
4. Zheng, C., Zhou, H., Meng, F., Zhou, J., Huang, M. Study of multiple-choice selection bias in large language models. ICLR 2024. Cited by VisionQ (ref. 29) for last-option bias; title paraphrased.
5. Guerdan, L., Barocas, S., Holstein, K., Wallach, H., Wu, Z. S., Chouldechova, A. Validating LLM-as-a-judge systems under rating indeterminacy. NeurIPS 2025. Cited by VisionQ.

FAQ

What does a VisionQ score actually measure?

Agreement with the source paper's authors about their own figure. In the 309 eligible VisionQ-Bench questions, we found the gold answer is the paper's own method or one of its own variants every time: 235 carry an Ours, our or proposed label, 43 match the paper title, 22 are named as the subject of the claim and 9 were identified by reading. Appendix D.1 of arXiv:2610.00666 states the winner_method field contains the paper's method name. A 63.1% score means the judge picked the authors' panel 63.1% of the time with names hidden.

Is the VisionQ data released and does it match the paper?

Partly. On 4 October 2026 the GitHub repo ReML-AI/visionq (one commit, 29 September 2026, MIT code, CC BY 4.0 annotations) and the Hugging Face dataset visionq-anon-2026/VisionQ-1k both return HTTP 200. The 332 benchmark questions, 21 prediction files and 3,911 data points match the paper, and re-scoring reproduces Table 5 exactly. The 4,524-question VisionQ-MCQ file the training script expects (dpo_v5_1.jsonl) is not on Hugging Face, and 308 of the 717 judge test questions reference at least one crop that is not in the dataset.

Were the VisionQ data points hand-annotated by two annotators?

The paper says each item was labelled by two annotators with a third breaking ties. The released data_points.jsonl tells a different story: of 3,911 records, 3,762 have validation.overall_status proposed, which the release's own docs/status-semantics.md defines as no human review yet; 5 are validated, 6 skipped and 138 excluded. The dataset card says a vision-language model draws the bounding boxes and that inter-annotator agreement is not yet computed. The paper's Appendix D.1 confirms questions were drawn from proposed and validated records.