The VLM standing in for a human preference annotator gets 65.1% of the unambiguous pairs right

What was published, and when

arXiv:2609.21767, posted 18 September 2026 by Lobna Joualy, Eric Demeester and Nikolaos Tsiogkas at KU Leuven and Flanders Make, is an engineering result about throughput. RAPID couples GPU-parallel rollout in IsaacLab with a single-request labelling prompt, a stability rule that stops querying once the reward model settles, and farthest-point sampling over the image pool. Across five Franka Panda manipulation tasks and 135 training runs it cuts a run from 9.18 hours and 19,840 VLM calls to 1.15 hours and 896, and raises aggregate final success from 86.3% to 98.7%. The engineering is real and the ablation is controlled.

Two details in that table deserve their own line. C4's call count is 1,888 with a standard deviation of 2,537, and C5's is 896 with 656, so a labelling budget planned from the mean will be wrong on individual runs by more than the mean itself. And C2, which is C1 with the cheaper single-request prompt at one environment, drops success from 86.3% to 77.0%. The cheap prompt only becomes the better choice once 50 environments are running.

The substitute annotator scores 65.1% on the easy half

The paper builds an offline benchmark to price the prompt formats: 1,000 image pairs, 200 per task, 150 clear and 50 deliberately ambiguous, with a scripted oracle that prefers success and then task progress. Accuracy is reported on the 750 clear pairs.

The model behind every headline training number is Gemma 3 12B with the single-request prompt, stated in the implementation section: "Policy-training runs use Gemma 3 12B through OpenRouter", with GPT-4.1 mini used only offline. That configuration is the least accurate of the four, at 65.1%. On pairs designed to be unambiguous, it gets roughly one in three wrong, and it is 15.1 points above a coin flip between the two available directions. The 250 ambiguous pairs are excluded from the accuracy figure entirely, so this is the optimistic half of the benchmark.

At 896 calls per run and 65.1% accuracy, a C5 run trains on roughly 313 wrong preference labels out of 896, if the clear-pair rate carries over to live rollout pairs, which are not filtered for clarity and are therefore probably harder. The policy still reaches 98.7% success.

That is not a contradiction, and the authors say why in one careful sentence: the preferred prompt format changes with the collection regime, so "downstream policy success cannot be attributed to VLM label accuracy alone". Our reading is stronger. Success here is scored by the simulator's own task-success signal, which is available throughout and is used "only for evaluation and to construct the offline oracle benchmark". The experiment shows that on five short-horizon tasks with a scripted success criterion, a noisy preference signal is enough to get a policy there. It does not show that the VLM is a substitute for a human annotator, because no human labelled anything in this paper.

A 16 by 16 thumbnail selects better pairs than a 6,144-dimensional embedding

The sampling ablation is the most transferable result in the paper and gets the least space in it.

Grayscale thumbnails of 16 by 16 pixels, 256 numbers per image, produce 98.7% success with 896 calls. Quadruple the resolution to 32 by 32 and success falls to 96.0% while calls rise to 2,373. At 64 by 64 it is 87.9% and 3,128 calls. The learned reward-encoder features, 6,144 dimensions, give 89.7% and 3,771 calls, and trigger the stability stop in only 10 of 15 runs against 15 of 15 for the thumbnail.

The selection job is to spread queries across visibly different states. A coarse representation measures gross configuration, which is what differs between a drawer half open and a drawer shut. A fine one measures shadows and texture, and spends the budget separating states that look different and mean the same. If you run any diversity-based query selection, whether the labeller is a model or a person, this is worth an afternoon: try the cheapest representation first, and measure whether more detail buys anything.

The entire annotation guideline is eleven lines

The repository is public at github.com/rapid-vlm/rapid-vlm-rl, created 17 September 2026, and source/rapid/prompt.py holds every instruction the labeller ever sees. For the drawer task it is: "The goal is to open the drawer. Is there any difference between Image 1 and Image 2 in terms of achieving the goal?" followed by a required output line, "LABEL: <0 or 1 or -1>", with 0 meaning image 1 is better, 1 meaning image 2 is better, and -1 meaning unsure or no difference. Five tasks, five goal strings, one template.

Set that beside the COCO label policy released the same week by a different group: 80 pages, one per class, with rules for objects seen in mirrors, objects inside photorealistic posters, and objects of any size. Both documents are the complete brief given to an annotator. The gap between them is the gap between an annotation programme and an API call, and it is the reason "the VLM replaces the annotator" is a claim about cost rather than about work.

Our arithmetic on the study's own totals: the 90 ablation runs spent about 934,560 VLM calls, the 45 sampling runs about 139,080, and the offline benchmark about 6,000 across two models and two prompt formats, for roughly 1,079,640 preference queries in total, to establish that 896 per run is enough. That is the shape of most labelling economics: the experiment that finds the cheap configuration costs a thousand times the configuration it finds.

What this means if you are pricing out human preference labels

An accuracy number needs a named reference set. "74.0% accurate" against a script that reads simulator state is a different claim from "74.0% agreement with a human rater", and only the second one supports replacing a rater. A benchmark that discards its ambiguous 25% reports on the cases nobody was worried about.

Measure agreement where the task is contested. The 250 ambiguous pairs are the interesting ones, and they are where a human panel's disagreement would tell you what the label even means. Keeping multi-rater disagreement, rather than collapsing it, is what makes an ambiguous pair usable instead of noise.

A task with a scripted success signal does not need a reward model to be validated. If you can score success automatically, you can measure label quality directly against it, and you should report both. The results that will transfer to real robots are the ones where that crutch is absent, which the authors list as future work: the study is entirely in simulation, with the reward model seeing one fixed camera and no sim-to-real claim.

Budget variance, not budget means. A query budget with a standard deviation larger than its mean is not a budget. If you are buying human labels on the same adaptive schedule, price the tail.

Check it yourself

The prompt and the training scripts are public, and the labelling instruction is one file:

git clone https://github.com/rapid-vlm/rapid-vlm-rl
sed -n '1,88p' rapid-vlm-rl/source/rapid/prompt.py     # every instruction the labeller receives
ls rapid-vlm-rl/scripts/                               # train_drawer_open.py, train_button_push.py, ...
grep -rn "success" rapid-vlm-rl/source/rapid/reward_labeling.py | head

The accuracy figures come from Section VI-C and Figure 4 of the paper, the configuration table from Table II, the sampling table from Table III. The floors are arithmetic: on a clear pair the oracle names one image, so a labeller sampling the two directional labels uniformly is right half the time, and one sampling all three labels uniformly is right a third of the time. To turn the comparison into an annotation result rather than a systems result, the missing experiment is small: take the 250 ambiguous pairs, have three people label each one, and report agreement between the VLM and the majority human label, plus agreement between the people.

What would prove this wrong

Our claim is that this paper prices a labelling pipeline and does not establish that a VLM can stand in for a human preference annotator. It would be wrong if someone runs the missing experiment and Gemma 3 12B with the single-request prompt agrees with a majority-of-three human label on the ambiguous pairs at a rate close to the humans' own pairwise agreement. We predict it does not, and specifically that its agreement with the human majority on the 250 ambiguous pairs lands below 60%, since it scores 65.1% on the pairs built to be clear. The prediction is checkable with 750 human judgements, which is a few hours of annotation on a public benchmark. If it is not run by 1 March 2027, the substitution claim stays unevaluated, which is the state it is in today.

Sources

1. Joualy, Demeester, Tsiogkas. Scaling Vision-Language Reward Learning for Robot Manipulation in Parallel Simulation. arXiv:2609.21767v1, 18 September 2026, KU Leuven and Flanders Make. Tables I to III, Sections V to VII.
2. rapid-vlm/rapid-vlm-rl on GitHub, created 17 September 2026. source/rapid/prompt.py holds the labelling instructions quoted here. Retrieved 21 September 2026.
3. Wang, Sun, Zhang, Xian, Biyik, Held, Erickson. RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback. ICML 2024. The two-stage baseline RAPID builds on.
4. Christiano, Leike, Brown, Martic, Legg, Amodei. Deep Reinforcement Learning from Human Preferences. NeurIPS 2017. The human-labelled formulation being automated.
5. Penquitt et al. Object Detection Benchmarks are Incomplete. arXiv:2609.21822, 18 September 2026. Source of the 80-page COCO label policy used for the comparison.

FAQ

Can a vision-language model replace a human preference annotator?

arXiv:2609.21767 opens with that claim and does not test it. Its labeller, Gemma 3 12B with a single-request prompt, scores 65.1% against a scripted oracle on 750 pairs the authors built to be unambiguous, and no human label appears anywhere in the paper. A labeller guessing between the two directional labels scores 50.0% on those pairs by construction. What the paper does establish is throughput: 896 API calls and 1.15 hours per run against 19,840 calls and 9.18 hours for the two-stage baseline.

How accurate are VLM preference labels?

On the clear half of a 1,000-pair benchmark in arXiv:2609.21767: GPT-4.1 mini 74.0% with a single request and 72.0% with two, Gemma 3 12B 70.9% with two requests and 65.1% with one. The 250 deliberately ambiguous pairs are excluded from those figures. The reference is a script that reads simulator success and task progress, not a human rater.

Why does a policy reach 98.7% success on noisy preference labels?

Because success is scored by the simulator's own task-success signal on five short-horizon manipulation tasks, and a reward model fitted to a noisy but directionally correct preference signal is enough to reach it. The paper notes that the preferred prompt format flips between collection regimes, so downstream success cannot be attributed to label accuracy alone. It is evidence about the tasks, not about label quality.

What image resolution should drive query selection?

The coarsest that separates the states you care about. In arXiv:2609.21767 Table III, farthest-point sampling on 16 by 16 grayscale pixels gives 98.7% success with 896 API calls, against 96.0% and 2,373 calls at 32 by 32, 87.9% and 3,128 at 64 by 64, and 89.7% and 3,771 for 6,144-dimensional learned features.