Re-annotating KITTI moves Grounding DINO from fourth place to first, and drops RT-DETR by 51 AP points

What was released, and when

arXiv:2609.21822, posted 18 September 2026 by Penquitt, Klees and colleagues at Osnabrück University, the University of Wuppertal and Wayve, re-annotates four object detection benchmarks end to end: KITTI (7,481 training images, 7 classes), Cityscapes (2,975 train plus 500 val, 8 classes), COCO 2017 (the 5,000-image validation split, all 80 classes) and Pascal VOC 2012 (11,540 images, 20 classes). The work is funded under the German BMFTR project RELiABEL, grants 16IS24019A and 16IS24019B.

The pipeline is a three-stage microtask plus post-processing. Class labels come from at least 11 independent annotators per box, which makes the output a soft label, a probability distribution over classes, rather than one class name.

The false negative rates are the part to keep. On KITTI, 41.5% of visible objects were missing from the original labels, and the re-annotation leaves 11.6% missing. Nobody gets to zero. The authors say so: "there is no perfect data".

Rankings hold on COCO and VOC, and break on KITTI and Cityscapes

The paper reports that "model rankings remain largely stable" when the six detectors are scored against the better labels. We recomputed that claim from its own Table 8a. Over the same six models, Spearman rank correlation between the original-label ordering and the validated-label ordering is +0.886 on COCO and +0.943 on Pascal VOC. On KITTI it is +0.657 and on Cityscapes +0.714, and on both of those the top model changes.

The KITTI column is the interesting one. RT-DETR-L falls from 78.5 to 27.3, YOLOv8-L from 74.7 to 26.7, Faster R-CNN from 65.3 to 26.7. Grounding DINO rises from 29.2 to 39.4 and takes first place. The split is exactly fine-tuned against open-vocabulary: models trained on KITTI's original annotation convention are being marked against a different convention, while zero-shot models were never fitted to the convention in the first place and convert previous false positives into true positives. The Pearson correlation between a dataset's missing-box error rate and its mean AP change across the six models is -0.784 over the four datasets, which with n=4 is a direction, not a result.

Our judgement: the phrase "rankings remain largely stable" is safe for COCO and VOC, and should not be carried over to a dataset whose original labels excluded whole regions. If you are using KITTI to pick a detector, the label convention decides the winner.

The annotator instructions are public, and part of the gap is written into them

Four Google Drive links inside the PDF resolve to the label policies handed to annotators, one per dataset. They download without a login. We read them.

The COCO policy is 80 pages, one per class, 35,021 characters, about 438 characters of instruction per class. Every one of the 80 class entries repeats four general rules: box the visible part only, label objects seen through glass, label objects in mirrors, label objects inside photographs, posters and signs when photorealistic, and label objects no matter how small they are. The Pascal VOC policy carries the mirrors and photorealistic rules on all 20 classes but never the size rule. The Cityscapes and KITTI policies carry neither, and KITTI's entire instruction set is 1,942 characters for seven classes, mostly class boundaries: a Car is a regular passenger car and not a van or a truck, a Pedestrian is walking or standing and not sitting in a café.

That matters for how the +40.0% on COCO should be read. A portion of the objects added to COCO are objects that the original COCO convention would not have asked for: tiny ones, reflections, objects inside a poster. The paper concedes the point in general terms, that "some differences arise from dataset-specific annotation conventions", and does not split the increase into convention change and genuine misses. Its false-negative audit does bound the genuine part: three expert reviewers found 13.1% of visible objects missing from original COCO and 6.1% from the validated version, so at least half the original miss rate was real. KITTI's +60.0% has a different cause again, since the audit counts objects inside the original "don't care" regions.

Ranking false positives by confidence beats every published label error detector

The second benchmark in the paper is label error detection on real errors rather than synthetic noise. Three published methods are compared against two baselines. The score baseline, which takes a detector's false positives with respect to the original labels and ranks them by confidence, has the best F1 on all four datasets.

The margin is not small: 59.29 against 36.89 on KITTI, 43.53 against 31.08 on COCO. The naive baseline recovers 98.3% to 100.0% of errors at 0.4% to 2.1% precision, which is the same as reviewing everything. Label error detection is a ranking problem for a review queue, and on this evidence the ranking signal that works is the detector's own confidence.

What this changes if you build or buy detection data

Four things follow.

Absolute AP on a public benchmark is a property of the label set, not only of the model. Grounding DINO on COCO is 56.9 or 52.1 depending on which ground truth you score against. Quoting one number without the annotation version is now an incomplete claim.

A relabelling job needs a written size and occlusion rule before it starts. The measurable difference between a +40% dataset and a +5% dataset sits in one line of a guideline. Write the rule, version it, and ship it with the data, the way these four policies are shipped.

Budget for disagreement, not for consensus. Eleven annotators per box is the cost of knowing that a box is ambiguous. The soft labels line up with expert judgement: objects experts call clearly correct average 0.737 to 0.843 probability, objects they call not correct average 0.169 to 0.285. A single annotation cannot produce that number at any price.

Test your label error tooling against a confidence baseline. If a vendor's error detector cannot beat "sort the model's false positives by score", it is adding cost and not recall.

Check it yourself

The rank recomputation is the paper's Table 8a and eight lines of Python.

python3 - <<'PY'
import statistics
M=["GroundingDINO","YOLOv8-World","OWL-ViT","RT-DETR-L","YOLOv8-L","FasterRCNN"]
T={"COCO":([56.9,44.5,28.7,52.4,51.7,47.1],[52.1,43.8,28.6,47.2,47.8,43.1]),
   "VOC":([80.0,69.6,52.8,70.3,74.4,61.2],[73.8,65.4,49.2,64.2,67.4,55.6]),
   "KITTI":([29.2,19.9,9.3,78.5,74.7,65.3],[39.4,24.4,9.6,27.3,26.7,26.7]),
   "Cityscapes":([34.2,24.1,11.6,38.2,38.1,35.9],[43.8,30.4,13.8,42.5,44.4,41.0])}
def rk(v):
    o=sorted(range(len(v)),key=lambda i:-v[i]); r=[0]*len(v)
    for p,i in enumerate(o): r[i]=p+1
    return r
def rho(a,b):
    ra,rb=rk(a),rk(b); ma,mb=statistics.mean(ra),statistics.mean(rb)
    n=sum((x-ma)*(y-mb) for x,y in zip(ra,rb))
    d=(sum((x-ma)**2 for x in ra)*sum((y-mb)**2 for y in rb))**.5
    return n/d
for k,(o,v) in T.items():
    print(k, round(rho(o,v),3), M[o.index(max(o))], "->", M[v.index(max(v))])
PY

The four annotator label policies are linked from the PDF and download directly:

# COCO (80 pages), Pascal VOC, KITTI, Cityscapes label policies
curl -L -o coco.pdf   "https://drive.usercontent.google.com/download?id=1lWBdhOV665ye4UBOMDaT2d0AUHiQy0l4&export=download"
curl -L -o voc.pdf    "https://drive.usercontent.google.com/download?id=17za9WVbpNLjipyt4mNTFsSnidjKPrbCj&export=download"
curl -L -o kitti.pdf  "https://drive.usercontent.google.com/download?id=1BrvtNaXgjywWEDLXoM31Eni_by8eXdYd&export=download"
curl -L -o cityscapes.pdf "https://drive.usercontent.google.com/download?id=134MEILCPpeqQQGcWIJ_cTwMi-DRSyd4k&export=download"
# count the size rule
python3 -c "from pypdf import PdfReader; t=''.join(p.extract_text() or '' for p in PdfReader('coco.pdf').pages); print(t.count('no  matter  how  small'))"   # 80

The paper says the code and the validated annotations "will be publicly available on GitHub". On 21 September 2026 the repository linked in the PDF, github.com/JonathanKlees/Uncertainty-aware-Object-Detection, returns HTTP 404. Until it opens, the validated ground truth itself cannot be checked, only the policies and the printed tables.

What would prove this wrong

Our claim is that on KITTI the change of leader is caused by the annotation convention rather than by model quality, and that the same relabelling applied to a dataset without "don't care" regions leaves the order alone. It fails if, when the repository opens, someone scores the six detectors on a KITTI subset restricted to objects that were inside the original labelled regions and outside "don't care", and Grounding DINO still ranks first. It also fails if a re-run of the COCO evaluation with the small-object rule switched off, that is with objects below 40 pixels dropped on both sides, still moves Grounding DINO's AP by more than two points. We expect neither, and both are checkable within a day of the data landing. If the repository is still 404 on 1 December 2026, treat every number in Table 8 as unreplicated.

Sources

1. Penquitt, Klees, van Betteray, Jashnieh, Stehr, Rottmann, Schmarje. Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty. arXiv:2609.21822v1, 18 September 2026. Tables 2, 3, 4, 6, 7, 8, 10.
2. Label policy documents for COCO, Pascal VOC, KITTI and Cityscapes, linked from the paper. Retrieved 21 September 2026.
3. Lin et al. Microsoft COCO: Common Objects in Context. ECCV 2014. The benchmark being re-annotated.
4. Geiger, Lenz, Urtasun. The KITTI Vision Benchmark Suite. CVPR 2012. Source of the "don't care" regions.
5. Northcutt, Jiang, Chuang. Confident Learning: Estimating Uncertainty in Dataset Labels. JAIR 70, 2021. The synthetic-noise tradition the label error benchmark departs from.

FAQ

How many objects are missing from COCO's labels?

Re-annotating the 5,000-image COCO 2017 validation split with at least 11 annotators per object raised the object count from 36,781 to 51,487, a 40.0% increase (arXiv:2609.21822, Table 2). An expert audit of 600 sampled images put the original false negative rate at 13.1% of visible objects and the re-annotated rate at 6.1%. Part of the increase is a convention change: the COCO label policy used in this work tells annotators to label objects no matter how small, and to label objects in mirrors and in photorealistic pictures.

Do better labels change which object detector wins?

On two of four datasets, yes. Recomputing Spearman rank correlation over the six detectors in Table 8a of arXiv:2609.21822 gives +0.886 on COCO and +0.943 on Pascal VOC, where the order barely moves, against +0.657 on KITTI and +0.714 on Cityscapes, where the leading model changes. On KITTI, RT-DETR-L falls from 78.5 to 27.3 AP50..95 and Grounding DINO rises from 29.2 to 39.4 and takes first place.

What is a soft label in object detection?

A probability distribution over classes for one bounding box, aggregated from many independent annotator votes instead of a single class name. In arXiv:2609.21822 each box carries a soft label from at least 11 annotators, with 11 more for ambiguous cases. Objects experts judge clearly correct average 0.737 to 0.843 probability on the proposed class, and objects experts judge incorrect average 0.169 to 0.285.

Which label error detection method works best on real errors?

None of the published ones. In the benchmark of arXiv:2609.21822, ranking a detector's false positives by confidence has the highest F1 on all four datasets: 59.29 on KITTI, 44.06 on Pascal VOC, 43.53 on COCO and 34.21 on Cityscapes. ObjectLab, MetaDetect and loss inspection all sit below it, despite performing well on synthetic noise in prior work.