Cleanlab flagged 12.70% of 34,924 retinal images, 42.49% of the flags were real errors, and its confidence score can't tell which (AUC 0.54 to 0.67)
A preprint posted 30 September 2026 ran Cleanlab's Datalab module on 34,924 images pooled from nine public retinal datasets. It flagged 4,434 images (12.70%) for human review, and four-stage ophthalmologist adjudication confirmed 1,884 of them (42.49% of the flags, 5.39% of all images) as real label errors, a roughly eight-fold enrichment over the base rate. But Cleanlab's own confidence score, the number meant to say which flagged image is most likely wrong, separated true errors from false alarms with an AUC of only 0.54 to 0.67 wherever the difference was even statistically significant, and the two ophthalmologists re-grading the same flags without seeing that score agreed with each other only moderately (Cohen's kappa 0.34 to 0.46).
What did the preprint test, and when?
On 30 September 2026, Yongleng Fang, Tian Lin, Haoyu Chen and seven co-authors from the Chinese University of Hong Kong, the Joint Shantou International Eye Center and Sun Yat-sen University's Zhongshan Ophthalmic Center posted Label Cleaning in Multimodal Public Retinal Image Datasets to Research Square (DOI 10.21203/rs.3.rs-10775714/v1). It is a preprint, not yet peer-reviewed, and the authors disclose no conflict of interest.
They pooled nine public datasets across three imaging modalities: color fundus photography (ODIR, MuReD, the Eye Diseases Classification set, here EDC), optical coherence tomography (OCT2017, OCTID, OCTDL) and ultra-widefield fundus imaging (TumorUWF, OpenUWF, TsukazakiUWF). After excluding ambiguous categories (ODIR's and MuReD's "other" class, TsukazakiUWF's undefined "AO" and "DM" labels) and de-duplicating OCT2017 to one foveal cross-section per patient, 34,924 of the original 114,715 images remained.
A Swin Transformer was trained separately on each dataset (batch size 32, learning rate 0.0001, 10 epochs, 3 folds, patience 2, Adam optimizer) purely to generate the out-of-fold predicted probabilities and feature embeddings that Cleanlab's confident-learning algorithm needs. Those went into Cleanlab's Datalab module, which ran three checks per dataset: near-duplicate detection on feature-space distance, a Label Issue check that estimates the confident joint between observed and latent labels and proposes a corrected label with a confidence score, and an Outlier check that flags atypical embeddings without proposing a replacement label.
Every flag went through a four-stage human pipeline. Two ophthalmologists (Reviewer A and B) independently graded each flagged image, seeing the original label and, for Label Issue flags, Cleanlab's suggested label, but blinded to the confidence score and to each other. Discordant cases went to a third senior ophthalmologist, blinded to the first two reviews. A majority vote across the three set the label (the senior reviewer's call broke three-way ties), and a senior consultant ophthalmologist confirmed every revised label in a final stage. Near-duplicates were checked separately, by perceptual hashing (pHash, Hamming distance ≤ 5) rather than by a human.
How precise is a Cleanlab flag, dataset by dataset?
Pooled, a flag was right 42.49% of the time. Split by dataset, precision ranges nearly 40-fold, from 1.50% to 57.14%, and three of the nine datasets sit on either side of that pooled rate by more than 10 points.
| Dataset | Modality | Images (n) | Flagged | Confirmed errors | Precision (PPV) | Suggested-label agreement | Source |
|---|---|---|---|---|---|---|---|
| ODIR | CFP | 5,684 | 1,310 (23.05%) | 515 (39.31% of flags) | 39.31% | 86.72% | Table 1 |
| MuReD | CFP | 2,060 | 484 (23.50%) | 231 (47.73% of flags) | 47.73% | 51.95% | Table 1 |
| EDC | CFP | 4,217 | 208 (4.93%) | 91 (43.75% of flags) | 43.75% | 60.29% | Table 1 |
| OCT2017 | OCT | 5,620 | 379 (6.74%) | 142 (37.47% of flags) | 37.47% | 65.41% | Table 1 |
| OCTID | OCT | 572 | 7 (1.22%) | 4 (57.14% of flags) | 57.14% | 75.00% | Table 1 |
| OCTDL | OCT | 2,064 | 107 (5.18%) | 35 (32.71% of flags) | 32.71% | 75.76% | Table 1 |
| TumorUWF | UWF | 2,031 | 133 (6.55%) | 2 (1.50% of flags) | 1.50% | 100.00% (n=1) | Table 1 |
| OpenUWF | UWF | 700 | 131 (18.71%) | 44 (33.59% of flags) | 33.59% | 70.73% | Table 1 |
| TsukazakiUWF | UWF | 11,976 | 1,675 (13.99%) | 820 (48.96% of flags) | 48.96% | 58.96% | Table 1 |
| Pooled | all | 34,924 | 4,434 (12.70%) | 1,884 (42.49% of flags) | 42.49% | not a simple pool | Table 1 |
"Flagged" and "Confirmed errors" are the Label Issue plus Outlier totals after removing overlap between the two modules. Precision (PPV) is confirmed errors divided by flagged images. Suggested-label agreement is the share of confirmed errors where Cleanlab's proposed replacement label matched the final, adjudicated label; TumorUWF's 100.00% rests on a single confirmed case. Source: Chen et al., Table 1, posted 30 September 2026.
Why a flag that is right 42% of the time still can't be ranked
Cleanlab's Datalab module does two separable jobs: decide whether to flag an image at all, and, for flagged images, assign a confidence score meant to say how likely the flag is a real error. The preprint's results show the first job working (flags are 7.9 times more likely to be real errors than a random image: 42.49% against the 5.39% base rate) and the second job not working (the confidence score barely separates real errors from false alarms within the flagged set).
The authors' own cross-check on this gap: they compared Cleanlab's confidence score between confirmed errors and false-positive flags, dataset by dataset and module by module, 18 cells in total (nine datasets times two modules; OCTID had no Outlier flags at all). Only four cells reached p < 0.05 (Mann-Whitney U), and their AUCs were 0.54 (MuReD Label Issue), 0.54 (ODIR Label Issue), 0.57 (TsukazakiUWF Label Issue, direction reversed: false positives scored higher) and 0.67 (EDC Outlier). OCTID's Label Issue AUC reads as a perfect 1.00, but on only 7 flagged images and 4 confirmed errors, so it is not statistically significant (p=0.06) and not evidence the score works there either. None of the 18 cells reaches an AUC that the authors would call useful for deciding an individual image's label, which is their own stated bar.
The 1,884 confirmed errors split three ways. 1,435 (76.17%) were in-distribution corrections, where the image simply belonged to another class already in the dataset's own taxonomy. 103 (5.47%) were multi-label cases, where more than one retinal condition was visible in the same image. 400 (21.23%) were out-of-distribution, a diagnosis the dataset's class list had no category for, including rare diseases and ungradable images. 54 images counted in both the multi-label and out-of-distribution groups, so after removing that overlap, 449 of the 1,884 confirmed errors, 23.83%, could not be represented at all within the original single-label scheme.
We cross-checked one number the paper states in two different places rather than together. The text says the Outlier module's 244 confirmed errors included 147 (60.25%) not caught by the Label Issue module; Table 1 separately gives Label Issue 1,737 confirmed and Outlier 244 confirmed, combining to 1,884 unique. Solving 1,737 + 244 − overlap = 1,884 gives an overlap of exactly 97, which equals 244 − 147. The two independently stated numbers agree, which is a check worth doing before trusting either count on its own.
What this means for anyone running a Cleanlab-style audit
Use the flag to triage, not the score to prioritize. The binary decision to flag an image is where Cleanlab earns its keep here: it cut the pool needing full manual re-annotation from 34,924 to 4,434, an 87.3% reduction, while catching the large majority of errors a from-scratch audit would find. The continuous confidence score attached to each flag is a separate claim, and on this evidence it should not decide review order within the flagged set. Budget reviewer time close to evenly across flags rather than skimming only the highest-confidence ones.
Run the Outlier module even though it has no replacement label to offer. It caught 147 of 244 of its own confirmed errors (60.25%) that the Label Issue module missed, plus out-of-distribution cases at more than double the rate of Label Issue (15.88% vs 7.70%, odds ratio 2.26) and rare-disease cases at nearly five times the rate (0.78% vs 0.16%, odds ratio 4.95). Skipping it to save a step throws away a meaningfully different error population, not a redundant one.
Do not transfer a precision number across tasks. A flag's hit rate here moved from 1.50% to 57.14% depending only on which dataset it ran on, same method, same thresholds, same model architecture. A pilot showing 40% precision on one annotation task is not a license to assume 40% on the next one; class separability and imbalance in the specific label space appear to dominate.
A single reviewer is not a verification step. The two ophthalmologists grading the same flagged images, blinded to each other and to Cleanlab's score, reached only moderate agreement (kappa 0.34 on Label Issue flags, 0.46 on Outlier flags). That is the agreement rate on images a model already picked out as suspicious, where you would expect disagreement to be lower than on a random sample, not higher. Treat a flagged-image audit as needing the same adjudication discipline as original labeling, not less.
Decide in advance what happens to cases outside the taxonomy. 449 of the 1,884 confirmed errors here (23.83%) were multi-label or out-of-distribution cases that a single-label classification scheme cannot hold, including 13 rare-disease cases and 32 images judged ungradable from poor image quality. A cleaning pipeline that only offers "accept Cleanlab's suggested class" or "keep the original class" will quietly misfile close to a quarter of what it correctly identifies as wrong.
For how differently two correction frameworks can read the same underlying noise, see our note on Dawid-Skene's calibration gap between majority vote and model-based consensus. For how an instrument choice alone can swing a measured noise rate by an order of magnitude, see HH-RLHF's label noise at 39% or 1.8%.
Check it yourself
The preprint and its four supplementary tables are open access (CC BY 4.0) at the DOI below. The arithmetic cross-check above (the 97-image overlap) uses only numbers printed in the paper; this recomputes it and the pooled enrichment factor from the published counts.
curl -sL -o main.pdf \
"https://assets-eu.researchsquare.com/files/rs-10775714/v1/fbdad34f-3725-433c-a00f-0a3d6dbe05ad.pdf"
pdftotext -layout main.pdf main.txt # Results section, Table 1, Section 3
python3 - <<'PY'
# every number below is copied from Chen et al., Table 1 and Section 3 (30 Sep 2026)
flagged_total = 4434
confirmed_total = 1884
images_total = 34924
li_flagged, li_confirmed = 3767, 1737
out_flagged, out_confirmed = 894, 244
out_only_confirmed = 147 # stated directly in Section 3
overlap_derived = li_confirmed + out_confirmed - confirmed_total
overlap_from_text = out_confirmed - out_only_confirmed
print("overlap via inclusion-exclusion:", overlap_derived) # 97
print("overlap stated another way: ", overlap_from_text) # 97, cross-checks
ppv_pooled = confirmed_total / flagged_total
base_rate = confirmed_total / images_total
print(f"pooled precision: {ppv_pooled:.4f} base rate: {base_rate:.4f} "
f"enrichment: {ppv_pooled/base_rate:.2f}x")
# pooled precision: 0.4249 base rate: 0.0539 enrichment: 7.88x
PYTo run Cleanlab's own checks on a new dataset, the module call is Datalab(data={"labels": y}).find_issues(pred_probs=probs, features=embeddings), documented at docs.cleanlab.ai; the preprint used Cleanlab's own default out-of-fold training recipe (3-fold, 10 epochs) to produce probs and embeddings for each of the nine datasets independently.
What would prove this wrong
If this preprint reaches peer-reviewed publication, the underlying model outputs that produced the AUC and PPV numbers will not change, since confident learning's flags are a deterministic function of the probabilities and embeddings already computed. We predict that a peer-reviewed version appearing by 31 December 2027 will report the same 4,434 flagged, 1,884 confirmed, and an AUC range that still sits below 0.70 for every statistically significant dataset-module cell. If a revised version reports any cell's confidence-score AUC above 0.75 with p<0.05, that would mean our reading of "the score doesn't rank what's flagged" rested on an error the authors later corrected, not a property of the method.
The broader claim, that Cleanlab's confidence score is weak for ranking flagged medical images generally, is falsifiable on a tenth dataset. If someone runs the same Datalab pipeline on a new public retinal or other medical-image dataset of comparable size (order 1,000 to 10,000 images) and reports a confidence-score AUC above 0.80 for distinguishing confirmed errors from false-positive flags, with the comparison pre-registered before manual review, that would contradict the pattern here and suggest this preprint's weak AUCs were a property of these nine datasets or their imbalance, not of the scoring method itself.
Sources
- Fang, Y., Lin, T., Chen, H. et al. Label Cleaning in Multimodal Public Retinal Image Datasets. Research Square preprint, DOI 10.21203/rs.3.rs-10775714/v1, posted 30 September 2026. Table 1, Supplementary Tables 1, 2, 3 and 4, Sections 2 to 4.
- Northcutt, C., Jiang, L., Chuang, I. Confident Learning: Estimating Uncertainty in Dataset Labels. Journal of Artificial Intelligence Research 70, 2021, p. 1373 to 1411. The method behind Cleanlab's Datalab module.
- Northcutt, C.G., Athalye, A., Mueller, J. Pervasive label errors in test sets destabilize machine learning benchmarks. NeurIPS Datasets and Benchmarks Track, 2021.
- Lin, T., Wang, M., Lin, A. et al. Efficiency and safety of automated label cleaning on multimodal retinal images. NPJ Digital Medicine 8(1), 2025, p. 10. Prior Cleanlab evaluation using synthetic 0 to 70% label noise on fundus and OCT data, cited by Fang et al. as reference 18.
- Wang, Y., Han, X., Li, C. et al. Impact of Gold-Standard Label Errors on Evaluating Performance of Deep Learning Models in Diabetic Retinopathy Screening: Nationwide Real-World Validation Study. Journal of Medical Internet Research 26, 2024, p. e52506.
- Cleanlab. Datalab quickstart documentation. Accessed 24 October 2026.
- BLOMEGA. Majority vote runs underconfident and Dawid-Skene runs overconfident on the same labels.
- BLOMEGA. HH-RLHF's label noise is 39% or 1.8%, depending on the instrument.