Active spot selection trails random at 5% and 10%, and the same table reads as a 40% label saving

What changed, and when

On 23 September 2026 Zheyu Zhu, Ruining Deng (corresponding, Weill Cornell), Mert R. Sabuncu, Yuankai Huo and six co-authors posted Benchmarking Active Spot Selection for Cost-Efficient Spatial Transcriptomics (arXiv:2609.27208v1). The question is an annotation-budget question in a lab costume: if measuring a spot is the expensive label, can a model choose which spots to measure better than a uniform draw?

The design is retrospective and pool-based. Two fully profiled legacy Spatial Transcriptomics cohorts from HEST-1k (HER2-positive breast cancer: 8 patients, 36 slides, 13,620 spots; cSCC: 4 patients, 12 slides, 8,671 spots, Table 1) have their expression vectors masked. Each strategy reveals spots over 6 acquisition rounds at budgets of 5%, 10%, 30% and 50% of the fold-wide training pool. Five strategies, four budgets and four patient-level folds give 80 configurations per cohort and 160 in total. All share one ST-Net-style DenseNet-121 predictor regressing 300 genes (Sections 2 and 3.1).

The same day, Ben McEwen, Shiqi Zhang and Dan Stowell posted Active Learning for Biodiversity Monitoring (arXiv:2609.27409v1), a review of 164 studies. Its central claim applies to any labelling programme: labels bought by a query strategy are a non-random sample, so they cannot double as validation, calibration or threshold-setting data, and the expert budget has to be split as B = Btrain + Bval (Section 2.2). We use the benchmark for the numbers and the review for what they imply.

How far below Random do the four strategies land, by budget?

Each cell is the active strategy's mean held-out PCC minus Random's, over four folds. HER2 figures and fold counts are the paper's. The cSCC pooled row is ours, computed from Table 5 means, because the paper gives a pooled row only for HER2.

Parentheses: folds out of four where the active strategy beat Random, as printed in Table 4. The paper does not print per-fold cSCC PCC differences, so cSCC fold counts are not reported here. Random reference means: HER2 0.1295, 0.1940, 0.2361, 0.2367; cSCC 0.0740, 0.0948, 0.1029, 0.1066.

Before using the table we checked that it adds up. Differencing the Table 3 means reproduces every Table 4 cell to within 0.0001, the pooled HER2 means to within 0.0001, and the "active better" counts of 5, 4, 10 and 12 out of 16 exactly. Table 2's delivered spot counts (486, 978, 2,940, 4,902) follow from q = floor(βN/6) with N = 9,810. The tables are internally consistent.

In relative terms the small-budget deficit is large. Our arithmetic: the pooled HER2 gap at 5% is 13.6% of Random's mean, and the cSCC gap is 12.1%. At 30% on cSCC the pooled difference is still negative (-0.0016); the crossover the abstract reports at 30% is a HER2 result. The positive gains that do exist are small against fold spread: +0.0056 at 30% on HER2 is 0.042 of Random's fold standard deviation of 0.1340.

How a flat Random curve turns a tie into a 40% saving

The most common headline in the active-learning literature is a threshold reduction: the strategy reaches Random's score at budget X with fewer labels. The review counts it as the dominant applied summary, with reductions reaching 89% and averaging 64% over n = 7 studies in its corpus that pair the figure with a random baseline (Section 4.4). Apply that convention to this benchmark's HER2 table and it produces a clean result: CoreSet (0.2442), TOD (0.2428), TypiClust-inspired (0.2418) and MC-dropout (0.2378) at 30% all sit at or above Random at 50% (0.2367). Every strategy "saves 40% of spots".

The same table explains why that number means little. Random at 30% scores 0.2361, which is 99.7% of Random at 50% and 97.7% of the full-label reference (0.2416). Any method that lands on the plateau at 30% will "match" Random at 50%, including a second Random draw with a lucky seed. The paper runs one seed per configuration and states that training randomness is not paired across strategies (Section 3.1), so a gap of 0.0011 between MC-dropout and Random-at-50% is within what reseeding alone could move.

cSCC is the control. There Random is still climbing between 30% and 50% (+0.0037, 0.1029 to 0.1066), and the best 30% strategy, TOD at 0.1056, falls short. Same strategies, same predictor, same schedule: the "saving" appears on the cohort whose baseline flattened and disappears on the one where it did not. The review makes this argument about camera-trap corpora, where a reduction above 99% "may reflect the redundancy of the pool" (Section 4.4). This is the same effect, measured on a single paper's tables.

Two confounds the authors name belong in any reading. Optimizer exposure grows with budget (715 SGD steps at 5% versus 3,156 at 50% on HER2 fold 1, Table 2), and the 5% and 10% models are still improving at their final epoch (Section 4.1). Part of the small-budget deficit may be under-training of models whose later rounds are chosen by an under-trained scorer, rather than bad selection as such. The paper does not separate the two.

What an annotation lead should change

Default to uniform sampling below about 10% of the pool. On both cohorts, at least three of four strategies trail Random at 5% and all four at 10%. Only TypiClust-inspired edges ahead at 5% (+0.0013 on HER2, +0.0034 on cSCC), consistent with the cold-start argument that typicality methods suit low budgets. Our judgement: that margin does not pay for a scoring pass of 71 to 1,584 seconds per round (Table 3), up to about 2.2 hours over five scored rounds at 50%, when Random needs none.

Report the slope of Random, not only the saving. A threshold reduction needs to be accompanied by how much Random gains over the saved interval. "Matches Random at 50% from 30%" alongside "Random gains 0.0006 from 30% to 50%" tells a reader the saving is about plateau; the review's alternative, area under the learning curve as a ratio to Random over a stated range (Section 4.4), avoids the threshold entirely. The same gap showed up in the object-detection results we compiled, where ten methods sat within one mAP50 point of random.

Pick the metric before you pick the sampler. On HER2 at 5% and 10%, CoreSet and MC-dropout have lower PCC than Random but higher expression-cluster agreement on all four folds (ARI +0.0063 to +0.0093, Table 6). On cSCC every active strategy has lower ARI than Random at both budgets (Table 7). A ranking that flips with the metric and then fails to replicate across cohorts is not a basis for procurement.

Budget validation labels separately and draw them at random. The benchmark gets this right by construction: held-out patients (2 per HER2 fold, 2,457 to 3,810 spots) are fully profiled and never selected. They are also free, because the cohort was already sequenced. In a prospective study that set is part of the spend. The review's recipe is to state Btrain and Bval and draw Bval representatively (Section 5.2), and to record the selection mechanism in dataset metadata, since actively chosen data "outlive models" (Section 5.1). The BioDCASE task we covered in our bioacoustics note is the first shared task whose framework (BaseAL v1.2) supports active testing for the evaluation labels.

Read v1's tables, not its Figure 1. The text layer of the v1 PDF carries what look like leftover editing notes inside Figure 1, including "The paper does not use ASIGN. The current panel incorrectly suggests that ASIGN is the evaluated model" and "The budgets are separate experiments, not successive acquisition rounds. The paper does not evaluate 100%." The results section and tables are unaffected, and we found no inconsistency in them. Expect a revised figure in a later version.

Check it yourself

Everything above comes from two tables in the PDF. The script differences them and prints the plateau comparison. No code or data release is linked from v1; the cohorts are public in HEST-1k (huggingface.co/datasets/MahmoodLab/hest, which answered HTTP 200 to /api/datasets/MahmoodLab/hest on 27 September 2026).

curl -sL https://arxiv.org/pdf/2609.27208v1 -o st.pdf
pdftotext -layout st.pdf st.txt
grep -n -A7 "Table 3. HER2" st.txt     # HER2 PCC means, 4 folds
grep -n -A7 "Table 5. cSCC" st.txt     # cSCC PCC means
pdftotext -raw -f 1 -l 2 st.pdf - | grep -n "ASIGN"   # the Figure 1 notes

python3 - <<'PY'
her2 = {"Random":[0.1295,0.1940,0.2361,0.2367],"CoreSet":[0.0883,0.1751,0.2442,0.2422],
        "MC-dropout":[0.1009,0.1811,0.2378,0.2439],"TOD":[0.1276,0.1870,0.2428,0.2418],
        "TypiClust":[0.1308,0.1857,0.2418,0.2417]}
cscc = {"Random":[0.0740,0.0948,0.1029,0.1066],"CoreSet":[0.0528,0.0862,0.0994,0.1068],
        "MC-dropout":[0.0594,0.0785,0.0977,0.1093],"TOD":[0.0705,0.0883,0.1056,0.1109],
        "TypiClust":[0.0774,0.0915,0.1024,0.1106]}
for name, t in (("HER2", her2), ("cSCC", cscc)):
    R = t.pop("Random")
    pooled = [sum(v[i]-R[i] for v in t.values())/4 for i in range(4)]
    hits = [k for k, v in t.items() if v[2] >= R[3]]
    print(name, "pooled", ["%+.4f" % p for p in pooled],
          "| Random 30->50 %+.4f" % (R[3]-R[2]), "| active@30 >= Random@50:", len(hits))
PY
# HER2 pooled ['-0.0176', '-0.0118', '+0.0055', '+0.0057'] | Random 30->50 +0.0006 | active@30 >= Random@50: 4
# cSCC pooled ['-0.0090', '-0.0087', '-0.0016', '+0.0028'] | Random 30->50 +0.0037 | active@30 >= Random@50: 0

The HER2 pooled values from means differ from the printed Table 4 row (-0.0117, +0.0056) in the fourth decimal because Table 4 averages paired fold differences, not rounded means. One trap to avoid: a naive one-sided sign test on HER2's 12 of 16 positive pairs at 50% gives p = 0.038 (our arithmetic, 2,517 / 65,536). The authors decline to report p-values because the 16 pairs share four Random runs and overlapping folds (Section 3.2), and they are right to: the pairs are not independent.

What would prove this wrong

Our reading is that the HER2 "saving" at 30% is a plateau effect rather than a selection effect. That predicts that repeating the HER2 runs with several seeds will show Random's own 30% to 50% gain and the active strategies' advantage at 30% to be of the same size and both within seed noise.

A dated prediction: by 30 June 2027, any multi-seed rerun of this benchmark (three or more seeds per configuration, the paper's HER2 folds, same predictor) will report a pooled active-minus-Random PCC at 30% within 0.01 of zero, with a seed-to-seed Random standard deviation at least as large as that pooled gain. If instead the pooled gain at 30% exceeds 0.02 with all four strategies positive on all four folds, selection is doing real work on this cohort and our plateau reading is wrong. A third cohort from HEST-1k where active selection at 30% beats Random at 50% while Random is still climbing would also count against us.

Sources

1. Zhu, Z., Zhu, J., Liu, F., Yao, T., Xu, G., Cannon, J., Yang, H., Huo, Y., Sabuncu, M. R., Deng, R. Benchmarking Active Spot Selection for Cost-Efficient Spatial Transcriptomics. arXiv:2609.27208v1, 23 September 2026. Tables 1 to 7, Sections 2 to 4.
2. McEwen, B., Zhang, S., Stowell, D. Active Learning for Biodiversity Monitoring. arXiv:2609.27409v1, 23 September 2026. Sections 2.2, 4.4, 5.1, 5.2.
3. Jaume, G. et al. HEST-1k: A dataset for spatial transcriptomics and histology image analysis. NeurIPS 2024. Source of both cohorts.
4. Hacohen, G., Dekel, A., Weinshall, D. Active learning on a budget: opposite strategies suit high and low budgets. ICML 2022. Basis of the TypiClust-inspired strategy.
5. BLOMEGA. Ten active-learning methods for object detection sit within one mAP50 point of random selection on MS-COCO. September 2026.
6. BLOMEGA. The best active learning sampler beat random by 26.4%, and two of the ten submissions lost to it. September 2026.