The best active learning sampler beat random by 26.4%, and two of the ten submissions lost to it

What changed, and when

On 14 September 2026 Ben McEwen, Rupa Kurinchi-Vendhan, Shiqi Zhang, Lukas Rauch, Marek Herde and Sara Beery posted BioDCASE: Active Learning for Bioacoustics (arXiv:2609.15255v1), the results paper for Task 4 of the 2026 BioDCASE challenge. Datasets were released in April 2026 and the submission deadline was 15 June 2026. Ten acquisition functions arrived from seven teams across twelve institutions in nine countries, and were ranked against four baselines on a held-out test set.

Active learning papers are usually not comparable to each other: different backbones, different budgets, different metrics, different data. This challenge removed all of those variables. Participants received pre-computed 1,536-dimensional PerchV2 embeddings rather than audio, a fixed two-layer MLP classification head, an oracle labeller and a budget of 500 labels. The only things they could edit were the acquisition function, an optional warm-up function for the first cycle, and the batch size. Every result is the average of five independent runs.

The four subsets are three terrestrial soundscape sets from BirdSet (HSN, 12,000 segments, 19 classes, 0.52 labels per segment; POW, 4,560 segments, 41 classes, 2.83 labels per segment; UHH, 36,637 segments, 25 classes, 1.05) and one marine set, the Acoustic Trends Blue Fin Library (ATBFL, 19,633 five-second segments, about 21.7 hours, 7 blue and fin whale call types, 79.8% of segments containing at least one call). Everything is cut into 5-second segments, so a "label" here is one annotator decision on one segment.

The evidence table

The full ranking on the held-out test set. Score is the area under the learning curve of macro mAP, averaged across the four subsets, higher is better. Compute is relative to the reference configuration; sampling time is seconds to score and select one batch; annotation cost is the challenge's proxy, the total number of labels attached to the segments a method chose, where more labels per segment means a slower annotator. Baselines are italicised, and the last two rows scored below random sampling.

Scores, compute, sampling time and annotation cost are from Table 3 of arXiv:2609.15255v1. The "vs random" column is ours, computed as (score minus 0.401) divided by 0.401.

Two things in that table are worth more than the ranking. The first is row 11: Margin, pure uncertainty sampling, scores 0.408 against random's 0.401, a gain of 1.7%, while selecting segments carrying 31% more labels than random (1263.7 against 966.1). On the challenge's own annotation-cost proxy, uncertainty sampling is the most expensive method to annotate and very nearly the least useful. The second is rows 13 and 14: two confidence-quantile strategies that a team submitted, tested and shipped, and that lose to picking segments at random.

Averaging hides where the gain comes from. Per subset, the best entry against random: 0.625 against 0.374 on HSN (+67.1%), 0.428 against 0.333 on UHH (+28.5%), 0.490 against 0.433 on POW (+13.2%), and 0.502 against 0.465 on the marine ATBFL (+8.0%).

The spread across the fourteen entries tells the same story from the other side. On HSN the range between best and worst is 0.251 AULC; on ATBFL it is 0.055. Choosing an acquisition function matters roughly five times more on the sparse bird soundscape than on the whale recordings. Rankings across the two domains are still broadly consistent, Spearman rho = 0.855 across ten submissions, so a method that works on birds is likely to work on whales; it just will not buy you as much.

Coverage carries a small budget, and uncertainty only pays once the classifier is worth trusting

All ten submissions were hybrids of diversity and uncertainty. None was pure uncertainty, which is the textbook default. The baselines explain why: CoreSet, a pure k-center diversity criterion with no notion of model confidence, scores 0.464, while Margin, a pure uncertainty criterion, scores 0.408. Every team placed at least one entry above CoreSet.

The top two methods agree on the schedule and disagree on how to trigger it. ADU-MMR keys the shift to the model: the uncertainty term contributes nothing until mean entropy over the unlabelled pool falls below 0.05, and its weight is then capped at 0.5 so diversity never becomes the minority signal, with batches diversified by greedy maximum marginal relevance at lambda = 0.3. CARE-DPP keys it to the budget instead, dropping its novelty weight from 0.65 to 0.25 at 250 labels and selecting each batch with a determinantal point process. Removing the determinantal point process costs CARE-DPP 0.038 AULC, from 0.502 to 0.464, which is larger than the entire spread between the first and eighth ranked entries.

The gap between first and second is 0.002 with an average standard deviation of 0.007, so the paper states plainly that it is not significant. The difference that is real is the bill: ADU-MMR halved the acquisition batch to 25, doubling the number of cycles and its relative compute to 2.0, the highest of any entry. CARE-DPP got statistically the same score at 1.1.

One component earned much less than expected. Parcerisas et al. compared warm-up procedures across ten acquisition strategies and found a 0.003 AULC difference between them, against 0.121 for the choice of acquisition function. Two other teams measured warm-up gains of 0.026 and 0.014 on the development set. Our reading: with a strong in-domain pretrained embedding, the first batch is nearly free to choose, and effort belongs in the acquisition rule.

What it means for anyone buying annotation by the hour

Price the batch, not the method. The challenge reports what almost no active learning paper does: a proxy for how expensive the selected items are to label. Margin sampling's segments carry 31% more labels each than random's, and TypiClust's carry 0.7% fewer. If your annotators are paid by time and your acquisition function keeps handing them the busiest items, an AULC gain on a per-label axis can be a loss on a per-hour axis. Ranking on labels acquired is the wrong unit for a budget conversation.

Run the pilot on your own data before you buy the method. A 67.1% gain and an 8.0% gain came from the same submission on two subsets of the same challenge. HSN, the sparsest BirdSet subset at 0.52 labels per segment, showed the widest separation between methods; the paper is careful to say four subsets are too few to attribute that to label density alone. The actionable version: measure your own gain against random at your own budget, because the published number is not transferable.

Random sampling is a real baseline, not a formality. Two submitted methods lost to it. Both were built on confidence quantiles, and both came from the team that also placed an entry above the strongest baseline, so this is not a weak team; it is a reminder that a plausible acquisition rule can be worse than nothing, and only a held-out comparison tells you which.

Budget the acquisition compute too. Sampling time across the ranked entries spans 0.001s (random) to 7.800s (Coreset Eigenvalues) per selection, and CoreSet's 5.838s is roughly 2,900 times Margin's 0.002s. That does not matter for 500 labels. For a passive acoustic deployment with millions of unlabelled segments, scoring the pool each cycle becomes the dominant cost, which is exactly why ADU-MMR's 2.0x relative compute for a statistically insignificant lead is a bad trade at scale. This is the same tradeoff we track for annotation budget allocation: cheap proxies that pick a near-optimal point beat expensive searches for the exact one.

Check it yourself

Both datasets, the framework and every technical report are public. The two ranked datasets are on Zenodo, the framework is versioned, and each submission's method description is a separate PDF on the challenge site.

# the results paper, with Tables 1 to 4
open https://arxiv.org/html/2609.15255v1

# the two challenge datasets and the framework
open https://doi.org/10.5281/zenodo.19133112   # ATBFL, marine, 19,633 segments
open https://doi.org/10.5281/zenodo.19340660   # BirdSet subsets HSN / POW / UHH
open https://doi.org/10.5281/zenodo.21806641   # BaseAL v1.2.0, the fixed AL loop

# recompute every gain quoted above from Tables 3 and 4
python3 - <<'PY'
rnd_overall = 0.401
print("overall +%.1f%%" % (100*(0.507-rnd_overall)/rnd_overall))
per = {"ATBFL":(0.502,0.465),"HSN":(0.625,0.374),"POW":(0.490,0.433),"UHH":(0.428,0.333)}
for k,(b,r) in per.items():
    print("%-6s best %.3f random %.3f  +%.1f%%" % (k,b,r,100*(b-r)/r))
print("margin +%.1f%%  coreset +%.1f%%" % (100*(0.408-.401)/.401, 100*(0.464-.401)/.401))
print("margin annotation cost vs random: %.1fx" % (1263.7/966.1))
PY
# overall +26.4%
# ATBFL  best 0.502 random 0.465  +8.0%
# HSN    best 0.625 random 0.374  +67.1%
# POW    best 0.490 random 0.433  +13.2%
# UHH    best 0.428 random 0.333  +28.5%
# margin +1.7%  coreset +15.7%
# margin annotation cost vs random: 1.3x

The two top methods also published standalone technical reports on arXiv, which is unusual for a challenge and makes them independently checkable: ADU-MMR (arXiv:2607.04868, 6 July 2026) and CARE-DPP (arXiv:2607.06063, 7 July 2026). CARE-DPP's own report gives a mean development AULC of 0.50 against 0.46 for the CoreSet baseline, consistent with the 0.505 and 0.464 in the organisers' held-out table. All ten reports are linked from biodcase.github.io.

One caveat on reproduction: participants submitted acquisition functions, not predictions, and the organisers re-ran them on hardware they controlled. Numbers in the participants' own reports are development-set numbers and will not match Table 3.

What would prove this wrong

The central claim here is not that ADU-MMR is the best sampler. It is that the ranking is budget-shaped and dataset-shaped, and that the per-label metric flatters methods that pick expensive items. That would be wrong if a second edition ranking on labelling cost rather than labels acquired produced the same order as Table 3.

A dated prediction: when BioDCASE publishes a 2027 edition of this task with an annotation-cost-aware ranking, the top-5 ordering will differ from the 2026 label-count ordering by at least two positions, and pure uncertainty sampling will still sit within 5% of random on the primary metric. If the 2027 results reproduce the 2026 top five in the same order under a cost-aware metric, this reading is wrong. Deadline: 31 December 2027.

Sources

1. McEwen, B., Kurinchi-Vendhan, R., Zhang, S., Rauch, L., Herde, M., Beery, S. BioDCASE: Active Learning for Bioacoustics. arXiv:2609.15255v1, 14 September 2026. Tables 1 to 4, Sections 2.3 to 4.2; HTML version.
2. Dubus, G., Magaldi, H., Gros-Martial, A. Adaptive Diversity-Uncertainty Active Learning with Redundancy Control for Bioacoustic Event Classification. arXiv:2607.04868v1, 6 July 2026. The rank-1 submission's own report.
3. Magaldi, H., Dubus, G. Determinantal point process sampling for bioacoustic active learning. arXiv:2607.06063v1, 7 July 2026. The rank-2 submission's own report.
4. Kurinchi-Vendhan, R., Zhang, S., McEwen, B. BioDCASE 2026 Task 4: ATBFL dataset. Zenodo, March 2026.
5. Rauch, L., Herde, M., McEwen, B. BioDCASE 2026 Task 4: BirdSet Dataset. Zenodo, 2026.
6. McEwen, B., Zhang, S. BaseAL: Release v1.2.0. Zenodo, August 2026. The fixed active learning loop all entries ran inside.
7. BioDCASE 2026 challenge, biodcase.github.io, including all ten Task 4 technical reports.
8. BLOMEGA. The near-optimal region of an annotation budget is wide.
9. BLOMEGA. Data annotation research: the latest.