SALUS scores F1 0.919 where flagging every task scores 0.716, and its 37% BIRD error rate restates 200 hand labels

What UC Irvine and Cornell released on 4 October 2026

Shiyuan Zhou, Ashwin Gerard Colaco and Sharad Mehrotra (University of California, Irvine) and Sainyam Galhotra (Cornell University) posted SALUS: Automated Auditing of NL-to-SQL Benchmarks through Weak Supervision of Multi-Agent Output as arXiv:2610.05540v1 (cs.DB) on 4 October 2026. It is the extended version of a paper accepted at ACM SIGMOD 2027 (Proc. ACM Manag. Data 4(6), Article 374, DOI 10.1145/3856369).

The method asks four LLMs (GPT-5, GPT-4o, Claude Opus 4.6, Gemini 2.5 Pro) to write SQL for each benchmark question once. It turns their execution results, AST differences and question cues into 44 labelling functions, and fits a Snorkel label model to them. The 250 most confident tasks per class (500 total) then train one depth-2 decision tree per agent that predicts when that agent can be trusted from 12 features of the gold SQL. No human label is used for training. A "Legacy" configuration swaps in GPT-4o-mini, GPT-4o, Claude 3 Haiku and Haiku 4.5.

The evaluation set, BIRD-Clean-xs, holds 298 BIRD dev tasks with manual labels: 166 erroneous, 132 correct (Section 4). Applied to whole dev sets, SALUS flags 656 of 1,534 BIRD tasks (earlier version), 571 of 1,534 (November 2025 release) and 325 of 1,034 Spider tasks. After calibration it reports about 39%, 37% and 27% annotation error rates (Table 12). From the 37% it argues that BIRD dev accuracy gaps under 5.09 points cannot be separated at 95% confidence.

The code is real. ucisharadlab/SALUS was created at 04:23 UTC on 6 October 2026 with one commit, 1edeefc "Initial release of SALUS". It ships the four label files, the recorded agent SQL and execution rows for all four configurations, saved Snorkel artifacts and per-task predictions. The GitHub API returns "license": null and the licence endpoint returns 404. The paper itself is CC BY 4.0.

Does 0.9194 survive a recount, and what does flagging everything score?

It survives. From reports/predictions/frontier.csv joined to src/data/ground_truth/BIRD_CLEAN_xs.csv we get 154 true positives, 15 false positives, 12 false negatives and 117 true negatives: precision 0.9112, recall 0.9277, F1 0.9194. Detection counts 656, 571 and 325 also match. What the paper does not print is the floor each number should be read against.

Positive class: annotation error (label 0 in the CSVs). Computed by BLOMEGA on 6 October 2026 from ucisharadlab/SALUS at commit 1edeefc. Random-flag F1 is the expectation 2pq/(p+q) for prevalence p and flag rate q (0.567 on BIRD-Clean-xs, 0.385 on the random 200).

SALUS clears every floor by a wide margin. On the random sample it sits 0.348 F1 above flag-everything and reaches MCC 0.777, 0.11 above the next best system (majority vote, 0.664). The detector works; the number to quote is 0.855 on a sample that looks like BIRD dev, not 0.919.

SQLDriller is the uncomfortable row. On BIRD-Clean-xs its F1 of 0.6456 is 0.070 below labelling every task an error. On the random sample it is 0.024 above that floor, and its accuracy (0.630) is lower than calling every task correct (0.660). Its MCC of 0.24 says it carries real signal, but F1 alone, at this base rate, does not show it. The paper reports SQLDriller at 0.6456 without a reference point, and readers could take that as a usable detector.

Why BIRD-Clean-xs inflates F1: 98 of its 298 tasks are errors BIRD already fixed

Section 4 builds BIRD-Clean-xs from two sources. One is a random 200 of the 1,534 dev tasks, hand-labelled at 68 errors and 132 correct. The other is 109 tasks whose gold SQL changed between the BIRD dev version SQLDriller used and the November 2025 release, with question and evidence unchanged and different execution results. All 109 were confirmed as errors. The paper gives 200 and 109 and a total of 298; the README states the overlap, 11 tasks.

We confirmed the consequences from the CSVs. GT_RS_200.csv is a subset of BIRD_CLEAN_xs.csv with identical labels, and the 98 extra tasks are all label 0, all typed Incorrect SQL. Every correct task in BIRD-Clean-xs comes from the random sample, so specificity (117 of 132) is identical on both sets. Only recall and precision move. The added 98 contribute 92 true positives and cannot contribute a false positive, which lifts precision from 0.805 to 0.911.

The enrichment also shifts the error taxonomy. The paper reports Incorrect SQL as 134 of 166 errors (80.7%). In the random sample it is 36 of 68 (53%), and the other four types (ambiguity, data inconsistency, domain semantics, evidence) make up 32 of 68, 47% of the errors a random audit would meet. SALUS catches 35 of the 36 sampled Incorrect SQL errors and 27 of the 32 others. Our reading: the 109 version-diff tasks were the easiest kind to confirm (BIRD's own maintainers changed them), and adding them makes both the score and the error mix look more SQL-centric than BIRD dev is.

Enriching an evaluation set with known positives is a standard way to get enough positives to measure recall. The cost is that F1 and precision then describe the enriched set, not the benchmark. The paper does report the random-200 numbers in Table 4(B) and uses them for calibration; the abstract quotes only 0.9194.

Is 37% a measurement of BIRD, or of the 200 hand labels?

Mostly of the hand labels. Table 12 applies the Rogan-Gladen correction, true rate = (flag rate - FPR) / (TPR - FPR), with TPR and FPR measured on a hand-labelled random 200 from each benchmark version. We recomputed it from GT_RS_200*.csv and the prediction files and got the paper's values: 39.3%, 36.8% and 27.0%, with delta-method intervals 33.2% to 45.5%, 30.2% to 43.3% and 21.4% to 32.6%.

The same 200 hand labels give a direct estimate without any detector: 68, 69 and 60 errors, so 34.0%, 34.5% and 30.0%. Their Wilson intervals are 27.8% to 40.8%, 28.3% to 41.3% and 24.1% to 36.7%, each about 13 points wide. The calibrated intervals are also about 12 to 13 points wide. Running SALUS over the other 1,334 BIRD tasks does not narrow the estimate, because the uncertainty comes from TPR and FPR, and those are measured on the same 200 tasks.

Where the two disagree, the gap comes from the detector's flag rate. On the earlier BIRD version SALUS flags 38.5% of the sampled 200 but 42.8% of all 1,534. Applying the correction to the sample's own flag rate returns exactly 34.0%, the hand count. The 5-point lift to 39% comes entirely from the rest of the set looking more flaggable than the sample, a difference of about 1.2 binomial standard errors. For a 37% headline, "34.5% of a random 200, hand-labelled" would have been an equally defensible and simpler statement.

The spread across audits is a definitions problem more than a sampling one. SQLDriller's 50.1% for the same BIRD dev version lies well outside SALUS's own hand-label interval of 27.8% to 40.8%, so the two teams are not counting the same thing as an error. NL2SQL-BUGs' 6.91% is far below both. SALUS's paper says this in one sentence ("differences in methodology and error definitions"); for anyone using these rates, the definition matters more than the detector.

Did BIRD's November 2025 update reduce errors in the sample? Net, no: 11 fixed, 12 newly wrong

The release labels the same 200 task IDs twice, once for the earlier BIRD dev and once for the November 2025 version. Crossing GT_RS_200.csv with GT_RS_200_nov_bird.csv: 57 tasks are wrong in both, 120 correct in both, 11 went from wrong to correct, and 12 went from correct to wrong. Errors move from 68 to 69. The paper reports that 47 of the 68 errors persisted unchanged. It does not report the 12 newly wrong tasks.

Diffing the packaged inputs (bird_dev_frontier.json.gz against bird_dev_nov_4agent.json.gz) shows why. Of the 12 newly wrong tasks, 11 had their question, evidence or gold SQL changed. Some changes are rewrites, not fixes: task 6 went from listing magnet schools to a "comprehensive performance analysis" with window functions. Across all 1,534 tasks, 474 gold SQL strings differ between the two files (the paper counts 475), and gold queries that open with a WITH clause go from 9 to 102.

Three of the 200 labels flipped on tasks whose question, evidence and gold SQL text are byte-identical in the two files (1 correct to wrong, 2 wrong to correct). That is 3 of the 115 sampled tasks with no text change. Database contents may have changed underneath; the files do not let us rule that out. If they did not, it is a small, honest measure of re-annotation noise in the reference labels themselves, about 2.6%.

What this means if you audit or build labelled benchmarks

Quote detector quality on a random sample, next to a flag-everything floor. For an error detector the floor is F1 = 2p/(1+p) at prevalence p: 0.716 at 55.7%, 0.508 at 34%. If your evaluation set is enriched with known positives, report precision and F1 on the random stratum separately, as SALUS does in Table 4(B), and put that number in the abstract. MCC or balanced accuracy travel better across base rates.

For a prevalence estimate, the hand-labelled random sample is the measurement. A detector calibrated on n hand labels cannot be more precise than those n labels on TPR and FPR. If you need a tighter benchmark-wide rate, label more random items; the detector's value is in ranking which items to send to adjudication, the use SALUS's own conclusion recommends. At a 34% rate, 200 labels give about plus or minus 6.5 points; 800 would give about plus or minus 3.3.

Treat a benchmark update as a new benchmark. In BIRD's November 2025 sample, fixes and newly introduced errors cancelled out (11 against 12). Re-audit after every release with the same random IDs, and track both directions. SALUS at $0.0332 per task on Frontier (Table 11, about $51 for BIRD dev) is cheap enough to run on every version as a triage pass.

Budget for the non-SQL error types. In a random audit of BIRD dev, 47% of errors were ambiguity, data, domain-semantics or evidence problems, which SQL substitution cannot fix. SALUS's automatic correction fixes 59 of the 134 Incorrect SQL errors (Table 16) and none of the others by design. Those cases need a human who can rewrite the question or the evidence, and here two annotators agreed at only kappa about 0.73 before adjudication.

Pick one error definition and publish it with the rate. Reported BIRD dev rates run from 6.91% to 50.1% depending on who counts. A rate without its rubric and its annotator agreement is not comparable to anyone else's.

Check it yourself

The release is about 195 MB cloned. No API keys, databases or model calls are needed for this check; we read only the label CSVs and the prediction CSVs, and did not rerun the pipeline (its pinned stack is Python 3.10, torch 2.10, snorkel 0.9.9).

# repo facts (checked 2026-10-06)
curl -s https://api.github.com/repos/ucisharadlab/SALUS | grep -E '"(created_at|pushed_at|license)"'
curl -s "https://api.github.com/repos/ucisharadlab/SALUS/commits?per_page=100" | python3 -c "import json,sys;print(len(json.load(sys.stdin)))"   # 1
git clone --depth 1 https://github.com/ucisharadlab/SALUS && cd SALUS

# files used
#   src/data/ground_truth/BIRD_CLEAN_xs.csv     query_id, db_id, difficulty, label (0 = error), error_type, correction
#   src/data/ground_truth/GT_RS_200.csv, GT_RS_200_nov_bird.csv, GT_RS_200_spider.csv   query_id, label
#   reports/predictions/frontier.csv, frontier_november.csv, frontier_spider.csv       task_id, predicted_label
python3 - <<'PY'
import csv, math
g=lambda f:{int(r['query_id']):int(r['label']) for r in csv.DictReader(open('src/data/ground_truth/'+f))}
p=lambda f:{int(r['task_id']):int(r['predicted_label']) for r in csv.DictReader(open('reports/predictions/'+f))}
def c(G,P):
    tp=sum(G[i]==0 and P[i]==0 for i in G); fp=sum(G[i]==1 and P[i]==0 for i in G)
    fn=sum(G[i]==0 and P[i]==1 for i in G); tn=len(G)-tp-fp-fn
    return tp,fp,fn,tn,round(2*tp/(2*tp+fp+fn),4)
cx,rs=g('BIRD_CLEAN_xs.csv'),g('GT_RS_200.csv'); pf=p('frontier.csv')
print(c(cx,pf), c(rs,pf))                     # (154,15,12,117,0.9194) (62,15,6,117,0.8552)
for G in (cx,rs): e=sum(v==0 for v in G.values())/len(G); print(round(2*e/(1+e),4))   # 0.7155, 0.5075
print(len([i for i in cx if i not in rs]), sum(cx[i]==0 for i in cx if i not in rs))      # 98 98
for gf,pfile,N in [('GT_RS_200.csv','frontier.csv',1534),('GT_RS_200_nov_bird.csv','frontier_november.csv',1534),('GT_RS_200_spider.csv','frontier_spider.csv',1034)]:
    G,P=g(gf),p(pfile); tp,fp,fn,tn,_=c(G,P); tpr,fpr=tp/(tp+fn),fp/(fp+tn)
    ed=sum(v==0 for v in P.values())/N; k=tp+fn
    print(gf, round((ed-fpr)/(tpr-fpr),4), k, '/200')    # 0.3934 68, 0.3676 69, 0.2697 60
nov=g('GT_RS_200_nov_bird.csv')
print(sum(rs[i]==0 and nov[i]==1 for i in rs), sum(rs[i]==1 and nov[i]==0 for i in rs))  # 11 fixed, 12 newly wrong
PY

Wilson intervals use z = 1.96 on k of 200. The delta-method intervals combine binomial variances of the full-set flag rate (n = 1,534 or 1,034), TPR (n = errors in the sample) and FPR (n = correct tasks in the sample). Our reconstruction returns the paper's Table 12 intervals to within half a point. The version diff reads question, evidence and gold_sql from the two gzipped JSON inputs in src/data/processed_query/; the README warns against joining versions on task ID alone, and our flip counts use the label files, which share IDs by design.

What would prove this wrong

Our claims are that SALUS's quality on a benchmark that looks like BIRD dev is about F1 0.85, not 0.92, and that its benchmark-wide rates are as uncertain as the 200 hand labels behind them.

First, if an independent team hand-labels a fresh random sample of at least 300 BIRD dev tasks (November 2025 release, none of the 200 used here) under SALUS's rubric, and SALUS's released predictions score F1 at or above 0.91 on it, the enrichment explanation is wrong and 0.9194 is representative. We predict F1 between 0.78 and 0.90 on such a sample.

Second, a dated prediction: by 30 September 2027, any published hand audit of a random sample of BIRD dev (November 2025) that uses an "intent not captured by the gold SQL" definition, with two or more annotators and adjudication, will report an error rate between 28% and 43%. A result outside that band, under a comparable definition, would mean the SALUS sample is unrepresentative or the rubric does not transfer between teams.

Sources

1. Zhou, S., Colaco, A. G., Galhotra, S., Mehrotra, S. SALUS: Automated Auditing of NL-to-SQL Benchmarks through Weak Supervision of Multi-Agent Output. arXiv:2610.05540v1, 4 October 2026; Proc. ACM Manag. Data 4(6), Article 374, DOI 10.1145/3856369. Sections 2.2, 4, 5.1 to 5.5, Tables 4, 8, 10, 11, 12, 16.
2. ucisharadlab. SALUS repository, commit 1edeefc, created 6 October 2026, no licence declared. Accessed 6 October 2026.
3. Jin, T., Choi, Y., Zhu, Y., Kang, D. Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards. arXiv:2601.08778, first posted 13 January 2026.
4. Liu, X., Shen, S., Li, B., Tang, N., Luo, Y. NL2SQL-BUGs: A Benchmark for Detecting Semantic Errors in NL2SQL Translation. arXiv:2503.11984, 15 March 2025; KDD 2025.
5. Yang, Y., Wang, Z., Xia, Y., et al. Automated Validating and Fixing of Text-to-SQL Translation with Execution Consistency (SQLDriller). Proc. ACM Manag. Data 3(3), Article 134, 2025, DOI 10.1145/3725271.
6. Li, J., Hui, B., Qu, G., et al. BIRD: A Big Bench for Large-Scale Database Grounded Text-to-SQLs. NeurIPS 2023; November 2025 dev update as packaged in the SALUS repository.
7. Rogan, W. J., Gladen, B. Estimating Prevalence from the Results of a Screening Test. American Journal of Epidemiology 107(1), 1978.

FAQ

How good is SALUS at finding NL-to-SQL annotation errors?

On the representative 200-task random sample of BIRD dev, SALUS (Frontier) catches 62 of 68 errors and wrongly flags 15 of 132 correct tasks: F1 0.855, MCC 0.777, against F1 0.508 for flagging every task. The 0.9194 headline is measured on BIRD-Clean-xs, where 166 of 298 tasks are errors and 98 of them are already-fixed Incorrect SQL items, so flagging every task already scores 0.716. BLOMEGA recomputed all of this from the released predictions and labels on 6 October 2026.

Is the 37% BIRD error rate estimate reliable?

It is consistent with the hand labels but adds no precision to them. SALUS's calibrated estimate for the November 2025 BIRD dev release is 36.8% with a 95% interval of 30.2% to 43.3%. The 200 hand labels it is calibrated on give 69 errors, 34.5%, Wilson interval 28.3% to 41.3%. Both intervals are about 13 points wide. For the earlier BIRD version the calibrated 39.3% sits above the hand-label 34.0% because SALUS flags 42.8% of the full set but only 38.5% of the sample.

Are the SALUS code and BIRD-Clean-xs released?

Yes. github.com/ucisharadlab/SALUS was created on 6 October 2026 with a single commit (1edeefc). It contains BIRD-Clean-xs labels, three more 200-task label files, recorded outputs of the four agents, saved weak-supervision artifacts and per-task predictions. The GitHub API reports no licence for the repository. BLOMEGA checked it on 6 October 2026.