Uber's RE-AD Checks Labels in 2.3 Seconds, 16x Faster Than Batch LLM Review

What changed, and when

Five researchers at Uber AI Solutions, Siddarth Malreddy, Ishan Nigam, Akshay Arora, Nikhil Mittal, and Subrat Sahu, posted "RE-AD: Real-Time Requirement Adherence for Data Labeling" to arXiv (2607.20455) on 14 May 2026. The paper was accepted to the Fifth Workshop on Generation, Evaluation and Metrics (GEM), held alongside ACL 2026 in San Diego in July, pages 148 to 154, DOI 10.18653/v1/2026.gem-main.17. Uber AI Solutions is the internal group that also sells Uber's uLabel and uTask annotation tools externally. RE-AD is a research write-up of a quality-control layer the team built for Uber's own labeling pipelines, not a new product announcement.

The problem RE-AD targets is specific: crowd-sourced annotation for training LLMs runs on long written Standard Operating Procedures (SOPs) that mix hard formatting constraints, such as wrapping code in triple backticks, with subjective judgment calls, such as flagging a response that reads as condescending. Annotators drift from these rules through misreading, fatigue, or genuine disagreement about what a rule means. The usual fix, a full LLM review pass once a labeling batch finishes, catches that drift only after the work is done and the annotator has already moved to the next item.

RE-AD changes two things: when the check runs, during labeling rather than after, and how it checks, with one validator per rule tier rather than a single model judging everything at once. An LLM agent first converts the SOP into a list of atomic rules through iterative self-reflection, applying an atomicity and orthogonality test adapted from Ji et al. (2023), and the paper reports under 5% variance in the resulting rule list across repeated decompositions of the same SOP. The rules then split into three tiers, Formatting (T_fmt), Simple (T_sim), and Subjective (T_sub), each validated by a different mechanism, all running in parallel with prefix caching so the system stays responsive while the annotator types.

The evidence table

The paper's benchmark, RE-AD-Eval, builds a 20-rule SOP (8 formatting, 5 simple, 7 subjective) for a conversational data-collection task, generates clean conversations that satisfy all 20 rules, then injects exactly one violation per sample using three synthetic annotator personas, Expert, Intermediate, and Novice, with tier-specific error rates. Ground truth for formatting rules comes from deterministic code; ground truth for simple and subjective rules comes from an independent LLM check, run on Gemini models the authors say they chose to avoid biasing results toward RE-AD's own validator.

The pattern across the tier rows is one number falling while another holds steady: precision is perfect on formatting (1.000), strong on simple (0.908), and collapses on subjective rules (0.402), while recall barely moves (1.000, 0.824, 0.876). RE-AD keeps catching most real subjective violations. It just surrounds each real catch with more false alarms.

Three validators per SOP, not one model checking everything

Formatting rules need no model call at all: a regex or string parser checks line breaks, placeholders, and markup, which is why that tier hits F1 1.000 at zero LLM latency. Simple rules, things like word-count ceilings or required disclaimers, go to Gemini 3 Flash with a zero-shot prompt. Subjective rules, tone, cultural appropriateness, whether an answer counts as genuinely helpful, go to Gemini 3 Pro with chain-of-thought prompting that produces a written rationale before a pass or fail verdict. All three run per rule and in parallel, and the system caches the SOP's system prompt and atomized rule list so each new annotator turn only costs a short incremental pass, which the paper credits for most of the latency gap against batch review.

Recall holds up across tiers; precision does not

Table 4 of the paper backs this with a direct comparison: running the same 20-rule check as one batched pass after labeling finishes, instead of RE-AD's tiered real-time pass, costs 5.8 seconds median with Gemini 3 Flash (total F1 0.628) or 36.9 seconds median with Gemini 3 Pro (total F1 0.742). RE-AD's 2.3-second median beats the faster batch option on both speed and F1, and comes within 0.007 F1 of the slower, roughly 16-times-more-expensive batch option (36.853 / 2.278 ≈ 16.2x, matching the paper's own claim). The one place batch review wins outright is the subjective tier, where it scores F1 0.612 against RE-AD's 0.551, because a single model reading the full conversation keeps context that RE-AD's rule-by-rule atomization throws away.

The same erosion shows up by annotator skill level, not just by rule type: more experienced synthetic annotators produce violations RE-AD catches with higher precision, but recall stays uniformly high regardless of skill.

What it means for someone doing this work

For a team running model-in-the-loop pre-labeling in the Snorkel, Argilla, or Label Studio mold, RE-AD's numbers argue for routing checks by rule type rather than running one judge model over an entire SOP. The two-thirds of a typical SOP that is formatting and simple rules, 13 of RE-AD-Eval's 20, can run on regex and a cheap zero-shot call for close to free and near-perfect accuracy, F1 0.864 to 1.000. The subjective third is where the cost and the judgment calls both concentrate, and RE-AD does not solve it so much as narrow it: a flag with precision 0.402 means three in five flagged subjective violations are not real, so the system is only useful there as a triage signal a human still has to adjudicate, not an auto-reject gate. That lines up with other work in this cluster showing disagreement concentrates in subjective labeling questions, not factual ones.

The headline production number, 82% of flagged errors accepted and fixed, is the weakest part of the paper by the authors' own account. It comes from one unnamed company workflow over roughly a month, with no annotator count, no label volume, and no isolation of RE-AD's effect from concurrent UI changes and the annotators' own learning curve. Flags were shown as non-blocking suggestions, which likely inflates acceptance since annotators were never forced to resolve a flag to proceed. Treat 82% as evidence the tool looked useful enough that Uber kept it running, not as a controlled quality lift.

Our judgement: the engineering comparison, F1 by tier, latency by tier, batch versus real-time, is specific enough to trust. It is reported with full tables, a stated benchmark construction method, and a baseline that reuses the same two model sizes RE-AD itself calls on. The production claim is a company anecdote, and the paper says so directly. We weight the latency and precision/recall numbers as reliable engineering findings, and the 82% figure as suggestive but not proof that real-time checking improves label quality, only that Uber's annotators engaged with it at a high rate when it was optional and non-blocking.

Check it yourself

# Primary source (preprint, submitted 14 May 2026)
curl -sL https://arxiv.org/abs/2607.20455 | grep -i "0.749"

# Published workshop version (GEM 2026, ACL Anthology)
curl -sI https://doi.org/10.18653/v1/2026.gem-main.17
curl -sL https://aclanthology.org/2026.gem-main.17.pdf -o re-ad.pdf

# Numbers to check against Table 3 and Table 4 of the PDF:
#   Formatting tier:    P 1.000  R 1.000  F1 1.000
#   Simple tier:        P 0.908  R 0.824  F1 0.864
#   Subjective tier:    P 0.402  R 0.876  F1 0.551
#   RE-AD p50 latency:          2.278 s
#   Batch (Gemini 3 Flash):     p50 5.823 s, total F1 0.628
#   Batch (Gemini 3 Pro):       p50 36.853 s, total F1 0.742
#
# 36.853 / 2.278 ~= 16.2x, matching the paper's own "over 16x" claim.
# Persona breakdown is in Table 2: Expert / Intermediate / Novice rows.

What would prove this wrong

RE-AD-Eval is synthetic end to end: an LLM wrote the SOP, an LLM wrote the clean conversations, and an LLM injected exactly one violation per sample. If Uber AI Solutions or an independent group publishes a follow-up that runs the same tiered, real-time approach against a public, human-collected labeling dataset with naturally occurring, not injected, violations, and subjective-tier precision comes in above 0.55 while recall stays near 0.85, that would show the 0.402 precision figure here is mostly an artifact of synthetic, single-violation construction rather than a hard limit on LLM judgment of subjective rules. We predict no such replication clears 0.55 subjective-tier precision on real annotator data before 31 December 2027, because RE-AD's own ablation already shows atomized per-rule checking loses context that a single full-context batch pass partly recovers, subjective F1 0.551 for RE-AD versus 0.612 for batch Gemini 3 Pro, on the same benchmark.

Sources

1. Malreddy, S., Nigam, I., Arora, A., Mittal, N., Sahu, S. "RE-AD: Real-Time Requirement Adherence for Data Labeling." arXiv:2607.20455, submitted 14 May 2026. arxiv.org/abs/2607.20455.
2. Malreddy, S., Nigam, I., Arora, A., Mittal, N., Sahu, S. "RE-AD: Real-Time Requirement Adherence for Data Labeling." Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM), ACL 2026, San Diego, pp. 148-154. DOI: 10.18653/v1/2026.gem-main.17.
3. BLOMEGA. LLM judge soft labels vs human disagreement. Related internal analysis on subjective-label agreement.
4. BLOMEGA. Decision models for text annotation cost and accuracy. Related internal analysis on annotation cost/accuracy tradeoffs.