The first decision model labels 44x cheaper and 11.6 F1 points worse, and its confident labels are 73.6 percent right

A typed-answer model shipped in September, and within days it had a pre-registered audit

Hazem Ibrahim and Yasir Zaki (Computer Science, New York University Abu Dhabi) posted Evaluating Decision Models for Text Annotation in Computational Social Science (arXiv:2609.24574v1, 47 pages) on 21 September 2026. All model runs are dated 20 September 2026 (Table 5), and the design, hypotheses and correction procedure were pre-registered as AsPredicted #312,511.

The product under test is Jev, from a company called TypeSafe, served through the OpenRouter decisions endpoint as model id typesafe/jev-1.13; every call resolved to jev-1.13-20260917. The paper describes it as the first commercially available decision model, released in September 2026 at $0.042 per million input tokens. A decision model takes a typed question with an enumerated label set and returns a choice, a probability distribution over every option and a scalar confidence. It never generates free text, so it cannot return an unparseable answer. The training method is called RLCD, reinforcement learning for calibrated decisions (unrelated to the 2023 alignment method with the same acronym).

Around it sits an open ecosystem released the same week, which the paper also runs: Mapika/decider (Qwen3.5 0.8B and 2B decoders), Laya (a 420M ModernBERT-large encoder), SemIf (a frozen Qwen3.5-4B logit readout with no training), Bespoke-Nimble-9B, and an RLCD-trained Qwen3-0.6B. A community leaderboard for Jev-style models, JevBench, lives at benchmarkheaven.com/jev-models.

The benchmark is the Ziems et al. (2024, Computational Linguistics) CSS suite: 18 closed-choice tasks, their released test splits of at most 500 items, their prompts verbatim. Three tasks (stance, implicit hate, discourse acts) were a pilot; the other 15 carry every headline number. The 19 baselines span GPT-5.6, Claude, Gemini, Llama, Qwen, DeepSeek, Mistral, GLM, Kimi and Gemma, and each was asked to append a 0 to 100 verbalized confidence.

Cheap, parse-proof, and 11.6 points behind: the numbers

All values from arXiv:2609.24574v1. Accuracies in the cascade rows are medians over the 15 evaluation tasks.

The per-task cost multiples in that chart are ours. Dividing each task's best-LLM price by Jev's in Table 7 gives a range of 21x (media ideology, where Gemini 3.8 Flash wins at $1.358 per 1,000) to 296x (emotion, where Opus 5 wins at $5.035), and the median of the 15 ratios is 44.5x, which matches the paper's "median 44 times". The deficit column also reproduces: the median of the 15 Table 21 values is -11.6. Only the dialect task has a positive point estimate (+4.2, interval [-3.9, +11.5]), and the authors flag that the comparator is an oracle: the best LLM is picked on the same test items, which favors the LLM side.

How one forward pass becomes a routing decision

A chat LLM labels by generating a string that a parser has to map back onto the codebook, and the confidence has to be asked for in words. Jev scores the declared options in a single pass and returns the distribution directly, so the routing signal comes free with every label. That is why the cascade is cheap: the first stage costs $0.027 per 1,000 items, so on the Gemini cascade almost all of the spend is the escalated share. At t = 0.8 the median task keeps Gemini's accuracy within 0.4 points (0.674 against 0.678) at 56 percent of Gemini's cost.

The cheap-partner cascade works for a different reason. Jev and Gemma 4 31B err on different items, so at t = 0.5 the combination beats Gemma alone, 0.638 against 0.626. The dialect task shows the same complementarity in item counts: of 266 items, Jev alone is right on 106 and the best LLM alone on 40. Table 18 shows where this breaks. When Jev's price doubles, the Gemma cascade costs 1.47 times Gemma alone at t = 0.5, because the two prices sit within a factor of three and routing overhead eats the saving. The frontier-partner result holds under every price scenario (cost fraction 0.54 to 0.61 at t = 0.8).

What the diagram cannot show is whether the confidence is right about which items to keep. That depends on calibration per construct, and here the paper's median-task summary hides the concentration.

Pooled over items, the 0.9 bar buys 73.6 percent accuracy, not 81.5

This section is our analysis. The paper reports routing on the median task: 0.815 accuracy at 0.376 coverage. A team labeling a mixed corpus experiences the pooled number instead: every confident label they keep, across every construct. The replication package at github.com/hazemibrahim97/decision-models-css (MIT, created 21 September 2026) ships analysis/items.csv, 237,192 rows with one record per item and model, including conf, conf_status and correct. We filtered to the 15 tasks marked confirmatory in analysis/cell_metrics.csv and counted.

Jev answers all 6,547 evaluation items. At confidence of at least 0.9, 2,644 items (40.4 percent) qualify, of which 1,945 are correct: a pooled accuracy of 0.736. The per-task counts match Table 22 exactly (for example empathy: 389 of 498 confident, 149 right, which is 0.781 coverage and 0.383 accuracy). The 699 confident errors are not spread evenly.

Empathy contributes 240 and toxicity prediction (Conversations Gone Awry) 125, together 365 of 699, 52.2 percent. Drop those two tasks and the remaining 13 give 1,934 confident items at 0.827 accuracy. Drop only empathy and it is 2,255 items at 0.796. Raising the bar does little: pooled accuracy moves from 0.736 at 0.9 to 0.742 at 0.95 while coverage falls from 0.404 to 0.302, because the empathy items sit at the top of the confidence scale.

Pooled over the 6,547 items of the 15 evaluation tasks. Coverage denominators include items with no usable confidence, which only matters for the LLMs (Jev has none). LLM confidence is verbalized, Jev's is the returned field.

The comparison row is the useful one for a routing decision. Opus 5's verbalized confidence at 0.9 keeps fewer items (26.0 percent) at higher pooled accuracy (0.802), but costs $6.080 per 1,000 items to Jev's $0.027. Gemini 3.8 Flash at 0.9 keeps 65.3 percent at 0.749. None of the three clears 0.85 pooled at any threshold we checked. Our judgement: a single global confidence cut is not a quality control on a multi-construct corpus with any of these models.

Where Jev pays for itself, and where a $0.05 open-weight LLM beats it

Jev alone is a substitute on two constructs, not fifteen. Ideological books and Indian English dialect are the tasks where the accuracy gap spans zero under both the fixed and the selection-aware bootstrap, at 35x ($0.016 against $0.568 per 1,000) and 44x ($0.048 against $2.135) lower cost. On dialect Jev also has the grid's lowest calibration error of 22 models, 0.063, which rejected the authors' pre-registered expectation that dialect text would be where calibration failed. Character tropes looked like a third exception until the selection-aware interval moved below zero, [-15.5, -1.4].

The 44x headline compares Jev to frontier prices; the cheap tier is closer. The paper notes that among six hosted open-weight models under $0.10 per 1,000 items, Gemma 4 31B (0.611) and Qwen3 235B (0.604) beat Jev's median macro-F1 of 0.581. Our per-task version from Table 7: the most accurate LLM priced under $0.10 per 1,000 beats Jev's accuracy on 10 of 14 tasks (tropes has no LLM that cheap), by a median 2.0 accuracy points, at a median 2.2 times Jev's cost. Jev wins clearly on dialect (Gemma 29.0 points behind) and media ideology (GPT-OSS 120B 7.7 behind). It is still the cheapest metered model on every one of the 15 evaluation tasks. The same oracle caveat applies: we picked the cheap winner on the test items.

Budget for a validation set per construct, and read it as a stop signal. On empathy (TalkLife), Jev predicts "no exploration" for 422 of 498 items (Table 13: 136 + 133 + 153 in the predicted-"none" column), recall 0.92 on that class and 0.02 on strong exploration, while stating confidence of at least 0.9 on 78 percent of items. Gemini 3.8 Flash collapses onto the same class (Table 20), and every open decision system scores between 19.8 and 35.7 macro-F1 there, so the failure belongs to the construct. A 100-item gold sample would have shown it. Politeness adds a second warning: no model's confidence tracks the spread of five human annotators' scores (|rho| at most 0.14, Table 23).

The calibration edge is thin once competitors are tuned. Fitting a single temperature on the three pilot tasks moves Jev's median ECE from 0.157 to 0.121. By our count from Table 10, 15 of the 19 LLMs end below 0.121 after the same one-parameter fix, led by Llama 4 Scout at 0.060. If a native distribution is the reason to adopt Jev, compare it to scaled verbalized confidence, not raw.

Human review is the expensive half of the pipeline. Routing everything under 0.9 to a reviewer assumed to be correct lifts median-task accuracy to 0.944, at $31.24 per 1,000 items if a human label costs $0.05 and $624.51 at $1 (Table 19). At that point Jev's $0.027 is a rounding error, and the question is only how many items the threshold sends to people. For our related notes on annotator agreement, see when LLM annotators agree with each other and with nobody else and reproducibility is not construct validity.

Check it yourself

The pooled routing numbers need one 23 MB download and no API key. The authors' own pipeline (analysis/routing_curves.py, analysis/cascade.py) reproduces their tables from the same files.

B=https://raw.githubusercontent.com/hazemibrahim97/decision-models-css/master/analysis
curl -sO $B/items.csv          # task,model,kind,conf_elicited,id,gold,pred,valid,correct,conf,conf_status,p_top,brier,cost
curl -sO $B/cell_metrics.csv   # per-cell metrics; column "split" marks confirmatory vs discovery

python3 - <<'PY'
import csv, collections
split = {r['task']: r['split'] for r in csv.DictReader(open('cell_metrics.csv'))}
n = collections.Counter(); k = collections.Counter(); ok = collections.Counter()
for r in csv.DictReader(open('items.csv')):
    if r['model'] != 'jev' or split[r['task']] != 'confirmatory': continue
    n[r['task']] += 1
    if r['conf_status'] == 'ok' and float(r['conf']) >= 0.9:
        k[r['task']] += 1; ok[r['task']] += int(r['correct'])
N, K, C = sum(n.values()), sum(k.values()), sum(ok.values())
print(N, K, round(K/N, 3), C, round(C/K, 3), 'wrong', K - C)
for t in sorted(k, key=lambda t: ok[t] - k[t])[:2]:
    print(t, 'wrong', k[t] - ok[t])
PY
# 6547 2644 0.404 1945 0.736 wrong 699
# talklife wrong 240
# conv_go_awry wrong 125

For the cost ratios, take the $/1k column of Table 7 in the PDF: for each evaluation task, divide the price of the LLM with the highest macro-F1 by Jev's price. The 15 ratios have a median of 44.5. For the cheap-tier comparison, restrict to LLMs at or under $0.100 and take the highest accuracy. arXiv: 2609.24574, Tables 2, 3, 7, 10, 19, 21, 22.

What would prove this wrong

The paper is a snapshot of one Jev version (jev-1.13-20260917) on public benchmarks that predate every evaluated model, so contamination cannot be excluded, and gold labels are single adjudicated answers on constructs trained annotators dispute. A better model, or a better gold standard, could change every number here.

A dated prediction: by 31 March 2027, a rerun of the released grid with the then-current Jev version (the full grid cost $0.21) will still show pooled accuracy below 0.80 on items at confidence 0.9 or above across the 15 evaluation tasks, and empathy will still be the largest single source of confident errors. If a later Jev reaches 0.85 pooled accuracy at 35 percent coverage or more, a global confidence threshold becomes defensible and the per-construct validation advice above weakens.

A second test: our cheap-tier result predicts that on a new CSS task outside the Ziems suite, the most accurate hosted open-weight LLM under $0.10 per 1,000 items will beat Jev's accuracy more often than not. If Jev wins on most new tasks, the 2.0-point median gap we computed is an artifact of choosing the winner on the test items.

Sources

1. Ibrahim, H., Zaki, Y. Evaluating Decision Models for Text Annotation in Computational Social Science. arXiv:2609.24574v1, 21 September 2026. Tables 1 to 3, 7, 10, 13, 18 to 22, 25; Sections 3 to 5.
2. Ibrahim, H. decision-models-css replication package. GitHub, created 21 September 2026, MIT. Files used: analysis/items.csv, analysis/cell_metrics.csv. Accessed 23 September 2026.
3. Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., Yang, D. Can Large Language Models Transform Computational Social Science? Computational Linguistics 50(1), 2024. Task suite and test splits: SALT-NLP/LLMs_for_CSS.
4. TypeSafe. Jev decision model, model id typesafe/jev-1.13, via the OpenRouter decisions API, September 2026 (paper reference 26).
5. Mapika. decider, GitHub, created 16 September 2026. Nandha Kishor M. Laya, GitHub, created 18 September 2026. Theo Lee. SemIf, GitHub, created 16 September 2026.
6. Maio, A. Qwen3-0.6B RLCD Decision Model. Hugging Face, 2026.
7. JevBench. Community leaderboard for Jev-style decision models, accessed 23 September 2026.
8. BLOMEGA. Five LLM runs agree at 0.995, and the measure correlates 0.029 with what it was built to measure. 22 September 2026.