A second JEV audit finds no cost edge over batch-priced GPT-6 Luna. The gap from "44x cheaper" comes down to comparator, batch pricing and question bundling

What did the Leiden paper test, and when?

Matthew DiGiuseppe (Institute of Political Science) and Steven Denney (Leiden Institute for Area Studies), both at Leiden University, posted JEV versus LLMs: Accuracy, Cost and Calibration on Seven Political Science Replications (arXiv:2610.06625v1, cs.CL and cs.CY, 71 pages, 14 tables) on 5 October 2026. It re-runs seven published studies that used LLMs for annotation, scaling or data generation: Gilardi et al. (2023) on tweet relevance, Ornstein et al. (2025) on tweet sentiment, Brandt et al. (2026) on terrorist attack type, Weidmann et al. (2026) on V-Dem indicators, Di Leo et al. (2025) on party positions, DiGiuseppe and Flynn (2026) on open-ended survey answers, and Le Mens and Gallego (2025) on manifestos and speeches.

JEV was called as typesafe/jev-1.13 through OpenRouter on 19, 22 and 23 September 2026, and every response resolved to jev-1.13-20260917 (Appendix E). The comparators were GPT-6 Luna through OpenAI's Batch API on 25 and 28 September, with reasoning off, and the open-weight Qwen3.8-27B through OpenRouter on one host (Parasail, FP8) on 30 September. Timing runs were 200 items per application from the Netherlands on 29 and 30 September.

The accuracy verdict is mixed rather than negative: JEV beats GPT-6 Luna on V-Dem (mean country correlation 0.680 against 0.628) and on BBC conflict F1 (0.321 against 0.139), trails it on tweet relevance and sentiment, and is level on party positions and the survey pairs (Table 9). The cost verdict is the new part. Our earlier note on arXiv:2609.24574 covered the accuracy and routing side of the first JEV audit; this one is about why two independent papers, using the same model at the same list price, reach opposite cost conclusions.

Same model, same $0.042 per million tokens: why is it 44x cheaper in one paper and dearer in the other?

The JEV side barely moves between the two papers. Ibrahim and Zaki measured $0.027 per 1,000 items on the Ziems et al. social science tasks; DiGiuseppe and Denney measured $0.012 to $0.027 per 1,000 decisions on their seven tasks. What changes is the thing JEV is divided by.

All dollar values are per 1,000 items (2609.24574) or per 1,000 decisions, one answer to one question about one item (2610.06625). Medians over the seven rows of Table 2 computed by BLOMEGA from the printed three-decimal values, so ties at $0.017 and $0.012 may hide a difference of under $0.001.

Read top to bottom, the chart is a reconciliation. The 44.5x headline divides JEV by whichever of 19 LLMs was most accurate on each task, for example Gemini 3.8 Flash on media ideology or Opus 5 on emotion, at real-time prices, with a 0 to 100 confidence line appended to every LLM prompt (more output tokens). Against the cheap Luna of that generation, GPT-5.6 Luna, the multiple was already only 4.5x, and against GPT-OSS 120B 1.6x. The Leiden paper then picks the cheapest current OpenAI model, asks for about four output tokens per answer, and applies the Batch rate, which is exactly half the Standard rate on every row of Table 2 within rounding. Halving a median 1.30x gives roughly the 0.64x.

Neither paper is wrong. They answer different questions: "how much cheaper than the LLM I would otherwise pick for accuracy" against "how much cheaper than the cheapest LLM that does the job overnight". The second question is the one a data-ops lead with a fixed corpus and no deadline faces. Our judgement: the Leiden comparison is the fairer floor, and the 44x figure is the fairer ceiling.

Table 2 confirms the text's two counts with one caveat. JEV is cheaper than Standard Luna on six of seven tasks; Di Leo et al. is the exception ($0.018 against $0.016). Batch Luna is strictly cheaper on five rows, and the Gilardi and DiGiuseppe rows tie at the printed precision ($0.017 and $0.012). The text says Luna is cheaper on six and 3% dearer on the seventh, so one of the two ties must favour Luna below the third decimal; the table alone cannot say which.

How does JEV bill a decision, and why does bundling decide the comparison?

TypeSafe charges only for input tokens, at $0.042 per million, and lists no batch or cached-input rate on its models page (checked 6 October 2026). Input means the state plus every question's instructions and every option description, on every call. The paper notes that JEV billed 1.2 to 3.3 times as many input tokens per decision as Luna, while Luna wrote about four output tokens per answer. A long codebook therefore costs JEV in full each time. The Luna figures use OpenAI's posted rates, under which a repeated prompt prefix can be cached; Appendix H gives an upper bound of 1.27x JEV if every uncached input token is instead billed at the posted cache-write rate.

The lever JEV does have is sharing the state. Several questions about one text go in one request and the text is billed once. The Leiden team used it: the three Gilardi questions (relevance, problem frame, solution frame) were "bundled in one request" (Appendix A), and Appendix E reports a schema-fixed output-token count of 97 for that bundle. Table 2 then divides by decisions, so the Gilardi cost is a three-way split of one request. TypeSafe's own cookbook makes the same point at a larger scale, reporting 13 questions in one call as 12.2 times cheaper than 13 separate calls.

We measured how much this matters on the Gilardi prompt. Asked alone, the relevance question costs a mean 827 input tokens per request over 500 tweets (range 778 to 949), or $0.0348 per 1,000 decisions. Bundled with the two framing questions on 50 of those tweets, a request costs 1,179 tokens for three decisions, 393 per decision, $0.0165 per 1,000. On the same 50 tweets the single-question mean was 824 tokens, so bundling cuts the per-decision bill by 52%. Single-question JEV on this prompt ($0.0348) is level with Standard Luna ($0.034) and twice Batch Luna ($0.017).

The paper's own bill confirms the accounting. Appendix C reports 12,156 bundled requests (four runs over the 2,559 plus 480 tweets) for $0.61, which at $0.042 per million is 1,195 tokens per request. We measured 1,179, 1.3% lower; dividing 1,195 by three gives 398 tokens and $0.0167 per 1,000 decisions, which rounds to Table 2's $0.017. Bundling did not change the answers in our sample: 49 of 50 relevance labels matched the single-question run, with a mean probability shift of 0.0126, in line with the paper's 0.0085 for bundled-versus-separate and 0.0120 for replays (Appendix E). Our output-token counts, 39 single and 95 bundled, track the paper's schema-fixed 38 to 97.

The same divisor moves the speed column. Gilardi's 0.09 seconds per decision in Table 2 is the fastest row, against 0.25 to 0.27 seconds for the single-question tasks; three decisions per request accounts for about that factor. On our route the bundled request took a median 0.44 seconds (0.15 per decision) against 0.40 seconds for one question.

Does JEV reproduce the paper's 89.0% on Gilardi tweets? Yes, 89.8% on 500

Gilardi et al. publish their replication archive on Harvard Dataverse (doi:10.7910/DVN/PQYF6M, CC0, released 19 July 2023). The file training_data_relevance_final.tab holds the 2,559 double-coded 2021 tweets with both research assistants' labels. Filtering to relevance_agreement_check == TRUE leaves 2,403 tweets (93.9% agreement), the exact gold set the paper reconstructs; 1,111 are relevant and 1,292 irrelevant.

Our run: model jev-1.13.0 through Cloudflare Workers AI, not the OpenRouter route the paper used; relevance question only, instruction and option text as printed in the paper's Appendix A with straight quotes. Two calls hit a gateway rate limit (code 2018) and were retried once successfully.

The accuracy reproduces within sampling error: the paper's 89.0% sits inside our interval. Calibration on this task is good. In the 0.9 to 1.0 probability bin, 342 answers average 0.984 stated and 0.962 observed; the 0.6 bin is under-confident (0.655 stated, 0.767 observed). Contrast the Ziems suite in arXiv:2609.24574, where only 73.6% of Jev's labels at confidence 0.9 or above were right when pooled across 15 tasks. A binary relevance task with a long, explicit codebook is close to JEV's best case, which our earlier note found was concentrated in a few constructs.

What should an annotation team budget from these two papers?

Price per decision, not per million tokens. JEV's list price is fixed, but the cost per label on our one prompt ran from $0.0165 to $0.0348 depending only on how many questions shared the text. Count your instruction length and your questions per item before comparing vendors. A 2,000-token codebook asked once per item costs $0.084 per 1,000 items on JEV with no discount.

Bundle every question you need about an item into one JEV call. If the schema has five fields per document, a single request with five questions bills the document once. Our sample showed no accuracy cost from bundling (49 of 50 identical answers). TypeSafe documents that questions are judged in isolation, and the paper bounds any bundling effect at about the size of replay noise.

If latency is not a constraint, compare against batch. OpenAI's Batch rate halves Luna's cost, and on the Leiden tasks that puts Luna at or below JEV on every task. JEV's real advantages in that paper are speed (0.09 to 0.27 seconds per decision against 0.75 to 1.04 for Luna) and a probability for every option without logprob parsing, which matters most where Luna returns near-certain probabilities: 74% of Luna's party-comparison probabilities fall below 0.01 or above 0.99, against 18% for JEV.

The money is in the human half. At these prices the full machine pass is cents per 10,000 items under any route in either paper. Routing even 10% of items to a human reviewer at $0.05 each is $5 per 1,000 items, about 300 times the $0.017 machine bill for the same 1,000 items. Our judgement: choose the model by which items it lets you send to people with confidence, using a gold sample per construct, and treat the machine price as noise.

Validate per construct. On this tweet task JEV's confident labels are 97% right; on TalkLife empathy in the earlier audit they were 38% right at 78% coverage. The same model and the same threshold behave differently by task. See also reproducibility is not construct validity.

Check it yourself

The cost reconciliation needs only the two PDFs. The accuracy rerun needs one 0.7 MB file from Dataverse and JEV access (our 500 calls cost $0.0174).

# 1. Gold labels (Gilardi et al. 2023, Harvard Dataverse doi:10.7910/DVN/PQYF6M, CC0)
#    file id 7194128 = training_data_relevance_final.tab
#    fields: status_id, Date, text, relevant_paula, relevant_fabio, relevance_agreement_check, relevant_ra
curl -sL -o rel.csv "https://dataverse.harvard.edu/api/access/datafile/7194128?format=original"
python3 - <<'PY'
import csv, json, random
r = [x for x in csv.DictReader(open('rel.csv', encoding='utf-8')) if x['relevance_agreement_check'] == 'TRUE']
print(len(r))                       # 2403
random.seed(2610); s = random.sample(r, 500)
with open('sample.jsonl', 'w') as f:
    for x in s: f.write(json.dumps({'gold': 'relevant' if x['relevant_ra'] == '1' else 'irrelevant', 'text': x['text']}) + '\n')
PY

# 2. JEV: one Choice question, instruction and options from arXiv:2610.06625 Appendix A (Gilardi relevance)
#    request body: {"state": "<tweet>", "questions": {"relevance": {"type": "choice",
#      "instructions": "<Appendix A text>", "criteria": {"relevant": "...", "irrelevant": "..."}}}}
jevq -f sample.jsonl -k text "$(cat questions.json)" -o jev_out.jsonl
#    result here: 500 answers, accuracy 0.898, mean usage.input_tokens 827.4, $0.0348 per 1,000
#    add the problem and solution frame questions to the same request: 1,179 tokens per request, 393 per decision

# 3. Paper's own bill: Appendix C, 12,156 requests for $0.61
python3 -c "print(0.61/12156/0.042e-6, 0.61/12156/3*1000)"   # 1194.8 tokens/request, $0.0167 per 1,000 decisions

# 4. Ratios: arXiv:2610.06625 Table 2 (Luna Std / JEV, Luna Batch / JEV) and arXiv:2609.24574 Table 3
python3 -c "
import statistics as s
j=[.017,.024,.027,.020,.018,.012,.014]; b=[.017,.012,.018,.012,.008,.012,.009]; d=[.034,.025,.035,.024,.016,.024,.019]
print(s.median([x/y for x,y in zip(d,j)]), s.median([x/y for x,y in zip(b,j)]), .122/.027, .043/.027)"
# 1.296 0.643 4.52 1.59

Release check, 6 October 2026: arXiv:2610.06625 refers to "the replication archive" and to files such as data/batch_inputs/lemens_prompts_used.json, but gives no URL. GitHub repository and user searches (JEV with DiGiuseppe or Denney; the file name) and a Harvard Dataverse search returned no matching archive; the arXiv abstract page lists no code or data link. The underlying benchmark data for Gilardi et al. is public as above. Prices in Table 2 for Luna are OpenAI's posted rates applied by the authors; the paper prints neither the per-token Luna rates nor per-task token counts, so the Luna side cannot be recomputed from the PDF.

What would prove this wrong

Our claim is that the two papers' cost conclusions differ because of comparator choice, batch pricing and the questions-per-request divisor, and not because JEV's price or token accounting changed. If the Leiden archive, once released, shows JEV input tokens per Gilardi request far from 1,179 to 1,195, or shows the Table 2 Gilardi figure was computed per request rather than per decision, our bundling explanation is wrong.

A dated prediction: by 31 March 2027, the next independent paper that prices JEV against a small commercial LLM at batch rates, on single-question classification with instructions longer than 500 tokens, will find the LLM cheaper or within 1.25x on most tasks, unless TypeSafe has by then published a batch or cached-input rate. If such a paper finds JEV more than twice as cheap on that setup, the instruction-billing mechanism we describe is not the binding cost driver.

A second, smaller test: rerunning our script on all 2,403 agreed tweets with jev-1.13.0 should land between 88% and 91% accuracy. A result outside that band would mean either the Cloudflare and OpenRouter routes serve different behaviour or our 500-tweet sample was unrepresentative.

Sources

1. DiGiuseppe, M., Denney, S. (Leiden University). JEV versus LLMs: Accuracy, Cost and Calibration on Seven Political Science Replications. arXiv:2610.06625v1, submitted 5 October 2026. Tables 2, 4 and 9; Appendices A, C, E and H.
2. Ibrahim, H., Zaki, Y. (New York University Abu Dhabi). Evaluating Decision Models for Text Annotation in Computational Social Science. arXiv:2609.24574v1, 21 September 2026. Tables 3 and 7.
3. Gilardi, F., Alizadeh, M., Kubli, M. Replication Data for: ChatGPT outperforms crowd-workers for text-annotation tasks. Harvard Dataverse, CC0, released 19 July 2023. File training_data_relevance_final.tab. Accessed 6 October 2026.
4. TypeSafe. JEV models page (price $0.042 per Mtok input, output free) and Parallel questions cookbook (13 questions in one call, 12.2x cheaper). Accessed 6 October 2026.
5. BLOMEGA. The first decision model labels 44x cheaper and 11.6 F1 points worse. 23 September 2026.
6. BLOMEGA. Reproducibility is not construct validity.
7. BLOMEGA. JEV rerun on 500 Gilardi tweets and bundled-question token measurement, jev-1.13.0, 6 October 2026.

FAQ

Is TypeSafe's JEV cheaper than GPT-6 Luna for text annotation?

Not at batch prices. arXiv:2610.06625 (Leiden University, 5 October 2026) finds GPT-6 Luna at OpenAI's Batch rate costs 0.44 to 1.0 times JEV per decision on seven political science tasks, a median 0.64 by BLOMEGA's count. At Luna's Standard rate JEV is cheaper on six of seven tasks, with Luna at a median 1.30 times JEV.

Why did an earlier paper find JEV 44 times cheaper?

arXiv:2609.24574 divided by the most accurate of 19 LLMs on each task at real-time prices, often a frontier model, with a verbalized confidence line on every prompt. Against GPT-5.6 Luna in that paper the multiple was 4.5x and against GPT-OSS 120B 1.6x. The Leiden paper compared against the cheapest current OpenAI model at the batch rate, which is half the standard rate.

How can I reduce JEV's cost per label?

Put every question about an item in one request. JEV bills the text plus every question's instructions as input, so the text is billed once when questions share it. On the Gilardi relevance prompt BLOMEGA measured $0.0348 per 1,000 decisions asked alone and $0.0165 with three questions bundled, with 49 of 50 answers unchanged.