Verdict-only training recovers ContractNLI evidence at span F1 0.564, but 30 seconds of TF-IDF on the span labels scores 0.709

What arXiv:2610.06962 claims, and on what data

Nishanth Nayakanti, Prasang Gupta, Ashutosh Bilthare and Kevin Paul (PricewaterhouseCoopers, U.S.) posted Verdicts Without Annotated Evidence: Rejection Sampling or Label-Only Post-Training for Evidence Recovery? as arXiv:2610.06962v1 (cs.CL, 16 pages) on 3 October 2026. It is the only version as of 7 October 2026.

The setup mirrors a common operational record: a reviewer keeps the verdict and never marks the passages behind it. ContractNLI (Koreeda and Manning, Findings of EMNLP 2021) supplies that shape: 607 non-disclosure agreements, 17 fixed hypotheses per contract, a three-way verdict per (contract, hypothesis) row, and a separate layer of evidence spans. The authors hide the span layer during training and use it only for scoring.

Two methods post-train the same 4-bit Qwen3.5-9B with QLoRA for one epoch. Label-only SFT trains on the bare verdict, a 71-character target, in 2.63 GPU-hours on one L40S. Rejection-sampling SFT (RSFT) generates 75,334 candidate traces from the student and a Gemma-4-31B sampler, keeps those whose verdict matches the record, ranks survivors by a string-grounding score, and trains on one trace per row: about 9.7 GPU-hours end to end. The headline results (Tables 6, 10, 12) are label-only accuracy 0.896 and span F1 0.564; RSFT 0.797 and 0.556; untrained 0.747 and 0.493. The paper also argues that verdict accuracy and evidence quality are "only weakly related" across six systems (Section 6, Pearson +0.12).

What the paper does not do is score any baseline on the evidence axis that skips the LLM, either with or without span labels. Its only span-supervised comparison, Span NLI BERT at 0.892, is on verdict accuracy. That gap is what we filled.

How far do no-model and cheap baselines get on the paper's own metric?

Everything below is on the canonical ContractNLI test split: 123 contracts, 2,091 rows, of which 1,188 carry gold Entailment or Contradiction and 2,404 gold evidence spans. Our counts reproduce the paper's: majority-class accuracy 0.463, hypothesis lookup 0.659, 559,215 binary span decisions in train, and 2,404 = tp + fn for every system in the paper's Section 5.

Paper rows from arXiv:2610.06962v1, Tables 10 and 12 (n/r: precision and recall not reported for the reference systems). BLOMEGA rows computed on 7 October 2026 from the ContractNLI release (contract-nli.zip, sha256 e03fc77b...6287b), pooled tp/fp/fn over the 1,188 gold E/C test rows, scored regardless of predicted verdict as in the paper's Section 2.4.

Two results sit side by side. The first supports the paper: a model that never saw an evidence label beats every no-label heuristic we tried by a wide margin. Picking the spans that share the most content words with the hypothesis peaks at 0.331 (top 2), and marking everything scores 0.047 because the test contracts average 81.8 candidate spans each. The untrained 9B student at 0.493 is already 0.16 above the best heuristic, and the paper's 0.071 fine-tuning gain is on top of that.

The second result reframes the paper's premise. A per-hypothesis TF-IDF (unigrams and bigrams) logistic regression, trained only on the train split's E/C rows and their span labels, reaches 0.709 at a threshold of 0.8 chosen on the dev split (dev F1 0.679). It reads no hypothesis text, no verdict and no neighbouring context; it scores each span on its own words. Its precision 0.736 is lower than the LLMs' 0.83 to 0.88, but its recall 0.683 is far above their 0.35 to 0.42. Forced to cite one span per row, it keeps the LLMs' precision (0.882, identical to label-only SFT) and still reaches 0.584.

On verdicts, the cheap route is weaker. A per-hypothesis TF-IDF logistic regression on the whole contract text, trained on verdicts only, scores 0.753 accuracy, level with the untrained student under the verdict-only prompt (0.753) and well below label-only SFT at 0.896. The LLM fine-tune earns its cost on the verdict axis; on the evidence axis, the span labels are worth more than the model.

Why a bag of words wins on evidence: 17 fixed hypotheses and a metric that only scores E/C rows

Three properties of the task explain the gap. The hypotheses are fixed: all 17 repeat across every contract, so a separate classifier per hypothesis learns the vocabulary of, say, a return-or-destroy clause from about 260 annotated rows on average (4,371 train E/C rows over 17 hypotheses) and never has to generalise to a new question. The authors make this repetition the point of their setup ("a fixed set of questions applied to every document"); it is also exactly the condition under which a small supervised model is hard to beat.

The evidence is redundant and the metric counts every gold span. ContractNLI's README says evidence spans "need to be comprehensively identified where they are redundant". The test set has 2,404 gold spans on 1,188 rows, about 2.0 per row, while the LLMs emit 1.11 to 1.19 quotations per citing row (paper Section 5). Their precision of 0.83 to 0.88 is high; their recall cannot climb much above 0.42 when they quote about one passage per row. The classifier trades precision for recall and wins on F1 by 0.145.

NotMentioned rows are outside the evidence score. Section 2.4 scores the 1,188 E/C rows "regardless of the model's predicted label", and the 903 NotMentioned rows contribute nothing. So a system that cites on every row pays no price for it, which is how our classifier is scored and is also how the LLMs behave (they cite on 92.5% to 96.5% of scored rows). This is a property of the protocol, not a flaw we exploit: the paper's own reading that evidence F1 is "mediated by the verdict" follows from it, since a wrong NotMentioned on an E/C row zeros that row's recall.

Our matching is no looser than the paper's. We predict whole candidate spans and count a hit only on an exact span index match. The paper's three-tier ladder also gives credit to quotations that are whitespace-normalised or 60% contained in one stretch, which can only add true positives for the LLMs.

How many evidence-annotated contracts does it take to match verdict-only training?

The paper motivates verdict-only training by cost: span labels are "far more expensive than retaining the verdict alone". We measured how much span annotation the cheap route needs. Holding the classifier to one citation per row, so it matches the LLMs' citation habit and precision, we trained on span labels from random subsets of the 423 training contracts, three draws each.

Five contracts give 0.390 (draws 0.440, 0.448 and 0.281). Ten give 0.475. Twenty-five give 0.531, already above the untrained student's 0.493. Fifty give 0.551. One hundred give 0.565 (draws 0.566, 0.566, 0.562), level with label-only SFT at 0.564 and above RSFT at 0.556. All 423 give 0.587. With the dev-tuned threshold instead of one span per row, the full training set reaches the 0.709 above.

Annotating 100 NDAs against 17 hypotheses is real work, and our curve says nothing about what that costs in reviewer hours; the paper does not report a cost per span either. The comparison it supports is narrower: on a fixed-question task, the evidence layer for about a quarter of the training contracts buys as much span F1 as one epoch of LLM fine-tuning on 6,477 training verdicts, and the full layer buys more than any model in the paper.

Is "weakly related" across six systems a measurable claim?

From Table 12 we get Pearson +0.113 between accuracy and span F1 (the paper reports +0.12, the difference is rounding in the table), Spearman -0.029 and Kendall 0.000. For accuracy against verbatim rate we get Pearson +0.199, matching the paper's +0.20, and Spearman +0.086. On the four off-the-shelf systems, accuracy against verbatim rate is -0.273, matching the paper's -0.27.

The arithmetic checks out; the inference is thin, which the authors partly concede by saying the coefficients "summarize this set rather than a population". With six points, |r| must exceed 0.811 for p below 0.05; the p-value for +0.113 is 0.83. Dropping one system at a time moves Pearson between -0.249 (drop untrained) and +0.274 (drop Gemma-4-31B). These data cannot separate "unrelated" from a moderate positive relation.

The rank reversal the paper leans on is solid as a pair of observations: gpt-5.2 has the top off-the-shelf accuracy (0.808) and ties the untrained student on span F1 (0.493), and the two evidence leaders, Gemma-4-31B and sonnet-4.6, are below RSFT on accuracy. Our baselines add a sharper version. The TF-IDF evidence classifier reads no verdict at all and leads on span F1; the TF-IDF verdict classifier sits at 0.753 accuracy with no evidence output. The two axes come apart because different inputs drive them, which a six-point correlation cannot show and a component baseline can.

Should you pay for evidence spans or keep only verdicts?

If the question set is fixed, price the span layer against a small supervised model, not against nothing. The paper's comparison is verdict-only LLM versus untrained LLM. On ContractNLI the relevant alternative for a data buyer is evidence labels plus a cheap per-question classifier, which beat every verdict-only system here at about 30 seconds of CPU. Our judgement: for checklist-style review (filings against a checklist, claims against a policy, the authors' own examples), annotating evidence on 100 to 400 documents is likely the better spend if reviewable citations are the product.

If questions change often, verdict-only fine-tuning is the reasonable default. A per-hypothesis classifier has nothing to say about an 18th hypothesis, while a fine-tuned LLM can at least be prompted with it. The paper does not test unseen hypotheses, and neither did we, so the size of that advantage is not measured.

Spend LLM effort on the verdict, labels on the evidence. Label-only SFT at 0.896 accuracy beats a bag-of-words verdict model (0.753) by 0.143; the TF-IDF span model beats it on evidence by 0.145. A hybrid, the LLM's verdict gating a span classifier's citations, is the obvious next measurement. On the paper's metric the gate would not change span F1, because NotMentioned rows are unscored; on a deployment metric that penalises citing on NotMentioned rows, it would matter.

Match the citation count to the annotation guideline. ContractNLI marks redundant evidence comprehensively, about 2.0 gold spans per scored row, and the LLMs quote about 1.1 to 1.2 passages. If your reviewers need every supporting clause, the LLM recall of 0.41 to 0.42 is the binding constraint, and a precision of 0.88 hides it. If they need one verifiable quote, the paper's verbatim rate (72.9% for label-only) is the better metric than span F1.

When selecting a model, measure both axes on your data. We agree with the paper's practical advice even though the correlation does not prove it: gpt-5.2 and the untrained 9B tie on span F1 at accuracies 0.808 and 0.747. One evaluation set of 100 contracts with marked evidence would settle the choice; this is a small, one-time annotation cost compared with labelling a training set.

Check it yourself

ContractNLI is public under CC BY 4.0 (Hitachi America terms file plus the Creative Commons licence in the zip). The download is 65 MB including the raw PDFs; only train.json, dev.json and test.json are used. Each document has text, spans (character offsets of candidate spans) and annotation_sets[0].annotations[hyp_id] with choice and spans (indices into spans).

mkdir -p cnli/data cnli/scripts && cd cnli
curl -sL -o data/contract-nli.zip https://stanfordnlp.github.io/contract-nli/resources/contract-nli.zip
shasum -a 256 data/contract-nli.zip   # e03fc77bbf8b53e2976a250e81d8a294bc3d5e5fb014521e477dee9340d6287b
(cd data && unzip -q contract-nli.zip)
python3 -m venv venv && ./venv/bin/pip -q install scikit-learn scipy numpy
# scripts/baselines.py: majority + hypothesis lookup accuracy, every-span and word-overlap
#   evidence baselines, per-hypothesis TF-IDF(1-2 gram, min_df 2, sublinear) + LogisticRegression
#   (C 4, class_weight balanced) on span text of train E/C rows; threshold picked on dev
#   from {0.5 ... 0.99}; pooled tp/fp/fn over the 1,188 gold E/C test rows by exact span index
./venv/bin/python -I scripts/baselines.py data/contract-nli
#  acc_majority 0.4629  acc_hyp_lookup 0.6595  n_ec_rows 1188  n_gold_spans 2404
#  ev_all_spans F1 0.0465   ev_lexical_top2 F1 0.3314
#  ev_tfidf_lr_spanlabels  tp 1643 fp 589 fn 761  P 0.7361 R 0.6834 F1 0.7088  (threshold 0.8)
#  ev_tfidf_lr_top1 F1 0.5835   acc_tfidf_lr_verdict_only 0.7532
./venv/bin/python -I scripts/curve.py data/contract-nli     # learning curve, min_df 1, seeds 0-2
./venv/bin/python -I scripts/corr.py                        # Table 12 correlations

Run time was 38 seconds for baselines.py and 42 seconds for curve.py on an Apple-silicon laptop (Python 3.14.7, scikit-learn 1.9.1, scipy 1.18.1, numpy 2.5.3). The curve script uses min_df=1 so that 5-contract subsets have a vocabulary; that is why its full-data point is 0.587 against 0.584 in the main script. The paper's pooled counts can be checked directly from Section 5: 844/(844+173) = 0.830 precision and 844/2,404 = 0.351 recall for the untrained student, and likewise 0.882 and 0.415 for label-only.

Release check, 7 October 2026: the paper gives no code or model URL. GitHub repository searches for "verdicts without annotated evidence", "contractnli rejection sampling" and the run name cnli-rsft return 0 results, and a Hugging Face model search for cnli-rsft returns none. The paper mentions that "our code carries a pre-registration", so code exists but is not public. The LLM rows above are therefore not independently reproducible today; our rows are.

What would prove this wrong

Our claim is that on ContractNLI, under the paper's pooled span F1, evidence-span labels plus a bag-of-words classifier outperform verdict-only LLM post-training, and that the gap comes from recall on redundant evidence.

Dated prediction: by 30 June 2027, no system trained without ContractNLI evidence labels will report span F1 above 0.709 on the 1,188 gold E/C test rows under this paper's pooled metric with string matching. That includes the authors' promised second seeds, stacked RSFT plus label-only runs, and preference finishing (Section 11). If one does, our "labels beat the model" reading is wrong for this task.

A second test: if the authors add the semantic evidence credit rule they plan (embedding or entailment matching) and their systems' recall rises above 0.60 without their precision falling below 0.80, the recall ceiling we attribute to one-quote-per-row citation was a matching artifact, and the LLM route would close most of the gap.

Sources

1. Nayakanti, N., Gupta, P., Bilthare, A., Paul, K. (PricewaterhouseCoopers, U.S.). Verdicts Without Annotated Evidence: Rejection Sampling or Label-Only Post-Training for Evidence Recovery? arXiv:2610.06962v1, 3 October 2026. Sections 2.1, 2.4, 4, 5, 6, 8, 11; Tables 6, 10, 12.
2. Koreeda, Y., Manning, C. D. ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts. Findings of EMNLP 2021, arXiv:2110.01799.
3. Stanford NLP. ContractNLI dataset page and contract-nli.zip (CC BY 4.0, Hitachi America, Ltd.). Downloaded 7 October 2026.
4. Stanford NLP. contract-nli-bert (Span NLI BERT, Apache-2.0, last push 10 March 2023). Accessed 7 October 2026.
5. BLOMEGA. A blind 50/50 split of the annotation budget missed the near-optimal region up to 80% of the time.
6. BLOMEGA. Two experts, 5.6 hours of rationale labels: an LLM codebook hits 64.9 percent.

FAQ

Can a model trained only on verdicts find the evidence in ContractNLI?

Partly. arXiv:2610.06962 reports span F1 0.564 for Qwen3.5-9B fine-tuned on bare verdicts and 0.556 for rejection-sampling fine-tuning, against 0.493 untrained, on the 1,188 test rows with gold evidence. BLOMEGA reran no-label baselines on the same rows and metric: marking every span scores 0.047 and the best hypothesis-overlap heuristic 0.331, so the LLM gain is real and not a metric artifact.

Is it worth paying for evidence-span labels on a fixed-question review task?

On ContractNLI, yes. A per-hypothesis TF-IDF logistic regression trained on the public train-split span labels, about 30 seconds on a laptop CPU, reaches span F1 0.709 on the test split, above every system in arXiv:2610.06962 (best 0.625, Gemma-4-31B). Restricted to one span per row it reaches 0.584, and with span labels from 100 training contracts it reaches 0.565, level with the verdict-only LLM. Computed by BLOMEGA on 7 October 2026.

Are verdict accuracy and evidence quality correlated across models?

Not measurably with six systems. From Table 12 of arXiv:2610.06962, BLOMEGA gets Pearson +0.11 between accuracy and span F1 (the paper says +0.12), Spearman -0.03 and Kendall 0.00. Dropping one system at a time moves Pearson from -0.25 to +0.27, and with six points a correlation needs to exceed 0.81 in absolute value to reach p below 0.05. The data are consistent with no relation and with a moderate one.