Goldsmith cuts gold loss 93% on its own 15 examples; on held-out OpenNER the optimized definition adds 0.48 F1 points

What Sun Yat-sen University posted on 7 October 2026

Yihan Li and Hanyi Zhang (equal contribution), with Xiaoxi Jiang and Man Guo, from Sun Yat-sen University and South China University of Technology, posted Goldsmith as arXiv:2610.09489v1 (cs.CL) on 7 October 2026. The comment line says 20 pages, 11 tables, accepted to the EMNLP 2026 main conference. The paper describes Goldsmith as gold-loss-guided definition optimization inside an agentic annotation pipeline, and that is the term we use below.

The method treats the operational definition (the prose that says what to label and where boundaries fall) as the only trainable object. The label set, JSON schema, parser, repair, retrieval and review policies stay fixed outside it. Each round, Goldsmith runs the current definition on all 15 gold items, scores a structured loss (boundary misses weighted 1.2, extras 1.0, boundary 0.75 and type 0.25, parse failure 1.25, all times a reporting scale of 100, Appendix Table 10), sends the highest-loss failures to an LLM critic and editor, and keeps a candidate only when gold loss decreases. Settings per Appendix B.5 and Table 7: GPT-5.5, 5 candidates per round, up to 30 rounds, patience 5, optimizer temperature 0.3, annotation temperature 0.0, accessed May 2026.

Three tasks: OpenNER typed spans (PER, ORG, LOC; a 52-language, 2,600-item slice, seed 9), CrossRE given-pair relation classification (17 relations plus no_relation; 39,764 directed pairs from 1,200 sentences, 35,275 of them no_relation) and a fixed-trigger PHEE V2 event-argument setting. The 15 gold items are chosen by a coverage-first selector (Appendix A.2 and A.4).

Which reported gains are on the 15 gold items, and which are held out?

Only the task-stage tables are held out. Every optimizer comparison, and both training traces, use the 15 items that also decide acceptance. The paper says so in Section 5.1 ("before any held-out task evaluation"); the abstract does not.

All paper numbers from arXiv:2610.09489v1, text extracted with pypdf on 9 October 2026. Deltas and percentages are BLOMEGA arithmetic on those numbers.

Two numbers frame the 0.0048. The paper's held-out subset is about 2,392 items with roughly 3,500 gold spans (our estimate, scaling 3,833 spans over 2,600 items). A plain binomial standard error on micro-F1 at that size is about 0.0075, so the definition-only gain sits inside one standard error. That is a rough, unpaired figure and a paired test would be tighter, but the paper reports no interval at all. Second, Table 8 runs the same optimized definition with 5 to 20 retrieved references on a 500-item sample; micro-F1 wanders between 0.7727 and 0.8026 (sd 0.0078), and moving from k = 14 to k = 15 alone costs 0.0299. Our judgement: a 0.0048 difference is below the noise the paper itself displays.

The Table 5 ablation reads very differently. Swapping the initial definition into the full reviewed pipeline drops F1 from 0.8174 to 0.6661, a 0.1513 effect, 31.5 times the zero-shot gap. But 0.6661 is 0.0574 below the initial definition with no retrieval and no review (0.7235, Table 2). Retrieval and gold-corrected review making the initial definition worse is possible, and the paper does not explain it. Until it does, the zero-shot pair is the cleaner held-out estimate of what the definition is worth, because it changes one thing.

Why accept-if-loss-drops inflates a 15-item score

The acceptance rule is the right instinct: an objective, executable loss is better than an LLM grading its own edits. The catch is the sample. When up to 150 candidate definitions are each scored on the same 15 items and the best is kept, the kept score is a maximum, and a maximum of noisy draws is biased upward even when no candidate is truly better. Each candidate's pass pattern on 15 items mixes real quality with idiosyncratic fit to those items; selection rewards both.

The paper's own Table 9 shows the effect on a baseline. From round 1 to 30, OPRO's per-round gold loss averages 18.69 with sd 1.65 and no visible trend. Its reported best, 14.59 at round 16, is 2.48 sd below that mean. The expected minimum of 30 independent normal draws is 2.04 sd below the mean (our simulation). So OPRO's "best" is about what picking the lowest of 30 equivalent draws would give. Our judgement: OPRO's Table 3 row reflects its selection, and the same mechanism applies to all five rows.

The same loss scale converts the Table 3 gaps into items. A gold item that is fully wrong costs at least 6.67 loss points (100 over 15) before penalties, and CrossRE's trace in Table 11 moves in exact multiples of 6.67. Adjacent optimizers in Table 3 differ by 1 or 2 passing items: 7, 9, 10, 12, 14.

How many gold items does selection add when nothing improves? 1 to 5

We wrote a 30-line simulation (code in the check section). Fifteen gold items; item i has a pass probability p_i that every candidate definition shares, so all candidates have the same expected held-out quality and the true improvement is zero. The p_i are drawn from a Beta distribution with mean 8/15, CrossRE's reported starting pass count, and intraclass correlation rho, which sets how similar two candidates' pass patterns are (rho 0: independent coin flips; rho near 1: every candidate passes the same items). The initial definition is scored once; each new candidate is accepted only if it passes strictly more items. 20,000 runs per cell, seed 20261009.

With 40 candidates, CrossRE's reported budget, the apparent gain is 4.03, 3.39, 2.84, 2.16 and 1.07 items at rho 0, 0.3, 0.5, 0.7 and 0.9. With 150 candidates, OpenNER's maximum, it is 4.85, 4.10, 3.46, 2.62 and 1.33. In pass-rate terms that is 7 to 32 points, from selection alone. The 10th to 90th percentile of the gain at rho 0.5 runs from 1 to 5 items, so two optimizers with identical true quality would often land 2 or more items apart. That is the size of every adjacent gap in Table 3.

What the null does not produce is a perfect score. Reaching 15 of 15 happens in at most 0.3% of null runs at 40 candidates and at most 1.2% at 150, so CrossRE's 8 to 15 jump is unlikely to be pure selection. Our reading: some of the gold-set gain is real fit to the task, some is selection, and with one run per optimizer and no held-out row for the baselines the paper cannot say how much is which. We do not know the true rho for GPT-5.5 prompt variants; LLM prompt variants tend to agree on easy items, which argues for the upper half of the range and the smaller inflation.

Do the traces match the stated protocol? Five places they do not line up

None of these changes the direction of the held-out OpenNER result. They do matter for anyone trying to reproduce the optimizer table: without the code, a reader cannot tell whether ties were accepted, whether the patience rule ran, or which definition text was deployed. The Goldsmith final definition's clauses about "explicit named or vocative references to a deity", harbor installations and embarkation points read, in our judgement, like patches for individual gold items, the kind of edit a 15-item objective rewards and a 2,392-item test barely notices.

What should an annotation lead take from Goldsmith?

Split your gold set before optimizing anything. If experts can give 15 examples, give the optimizer 10 and hold 5, or better, ask for 30 and hold 15. Report the held-out pass count next to the training one for every candidate method. On 15 items, a two-item lead (13.3 points) is within what our simulation produces between equivalent methods.

Budget review before you budget definition editing. In Goldsmith's own OpenNER numbers, 193 gold-corrected review items moved held-out F1 by 0.0776, 16 times the definition-only gain. If the money for expert time is fixed, the paper's data say it goes further on reviewing the lowest-scoring 7% of items (193 of 2,600) than on polishing prose. That ratio may differ on your task; measure it with the same zero-shot pair the paper ran.

Treat the optimized definition as a guideline draft, not a model. The part we think is durable is the artifact: a written, inspectable boundary policy with a trace of which gold failures shaped it. Have the experts read it. Clauses that name single examples (sacred figures, harbors) are where to start editing, because they encode the 15 items rather than the concept. This is the same pattern we found when an LLM-drafted codebook beat a six-month expert version: the gain came with a held-out test.

Keep the loss, add a stopping rule that penalizes length. The paper notes it has no prompt-length penalty (App. B.6), and the accepted OpenNER definition grew from 433 to 1,444 tokens (3.3 times) over 30 rounds. A length penalty or a minimum improvement of one full item before acceptance cuts both cost and overfit.

Read relation-task accuracy against the majority class. With 88.7% no_relation pairs, any CrossRE-style system should be reported against all-no_relation accuracy and positive-class F1. Goldsmith's positive F1 of 0.5439 against the zero-shot LLM's 0.2470 is the real CrossRE result; it also includes retrieval and review, so the definition's share is unknown.

Check it yourself

Everything here comes from the PDF plus two short scripts. No model calls, no downloads beyond 0.9 MB.

# paper text (pdftotext missing on our Mac, so pypdf)
curl -sL https://arxiv.org/pdf/2610.09489 -o p.pdf      # 914,577 bytes, v1
python3 -m venv .venv && .venv/bin/pip -q install pypdf numpy
.venv/bin/python -c "import pypdf;r=pypdf.PdfReader('p.pdf');open('p.txt','w').write('\n'.join(p.extract_text() for p in r.pages))"
# Table 2 rows: 'Initial-prompt zero-shot 0 gold ... 0.7235', 'Goldsmith zero-shot 15 gold ... 0.7283'
# Table 9: per-round losses; Table 11: CrossRE trace; App. B.2: 2,600 / 15 / 193 split

# OPRO selection check (Table 9, rounds 1-30)
python3 -c "import numpy as np;o=np.array([19.5303,23.0152,17.1970,20.6667,19.1515,19.5303,17.3485,17.5303,16.9242,19.5303,18.9242,21.1212,19.3788,18.9242,18.0606,14.5909,19.5303,19.0152,19.5303,20.6667,16.2121,19.3485,18.9242,17.3485,17.7273,17.8788,19.5303,16.5455,19.1515,17.7273]);print(o.mean(),o.std(ddof=1),(o.mean()-o.min())/o.std(ddof=1))"
# 18.6854 1.6506 2.48

# selection-effect Monte Carlo (mc.py)
import numpy as np
rng=np.random.default_rng(20261009); n=15; sims=20000
def run(p0,rho,B):
    p=np.full((sims,n),p0) if rho==0 else rng.beta(p0*(1-rho)/rho,(1-p0)*(1-rho)/rho,size=(sims,n))
    init=(rng.random((sims,n))<p).sum(1); cur=init.copy()
    for _ in range(B): cur=np.maximum(cur,(rng.random((sims,n))<p).sum(1))
    return (cur-init).mean(), np.mean(cur==15)
for B in [40,150]:
    for rho in [0,.3,.5,.7,.9]: print(B,rho,run(8/15,rho,B))
# 40: 4.03 3.39 2.84 2.16 1.07   150: 4.85 4.10 3.46 2.62 1.33 (items)

# release check, 9 October 2026
#  PDF link annotations: only arxiv.org and citation DOIs, no code or data URL
curl -s "https://api.github.com/search/repositories?q=goldsmith+gold+loss"           # total_count 0
curl -s -o /dev/null -w "%{http_code}" https://huggingface.co/api/papers/2610.09489  # 404
for r in mainlp/CrossRE ZhaoyueSun/PHEE bltlab/open-ner; do curl -s https://api.github.com/repos/$r | grep -E '"spdx_id"|"pushed_at"'; done
# CrossRE GPL-3.0 (pushed 2024-08-20); PHEE MIT (2025-02-16); bltlab/open-ner no licence detected (2026-05-14)
curl -s "https://huggingface.co/api/datasets?search=open-ner-standardized"           # tag license:cc-by-4.0

Fields to look at: Table 2 columns P, R, F1, Macro, Exact; Table 9 columns Rewrite loss, OPRO loss, Goldsmith loss, Goldsmith best, Def. tok.; Table 10 parameters (reporting scale 100, alpha_miss 1.2, lambda_parse 1.25). The binomial standard error (0.0075) uses an estimated 3,526 held-out spans and is a rough unpaired figure. The OpenNER source collection carries per-corpus licences; the Hugging Face tag covers the standardized release only, and the paper reminds users to follow each source's terms.

What would prove this wrong

Our claim: Goldsmith's optimizer ranking is a gold-set result inflated by selection, and the optimized definition alone adds little on held-out OpenNER. Two results would overturn it.

First, a dated prediction: by 30 June 2027, if the authors or anyone else release the five final OpenNER definitions and score each zero-shot on the 2,392-item held-out subset, the five micro-F1 scores will span less than 0.015, and Goldsmith's will not beat PromptBreeder's by more than 0.010. A spread above 0.03 with Goldsmith on top, repeated over three optimizer seeds, would show the Table 3 ranking carries to held-out data.

Second, if a rerun with a split gold set (optimize on 10, hold 5, repeated over seeds) shows Goldsmith's held-out pass count within one item of its training pass count, the selection inflation we simulated is small in practice for this method, and our upper-range estimates are wrong.

Sources

1. Li, Y., Zhang, H., Jiang, X., Guo, M. Goldsmith: Gold-Loss-Guided Definition Optimization (full title paraphrased). arXiv:2610.09489v1, 7 October 2026, Sun Yat-sen University and South China University of Technology. Sections 3 to 5, Tables 2 to 11, Appendices A, B, D. Accessed 9 October 2026.
2. Palen-Michel, C., et al. OpenNER 1.0, EMNLP 2025. Dataset: bltlab/open-ner-standardized (tag cc-by-4.0); code bltlab/open-ner (last push 14 May 2026). Accessed 9 October 2026.
3. Bassignana, E., Plank, B. CrossRE, Findings of EMNLP 2022. mainlp/CrossRE (GPL-3.0, last push 20 August 2024). Accessed 9 October 2026.
4. Sun, Z., Pergola, G., Wallace, B. C., He, Y. Pharmacovigilance event extraction with ChatGPT, EACL 2024. ZhaoyueSun/PHEE (MIT, last push 16 February 2025). Accessed 9 October 2026.
5. Yang, C., et al. Large Language Models as Optimizers (OPRO), ICLR 2024.
6. Zhou, Y., et al. Large Language Models Are Human-Level Prompt Engineers (APE), ICLR 2023.
7. Fernando, C., et al. Promptbreeder, ICML 2024.
8. Pryzant, R., et al. Automatic prompt optimization with "gradient descent" and beam search (ProTeGi), EMNLP 2023.
9. BLOMEGA. Two experts, 5.6 hours of rationale labels: an LLM codebook hits 64.9 percent.

FAQ

Is Goldsmith's win over OPRO, APE and PromptBreeder measured on held-out data?

No. Table 3 of arXiv:2610.09489 reports pass counts (7, 9, 10, 12 and 14 of 15) and gold loss on the same 15 calibration examples every optimizer used for acceptance. The paper states the table comes before any held-out evaluation, and it reports no held-out score for the OPRO, APE or PromptBreeder definitions. On a 15-item set, one example is 6.7 points of pass rate.

How much does the optimized definition alone improve held-out annotation?

On OpenNER, Table 2 gives initial-prompt zero-shot micro-F1 0.7235 and Goldsmith zero-shot (optimized definition, no retrieval, no review) 0.7283, a gain of 0.0048 on the held-out subset of about 2,392 items. Retrieval adds 0.0115 and score-routed review with 193 gold-corrected items adds 0.0776. Of the total 0.0939, the definition is 5.1%, retrieval 12.2%, review 82.6% (BLOMEGA arithmetic from the paper's table).

How much gold-set gain does accept-if-loss-drops produce with no real improvement?

In a BLOMEGA Monte Carlo with 15 gold items, a starting pass rate of 8 of 15 and candidates that are all equally good in expectation, greedy acceptance gains 1.1 to 4.0 items after 40 candidates and 1.3 to 4.9 items after 150, depending on how correlated candidates are. That is 7 to 32 points of pass rate from selection alone. It does not explain CrossRE reaching 15 of 15, which the null produces in 0.3% or fewer runs at CrossRE's 40-candidate budget.