PaperlayerView exact revision on arXiv

Source-linked AI summary

Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness

Yihan Li, Hanyi Zhang, Xiaoxi Jiang, Man Guo

arXiv:2610.09489v1cs.CL

TL;DR

Annotation projects can begin before experts have stable guidelines or enough labels for task-specific models. Goldsmith learns an operational definition from expert gold examples while an external harness handles annotation controls. Its optimized definition improves annotation across three structured tasks when paired with retrieval, routing, and human review.

  • Problem

    Many annotation projects begin before experts have a stable guideline or enough labels to train a task-specific model.

    Source: §1 · p. 1

  • Method

    Goldsmith accepts LLM-proposed operational-definition revisions only when they improve structured loss on expert gold examples, while an external harness manages annotation controls.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    The resulting definition improves OpenNER typed span labeling, CrossRE positive-relation classification, and PHEE V2 fixed-trigger event-argument extraction when combined with retrieval, judge routing, and human review.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Long-context LLM annotation depends on a harness that tests definition edits against gold loss, audits outputs, and routes uncertain cases to human review; it does not replace supervised PLM pipelines once large stable datasets exist.

    Source: §6 · p. 9

    Goldsmith is designed for tasks whose boundaries can be expressed through natural-language definitions and examples, and may be less suitable when private expert knowledge is absent from the gold examples or labels are highly ambiguous.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Many annotation projects begin before experts have a stable guideline or enough labels to train a task-specific model. We present Goldsmith, an agentic pipeline that turns a small gold set---expert-annotated calibration examples representing the intended task boundaries---into a reusable structured annotation definition. Goldsmith treats this definition as a trainable textual object. Candidate definitions are run on the same gold examples and scored with an executable structured loss, while the output schema, formatting, retrieval, repair, judging, and human review remain in an external harness. A large language model (LLM) editor converts the highest-loss failures into textual-gradient revisions, which are accepted only when the measured loss decreases. In prompt-optimization comparisons, Goldsmith improves over direct rewriting, OPRO, APE, and PromptBreeder under matched evaluation protocols. The resulting definition also improves downstream annotation when combined with retrieval, score-based routing, and human review across typed span, pair-level relation, and fixed-trigger event-argument tasks. These results show that scarce expert supervision can support both task-definition learning and scalable annotation.

1 Introduction

Annotation projects often start before experts have settled task boundaries or gathered enough labels for a task-specific model. Goldsmith uses expert examples to refine an operational definition while keeping the annotation harness separate from that trainable text.

1 Introduction

  • Experts may recognize an annotation phenomenon before they have a stable boundary policy, complete manual, or enough labels for a task-specific model.Parameter-efficient updates reduce training cost but not the cost of deciding what the task means.Source: §1 · p. 1
  • Carefully chosen gold examples can specify task boundaries more concretely than an initial prose guideline, while unselected or unvalidated examples can dilute expert signal.Long-context models can fit expert examples, failures, and revision history within the working context.Source: §1 · p. 1
  • Goldsmith accepts definition revisions only when they improve structured loss on expert gold examples.The LLM editor turns high-loss failures into textual-gradient edit directions.Source: §1 · p. 2
  • The harness keeps schema, formatting, retrieval, repair, judge routing, and human review outside the trainable definition.This makes the optimized definition usable for batch annotation.Source: §1 · p. 2
  • The evaluation spans typed multilingual NER, pair-level relation classification, and controlled event-argument extraction, against LLM-only, prompt-optimization, and PLM baselines.The tasks are OpenNER, CrossRE, and PHEE V2.Source: §1 · p. 2

2 Related Work

Prior work studies LLM-assisted annotation, human checks, structured-output validation, and prompt optimization. Goldsmith makes the operational definition the only trainable artifact and uses structured gold loss to accept candidate revisions.

2.1 LLM Annotation and Task Definition

  • LLM annotation and analyst-in-the-loop systems use human checks to validate generated labels, while structured outputs remain sensitive to guidelines, schemas, parsing, and repair (Wang et al., 2021; Gilardi et al., 2023; Zhang et al., 2023; Dai et al., 2023; Tan et al., 2024; Xu et al., 2024; Sainz et al., 2024; Huang et al., 2024).Long-context prompting can include examples and revision context, but demonstrations should be selected and validated (Brown et al., 2020; Abbas et al., 2024; Bertsch et al., 2025; Zou et al., 2025; Zhang et al., 2025b,a).Source: §2.1 · p. 2

2.2 Prompt Optimization

  • Unlike APE, PromptBreeder, OPRO, ProTeGi, and TextGrad, Goldsmith trains only the operational definition and accepts candidates by structured gold loss under a fixed annotation interface (Zhou et al., 2023; Fernando et al., 2024; Yang et al., 2024; Pryzant et al., 2023; Yuksekgonul et al., 2025).The cited methods generate or evolve instructions, optimize prompt-score histories, or use textual-gradient feedback.Source: §2.2 · pp. 2–3

3 Method

Goldsmith optimizes an operational definition on a fixed gold set, then applies it to unlabeled data through an external annotation harness. Structured gold loss evaluates candidate definitions against expert annotations, while the harness manages the output protocol and review workflow.

3 Method

  • Goldsmith trains and selects an operational definition on a fixed gold set, then applies it to an unlabeled corpus through retrieval, format checking, quality routing, and human review.Only the definition is optimized; the label inventory, schema, parser, format repair, retrieval, and review policies remain with the external harness.Source: §3 · p. 3

3.1 Gold Loss for Structured Annotation

  • Structured gold loss averages item-level errors after candidate definitions are run on gold examples and their parsed outputs are checked against expert annotations.For typed spans, the loss accounts for boundary and type errors; relation tasks use label disagreement, and joint span-relation tasks combine the corresponding losses.Source: §3.1 · pp. 3–4

3.2 Gold-Loss-Guided Definition Training

  • Goldsmith selects high-loss failures to guide LLM-proposed definition edits, reruns candidates through the same parser and schema, and accepts edits only when measured gold loss decreases.The stateful trace records accepted and rejected candidates and remaining failures; training stops when all gold items pass or the patience budget is exhausted.Source: §3.2 · pp. 4–5

3.3 Batch Annotation and Review Harness

  • For batch annotation, Goldsmith retrieves references and applies the optimized definition; parseable predictions can be routed by judge scores, with high-risk items prioritized for human review.Format-only failures first receive conversational repair, and reviewed annotations may become references for later annotation.Source: §3.3 · p. 5

4 Experiments

The experiments test Goldsmith across three structured annotation tasks, using matched prompt-optimization comparisons and task-stage baselines. They also specify shared evaluation subsets and task-specific metrics.

4.1 Tasks and Datasets

  • OpenNER evaluates multilingual typed spans, CrossRE classifies marked entity pairs, and PHEE V2 predicts argument spans and relations for a supplied event trigger.The experiments reserve 15 calibration examples where applicable and exclude calibration and reviewed items from the held-out evaluation subset.Source: §4.1 · p. 5

4.2 Prompt-Optimization Baselines

  • Prompt-optimization baselines include rewrite-only, OPRO, APE, PromptBreeder, and Goldsmith, with Goldsmith accepting edits only when gold loss decreases.The shared comparison setup uses the same initial description, gold set, candidate budget, gold loss, and acceptance rule; the supplied passages do not report comparative outcomes.Source: §4.2 · pp. 5–6

4.3 Task-Stage Baselines

  • The task-stage baselines compare optimized-definition workflows with zero-shot prompting and low-budget supervised PLMs across OpenNER, CrossRE, and PHEE V2.OpenNER includes review and no-review settings; CrossRE uses retrieved references and reviewed feedback, while PHEE V2 compares initial zero-shot prompting with score-routed review.Source: §4.3 · p. 6

4.4 Metrics

  • Metrics are task-specific: OpenNER reports typed precision, recall, F1, and exact match; CrossRE reports classification metrics; PHEE V2 reports argument and relation F1, trigger pass rate, and joint exact match.CrossRE macro-F1 excludes the dominant negative class.Source: §4.4 · p. 6

5 Results and Analysis

Goldsmith improves on competing optimizers under a shared gold-loss objective and supports downstream annotation across typed spans, relations, and event arguments. Ablations attribute the main OpenNER gain to definition optimization, with retrieval contributing additional stability.

5.1 Prompt Optimization Results

  • Goldsmith solved 14/15 OpenNER gold items at loss 1.8030, versus 12/15 for PromptBreeder, 10/15 APE, 9/15 OPRO, and 7/15 direct rewriting.All optimizers shared the initial definition, 15 gold examples, and candidate budget.Source: §5.1 · p. 7, Table 3 · p. 7

5.2 Task Results

  • Score-routed review reaches 0.8174 micro-F1, 0.8919 macro-F1, and 0.8140 typed exact match on OpenNER.The reviewed second pass evaluates on the same held-out subset as the first pass.Source: §5.2 · p. 7
  • Goldsmith reaches accuracy 0.8650, positive macro-F1 0.4273, positive recall 0.7273, and positive F1 0.5439 on CrossRE.Accuracy is not the decisive metric because no_relation dominates the test pairs.Source: §5.2 · p. 8, Table 4 · p. 8
  • PHEE V2 score-routed review reaches argument micro-F1 0.7471, relation micro-F1 0.7427, trigger pass rate 0.9988, and joint exact match 0.4500.Without review, the corresponding scores are 0.7111, 0.7066, 1.0000, and 0.3958.Source: §5.2 · p. 8

5.3 Ablation Experiments

  • Removing prompt optimization drops OpenNER F1 from 0.8174 to 0.6661, while replacing structured retrieval with random references reduces it to 0.7915.The ablations distinguish the main definition-training gain from additional stability supplied by retrieval.Source: §5.3 · p. 8, Table 5 · p. 8
  • GPT-5.5 has the best reviewed F1 and exact match, while DSv4p leads without review but loses that lead after review.Reviewed scores are relatively close across capable backends, though the harness does not eliminate model differences.Source: Table 6 · p. 8, §5.3 · p. 9

6 Conclusion

Goldsmith uses expert gold examples to learn an operational definition and reports gains in prompt optimization and downstream annotation. Its harness tests and audits definition edits while keeping schema and review policies fixed.

6 Conclusion

  • Gold-loss-guided editing produces a stronger operational definition than direct LLM rewriting or OPRO, and the resulting definition improves three downstream annotation tasks.The reported tasks are OpenNER typed span labeling, CrossRE positive-relation classification, and PHEE V2 fixed-trigger event-argument extraction under low-label budgets.Source: §6 · p. 9
  • Goldsmith uses a small set of authoritative examples as executable supervision while the operational definition is still being formed.The conclusion describes this as a stage where experts know the phenomenon but have few examples and no stable definition.Source: §6 · p. 9
  • The harness fixes the schema, tests edits against gold loss, repairs and audits outputs, and routes uncertain cases to human review.Goldsmith is intended to reach a stable task definition before supervised PLM pipelines are used for large stable datasets.Source: §6 · p. 9

Limitations

Goldsmith is intended for tasks whose boundaries can be expressed through natural-language definitions and examples. Its suitability is constrained by the available expertise, label ambiguity, API restrictions, and scale.

Limitations

  • Goldsmith is designed for tasks whose boundaries can be expressed through natural-language definitions and examples.It may be less suitable when private expert knowledge is absent from the gold examples, labels are highly ambiguous, or LLM API calls are prohibited.Source: Limitations · p. 9
  • At very large scale, token costs, provider limits, and retry overhead can make direct LLM annotation inefficient.The paper suggests using Goldsmith to build a small, high-quality labeled set and stable definition before training a task-specific model for large-scale labeling.Source: Limitations · p. 9

Ethical Considerations

Goldsmith records workflow decisions and requires human review to correct model behavior, especially in high-risk domains. Its hosted API workflow also makes data governance and dataset licensing relevant to use.

Ethical Considerations

  • Human review remains necessary because model predictions and judge scores are not authoritative labels, particularly in high-risk settings.The passage names medical, legal, and safety-critical domains among those settings.Source: Ethical Considerations · p. 9
  • Hosted LLM APIs receive task text, prompts, retrieved references, model predictions, and judge inputs, so sensitive corpora require governance and institutional review.The passage calls for data-governance, provider-policy, and institutional review before use of private, restricted, or sensitive corpora.Source: Ethical Considerations · p. 9
  • The workflow records model calls, prompt revisions, reviewer decisions, and export artifacts so users can inspect and correct model behavior.Source: Ethical Considerations · p. 9
  • Experiments use derived slices of OpenNER, CrossRE, and PHEE V2, and users must follow the source datasets’ licenses and redistribution terms.Source: Ethical Considerations · p. 9

A Data Construction and Gold Selection

The appendix describes deterministic task slices and 15-item calibration sets selected for coverage. It details the construction and scope of OpenNER, CrossRE, and PHEE V2 packages.

A Data Construction and Gold Selection

  • All three task packages use deterministic task slices and reserve a 15-item calibration set that maximizes coverage before matching the full-pool distribution.The scripts write JSONL records with source text, spans or relation labels, task identifiers, and metadata.Source: A Data Construction and Gold Selection · p. 12

A.1 OpenNER Typed Span Slice

  • With seed 9, the OpenNER slice contains 52 languages, 2,600 tasks, and 3,833 typed spans: 1,424 PER, 1,367 LOC, and 1,042 ORG.Binary NE conversion is used only for OpenNER-derived PLM diagnostics, not as the official OpenNER task.Source: §A.1 · p. 12

A.2 OpenNER Gold Selection

  • The OpenNER gold selector covers 15 language groups, all four label states, and all four length bins in 15 selected rows from 2,600 candidates.It leaves 2,585 pool rows and no overlap between gold and pool files.Source: §A.2 · p. 12

A.3 CrossRE Pair-Level Slice

  • The CrossRE slice yields 1,200 sentence-level tasks and 39,764 directed entity-pair classification items, including 35,275 no_relation items.The prevalence of no_relation makes positive-class metrics necessary alongside accuracy.Source: §A.3 · p. 12

A.4 CrossRE Gold Selection

  • CrossRE’s 15 selected calibration items cover all six domains, 15 relation labels, and all four length bins among 39,764 candidates.The selector uses greedy coverage followed by exhaustive single-swap search; 39,749 items remain in the pool.Source: §A.4 · p. 12

A.5 PHEE V2 Event-Argument Setting

  • PHEE V2 fixes an event-trigger anchor and requires Subject, Treatment, and Effect argument spans plus trigger-to-argument relations.The setting reports argument and relation micro-F1, trigger pass rate, and joint exact match for zero-shot and Goldsmith variants, with and without score-routed review.Source: §A.5 · p. 13

B Evaluation Evidence Package

The evaluation package documents model and runtime settings alongside how the OpenNER and CrossRE evaluation data and reviewed feedback are constructed. It specifies reproducibility details and the scopes used for reported metrics.

B.1 Model and Runtime Details

  • Table 7 records model identifiers, access months, and decoding/runtime settings; hosted models may change, so configurations include provider-facing identifiers and run parameters.Platform-call failures are retried up to a run bound, while successful calls with invalid structured output enter format repair before requeueing.Source: §B.1 · p. 13

B.2 OpenNER Split Accounting

  • OpenNER’s score-routed review run uses 2,600 target items, 15 calibration items, and 193 reviewed feedback items, with metrics computed on a held-out subset excluding calibration and reviewed items.After pass1, the lowest judge-score items are corrected by human reviewers and promoted to reviewed references before annotation is rerun.Source: §B.2 · p. 13

B.3 CrossRE Scope Accounting

  • CrossRE metrics use the same 29,171-pair evaluation slice for Goldsmith and low-budget PLM references.The PLM references train on 200 sentences expanded into 10,593 pairs; Goldsmith uses reviewed feedback and retrieved references during annotation.Source: §B.3 · p. 13

B.4 GT-only RAG Reference Count Sweep

The appendix reports a retrieval-count diagnostic, optimization traces, and loss-configuration details. The CrossRE trace reaches zero gold loss by round 8, while the OpenNER diagnostic shows performance peaking at a moderate retrieval count.

B.4 GT-only RAG Reference Count Sweep

  • 0.8026 micro-F1 is reached at k = 14 in the OpenNER GT-only retrieval sweep, after substantial improvement from k = 0 and slight decline at larger k.The sweep uses a fixed 500-example evaluation sample, the optimized definition, and oracle gold-truth examples as its retrieval pool; it excludes reviewed model predictions.Source: §B.4 · p. 13, Table 8 · p. 14

B.5 Prompt-Optimization Trace

  • The OpenNER optimizer trace compares rewrite-only, OPRO, and Goldsmith per-round raw loss, with Goldsmith’s running best loss; its summary also includes APE and PromptBreeder.All five optimizers share the same initial definition, 15 gold examples, candidate count 5, maximum 30 rounds, patience 5, and gold loss.Source: §B.5 · p. 13

B.6 Gold-Loss Parameter Configuration

  • Gold-loss rate weights are normalized by their sum, while λB + λT = 1 and λS + λR = 1 make component weights relative contributions; ρ changes displayed loss values but not candidate ordering or acceptance.The implementation adds no prompt-length penalty to the loss.Source: §B.6 · p. 13

B.7 CrossRE Prompt-Optimization Trace

  • CrossRE calibration passes rise from 8/15 to 15/15 with zero gold loss at round 8, after evaluating 40 candidate prompts.The complete trace is reported in Table 11 under the OpenNER gold-loss protocol.Source: §B.7 · p. 14

C.1 Task-Stage Baselines

This section defines the task-stage and prompt-optimization baselines, documenting agent involvement in code construction and the optimizer templates and operational definitions used in the experiments. It also describes conservative OpenNER span labeling and directed CrossRE relation classification, both constrained to evidence explicit in the supplied text.

C.1 Task-Stage Baselines

  • OpenNER baselines compare zero-shot, static in-context learning, and first- versus second-pass Goldsmith, while CrossRE uses pair-level relation classification with typed-marker PLM anchors.The OpenNER PLM baselines are low-label supervised anchors trained on 208 examples; CrossRE PLMs train on 200 sentences expanded to 10,593 pairs and test on 29,171 pairs.Source: §C.1 · p. 14

C.2 Prompt-Optimization Baselines

  • Five optimizers are compared under a shared search protocol; Goldsmith uses high-loss errors as textual-gradient directions and accepts revisions only when gold loss decreases.The comparison includes LLM rewrite-only, OPRO, APE, PromptBreeder, and Goldsmith, with matched examples, candidate count or search budget, loss, acceptance rule, and stop policy.Source: §C.2 · pp. 14–15

C.3 Agent-Assisted Code Work

  • The authors used Codex with GPT-5.5 for part of code construction, reviewed the resulting code, and remain responsible for the final submission.Source: §C.3 · p. 15

D Prompt Artifacts

  • The prompt artifacts document optimizer templates and definitions while keeping fixed schemas, parser contracts, and JSON protocols in the harness.Concrete failed-example blocks are placeholders because they are training-only feedback; Goldsmith's template separates textual-gradient generation from definition editing.Source: D Prompt Artifacts · pp. 15–16

OpenNER

  • OpenNER labels explicit PER, ORG, and LOC mentions, excluding ambiguous or generic spans and preserving exact source-text forms without normalization.It prefers bare geopolitical names within broader descriptive phrases and returns no spans when no target mention is explicit.Source: D Prompt Artifacts · p. 19

CrossRE

  • CrossRE classifies only the provided directed HEAD → TAIL pair, using fixed entity spans and forbidding entity discovery, direction reversal, or background-knowledge inference.The deployed prompt treats the sentence and entity spans as fixed anchors.Source: D Prompt Artifacts · p. 19
  • CrossRE requires clear local sentence evidence for positive relations, prefers no_relation over weak positives, and selects the most specific supported label.The prompt defines boundaries for labels including role, physical, artifact, and cause-effect; vague association alone does not justify related-to.Source: D Prompt Artifacts · pp. 19–20

Related papers

Loading 2610.09489v1…