PaperlayerView exact revision on arXiv

Source-linked AI summary

MetaCurate: Evolving Curation Filters for High-Quality Agentic Coding Data

Donghoon Kim, Seunghyuk Oh, Kevin Galim, Minseo Kim, Minjae Lee, Younghyun Kim, Wooseok Seo, Hyung Il Koo, Wonjun Kang

arXiv:2610.23134v1cs.LG

TL;DR

Selecting useful coding-agent training trajectories requires judging their reasoning as well as their outcomes, while designing and validating filters can require expertise and costly training runs. MetaCurate evolves executable filters using pairwise LLM judgments and deploys them as pointwise scorers. Across two trajectory pools, its evolved filter achieves the highest observed mean downstream scores among compared methods.

  • Problem

    Trajectory outcomes alone do not establish training-example quality, while filter design requires criteria and validation that can be costly to develop.

    Source: §1 · pp. 1–2

  • Method

    MetaCurate iteratively revises executable curation filters using pairwise LLM preferences and disagreement feedback, then scores trajectories independently.

    Source: §1 · pp. 2–3

  • Results

    64.0% on the DeepSeek V4 Pro pool and 56.5% on Nemotron v3 are the highest observed mean SWE-bench Verified scores among compared methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results support using pairwise judgments to guide pointwise selection of training data without placing model training inside the search loop.

    Source: §1 · p. 3, §6 · p. 9

    The experiments focus primarily on downstream data-selection improvements rather than interpreting the curation principles that emerge during search.

    Source: §6 · p. 9

Abstract

from arXiv · show

Supervised fine-tuning on agent trajectories is an effective way to improve coding agents, but selecting useful training examples requires more than identifying successful runs. A resolved patch may follow inefficient reasoning or unnecessary tool use, while a failed run may still contain useful intermediate steps. Training pipelines therefore rely on rules and qualitative assessments to filter trajectories, yet designing these filters typically requires substantial human expertise, and validating improvements through repeated training and evaluation is expensive. We introduce MetaCurate, a training-free framework that evolves executable trajectory curation filters using pairwise LLM judgments. Disagreements between the filter's rankings and the judge's preferences provide targeted feedback for jointly revising its rules, rubric, and aggregation policy. The resulting filter scores trajectories independently, so pairwise comparisons are required only during search. Starting from a blank filter, MetaCurate autonomously discovers a curation strategy that achieves competitive downstream performance without task-specific handwritten criteria. We validate downstream utility through controlled SFT experiments on our uncurated trajectory pool and the Nemotron v3 SWE dataset, with random selection and hand-authored filtering as baselines. In our experiments, MetaCurate achieves the highest mean SWE-Verified score among the compared methods, including hand-authored rule-based filtering and a training-loop baseline whose search costs 3.5x as much.

1 INTRODUCTION

Trajectory curation must assess both problem-solving process and outcome, but existing filters require advance criteria or costly training-based validation. MetaCurate instead revises an executable filter using pairwise LLM judgments, then scores trajectories independently; its evolved filter achieves the highest observed mean downstream scores among compared methods on two pools.

1 INTRODUCTION

  • Useful SFT examples require judging problem-solving quality as well as outcome, since successful runs can include waste and failed runs can contain useful reasoning.Outcome-only filtering can retain poor reasoning leading to a correct patch and discard useful reasoning ending in failure.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Handwritten rules cover only prespecified behaviors, while LLM filters depend on their prompt and rubric; these filters are often applied without target-pool validation.Training-based revisions provide direct utility feedback but require another training run for each candidate.Source: §1 · p. 2
  • MetaCurate compares candidate filter rankings with an LLM judge’s pairwise preferences, using disagreements and rationales to revise the filter’s rules, rubric, and aggregation.The search does not train a target model; the final filter scores trajectories independently without pairwise judgments at deployment.Source: §1 · pp. 2–3
  • 64.0% on the DeepSeek V4 Pro trajectory pool and 56.5% on Nemotron v3 are the highest observed mean SWE-bench Verified scores among compared methods.The experiments used controlled SFT and random selection as a control.Source: §1 · p. 3

2 RELATED WORK

Prior curation approaches combine outcome labels, reference-patch checks, transcript heuristics, and LLM judgments, but each signal has limitations. Recent work optimizes curation procedures using downstream training performance; MetaCurate instead evaluates filters through judge agreement without training for each candidate.

Coding agents

  • Coding-agent SFT pipelines select trajectories because passing tests alone does not establish training usefulness, and successful runs may still contain unnecessary steps.Repository-issue agents inspect code, run commands, edit files, and revise changes using test results.Source: §2 · p. 3

Data curation

  • Curation signals include test outcomes, reference-patch verification, handwritten heuristics, and LLM judgments, each with potential limitations such as model-dependent errors or bias.Some selection methods offer little improvement over random sampling at scale.Source: §2 · p. 3
  • Recent approaches evolve curation code using trained-model performance, whereas MetaCurate jointly evolves rules, rubric, and aggregation using judge agreement without training each candidate.Source: §2 · p. 3

3 CURATION SIGNALS AND SCALABLE FEEDBACK

Pairwise judgments provide relative references but remain costly when repeated across large pools. Pointwise selection scales linearly, yet outcome labels and LLM scores do not by themselves establish training value, motivating validation feedback that can guide revisions without repeated model training.

3.1 PAIRWISE COMPARISON PROVIDES A RELATIVE REFERENCE

  • Pairwise comparisons give each trajectory a relative reference and avoid requiring independently calibrated absolute quality scores, but repeated ranking remains costly as pools grow.Exhaustive comparison requires judgments for N trajectories; sparse comparisons and tournaments reduce, but do not eliminate, evaluation expense.Source: §3.1 · p. 3

3.2 POINTWISE SELECTION SCALES, BUT REQUIRES VALIDATION

  • At scale, fixed pointwise scoring requires evaluations that grow linearly with pool size, while final task success alone does not establish the training value of a trajectory.In the DSv4Pro dataset, 90.1% of issues had the same outcome label across attempts; judges preferred the failed trajectory in 28.3% of audited pairs with different labels.Source: §3.2 · p. 4
  • LLM trajectory scores can capture qualitative properties, but score reliability and training utility require validation against real trajectories and suitable criteria.Without explicit criteria or reference examples, repeated scores may vary and judges may disagree.Source: §3.2 · p. 4
  • Training feedback costs 3.5 times as much and evaluates only 3 candidates against 45, motivating feedback that can assess and revise selection without repeated model training.Source: Figure 2 · p. 4, §3.2 · p. 4

4 METACURATE: EVOLVING THE CURATION FILTER

MetaCurate turns pairwise judge preferences into a reusable pointwise trajectory filter. It iteratively proposes and evaluates revisions, using separate exploration feedback and fixed selection pairs for promotion, then scores the full pool without further pairwise judgments.

4 METACURATE: EVOLVING THE CURATION FILTER

  • A curation filter is an executable scorer composed of deterministic rules, LLM-evaluated rubric criteria, and a score-aggregation policy.MetaCurate revises these components and uses pairwise judge preferences as supervision for a pointwise filter.Source: §4 · pp. 4–5

4.1 PAIRWISE JUDGMENTS FOR CURATION FILTER EVALUATION

  • Candidate filters are evaluated by concordance between their trajectory rankings and the judge’s pairwise preferences, rather than by training on selected data.The fixed selection pairs serve as an internal promotion signal, while trajectories and rationales from exploration are kept separate from that set.Source: §4.1 · p. 5

4.2 CURATION FILTER EVOLUTION LOOP

  • Each iteration uses accumulated exploration feedback to propose candidates, evaluates them on sampled trajectories, and promotes a filter using fixed selection pairs.Exploration pairs whose rankings differ receive judge verdicts and rationales that feed subsequent proposals; the run directory retains feedback and promotion records.Source: §4.2 · pp. 5–6

4.3 IMPLEMENTATION DETAILS

  • Diagnostics flag low-variation or redundant criteria, while draft probing checks candidate rules and rubric scores on exploration trajectories before submission.Pairs are presented in both orders and retained only when the judge identifies the same winner; apparent gains are screened with fresh scoring draws and stricter thresholds for new criterion-health defects.Source: §4.3 · p. 6
  • After search, the final filter ranks the full trajectory pool using its deterministic rules, rubric, and aggregation policy, selecting the highest-scoring subset without further pairwise judging.Source: §4.3 · pp. 6–7
  • Table 1 reports Qwen 3.5 9B SWE-bench Verified resolved rates across curation methods and two pools, with a default training budget of 1,000 trajectories.Its second column lists hand-authored guidance or oracle information required for curation.Source: Table 1 · p. 7

5 EXPERIMENTS

Controlled SFT experiments compare trajectory filters on two pools, then test model scale and curation-loop choices. The blank-seed filter achieves the highest observed mean scores in the main comparison, with ablations examining seeds, iterations, model roles, and data budget.

5.1 SETUP

  • The Qwen 3.5 9B comparison uses a 1,000-trajectory budget and evaluates each trained checkpoint three times on all 500 SWE-bench Verified instances.Experiments use the DSv4Pro corpus and the Nemotron v3 mini-swe-agent subset, with a shared fine-tuning recipe within each pool.Source: §5.1 · p. 7

5.2 MAIN COMPARISON

  • At 1,000 trajectories, blank-seed METACURATE yields 64.0% on DSv4Pro and 56.5% on Nemotron v3, versus CurateEvo's 62.7% and 53.6%, the highest observed means.On DSv4Pro, its selected subset also performs comparably to training on all 4,286 trajectories (63.1%).Source: §5.2 · p. 7

5.4 ABLATION STUDIES

  • After five iterations, Sol Pro produced higher-concordance filters than Luna; cross-model judge agreement was 0.90–0.96, while solver rankings agreed on 0.88–0.91 untied pairs.Luna costs approximately $0.004 per judge call versus $0.14 for Sol Pro, and $0.004 per scoring call versus $0.09 for Sol Pro.Source: Figure 4 · p. 8, §5.4 · p. 9

5.3 MODEL SCALE

  • On Qwen 3.5 27B, Ours reaches 74.3% mean resolved rate on SWE-bench Verified, versus 73.3% for Resolved, 73.1% for Rule filter, and 72.7% for Random.The methods use the same training subsets as for 9B, without repeating filter search.Source: Table 2 · p. 8, §5.3 · p. 8

5.4 ABLATION STUDIES

  • Best observed concordance reached 0.916 for the handwritten seed and 0.912 for the blank seed; blank-seed SFT means were higher on both pools: 64.0% vs. 63.3% and 56.5% vs. 54.7%.The blank seed contains one generic criterion, while the handwritten seed uses six criteria.Source: §5.4 · p. 8
  • As Selection C rose from 0.845 to 0.912, mean resolved rate rose from 62.4% to 64.0%, while Test C rose from 0.830 to 0.896.Selection and test concordance improved by +0.067 and +0.066, respectively; SWE-bench Verified instances were excluded from filter search.Source: §5.4 · pp. 8–9, Table 3 · p. 8
  • Presentation-order flips occur more often for closely ranked trajectories, supporting judging each pair in both orders and retaining only consistent preferences.Source: Figure 4 · p. 8, §5.4 · p. 9
  • With 200 DSv4Pro trajectories, Ours reaches 60.3%, versus 58.7% for Random and 59.8% for the strongest baselines.Source: Table 4 · p. 9, §5.4 · p. 9

6 CONCLUSION

MetaCurate evolves trajectory curation filters through pairwise LLM feedback and, across two coding-trajectory datasets, its selected subsets yield the highest observed mean downstream scores among the compared methods. The authors identify interpreting the curation principles that emerge during search as an important direction for future work.

6 CONCLUSION

  • Across two coding-trajectory datasets, blank-seed MetaCurate selects subsets yielding the highest observed mean downstream scores among the compared methods.The authors suggest pairwise judgments can guide pointwise data selection.Source: §6 · p. 9
  • The experiments primarily assess downstream data-selection improvements rather than interpret the curation principles that emerge during search.Future work could analyze how rules and rubric criteria change and relate those changes to training outcomes.Source: §6 · p. 9
  • The authors envision that such analyses may help extract empirical principles for collecting, generating, and curating future agent trajectories.This is presented as a possible longer-term outcome of studying filter changes and downstream results.Source: §6 · pp. 9–10

A EXPERIMENTAL SETUP DETAILS

The experiments compare curation methods under controlled SFT settings on a newly collected DeepSeek V4 Pro trajectory pool and a mini-swe-agent subset of Nemotron v3. Most curated datasets use a 1,000-trajectory budget, and all methods are trained with Qwen 3.5 9B.

DSv4Pro dataset

  • The DSv4Pro pool contains 4,286 DeepSeek V4 Pro trajectories from 1,415 instance IDs, with three attempts targeted per source instance.The source datasets were collected independently without deduplicating overlapping GitHub issues; 1,369 IDs have three trajectories, 23 have two, one has one, and 22 have six.Source: A EXPERIMENTAL SETUP DETAILS · p. 15

Nemotron v3 dataset

Baselines

  • Other baselines are the uncured full pool, a random 1,000-trajectory sample, resolved trajectories, and trajectories with reference-patch line recall of at least 0.5.The Gold-patch recall criterion yields 629 DSv4Pro training rows, all used, while 1,000 qualifying Nemotron trajectories are sampled.Source: A EXPERIMENTAL SETUP DETAILS · p. 15
  • The comparison includes a seven-check deterministic rule filter and an LLM filter that directly judges each trajectory’s training suitability.The rule checks cover submission validity, disallowed commands, repetitive loops, exploration, malformed calls, debugging artifacts, and edits without tests.Source: A EXPERIMENTAL SETUP DETAILS · p. 15
  • CurateEvo evolves a curation program using validation scores from models trained inside its optimization loop, then samples 1,000 accepted trajectories for downstream training.It reserves 10% of the training set for validation and uses Luna as scorer.Source: A EXPERIMENTAL SETUP DETAILS · p. 15

Training and evaluation details

  • Table 6 presents training and evaluation settings, listing shared settings separately from each model’s fine-tuning method and learning rate.Source: Table 6 · p. 16

B LOOP DETAILS

The search iteratively proposes and screens filter revisions using exploration feedback and pairwise judgments, while holding out SWE-bench Verified from the curation loop. Candidates are promoted based on comparison with the current filter, with constraints on size, criterion health, and evidence of improvement.

Runs and evidence split

  • The blank-seed and handwritten-seed DSv4Pro runs use different search settings, with the blank run receiving more trajectories and pair judgments per iteration but fewer rounds.The handwritten run uses 20 iterations, 50 exploration trajectories, 50 exploration pairs, and a 0.15 complexity weight; the blank run uses 12 iterations, 200 trajectories, 200 pairs, and no complexity weight.Source: B LOOP DETAILS · p. 16
  • Search uses randomly sampled exploration trajectories and a disjoint 220-trajectory selection set, while SWE-bench Verified is reserved as an external test set.The judge considered 6,673 selection pairs in both presentation orders; 5,674 pairs with the same winner in both orders form Psel, hidden from the proposer.Source: B LOOP DETAILS · p. 16

Proposal and exploration feedback

  • Each iteration has the proposer submit three candidates after probing drafts; a precheck evaluates surviving candidates before the solver scores them on exploration and selection trajectories.The proposer can revise once before full scoring, and the judge spends the run-specific pair budget on exploration trajectories.Source: B LOOP DETAILS · p. 16

Candidate screening

  • Candidate screening enforces hard size caps of nine rubric criteria and 13 rule functions, while dead criteria cover over 90% of rows and redundant pairs exceed 0.7 absolute tie-aware Spearman correlation.Soft-cap growth and newly dead or redundant criteria require stronger promotion evidence rather than automatic rejection.Source: B LOOP DETAILS · p. 16

Promotion

  • Promotion compares strict correctness on selection pairs: b counts pairs only the candidate orders correctly, while c counts pairs only the current filter orders correctly.An ordinary candidate needs at least five discordant pairs; a smaller filter can qualify as a simplification when b ≥ c, but an eligible improvement takes priority.Source: B LOOP DETAILS · p. 17
  • The promotion screen uses a one-sided binomial-tail statistic with a nominal z = 2 threshold adjusted across candidates, but it does not guarantee a full-search error rate.Pairs share trajectories and are reused across iterations; the selected leader is rescored with fresh LLM draws before promotion.Source: B LOOP DETAILS · p. 17

Stopping

  • A run stops at its iteration limit or after five consecutive iterations without promotion, once at least four iterations have run.Figure 5 illustrates how nearly constant criteria barely change rankings and how aggregation policies can produce different rankings from shared scores.Source: B LOOP DETAILS · p. 17, Figure 5 · p. 17

C CHECKING HAND-WRITTEN FILTERS ON DATA

The analysis checks whether handwritten filter criteria affect rankings and whether aggregation choices alter selected examples. It finds that some criteria are nearly inactive and that plausible policies can select substantially different top-50 sets.

Checking handwritten filters

  • The analysis checks criteria’s ranking effects and which examples the final policy selects, because written distinctions alone do not ensure influence on selection.The authors argue that inspecting scores, rankings, and selections manually for every candidate is costly, motivating an automated evaluation and revision loop.Source: C CHECKING HAND-WRITTEN FILTERS ON DATA · pp. 17–18

Dead criteria

  • Two of six handwritten criteria score nearly identically across 200 trajectories, and removing either barely changes the ranking.The Gold-Patch Relation criterion is effectively inactive in this setup because reference gold patches were not provided to the solver and judge.Source: C CHECKING HAND-WRITTEN FILTERS ON DATA · pp. 17–18

Score aggregation

D FILTER EVOLUTION

The blank-seed search makes larger scoring changes than the handwritten-seed search, whose history remains comparatively close to its starting filter.

D FILTER EVOLUTION

  • The blank-seed run produces more pronounced score departures in both directions, while the handwritten-seed run shows mostly pale cells and fewer large differences.The authors describe this pattern as consistent with greater inertia around an established handwritten filter and more room for change from a blank seed.Source: D FILTER EVOLUTION · p. 18
  • Figure 7 shows each iteration’s current filter and proposed candidates; colors encode concordance differences, and green outlines mark promoted candidates.Orange dashed outlines mark eligible candidates when a provisionally selected candidate fails re-evaluation and the current filter remains unchanged.Source: Figure 7 · p. 19
  • Figure 8 uses the same notation and color scale as Figure 7 to show the handwritten-seed search over twenty iterations.Its mostly pale cells indicate few large departures from the current filter.Source: Figure 8 · p. 20

E ROLE PROMPTS

The role prompts divide filter revision, candidate execution, and pairwise judging across LLM roles. The solver scores criteria separately, and the judge compares trajectories as training data.

E ROLE PROMPTS

  • The solver evaluates each criterion in a separate LLM call using the rubric and returns criterion scores with concrete evidence.The prompt specifies a +2 to -2 scale; absent or ambiguous evidence receives a score of 0.Source: E ROLE PROMPTS · p. 21
  • The role prompts were not extensively tuned for these experiments, yet the reported runs show effective filter evolution with this fixed configuration.The solver’s separate criterion calls are intended to reduce conflation across assessment dimensions.Source: E ROLE PROMPTS · p. 21
  • The proposer receives misordered pairs, judge rationales, and criterion diagnostics, then returns candidate replacements for the rubric, rules, and policy.The runtime evidence includes current filter files, prior diagnoses, and promotion records.Source: E ROLE PROMPTS · p. 21
  • The judge selects which trajectory offers a strong SWE model more good behavior to imitate, considering diagnosis, patch quality, verification, and efficiency.Each pair is presented in both orders.Source: E ROLE PROMPTS · p. 21

F SEED FILTER EXAMPLES

The seed examples contrast a handwritten six-criterion filter with a blank filter containing one generic criterion and a minimal rejection rule. Both use executable scoring policies to rank trajectories.

F SEED FILTER EXAMPLES

  • The handwritten seed is the six-criterion filter from Appendix C, with rubric anchors, deterministic rules, and a scoring policy.Its rubric uses a symmetric +2 / +1 / 0 / -1 / -2 scale and supports false-positive and false-negative critics.Source: F SEED FILTER EXAMPLES · pp. 22–23
  • Two handwritten criteria consult the task’s problem statement and gold patch, scoring their quality and the agent’s relation to the reference rather than similarity alone.When reference fields are missing, the score is 0 with a note that the reference was unavailable.Source: F SEED FILTER EXAMPLES · pp. 22–23
  • The handwritten rubric assesses trajectory quality through criteria including verification completeness, error recovery, task grounding, query quality, and relation to the gold patch.Anchors describe both good and bad trajectories; scores require concrete evidence, with missing or ambiguous evidence receiving 0.Source: F SEED FILTER EXAMPLES · p. 22
  • Its nine deterministic rules reject cases such as invalid submissions, repeated editing after tests, excessive malformed commands, debug artifacts, or edits without testing.The exploration rule also rejects trajectories with at least 20 actions when at least 85% are reads or searches and at most two are edits.Source: F SEED FILTER EXAMPLES · p. 23
  • The handwritten policy combines available rubric scores, rejection-rule firings, positive labels, and any -2 score into a clipped 0–100 ranking score.A score of at least 65 receives ACCEPT; ranking uses the continuous score.Source: F SEED FILTER EXAMPLES · p. 23
  • The blank seed starts with one generic criterion and rejects only trajectories with zero turns, without a task-specific repair heuristic.Its policy uses 50 + 12.5s plus a rule adjustment, clips the result to [0, 100], and accepts scores of at least 50.Source: F SEED FILTER EXAMPLES · p. 23
Loading 2610.23134v1…