PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Agentic Data for Long-Horizon Terminal Intelligence

Shun Zou, Ziyu Ma, Yi Zou, Yong Wang, Lin Chen, Zehui Chen, Guanhua Chen, Xiangxiang Chu, Feng Zhao

arXiv:2610.13166v1cs.CL

TL;DR

Short terminal trajectories do not fully capture how agents gather evidence, revise strategies, and complete sustained professional work, while rich long-horizon training data remains scarce. TerminalHorizon independently tests and reconstructs tasks, then expands them under the same goal. Training on the resulting trajectories improves performance across 12 benchmarks, and RSI case studies show feedback-driven improvement retained in final deliverables.

  • Problem

    High-quality terminal training data is scarce, and short trajectories offer limited experience of sustained problem solving and connected agentic behaviors.

    Source: §1 · pp. 1–2

  • Method

    TerminalHorizon independently executes tasks to identify construction gaps, guides environment reconstruction, and expands tasks under the same goal so added work builds on prior results.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    TerminalHorizon-27B improves performance across 12 benchmarks, including a 14.2% improvement over the base model on Terminal-Bench 2.0.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    RSI case studies show continued experimentation, feedback-driven method refinement, and effective improvements retained in final deliverables.

    Source: §1 · p. 3, §3 · p. 5

    Verifier completion requirements must be supported by the public instruction and discoverable environment evidence, though test data may remain private.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

Terminal agents are moving beyond local operations toward completing real-world professional tasks. Short trajectories can demonstrate commands and tool calls, but they do not fully show how agents accumulate evidence through exploration, use it to inform judgments, continually adjust strategies based on feedback, and turn intermediate results into final deliverables. High-quality long-horizon trajectories capture these agentic behavioral patterns and their connections within a single problem-solving process, providing richer experience for sustained autonomous work. However, environment synthesis may complete or simplify the key work underlying these patterns in advance and thus remove the need for exploration, implementation, and verification that the agent was meant to perform. To address this, we introduce TerminalHorizon, a fully automated data synthesis engine for long-horizon terminal agents. Starting from real-world professional work, it uses independent execution to identify key work omitted during construction and uses the resulting evidence to guide environment reconstruction. To extend these patterns across stages, it progressively expands tasks under the same goal, introducing new requirements that build on prior results and call for further exploration and improvement. Using this engine, we construct 1.5K task environments and perform supervised fine-tuning of Qwen3.5-27B on the resulting dataset to obtain TerminalHorizon-27B. We achieve substantial performance gains on all 12 benchmarks spanning coding-agent tasks, agentic tasks in real-world production settings, and general reasoning. Multiple RSI case studies, each with a 12-hour research budget, further demonstrate the model's ability to continue experimenting beyond an initial solution, improve its methods based on feedback, and retain effective improvements in its final deliverables.

1 INTRODUCTION

Long-horizon terminal tasks require agents to connect exploration, strategy revision, and verification, but high-quality training data for sustained problem solving is scarce. TerminalHorizon reconstructs and expands tasks to preserve this work, and training on its trajectories improves benchmark performance.

1 INTRODUCTION

  • Short trajectories offer limited experience in sustained problem solving, where agents may revise plans as evidence accumulates and revisit work after verification reveals problems.The authors motivate trajectories that show how these behavioral patterns connect across multiple stages of a task.Source: §1 · p. 2
  • Task synthesis can complete or simplify required work in advance, while adding difficulty or turns alone does not ensure behavioral patterns extend across stages.Added requirements should build on earlier judgments, actions, and results while preserving the same goal.Source: §1 · p. 2
  • TerminalHorizon independently executes candidate tasks to find omitted work, guides environment reconstruction, and progressively adds same-goal requirements that depend on prior results and feedback.Its two stages are challenge-guided long-horizon synthesis and progressive horizon expansion.Source: §1 · p. 2
  • TerminalHorizon-27B improves over the base model by 14.2% on Terminal-Bench 2.0 and 13.85% on Terminal-Bench 2.1, with gains across 12 benchmarks.The benchmarks span coding, real-world productivity, and general reasoning; 12-hour RSI case studies show continued experimentation, feedback-driven refinement, and improvements retained in final deliverables.Source: §1 · pp. 2–3

2 METHOD

TerminalHorizon constructs executable terminal tasks to preserve exploration, diagnosis, implementation, verification, and strategic coordination. It checks whether required work remains necessary during execution, then expands accepted tasks under the same goal so later work builds on earlier progress.

2 METHOD

  • Independent execution reveals work simplified or completed during construction and guides reconstruction; progressive expansion adds same-goal work that depends on prior results and feedback.The method presents these as two stages for eliciting agentic behavioral patterns in executable environments.Source: §2 · p. 3

2.1 TASK FORMULATION AND DESIGN OBJECTIVE

  • Task packages contain a public instruction, executable environment, reference solution, private verifier, and execution configuration, while agents access only the instruction and environment.The design aims to preserve evidence to explore, unfinished analysis and implementation, and feedback for revising judgments across 27 behavioral patterns in five families.Source: §2.1 · p. 3

2.2 CHALLENGE-GUIDED LONG-HORIZON SYNTHESIS

  • Batch planning selects uncovered professional work from existing tasks and feedback, then research briefs guide investigation of real-world needs, constraints, evidence, and dependent work.Construction creates a controlled task instance with available materials, unfinished work, and completion criteria, without completing the intended work in advance.Source: §2.2 · pp. 3–4
  • An independent challenger uses only public task information; trajectory and scoring evidence help diagnose omissions, and reconstruction restores required analysis, implementation, or debugging for another execution check.The review distinguishes efficient solutions from work already completed during construction.Source: §2.2 · pp. 4–5
  • Accepted tasks and review records feed subsequent batch planning, so execution gaps inform both current-task reconstruction and future research briefs.The planner uses accumulated records to identify repeated work and choose behavioral patterns for later tasks.Source: §2.2 · p. 5

2.3 PROGRESSIVE HORIZON EXPANSION

  • Progressive expansion proposes deeper same-goal requirements and designs dependencies through evidence, tool feedback, or instance variation; accepted candidates combine original and added requirements.The evolution agent checks whether the parent solution can directly satisfy proposed requirements and identifies further work needed.Source: §2.3 · p. 5

3 ANALYSIS OF SYNTHESIZED DATA

TerminalHorizon produces diverse, sustained trajectories that connect terminal interaction to multi-stage problem solving. The trajectories span professional domains, preserve feedback-linked action chains, and capture behavioral patterns beyond individual commands.

3 ANALYSIS OF SYNTHESIZED DATA

  • The analysis characterizes training experience as organizing sustained work toward a goal, observing feedback, revising strategies, and delivering final results.Trajectory collection retained 2,852 of two generated trajectories per each of 1,535 task environments after filtering and benchmark decontamination.Source: §3 · pp. 5–6
  • 25 professional domains are represented in TerminalHorizon-3K, bringing work in areas such as chips, physics, and astronomy into a unified terminal format.The Planner Agent schedules and coordinates the automated construction process.Source: §3 · p. 6
  • Trajectories average 52.8 interaction turns, reach 98 turns at the 90th percentile, and have a median feedback-linked action chain of 25 turns.The interaction lengths substantially exceed those of compared methods.Source: §3 · p. 6
  • Trajectories contain an average of 8.6 core behavioral patterns drawn from five families spanning exploration, construction, error correction, and verification.The taxonomy comprises 27 patterns, including information exploration and judgment, diagnosis and error correction, construction and work progress, verification and assurance, and strategy and execution coordination.Source: §3 · p. 6

4 EXPERIMENTS

TerminalHorizon-27B outperforms its Qwen3.5-27B base model across all 12 evaluated benchmarks. Analyses report gains across task categories and model sizes, while RSI-Exam cases illustrate feedback-driven revisions retained in final submissions.

4.1 EXPERIMENTAL SETUP

  • Evaluation covers 12 benchmarks across terminal task solving, real-world productivity, and general reasoning; Qwen3.5-27B is fine-tuned on TerminalHorizon-3K.The reported implementation uses a 262,144-token maximum sequence length and trains for three epochs.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • TerminalHorizon-27B outperforms Qwen3.5-27B on all 12 benchmarks spanning terminal tasks, real-world productivity, and general reasoning.Tables 1 and 2 report the main results.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 8
  • Terminal-Bench 1.0, 2.0, and 2.1 scores are 55.0%, 55.8%, and 60.67%, improving over the base model by 8.7, 14.2, and 13.85%, respectively.TerminalHorizon-27B also outperforms Terminal-Universe-27B by 3.0% using fewer than one-tenth as many trajectories.Source: §4.2 · p. 7, Table 1 · p. 7
  • AutomationBench Pass Rate and Partial scores improve by 11.7% and 23.3%, VitaBench by 11.3%, and τ 2-bench Airline and Retail by 5.5% and 2.6%.The reported gains span all four agentic benchmarks.Source: §4.2 · p. 8, Table 2 · p. 8
  • MATH-500 reaches 98.9%, while AIME 2024 and AIME 2025 reach 96.7% and 93.3%, improving over the base model by 3.4% and 6.6%, respectively.GSM8K and KOR-Bench also improve by 1.0% and 1.1%.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 ANALYSIS AND ABLATION STUDIES

  • DeepSeek-V4-Pro produces longer feedback chains and less exact action repetition than GLM-5.2, with these interactions associated with higher task completion.This comparison informed its selection as the trajectory-collection teacher.Source: §4.3 · p. 8, Table 3 · p. 8
  • Across three model configurations, TerminalHorizon training yields double-digit percentage-point gains over the corresponding base models.The reported gains are 13.85%, 13.11%, and 14.98%.Source: §4.3 · p. 8, Figure 5 · p. 9
  • Scaling from 1,000 to approximately 3,000 training trajectories raises Terminal-Bench 2.1 performance to 60.67%, with no sign of saturation.The authors suggest further expansion may yield additional gains.Source: §4.3 · p. 8, Figure 6 · p. 9

5 RELATED WORK

Prior work addresses long-horizon continuity through agent harnesses and terminal-task synthesis through coverage, evidence, or complexity. This paper emphasizes connections among behavioral patterns across stages of a shared goal.

5 RELATED WORK

  • Long-horizon agent harnesses support continuity through context, memory, and state management, while this work studies training for sustained problem solving.Its focus is on patterns connecting exploration, implementation, and verification around a shared goal.Source: §5 · p. 9
  • Terminal-task synthesis uses skill composition, technical evidence, real-world needs, and recursive reference-solution expansion to broaden coverage or complexity.The passage cites prior approaches using these strategies.Source: §5 · p. 9
  • The paper emphasizes less-studied cross-stage connections: findings inform decisions, feedback guides strategy revision, and intermediate results support delivery.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents TerminalHorizon as an automated engine for synthesizing long-horizon trajectories, with reported benchmark gains and RSI-Exam evidence of sustained, feedback-driven work. The section also records the study’s AI-use disclosure, ethical-scope statement, and reproducibility pointers.

6 CONCLUSION

  • TerminalHorizon links exploration, strategy revision, and verification around a shared goal; training on its trajectories improves performance on all 12 benchmarks.RSI-Exam cases show sustained experimentation, feedback-driven improvement, and progress retained through final delivery.Source: §6 · p. 10
  • Generative AI was used for automated environment construction, trajectory generation, language polishing, and selected illustrations, with manual review of AI-assisted content.The authors state they take responsibility for the final content.Source: AI USE STATEMENT · p. 10
  • The authors report no significant ethical concerns requiring specific disclosure within the scope of the study.Source: ETHICS STATEMENT · p. 10
  • Reproducibility materials include synthesis and trajectory procedures, training configuration, benchmark protocols, filtering, decontamination, prompts, and RSI case-study details.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 TRAINING DETAILS

  • A shared SFT hyperparameter table is provided for all three model sizes.Source: Table 4 · p. 16

A.2 EVALUATION PROTOCOLS

The evaluation protocols use benchmark-specific task sets, agent frameworks, repetition, and scoring procedures to assess terminal work, real-world automation, and general reasoning.

Terminal-Bench

  • Terminal-Bench 1.0, 2.0, and 2.1 use independent task runs and mean pass@1, with three hours allowed per task.The versions contain 80, 89, and 89 tasks, respectively; the 2.0 and 2.1 evaluations use E2B sandboxes.Source: §A.2 · p. 16

AutomationBench

  • AutomationBench reports mean Partial score and Pass Rate across 600 public-set rollouts, with three independent model runs.Trajectories have a 50-step limit; samples remaining incomplete after retries receive zero.Source: §A.2 · p. 16

VitaBench and τ 2-bench

  • VitaBench Cross-domain reports Avg@4 over 400 runs, while τ 2-bench reports mean task success across repeated runs in Airline and Retail.VitaBench covers 100 tasks; τ 2-bench covers 50 Airline and 114 Retail tasks.Source: §A.2 · pp. 16–17

General reasoning

  • General reasoning evaluation reports mean accuracy over three runs on MATH-500, GSM8K, AIME 2024, AIME 2025, and KOR-Bench Cipher.Each problem receives one generated answer per run, scored using the benchmark’s answer-matching procedure.Source: §A.2 · p. 17

B.1 PERFORMANCE ACROSS TASK CATEGORIES

TerminalHorizon-27B’s Terminal-Bench 2.1 gains span task categories, and its Terminal-Bench 2.0 score is high relative to the training-trajectory count.

B.1 PERFORMANCE ACROSS TASK CATEGORIES

  • TerminalHorizon-27B matches or exceeds Qwen3.5-27B in all 16 Terminal-Bench 2.1 categories, improving in 11, with gains of 46.7, 25.0, and 22.2 percentage points in debugging, data science, and machine learning.The reported gains also extend to system administration, security, and data processing.Source: Figure 8 · p. 17, §B.1 · p. 18

B.2 DATA EFFICIENCY

  • 55.8%: TerminalHorizon-27B achieves the highest Terminal-Bench 2.0 score among shown models with approximately 3k trajectories, exceeding Terminal-Universe-27B by 3.0 percentage points with fewer than one-tenth as many trajectories.The closer comparison uses the same Qwen3.5-27B base model and Terminus-2 harness: approximately 3k versus 32k trajectories.Source: Figure 9 · p. 18, §B.2 · p. 18

C ADDITIONAL LONG-HORIZON TASK APPLICATIONS

The four RSI-Exam6 cases examine whether agents sustain research by using evidence to revise methods and preserve effective changes in final submissions. The section also describes domain coverage and a behavioral-pattern taxonomy.

C ADDITIONAL LONG-HORIZON TASK APPLICATIONS

  • The four case studies use one 12-hour research run per model per case, and final scores are normalized rewards scaled to 0–100.Agents can inspect materials, revise implementations, and run development experiments, but hidden evaluation results are withheld during research.Source: C ADDITIONAL LONG-HORIZON TASK… · pp. 18–19, Table 5 · p. 19

C.1 WAREHOUSE ROBOT ROUTING AND MACRO OPTIMIZATION

  • 39.72 versus 14.63: TerminalHorizon-27B’s final hidden-set normalized score exceeds the base model’s in warehouse routing and macro optimization.The base model records a best public cost of 286.05 but submits at 466.35; TerminalHorizon-27B submits at 281.11 public mean cost after selecting routes by compressed button count.Source: §C.1 · p. 19, Figure 10 · p. 20

C.2 TIDAL SEABED FRICTION FIELD INVERSION

  • 0.009386 versus 0.021873: TerminalHorizon-27B’s hidden-case mean tidal-elevation RMSE is lower than the base model’s, a reduction of approximately 57.1%.Its method combines spatial coefficients and depth features, adjacent-coefficient regularization, Huber loss, and removal of two high-residual gauges.Source: §C.2 · pp. 20–21, Figure 11 · p. 21

C.3 2048 STRATEGY OPTIMIZATION

  • 38,626.25 versus 23,183.75: TerminalHorizon-27B’s mean score on hidden 2048 games exceeds the base model’s, with normalized scores of 45.25 and 37.54.The agent revises the evaluation function and selectively reverts candidates that regress on the full public test set.Source: §C.3 · pp. 22–23, Figure 12 · p. 22

C.4 MULTI-VIEW POINT-CLOUD FUSION

  • 77.68 versus 48.04: TerminalHorizon-27B’s mean hidden-scene point-cloud F-score exceeds the base model’s on a 0–100 scale.The solver uses cross-view alignment and refinement rather than restricting added points to the first view; the figure shows five additional scenes.Source: Figure 13 · p. 23, §C.4 · p. 24

C ADDITIONAL LONG-HORIZON TASK APPLICATIONS

  • Across the cases, sustained research involves optimizing coupled objectives, revising models, testing and reverting heuristics, and identifying misleading proxy metrics.The cases connect observations and subsequent decisions to the implementation ultimately submitted; longer interaction is useful when supporting those connections.Source: §C.4 · p. 24

D PROFESSIONAL DOMAIN COVERAGE

  • The 1,535 environments cover 25 professional domains, extending beyond software and infrastructure to education, agriculture, healthcare, finance, and scientific research.The terminal is a shared interaction interface; domain coverage describes the work agents do, not how they organize it.Source: D PROFESSIONAL DOMAIN COVERAGE · p. 24, Figure 14 · p. 25

E.1 DEFINITIONS AND TAXONOMY

  • The paper defines an agentic behavioral pattern as a structure connecting actions, observations, and subsequent decisions around a goal, and groups 27 patterns into five categories.The categories describe aspects of problem-solving behavior, not mutually exclusive task types or fixed execution stages.Source: §E.1 · p. 25

E.2 ORGANIZATION OF BEHAVIORAL PATTERNS WITHIN TRAJECTORIES

Three annotated trajectories illustrate how observations, decisions, and subsequent work connect behavioral patterns within problem solving; these examples are qualitative, not frequency or duration estimates.

E.2 ORGANIZATION OF BEHAVIORAL PATTERNS WITHIN TRAJECTORIES

  • Figure 15 links selected episodes to behavioral patterns across a TLS gateway repair, a neutron-diffraction study, and an event-driven staffing task.Horizontal spacing represents episode order, not duration.Source: §E.2 · p. 25, Figure 15 · p. 26
  • In the TLS repair, a logged 84-byte response with no downstream body redirects investigation toward request consumption and connection closure.The final evidence is a passing local regression suite, not an exhaustive check of every TLS requirement.Source: §E.2 · p. 27
  • In the neutron-diffraction case, new warming and cooling measurements guide tests of thermal-path dependence and model selection before the fit is committed with all 13 run identifiers.The trajectory does not expose the independent verifier’s refit results.Source: §E.2 · p. 27
  • In the staffing case, six successive schedule windows accumulate constraints, and validation plus a separate SQL calculation identify overloaded panels before correction and revalidation.The final status records all six windows as committed; the SQL calculation and validator use different checking paths but share database inputs.Source: §E.2 · p. 27

Verification and result establishment

  • Verification patterns include combining distinct checks, comparing results with explicit expectations, cross-checking through distinct derivations, and repeating fixed-setup runs.The taxonomy also includes revalidation after key conditions change, requirement reconciliation, and matching actual results to requirements or gaps.Source: Verification and result establishment · p. 28

Strategy and execution coordination

  • Strategy and coordination patterns include revising methods in response to information, branching and backtracking, coordinating linked objects, managing asynchronous jobs, and adapting actions to changing conditions.Across cases, episodes and patterns can connect without following a fixed sequence of behavioral stages.Source: Strategy and execution coordination · p. 28

F TRAJECTORY FILTERING CRITERIA

The section defines process-oriented criteria for selecting useful multi-turn trajectories and a task-level procedure for checking training tasks against benchmarks for reuse.

F TRAJECTORY FILTERING CRITERIA

  • Trajectory review weighs integrity, environment-grounded progress, and connected multi-turn learning value rather than task success or turn count alone.Recoverable local errors and unfinished work can be retained; systematic defects, prohibited shortcuts, and unproductive interactions are grounds for rejection.Source: F TRAJECTORY FILTERING CRITERIA · pp. 28–30

G DETAILED BENCHMARK DECONTAMINATION

  • Decontamination compares all 1,535 training tasks with benchmark instances using full task instructions and problem statements or user requests.The comparison excludes reference answers, private scoring criteria, and expected execution outcomes.Source: G DETAILED BENCHMARK DECONTAMINATION · p. 30, §G.1 · p. 30

G.1 COMPARISON CORPORA

  • Comparison corpora preserve task objectives and context while excluding answers and private evaluation information.They cover 3,251 evaluation instances across 12 benchmark groups.Source: §G.1 · p. 30

G.2 TEXT OVERLAP AND NEAR-DUPLICATE RETRIEVAL

  • Exact full-text and 13/14-token overlap identify verbatim spans, while 5-gram Jaccard and TF–IDF cosine retrieve near-duplicate candidates.Math-aware 8-gram overlap supplements retrieval; it does not establish contamination, and unigram JSD is descriptive only.Source: §G.2 · p. 30

G.3 SEMANTIC ASSESSMENT OF CANDIDATE PAIRS

  • Semantic review compares complete task descriptions, treating direct copies or minor paraphrases as contaminated but shared tools or domain terminology alone as insufficient.Multiple retrieval methods select candidate pairs for review, which may be labeled non-contaminated, contaminated, or uncertain.Source: §G.3 · pp. 30–31

G.4 DECONTAMINATION RESULTS

Decontamination review finds no evidence of benchmark-task reuse in the training set. Three reconstruction case studies show how independent challenges exposed work that environments had initially completed, simplified, or failed to trigger.

G.4 DECONTAMINATION RESULTS

  • 190 candidate pairs underwent semantic review, with none classified as contaminated or uncertain; the authors retain all 1,535 training tasks.Across 12 benchmark groups, there were no exact full-text or 13/14-token matches; the maximum 5-gram Jaccard and TF–IDF cosine similarities were 0.006024 and 0.083445, respectively.Source: §G.4 · p. 31

H CASE STUDIES OF CHALLENGE-GUIDED RECONSTRUCTION

H.1 RADIO-INTERFEROMETRIC CALIBRATION

  • The initial calibration task prepackaged numerical fitting, so a challenger passed without implementation; reconstruction left fitting to the agent, whose independent run revised parameters from residual feedback and passed validation.The reconstructed run's weighted RMS was 0.896 on fitted calibrators, 1.784 on holdout, and 1.944 on target, all below the public threshold of 2.8; these are not a comparison between task versions.Source: §01 · p. 32, §02 · p. 32, §03 · p. 32

H.2 RABBITMQ RECOVERY ACROSS BROKER AND APPLICATION STATE

  • RabbitMQ recovery initially reduced to healing a partition and restarting a broker; successive redesigns linked broker repair to ordered application workflows, but repeated challenges still avoided reconciliation until the fault exposed a missing predecessor.In the final run, all 216 outbox rows were confirmed, every workflow reached step 8, and no duplicate business effects were observed.Source: §01 · p. 32, §02 · p. 33, §03 · p. 33

H.3 FAMILY ENROLLMENT CHANGES AND PAYROLL RECONCILIATION

  • Benefits enrollment began as a fixed sequence of updates; versioning alone did not elicit evidence analysis, so reconstruction added conflicting claims whose review and validation changed eligibility decisions and downstream records.The final run approved D-4, denied D-3 and D-5, preserved age-bounded coverage for D-2, applied a $312 payroll adjustment, and obtained insurer acceptance while retaining version 5.Source: §01 · p. 33, §03 · p. 34, §04 · p. 34

I.1 TWO BRANCHES FROM THE SAME PHOTO-LIBRARY TASK

The examples show progressive task evolution adding decisions that depend on earlier work, while the appendices provide prompts for trajectory filtering and benchmark decontamination.

I.1 TWO BRANCHES FROM THE SAME PHOTO-LIBRARY TASK

  • Two photo-library branches extend a shared reconstruction task in different directions: one adds state reconciliation and file-group decisions, while the other adds identity recovery across systems.The final identity branch requires matching files before deciding which edits to merge, then tests stability through native indexing and rescanning.Source: §I.1 · pp. 34–35, Table 10 · p. 35

I.2 HOTEL EVENT SCHEDULING

  • Hotel-planning generations add attendee flows, incident recovery, and resource logistics while retaining the same ten activities.A venue closure led to three coordinated venue changes; equipment readiness later forced a rejected schedule and a revised start time.Source: §I.2 · pp. 35–36, Table 11 · p. 36

J.1 TRAJECTORY FILTERING PROMPT

  • The trajectory-filtering prompt directs reviewers to inspect complete trajectories and make evidence-backed KEEP/DROP decisions.It implements the three filtering criteria described in Appendix F.Source: §J.1 · p. 36

J.2 BENCHMARK DECONTAMINATION PROMPT

  • The decontamination prompt compares complete task descriptions and treats similarity scores as retrieval signals, not verdicts.Its criteria distinguish direct copies or substantially equivalent renamed tasks from shared domain, tools, or terminology.Source: §J.2 · p. 38

Output

  • The review output requires one record per pair with its exact pair ID, a CLEAN, CONTAMINATED, or UNCERTAIN verdict, and comparative evidence.The instructions require an atomic file rename and checks for parseability, missing records, and duplicates.Source: Output · pp. 38–39

Related papers

Loading 2610.13166v1…