PaperlayerView exact revision on arXiv

Source-linked AI summary

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

Fanzhe Meng, Guoxin Chen, Jiale Zhao, Shuang Sun, Zhiyu Lin, Wayne Xin Zhao, Ruihua Song, Ji-Rong Wen, Kai Jia

arXiv:2608.06352v1cs.LGcs.CL

TL;DR

Executable validation does not show whether terminal tasks are appropriately challenging for learning. CalibForge uses adversarial solver calibration to construct solver-relative tasks, and its calibrated supervision outperforms authoring and validation alone or single-solver feedback.

  • Problem

    Executable validity alone does not reveal whether a terminal task is appropriately challenging for learning rather than trivial or effectively unsolvable.

  • Method

    CalibForge uses verified solver outcomes and trajectories in a constrained adversarial author–solver loop to revise tasks toward multi-solver disagreement or strong-pass/weak-fail criteria.

  • Results

    Adversarial calibration improves downstream supervision over authoring and validation alone or single-solver feedback, with calibrated models reaching 32.58% and 47.57% on Terminal-Bench 2.0.

  • Takeaways & Limitations

    Solver-relative learnability is a practical target for constructing effective and transferable terminal-agent training data.

Abstract

from arXiv · show

Training terminal agents requires executable and verifiable tasks that are not merely solvable, but appropriately challenging for learning. Executable validation establishes feasibility, yet does not reveal how a task behaves relative to a given solver setting. In this paper, we present CalibForge, an autonomous terminal-task synthesis system that uses verified solver behavior to revise candidate tasks through adversarial solver calibration. Multi-solver calibration targets disagreement within a heterogeneous solver pool, whereas contrastive solver calibration targets a designated strong-pass/weak-fail relation; both operationalize a solver-relative learnable zone anchored in demonstrated solvability. Using CalibForge, we construct 5,431 calibrated terminal tasks. Our ablations show that both strategies yield more effective supervision than authoring and validation alone or ordinary single-solver feedback. Models trained on the full collection achieve 32.58% and 47.57% on Terminal-Bench 2.0. The largest improvements over the corresponding base model reach 24.71 percentage points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo. Together, these results support solver-relative learnability as a practical target for constructing effective and transferable agent training data.

1. Introduction

CalibForge treats terminal-task construction as an adversarial author–solver loop because executable validity alone does not establish solver-relative learnability. It uses multi-solver or contrastive calibration to revise validated candidates based on solver behavior toward desired outcome patterns.

  • Motivation: Executable validity alone does not reveal whether a terminal task is appropriately challenging for learning rather than trivial or effectively unsolvable.This motivates construction-time calibration based on solver behavior.
  • CalibForge: CalibForge jointly authors task instructions, execution environments, and verification tests, then structurally validates and self-solves candidates before adversarial calibration.Solver outcomes and trajectories guide multi-round revision and re-probing.
  • Calibration strategies: Multi-solver calibration targets disagreement within a heterogeneous solver pool, while contrastive calibration targets a designated strong-pass/weak-fail relation.Both strategies use verified solver behavior to determine retention and guide revision.
  • Results: 19% of candidates initially satisfy the contrastive target relation, but revision and re-probing increase cumulative acceptance to 96%.All candidates entering contrastive calibration had already passed structural validation and self-solving.

2. Methodology

CalibForge begins by constructing and validating a candidate terminal task, then attempts to solve it before entering an adversarial author–solver calibration loop. The loop is governed by a calibration specification and returns structured feedback with verified solver outcomes.

  • Candidate authoring: The authoring agent constructs a candidate terminal task τ from a clue and attempts to solve it.This initial construction-and-solving stage precedes adversarial calibration.
  • Candidate validation: V(τ) indicates whether the candidate passes both structural validation and the solving attempt.Candidates enter calibration only after these checks pass.
  • Adversarial calibration: Valid candidates enter an author–solver loop for adversarial solver calibration governed by specification γ.The specification defines the solver setting and behavioral retention criterion Cγ.
  • Calibration feedback: Each calibration round returns structured feedback summaries containing verified pass/fail outcomes.These outcomes provide the solver-behavior feedback used within the calibration loop.

I. Task Authoring & Validation

CalibForge authors candidate terminal tasks from researched engineering problems by jointly constructing the instruction, execution environment, and verification tests. Candidates undergo structural validation and self-solving, with failed candidates repaired and reevaluated before proceeding.

  • Task Authoring: The authoring agent researches concrete engineering problems using official documentation, GitHub repositories, issue discussions, and Stack Overflow.It works in a draft sandbox equipped for web research, shell execution, and file editing.
  • Task Authoring: Each candidate combines a task instruction, Dockerfile-based execution environment with initial files, and verification tests.The initial environment must not expose solution artifacts, and tests must match the instruction without unstated requirements.
  • Validation: Candidates pass two validation stages—structural validation and self-solving—before entering solver calibration.Structural checks cover required build-context files, environment construction and initialization, while self-solving verifies executable solutions and consistency among instruction, environment, and tests.
  • Validation: Failed candidates are repaired and reevaluated through both stages; V(τ) = 1 if and only if both structural validation and self-solving pass.Only candidates with V(τ) = 1 proceed to adversarial solver calibration.

3. Experiments

CalibForge produces broad, calibrated terminal-task supervision that improves Terminal-Bench performance and transfers to repository-level software-engineering benchmarks. Adversarial solver calibration outperforms single-solver feedback by targeting solver-relative difficulty rather than validity alone.

  • Terminal-Task Performance: 32.58% and 47.57%: CalibForge-30B-A3B and CalibForge-35B-A3B outperform the strongest shared-protocol baselines by 6.36 and 6.75 points on Terminal-Bench 2.0.All task sets use the same distillation and training protocol, making training-data source the primary experimental difference.
  • Cross-Benchmark Transfer: 27.68 and 3.03 points: CalibForge-30B-A3B and CalibForge-35B-A3B improve over their respective base models on SWE-bench Pro.The corresponding Doc2Repo improvements are 30.04 and 3.85 points.
  • Scale and Domain Coverage: 5,431 tasks: the collection contains 1,263 multi-solver-calibrated and 4,168 contrastively calibrated tasks spanning all 16 taxonomy categories.The composition broadens terminal supervision beyond narrow domain concentrations in several existing task sets.
  • Capability Diversity: 3,885 distinct capability tags: tasks have a median of five tags, with 51.6% of tags appearing in only one task, indicating long-tailed capability diversity.Domain coverage, capability variety, and environment and verification structure capture complementary dimensions of terminal-task diversity.
  • Ablations: 8.62 and 6.74 points: contrastive and multi-solver calibration improve Terminal-Bench 2.0 accuracy, compared with 1.87 points for single-solver feedback.The gains are not explained by trajectory volume alone, because multi-solver calibration produces fewer retained SFT trajectories than the no-solver variant.
  • Calibration Analysis: 19%: only this share of structurally validated and self-solved candidates initially exhibit the target strong-pass/weak-fail relation, while 81% do not.Calibration revisions correct many mismatches early, with 53% of retained runs completed within five probes and 93% within twenty.

4. Related Work

Related work spans executable terminal-agent benchmarks, verifiable task synthesis, and behavioral feedback for adaptive data construction. CalibForge differs by using verified solver outcomes and interaction feedback to revise the task itself.

  • Terminal Agents and Benchmarks: Terminal-agent benchmarks evaluate interactive software agents with executable tasks and verifiers.These environments include code, repositories, and stateful command-line workspaces.
  • Terminal Agents and Benchmarks: Executable terminal tasks jointly specify instructions, initial files, dependencies, and verification tests, making their construction and validation essential.
  • Verifiable Terminal-Task Synthesis: Existing synthesis methods generate terminal tasks from specifications, capability taxonomies, software artifacts, repositories, agent trajectories, skills, and skill graphs.
  • Behavioral Feedback for Data Construction: CalibForge applies solver behavior to the task under construction rather than only adapting benchmarks, using verified pass/fail outcomes for retention and feedback for revision.Feedback includes summaries and full interaction trajectories.

5. Conclusion

CalibForge uses solver behavior as construction-time feedback for autonomous terminal-task synthesis through multi-solver and contrastive adversarial calibration. Fine-tuning on trajectories from calibrated tasks improves Terminal-Bench 2.0 performance and transfers to repository-level software-engineering benchmarks, with ablations attributing gains to calibration strategies rather than data volume or single-solver feedback.

  • CalibForge is an autonomous terminal-task synthesis system that turns solver behavior into construction-time feedback.
  • Its two adversarial calibration strategies are multi-solver calibration and contrastive calibration.
  • Fine-tuning two Qwen backbones on trajectories distilled from 5,431 calibrated tasks improves Terminal-Bench 2.0 performance and transfers to repository-level software-engineering benchmarks.
  • Matched ablations attribute the gains to the two calibration strategies rather than trajectory volume or single-solver feedback.

A. From a Clue to a Calibrated Task … Media and network formats

CalibForge develops a terminal task from a data-processing clue by searching broadly across recovery directions and selecting candidates based on depth, novelty, tool specificity, and domain fit. The example proceeds through tabular, database, archive, and media/network possibilities before retaining a candidate for contrastive solver calibration.

  • A. From a Clue to a Calibrated Task: The clue targets data processing and ETL, with pipelines, schema evolution, log analysis, data I/O, binary parsing, and data recovery using xxd or hexdump.This clue is the starting point for developing a candidate terminal task.
  • A. From a Clue to a Calibrated Task: The authoring agent issued 24 search calls spanning major directions before selecting a task-development path.The process is described as wide search and direction selection.
  • Tabular and columnar data: Parquet repair appeared too shallow, while searches considered CSV edge cases and corrupted Parquet metadata.These examples represent the tabular and columnar data direction.
  • Database and repository recovery: Database and repository recovery considered SQLite/WAL salvage and damaged Git objects.The alternatives span database recovery and repository-object recovery.
  • Archive and text recovery: Git recovery was too close to existing tasks, SQLite/WAL recovery was too tool-specific, and archive/text options included truncated gzip, mixed encodings, and malformed NDJSON.These judgments narrowed the candidate space while retaining several archive and text-recovery directions.
  • Media and network formats: Media and network options included PCAP extraction, EXIF/JPEG repair, and MP3/ID3 recovery, but format-specific repair risked a single utility invocation and PCAP seemed too security-oriented.The example ultimately concerns retaining a candidate through contrastive solver calibration.

Legacy and heterogeneous data · Solver self-assessments · B. How Solver Feedback Revises Tasks

The paper presents a calibrated legacy-format ETL task that recovers sensor readings from corrupted binary data under explicit integrity and validity checks. Stronger and weaker solvers respectively pass and fail, with parsing and combined CRC/range validation identified as the main difficulty.

  • Legacy and heterogeneous data: The task recovers environmental readings from a corrupted binary sensor log using record synchronization and checksum validation.It targets fixed-width/COBOL-style, multi-format ETL through a search-grounded recovery problem.
  • Legacy and heterogeneous data: The required output is /app/recovered_readings.csv with timestamp, temperature, humidity, pressure, and co2 columns sorted by timestamp.The instruction says not to trust the header count and to discard invalid-checksum or physically impossible records.
  • Legacy and heterogeneous data: The input includes intact, truncated, and overwritten records plus a format specification defining layout, synchronization, CRC-16, and physical validity ranges.The environment places the binary log and specification under /app in Ubuntu 24.04 with Python available.
  • Legacy and heterogeneous data: Verification uses 11 tests covering schema, record count and order, overwritten-interval exclusion, value ranges, boundary cases, and selected expected records.The verifier checks both structural output requirements and recovery correctness.
  • Legacy and heterogeneous data: DeepSeek-V4-Pro passes the task while DeepSeek-V4-Flash fails, establishing the intended contrastive solver relation.The calibration feedback labels the former as the stronger solver and the latter as the weaker solver.
  • Solver self-assessments: The stronger solver rates the task Easy–medium, whereas the weaker solver rates it Medium.The stronger solver found the specification clear but encountered an inconsistent example CRC; the weaker solver struggled with binary parsing and combined CRC/range validation.
  • B. How Solver Feedback Revises Tasks: Structured feedback and full interaction trajectories guide task revisions during adversarial solver calibration, followed by revalidation and re-probing.Each revision case records initial outcomes, feedback used by the authoring agent, the revision, and post-revision outcomes.

B.1. Removing Procedural Hints after Both Solvers Pass … CalibForge-Eval User Prompt Template

The calibration cases revise tasks by diagnosing solver behavior, ambiguity, or verifier brittleness, while CalibForge-Eval supplies a compact terminal-agent interface and prompt framework for verified execution. Across revisions, outcomes are retained only when they meet the intended contrastive or multi-solver criterion.

  • B.1. Removing Procedural Hints after Both Solvers Pass: Both solvers passed the transaction-repair task, so removing disclosed corruption patterns and the worked repair path made success depend on diagnosing malformed records.DeepSeek-V4-Pro passed in 8 steps and DeepSeek-V4-Flash in 17 steps before revision; afterward, the stronger solver passed and the weaker failed.
  • B.2. Clarifying Comparison Semantics after All Solvers Fail: All three solvers failed the customer-export comparison because the instruction underspecified whether extra columns constituted record modifications or schema changes.The task compared normalized reference and current exports, but all solvers produced the same extra record-level classification.
  • BEFORE: After clarifying that only shared fields determine record modifications, while one-sided columns are schema changes, GLM-5 and DeepSeek-V4-Flash passed but Kimi-K2.5 failed.This mixed outcome resolved ambiguity without simplifying the underlying data-processing problem and satisfied the multi-solver retention criterion.
  • B.3. Generalizing an Overly Prescriptive Verifier after an Inverted Outcome: In the password-vault task, the stronger solver failed while the weaker passed because an overly prescriptive verifier rejected a valid whole-record encryption layout.The verifier expected separately encrypted named fields, whereas the stronger solver used authenticated encryption over a combined record.
  • BEFORE: Generalizing the verifier to discover encrypted payloads independently of field names preserved security checks and legacy migration while allowing alternative valid vault layouts.A subsequent round produced a stronger-solver pass and weaker-solver failure, satisfying the contrastive retention criterion.
  • C.1. Tool Interface: CalibForge-Eval is a minimal AweAgent-based code-agent scaffold for trajectory distillation and Terminal-Bench 2.0 evaluation, exposing bash, file editing, and finish tools.Its compact interface follows the DeepSeek-V4 code-agent evaluation setting.
  • CalibForge-Eval System Prompt: The system prompt directs agents to inspect requirements and the environment, use existing workflows, edit precisely, handle long-running or missing-dependency operations, and verify artifacts before finishing.It also instructs agents to debug failed checks, avoid destructive or hard-coded solutions, and call the finish tool only after reaching a verified final state.
  • CalibForge-Eval User Prompt Template: The user prompt template specifies the working directory and states that the runtime environment is evaluated after the agent calls finish.This template provides the task-specific execution context without adding further procedural requirements.

D. Benchmark Decontamination · E. Supervised Fine-Tuning Details

CalibForge removes candidates showing textual, code, or structural overlap with evaluation benchmarks before training. The student models are trained with full-parameter multi-turn SFT using shared hyperparameters and hardware, with final checkpoints reported after 10 epochs.

  • D. Benchmark Decontamination: Candidates with an exact 14-gram instruction match to evaluation instances are removed before trajectory distillation and model training.The comparison covers Terminal-Bench 2.0, SWE-bench Pro, and Doc2Repo.
  • D. Benchmark Decontamination: CalibForge computes 5-shingle Jaccard similarity over task instructions and available verifier or test code.Instructions are lowercased before shingle computation.
  • D. Benchmark Decontamination: Structural contamination checks include shared output paths, overlapping test functions, and high-risk task-family matches.These signals complement surface-level instruction similarity.
  • D. Benchmark Decontamination: Any candidate flagged against at least one evaluation benchmark is removed before trajectory distillation and model training.The three benchmarks are Terminal-Bench 2.0, SWE-bench Pro, and Doc2Repo.
  • E. Supervised Fine-Tuning Details: Qwen3-30B-A3B-Instruct and Qwen3.5-35B-A3B are trained with full-parameter, multi-turn supervised fine-tuning implemented using LLaMA-Factory.Both models use the shared configuration in Table E1.
  • E. Supervised Fine-Tuning Details: 10 epochs are used, and the final checkpoint is reported for each student model.The shared training configuration applies to both models.
  • E. Supervised Fine-Tuning Details: Table E1 lists the supervised fine-tuning hyperparameters and hardware used for both CalibForge student models.The table provides the shared implementation configuration referenced for the two models.

F. Failure Analysis of Trained Models … F.3. Changing State before Preserving Recovery Evidence

Failure analysis of three Terminal-Bench 2.0 evaluation runs reveals distinct mechanisms behind aggregate failures: agents explored solutions or committed partial evidence without producing valid, verified artifacts. Larger-model runs passed by materializing required files, integrating evidence, and checking all constraints.

  • F. Failure Analysis of Trained Models: Three evaluation runs of CalibForge-30B-A3B and CalibForge-35B-A3B expose failure mechanisms hidden by aggregate benchmark scores.The trajectories were examined under the same CalibForge-Eval setting.
  • F.1. Reasoning without Producing the Required Artifact: CalibForge-30B-A3B repeatedly explored regex candidates but never wrote /app/regex.txt before timing out, whereas CalibForge-35B-A3B wrote and tested the selected expression.The successful runs used Python re.findall, including multiple dates and invalid IPv4-like strings.
  • Case G1: regex-log: All three 30B regex runs failed because /app/regex.txt was absent, while all three 35B runs passed the verifier.This result directly distinguishes artifact production from solution exploration.
  • Diagnosis: Useful intermediate regex reasoning was insufficient because the agents did not commit the final artifact and check it through the specified interface.The case separates solution exploration from task completion.
  • F.2. Committing to Partial Forensic Evidence: Two 30B password runs misread the 23-character constraint by including PASSWORD=, while a third produced no output file before timing out.The incorrect artifacts contained PASSWORD=8XDK7VB3BV4W54.
  • F.3. Changing State before Preserving Recovery Evidence: Successful password recovery required integrating evidence from separate image regions and rechecking the combined result against every stated constraint.The failed runs committed to a locally plausible fragment after misreading a global length constraint.

Case G3: db-wal-recovery

In db-wal-recovery, all solver runs failed because opening the database before preserving and decrypting its WAL destroyed access to six additional records. The case demonstrates that plausible partial progress cannot substitute for verifier-confirmed recovery.

  • Task requirement: The task required repairing an encrypted SQLite WAL and writing all 11 records to /app/recovered.json, rather than only the five base-database records.The recovery artifact had to contain the complete record set.
  • Solver outcomes: CalibForge-30B-A3B and CalibForge-35B-A3B both failed in all three reported runs.The 30B runs timed out, while the 35B runs attempted reconstruction but did not recover the WAL contents.
  • Failure mechanism: Opening main.db first returned the five base records and removed the unreadable main.db-wal, preventing later inspection or repair of the six additional records.This premature application query altered the state needed for recovery.
  • Verifier evidence: The 30B outputs lacked the required recovery artifact, while the 35B outputs passed JSON and schema checks but failed record-completeness and WAL-decryption checks.Basic structural validity was therefore insufficient for verifier-confirmed recovery.
  • Failure analysis: This case illustrates premature commitment to partial evidence and mutable-state modification as distinct failure modes that trajectory inspection can expose beyond aggregate scores.The broader failure pattern also includes failing to produce the required artifact.
Loading 2608.06352v1…