PaperlayerView exact revision on arXiv

Source-linked AI summary

FACET: Preserving Source Intent and Executable State in Terminal Task Synthesis

Kou Shi, Zun Wang, Qisheng Su, Shiting Huang, Ziao Zhang, Zhen Fang, Qingnan Ren, Jin Liu, Yu Zeng, Yiming Zhao, Lin Chen, Zehui Chen, Feng Zhao

arXiv:2608.18580v1cs.AIcs.PL

TL;DR

Synthesizing executable terminal tasks risks losing source structure and producing inconsistent instructions, environments, solutions, and verifiers. FACET reconstructs scenarios and grounds all artifacts in a realized environment, while fine-tuning on its successful trajectories consistently improves Terminal-Bench 2.1 performance across Qwen3.5 model scales.

  • Problem

    Terminal-task synthesis can lose source capabilities, dependencies, states, and constraints while producing inconsistent artifacts that make tasks invalid.

  • Method

    FACET reconstructs coherent scenarios, realizes and repairs environments, and generates instructions, solutions, and verifiers against shared executable state.

  • Results

    Fine-tuning on FACET trajectories consistently improves Terminal-Bench 2.1 performance across Qwen3.5 models from 4B to 27B, including 8.24 points for 9B.

  • Takeaways & Limitations

    The results support source-intent preservation and shared executable-state grounding as principles for data-efficient terminal-agent supervision.

Abstract

from arXiv · show

Training terminal agents requires scalable executable supervision, yet synthesizing high-quality terminal tasks remains challenging. Each task couples an instruction, an initialized environment, a reference solution, and an executable verifier; if these artifacts are generated from inconsistent assumptions, the resulting task may be unsolvable or incorrectly evaluated. Meanwhile, multi-stage synthesis can discard the goals, dependencies, state transitions, and procedural constraints encoded in the original sources. We present FACET (Fine-grained Agentic Construction of Executable Tasks), a framework that addresses both information preservation and cross-artifact consistency. FACET reconstructs related agent skills into coherent, information-rich scenarios, then realizes and repairs the execution environment before generating the final task artifacts. The resulting container state serves as shared grounding for the instruction, solution, and verifier, while execution-based validation and targeted repair correct artifact-specific failures without unnecessarily regenerating valid components. FACET produces complex terminal tasks with dense executable checks, and successful trajectories collected from these tasks provide effective, data-efficient supervision. Fine-tuning models across multiple scales consistently improves performance on Terminal-Bench 2.1, while analyses of alternative generation schemes support the importance of environment-grounded construction for task validity and solution-verifier alignment. These results establish source-intent preservation and shared executable-state grounding as key principles for scalable terminal-task synthesis.

1 INTRODUCTION

FACET addresses two central problems in terminal-task synthesis: loss of source information and inconsistencies among tightly coupled task artifacts. It reconstructs source-grounded scenarios and coordinates instruction, environment, solution, and verifier through a shared executable state.

  • Challenges: Terminal tasks are tightly coupled bundles of instructions, initialized environments, reference solutions, and executable verifiers, so inconsistencies among components can invalidate the entire task.This challenge persists even as synthetic pipelines expand the number and diversity of source materials.
  • Challenges: Multi-stage synthesis can discard source capabilities, dependencies, intermediate states, input–output contracts, and procedural constraints, leaving tasks with only part of their original structure.The paper identifies source-information loss as a key challenge in preserving task intent.
  • FACET: FACET reconstructs coherent scenarios, cross-skill dependencies, intermediate states, and solution workflows before constructing and repairing the execution environment.This framework synthesizes complex, verifiable terminal tasks from heterogeneous reusable agent skills while retaining information from the original sources.
  • FACET: FACET coordinates task artifacts through a shared, realized environment, replacing independent generation of plausible components with construction of a coherent executable task as a whole.The executable-state-grounded paradigm provides common grounding for the task’s coupled artifacts.
  • FACET: FACET’s complete synthesis and validation pipeline produces reliable terminal-agent supervision, with analyses of generation design and post-training experiments across multiple model scales evaluating the resulting data.The contribution includes extensive analyses of task quality and experiments demonstrating the effectiveness of the synthesized supervision.

2 FACET

FACET preserves source intent by reconstructing rich, aligned scenarios and grounds all task artifacts in a shared executable environment. Execution-based validation and targeted repair enforce buildability, solvability, and verifier alignment.

  • Task formulation: A valid task bundle includes an instruction, environment, reference solution, executable verifier, and metadata, requiring a buildable environment and verifier-accepted final state.Acceptance also requires a non-trivial initial state and an executable reference solution.
  • Skill and scenario repository: FACET filters public skills into structured records and groups complementary, executable combinations into a validated scenario–skill repository.The process yields more than 71K valid skills and organizes retained tasks into nine families.
  • Scenario reconstruction: Stage 2 reconstructs richer compositional scenarios across five dimensions, preventing shallow workflows from omitting dependencies, intermediate states, and procedural constraints.The dimensions cover goals, context, capabilities, state, and required inputs, outputs, tools, services, and dependencies.
  • Reference alignment: FACET derives solution and instruction references from the integrated scenario, then checks shared initial states, target outcomes, supported requirements, and observable final effects.The references are generated as RS = fS(C) and RI = fI(C, RS).
  • Environment construction: Stage 3 materializes, initializes, and repairs the environment from failure traces before generating the executable task bundle.Repair continues until the environment is valid or the repair budget is exhausted.
  • Executable-state sharing: The realized container state grounds instruction, solution, and verifier generation, propagating environment changes to prevent inconsistent implicit versions across artifacts.Generators share access to actual files, services, schemas, resources, paths, ports, versions, and fixture contents.

3 EXPERIMENTS

FACET constructs dense, executable terminal tasks whose strict conjunctive checks expose near-successful failures and compositional difficulty. Fine-tuning on 1.2K successful trajectories improves Terminal-Bench 2.1 performance across model scales, while forward artifact generation yields stronger validity and cross-artifact alignment.

  • Dataset characteristics: FACET contains 1.2K training trajectories averaging 11.86 turns and 6,078 validated tasks averaging 22.77 executable tests per task.The trajectory length is close to TerminalWorld’s 11.94 turns, while executable-test density exceeds all compared datasets.
  • Terminal-Bench 2.1 performance: Qwen3.5-4B, 9B, and 27B improve by +7.12, +8.24, and +6.75 points, respectively, after fine-tuning on 1.2K successful trajectories.The 9B model has the largest absolute gain, while the 4B model has the largest relative improvement at 40.5%.
  • Terminal-Bench 2.1 performance: The fine-tuned Qwen3.5-27B reaches 47.57, just 1.49 points below Qwen3.5-397B’s 49.06 under the same evaluation setting.This result comes from a model roughly 15× smaller than Qwen3.5-397B.
  • Failure analysis: Although teacher rollouts satisfy 89.40% of individual checks, only 20.94% achieve full task success because task criteria are conjunctive.Among unsuccessful rollouts, 54.00% fail only one or two verifier checks, often due to a residual field, constraint, or deliverable.
  • Failure analysis: Task performance generally declines as instructions lengthen and verifier coverage broadens, with structured-data tasks solved more reliably than narrative-document tasks.The analysis attributes this pattern to longer instructions introducing more atomic, cross-file, and interdependent requirements.
  • Artifact-generation order: Forward generation initially validates 46.5% of tasks and reaches 83/100 after repair, outperforming Reverse at 24.2% initially and 63/100 finally.In paired comparisons, Forward succeeds where Reverse fails on 29 versus 9 pairs (p = 0.0017), supporting solution-before-verifier generation for cross-artifact alignment.

4 RELATED WORK

Prior work establishes realistic, containerized terminal benchmarks, shared task-packaging formats, and scalable pipelines for synthesizing terminal tasks. FACET builds on these directions by reconstructing heterogeneous skills into complex scenarios and coordinating task artifacts through a shared, realized environment state.

  • Terminal benchmarks and environments: Terminal-Bench 2.0 and 2.1 evaluate realistic containerized command-line tasks with execution-based verification, while TerminalWorld validates tasks from real terminal recordings.Harbor standardizes task packaging, agent execution, and rollout collection; FACET-generated tasks directly follow this format.
  • Scalable task synthesis: Existing scalable synthesis pipelines use procedural generation, domain specifications, seed datasets, skill composition, and requirement-driven retrieval.Agent Skills provide portable procedural knowledge, and SkillSynth organizes skills into scenario-mediated graphs.
  • Scalable task synthesis: FACET reconstructs complex scenarios from heterogeneous skills and coordinates task artifacts through a shared, realized environment state.This direction builds on prior work studying environment-grounded trajectory quality.

5 CONCLUSION · A DATASET DETAILS · A.1 SOURCE-SKILL TAXONOMY

FACET synthesizes complex, verifiable terminal tasks by preserving source information and grounding all task artifacts in a shared executable state. Experiments support environment-grounded construction and successful-trajectory supervision, while the dataset organizes retained skills and validated tasks into structured taxonomies.

  • 5 CONCLUSION: FACET reconstructs related skills into coherent scenarios, repairs the execution environment, and grounds instructions, solutions, and verifiers in one executable state.Targeted validation and repair improve consistency across task components while preserving source information.
  • 5 CONCLUSION: Sequential, environment-grounded construction achieves the highest task yield and improves solution–verifier alignment over generating the verifier before the solution.The comparison concerns alternative generation schemes analyzed in the experiments.
  • 5 CONCLUSION: Supervised fine-tuning on successful trajectories consistently improves Qwen3.5 models from 4B to 27B on Terminal-Bench 2.1.The resulting execution-based supervision is effective across multiple model scales.
  • 5 CONCLUSION: Coordinating scenario reconstruction, executable-state grounding, and artifact-level validation provides data-efficient supervision without relying solely on large-scale task generation.FACET is also planned for broader procedural sources, diverse interactive environments, reinforcement learning, and continued agent improvement.
  • AI USE STATEMENT: The authors used generative AI tools for drafting, language editing, and LaTeX preparation while retaining responsibility for source attributions, experimental records, numerical claims, and generated text.The statement says the authors checked these materials and take responsibility for the paper’s final content.
  • A.1 SOURCE-SKILL TAXONOMY: The dataset retains 71,341 skills organized into five top-level families and 34 fine-grained categories, with category counts and proportions computed over the complete corpus without sampling.The 6,078 validated tasks are separately classified into nine task families to characterize synthesized-dataset coverage.
  • A.1 SOURCE-SKILL TAXONOMY: Table 3 reports the distribution of source skills across the top-level categories.Its caption identifies the table as the top-level source-skill distribution.

A.2 TASK AND ROLLOUT ACCOUNTING

The final dataset comprises 6,078 tasks, with 6,074 recorded rollouts and a 20.94% teacher success rate after excluding infrastructure failures. Of the successful rollouts, 1,200 complete trajectories are selected for supervised fine-tuning, while 6,064 parseable runs support check-level analysis.

  • Task and rollout accounting: 6,078 tasks comprise the final dataset, including 6,074 rollout records; four jobs fail to produce records.Among recorded runs, 1,270 receive reward 1, 4,796 fail at least one requirement, and eight terminate from infrastructure errors.
  • Task and rollout accounting: 20.94% is the teacher success rate among 6,066 completed runs after excluding infrastructure failures.The rate is calculated as 1,270/6,066.
  • Task and rollout accounting: 6,064 completed runs have parseable pytest collection summaries and are included in the check-level analysis.Failed trajectories remain available for error analysis but are excluded from supervised fine-tuning.
  • Task and rollout accounting: 1,200 complete trajectories are selected from successful rollouts for the final supervised fine-tuning dataset.

A.3 TASK OUTCOMES AND SKILL-TAG VARIATION · A.4 CONSTRUCTION FUNNEL · A.5 DATASET-COMPARISON DETAILS

Task outcomes vary substantially across skill tags and task characteristics, while the construction funnel excludes candidates for validity and executability issues. Dataset comparisons separate statistics by their underlying units and computation procedures.

  • A.3 TASK OUTCOMES AND SKILL-TAG VARIATION: Task-level success is examined across skill tags and broader characteristics including output structure, instruction length, and verifier breadth.These groupings complement the aggregate task- and check-level results reported in Section 3.4.
  • A.3 TASK OUTCOMES AND SKILL-TAG VARIATION: Observed skill-tag pass rates range from 7.14% to 35.00%, with wide confidence intervals for some modestly sampled groups.Each task has seven overlapping tags, and tags co-vary with topic, output type, instruction length, and other tags, so estimates are descriptive rather than statistically independent.
  • A.3 TASK OUTCOMES AND SKILL-TAG VARIATION: Structured-data tasks have higher observed pass rates than narrative-document tasks, while success generally decreases as instructions lengthen or verifier suites broaden.Longer instructions add atomic requirements, output constraints, and cross-file dependencies; broader verifiers make single omissions more likely to fail.
  • A.4 CONSTRUCTION FUNNEL: Before validation, 58 candidate tasks are excluded for weak verifiers, ambiguous deliverables, unresolved external dependencies, or workflows permitting trivial bypasses.Table 5 summarizes candidates retained through environment construction and task validation.
  • A.4 CONSTRUCTION FUNNEL: Figure 4 presents strict task-level pass rates for the 20 most frequent skill tags among 6,066 completed rollouts, with Wilson 95% confidence intervals.Tags are selected by frequency and ordered by observed pass rate; the dashed line denotes the overall pass rate.
  • A.4 CONSTRUCTION FUNNEL: Table 5 reports the task-construction funnel, with percentages in its last column computed from the immediately preceding comparable stage.The table contextualizes candidate retention across environment construction and task validation.
  • A.4 CONSTRUCTION FUNNEL: The 58 exclusions are reported as an aggregate because their exclusion criteria may overlap and are not a mutually exclusive category breakdown.This reporting choice avoids implying that each excluded candidate belongs to exactly one reason category.
  • A.5 DATASET-COMPARISON DETAILS: Table 1 compares trajectory statistics, task-level verifier statistics, and common-solver evaluations across existing terminal-agent datasets.Because these quantities use different underlying units, their data sources and computation procedures are described separately.

A.6 DATASET-COMPARISON DETAILS · A.7 COMMAND-LEVEL BEHAVIOR IN SUCCESSFUL TRAJECTORIES

The dataset comparison separates trajectory-level, task-level, and common-solver measurements, while command analysis finds concentrated vocabularies and an observe–act–verify pattern in successful teacher trajectories. These comparisons and behavioral findings are descriptive and do not establish causal signatures of success.

  • A.6 DATASET-COMPARISON DETAILS: Trajectory- and task-level statistics use distinct independently selected samples, so they describe dataset characteristics rather than paired measurements.A task contains an instruction, initial environment, and verifier; a trajectory records an agent attempt and may not pair one-to-one with a task.
  • A.6 DATASET-COMPARISON DETAILS: Task-level analysis uses up to 15,000 executable bundles per dataset, including complete collections of 2,492 Endless-Terminals, 1,530 TerminalWorld, and 6,078 FACET tasks.Verifier tests are collected through Harbor and averaged as executable checkpoints per task.
  • A.6 DATASET-COMPARISON DETAILS: P@1 and P@3 evaluate 100 sampled tasks per dataset with three clean-environment attempts by DeepSeek-V4-Pro using Terminus-2.P@1 is computed over 300 individual attempts, whereas P@3 counts tasks solved at least once across three attempts.
  • A.6 DATASET-COMPARISON DETAILS: The shared scaffold, solver, attempt count, and sampling procedure reduce evaluation-side differences, but domain, construction, task-distribution, and verifier differences limit interpretation to descriptive comparisons.The comparison does not claim that one dataset’s construction or verifier is superior based solely on these results.
  • A.7 COMMAND-LEVEL BEHAVIOR IN SUCCESSFUL TRAJECTORIES: Among 1,270 parseable reward-1 teacher trajectories, analysis covers 15,075 assistant turns and 39,136 shell-command occurrences classified by command names and execution contexts.Reads, searches, listings, and checks are observations; writes, installation, movement, deletion, and builds are actions.
  • A.7 COMMAND-LEVEL BEHAVIOR IN SUCCESSFUL TRAJECTORIES: The commands cat, python3, and ls account for 69.5% of command occurrences, while the ten most frequent commands account for 88.4%.The concentrated vocabulary reflects recurring file inspection, script-based artifact construction, and output verification across heterogeneous tasks.
  • A.7 COMMAND-LEVEL BEHAVIOR IN SUCCESSFUL TRAJECTORIES: Successful trajectories exhibit an observe–act–verify loop: after an action-only turn, 53.1% of transitions return to observation, versus 28.7% continuing to another action-only turn.Most trajectories begin with observation-only turns, and consecutive observation turns often precede environment modification.
  • A.7 COMMAND-LEVEL BEHAVIOR IN SUCCESSFUL TRAJECTORIES: These command patterns characterize successful trajectories but do not establish which behaviors cause success or reliably distinguish successful from failed trajectories.The parser also cannot fully recover dynamically generated shell operations; failed-trajectory comparisons would be needed for stronger behavioral conclusions.

B GENERATION-SCHEME FAILURE ANALYSIS · C TASK CONSTRUCTION AND VALIDATION DETAILS

FACET compares three artifact-generation schemes under shared semantic paths, showing that staged generation achieves the strongest repaired-task yield and paired advantage over contract-first generation. The analysis also defines validation criteria and exposes differences in oracle feasibility, verifier discrimination, and generation-cost proxies.

  • B GENERATION-SCHEME FAILURE ANALYSIS: The three schemes start from the same 100 semantic path IDs: Base is staged, Hint-bundle v2 is contract-first, and Single-bundle v3 is joint generation.The original scheme names are retained for consistency with Figure 2(b) and its associated result table.
  • B GENERATION-SCHEME FAILURE ANALYSIS: Successful teacher-trajectory statistics use all 39,136 command occurrences, while transition probabilities are normalized within the current turn state.These statistics characterize command- and turn-level behavior rather than artifact-generation outcomes.
  • B GENERATION-SCHEME FAILURE ANALYSIS: Table 7 evaluates three artifact-generation orders on 100 shared semantic paths, with initial validity restricted to tasks reaching validation and final yield computed over all selected paths.This reporting convention distinguishes pre-repair validation from end-to-end selected-path yield.
  • B GENERATION-SCHEME FAILURE ANALYSIS: Initial validity requires complete pre-repair validation, oracle feasibility requires clean-state reference-solution execution, and negative discrimination requires incomplete solutions to fail verification.Final yield counts tasks recovered within each scheme’s repair budget across all selected paths.
  • B GENERATION-SCHEME FAILURE ANALYSIS: Staged generation repairs 37 of 53 initial failures and reaches 83/100 final yield, compared with 63/100 for contract-first and 65/100 for joint generation.Repair budgets are five rounds for staged generation and three rounds for the other two schemes.
  • B GENERATION-SCHEME FAILURE ANALYSIS: On 88 paths reaching validation under all schemes, staged beats contract-first alone on 29 paths versus 9, with p = 0.0017, but beats joint only 27 versus 18, with p = 0.233.These are two-sided exact sign tests from paired comparisons.
  • B GENERATION-SCHEME FAILURE ANALYSIS: Observed oracle feasibility is similar for staged and joint generation—46 of 99 versus 45 of 96—while joint exposes nine weak-verifier cases.Partial-solution coverage is 95/96 for joint, 7/99 for staged, and 9/91 for contract-first, so negative-discrimination results are descriptive.
  • B GENERATION-SCHEME FAILURE ANALYSIS: Joint generation uses one model call at 2.80 minutes per task, versus three calls averaging 4.14 minutes for staged and 5.19 minutes for contract-first generation.These are artifact-generation latency proxies excluding shared-prefix processing, validation, repair, and complete end-to-end runtime.

C.1 HARBOR TASK LAYOUT · C.2 SHARED-STATE CONSTRUCTION AND DOCKER VALIDATION

FACET organizes each task in a standard Harbor layout and grounds artifact generation in a realized, repaired environment state. Docker round-trip validation enforces clean, executable consistency, while verifiers assess required final behavior rather than exact procedures.

  • C.1 HARBOR TASK LAYOUT: Each generated task follows the Harbor directory structure for its instruction, metadata, environment, solution, and verifier artifacts.The layout separates user-visible requests, runtime metadata, Docker fixtures, executable reference solutions, and test assets.
  • C.1 HARBOR TASK LAYOUT: Table 8 summarizes shared-state task construction and Docker round-trip validation.The procedure links environment-grounded generation with execution-based checks.
  • C.2 SHARED-STATE CONSTRUCTION AND DOCKER VALIDATION: The environment is built and repaired before artifact generation, then recorded as read-only initial state e0 for all three generators.The record includes files, directories, schemas, dependencies, and local services, ensuring consistent references without mutable-container sharing.
  • C.2 SHARED-STATE CONSTRUCTION AND DOCKER VALIDATION: Baseline and oracle validation run in independent clean containers to prevent state leakage between trials.This is part of the completed task’s Docker round-trip procedure.
  • C.2 SHARED-STATE CONSTRUCTION AND DOCKER VALIDATION: A task is accepted only when its image builds, its verifier rejects the untouched initial state, and its reference solution passes from a clean environment.Every repair reruns the complete Docker lifecycle rather than relying on an incrementally modified debugging container.
  • C.2 SHARED-STATE CONSTRUCTION AND DOCKER VALIDATION: Verifiers evaluate observable final-state content and behavior, allowing alternative correct solutions instead of requiring exact command sequences or intermediate files.Nondeterministic values and irrelevant ordering are normalized when outside task requirements.
  • C.2 SHARED-STATE CONSTRUCTION AND DOCKER VALIDATION: Local services use deterministic interfaces, and localized verifier-failure messages can be routed to corresponding repairs.These design choices support targeted correction of artifact-specific failures.

D TRAINING AND EVALUATION CONFIGURATION

Training uses a standardized eight-GPU, three-epoch supervised fine-tuning recipe across Qwen3.5 scales, while evaluation uses the Terminus-2 scaffold with fixed attempts and timeouts. FACET fine-tuned models use a 32,768-token context limit, whereas other models follow their official limits.

  • Training configuration: Full-parameter supervised fine-tuning uses LLaMA-Factory on eight NVIDIA H200 GPUs, with all three Qwen3.5 scales trained for three epochs.The three model scales share the same training recipe.
  • Evaluation configuration: All evaluations use the Terminus-2 agent scaffold with three attempts per task and a two-hour timeout per attempt.These settings apply across the evaluated models.
  • Evaluation configuration: FACET fine-tuned models use a maximum context length of 32,768 tokens, while other models use the limit supported by their official checkpoints or interfaces.The context-length policy distinguishes FACET fine-tuned models from all other models.

E END-TO-END PIPELINE ABLATION

Across 500 common skill-pair inputs, FACET’s environment-grounded construction and targeted repair produced more validated tasks than the comparison pipelines, while retaining greater difficulty. Its gains reflect improved artifact consistency rather than easier workflows, though cross-pipeline difficulty differences are descriptive because retained inputs differ.

  • Experimental setup: The ablation compares Baseline, TW, and FACET end-to-end under the same model, Harbor format, and 500 accepted skill-pair inputs.The variants differ in scenario construction, artifact organization, information flow, validation, and repair.
  • Pipeline designs: FACET reconstructs executable scenarios, preserves requirements through instruction and solution references, and stages artifact generation around an explicitly built and repaired environment.Shared references propagate across instruction, solution, and verifier generation.
  • Results: 70.0% end-to-end yield: FACET validates 350 tasks, exceeding TW by 42.2 percentage points and Baseline by 54.4 points despite initially producing fewer complete packages.The comparison pipelines’ package-to-validation gaps show that complete files do not guarantee consistency among the environment, instruction, solution, and verifier.
  • Results: 182 FACET tasks pass initial validation and 168 more are recovered through targeted repair, converting inconsistent candidates into valid tasks while preserving scenarios and requirements.Execution feedback enables recovery without regenerating the underlying scenario and requirements.
  • Results: FACET’s validated tasks have the lowest P@1 and P@3 and the highest average terminal-command count, indicating that its yield advantage does not come from easier workflows.Retained tasks remain challenging under the same solver and agent scaffold.
  • Scope and limitations: TW is a best-effort reproduction, and P@1 and P@3 differences are descriptive rather than controlled causal estimates because adapters may alter implementations and pipelines retain different input subsets.The shared setting improves comparability but does not eliminate implementation and selection differences.
Loading 2608.18580v1…