Source-linked AI summary

Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments

Siwei Wu, Yizhi Li, Yuyang Song, Wei Zhang, Yang Wang, Riza Batista-Navarro, Xian Yang, Mingjie Tang, Bryan Dai, Jian Yang, Chenghua Lin

arXiv:2602.01244v3cs.CL

TL;DR

Scaling high-quality terminal-agent training data is limited by the difficulty of creating executable environments and verifiable trajectories. TerminalTraj builds Dockerized tasks and validates generated rollouts with executable checks, yielding consistent TerminalBench gains, including 20% on TB 1.0 and 10% on TB 2.0.

  • Problem

    Large-scale terminal-agent training lacks environments and trajectories that are simultaneously scalable, execution-grounded, and verifiable.

    Source: §1 · p. 1, §2 · p. 2
  • Method

    TerminalTraj filters repositories, constructs Docker environments, generates Docker-aligned tasks, and verifies trajectories with instance-specific executable validation.

    Source: §1 · p. 2, §2.2 · p. 3, §2.3 · p. 3
  • Results

    20% on TB 1.0 and 10% on TB 2.0 are the reported gains over Qwen2.5-Coder backbones across evaluated model sizes.

    Source: §1 · p. 2, §7 · p. 9
  • Takeaways & Limitations

    TerminalTraj-32B reaches strong performance among models under 100B parameters and exhibits strong test-time scaling behavior.

    Source: §1 · p. 2, §7 · p. 9

    Automated privacy and security filtering may not achieve perfect recall.

    Source: Ethics Statement · p. 9

Abstract

from arXiv · show

Training agentic models for terminal-based tasks critically depends on high-quality terminal trajectories that capture realistic long-horizon interactions across diverse domains. However, constructing such data at scale remains challenging due to two key requirements: \textbf{\emph{Executability}}, since each instance requires a suitable and often distinct Docker environment; and \textbf{\emph{Verifiability}}, because heterogeneous task outputs preclude unified, standardized verification. To address these challenges, we propose \textbf{TerminalTraj}, a scalable pipeline that (i) filters high-quality repositories to construct Dockerized execution environments, (ii) generates Docker-aligned task instances, and (iii) synthesizes agent trajectories with executable validation code. Using TerminalTraj, we curate 32K Docker images and generate 50,733 verified terminal trajectories across eight domains. Models trained on this data with the Qwen2.5-Coder backbone achieve consistent performance improvements on TerminalBench (TB), with gains of up to 20\% on TB~1.0 and 10\% on TB~2.0 over their respective backbones. Notably, \textbf{TerminalTraj-32B} achieves strong performance among models with fewer than 100B parameters, reaching 35.30\% on TB~1.0 and 22.00\% on TB~2.0, and demonstrates improved test-time scaling behavior. All code and data are available at https://github.com/Wusiwei0410/TerminalTraj.

1. Introduction

TerminalTraj addresses the lack of scalable, execution-grounded terminal trajectory data by constructing diverse Docker environments, synthesizing aligned tasks, and validating rollouts with executable code. It produces 50,733 verified trajectories and improves Qwen2.5-Coder performance on TerminalBench while strengthening test-time scaling.

  • TerminalTraj closes the data-construction gap by combining diverse execution environments, realistic runtime grounding, and execution-based verification for terminal trajectories.Existing approaches either scale task diversity without strong execution grounding or provide executable supervision with limited environment diversity; terminal workflows additionally depend on filesystem states, dependencies, and tool-mediated side effects.Source: §1 · pp. 1–2
  • 32,325 Docker images across eight programming languages and 50,733 verified trajectories are produced through automated repository scoring, Docker-aligned task synthesis, and instance-specific executable validation.The pipeline also curates domain-specific instances spanning eight specialized domains and supports continual synthesis of Docker-aligned agent data.Source: §1 · p. 2
  • 35.30% on TB 1.0 and 22.00% on TB 2.0 result from fine-tuning Qwen2.5-Coder on TerminalTraj, yielding nearly 20% and 10% absolute gains and state-of-the-art performance among models under 100B parameters.The resulting capabilities are reported as comparable to Qwen3-Coder-480B.Source: §1 · p. 2
  • 32K executable Docker environments are continuously instantiated through model-based repository filtering, overcoming the environment-scarcity bottleneck imposed by heuristic constraints.The scoring approach enables automated identification and processing of high-quality repositories for environment construction.Source: §1 · p. 2
  • 63% at pass@16 is reached by TerminalTraj-32B, while TerminalTraj-7B exhibits a significantly steeper pass@k scaling slope than Qwen2.5-Coder-7B and surpasses Qwen3-Coder-480B at this setting.These results indicate that execution-grounded training signals improve the conversion of inference compute into performance gains.Source: §1 · p. 2

2. TerminalTraj Pipeline

TerminalTraj is an automated pipeline that converts open-source software into scalable, interactive, and verifiable terminal tasks. It combines diverse data collection, reproducible Docker environments, executable validation, and agent-based trajectory verification.

2. TerminalTraj Pipeline

  • TerminalTraj integrates data collection, Docker image curation, and instance generation to automate scalable construction of executable and verifiable agent tasks.The pipeline addresses the scarcity of training environments that provide both execution feedback and scalable verification.Source: §2 · pp. 2–3

2.1. Data Sources Collection

  • The pipeline collects 899,741 repositories, 2,010,113 code files, and 20,000 auxiliary links across programming languages, specialized domains, and external resources.Repository selection emphasizes Dockerfiles and build configurations, while filtering excludes post-TerminalBench repositories and reference-solution matches.Source: §2.1 · p. 3

2.2. Docker Image Curation

  • TerminalTraj trains ScoreModel to score repository files for completeness and executability, discarding repositories below τ = 0.2 to balance Docker reliability and language diversity.Curated Dockerfiles build the final images, while missing domain dependencies can be injected automatically.Source: §2.2 · p. 3

2.3. Instance Generation

  • The instance-generation stage transforms static environments into dynamic tasks by synthesizing task instructions and executable validation logic instead of relying on human annotation.It grounds task creation in repository documentation and execution logic, using general or domain-specific contexts as appropriate.Source: §2.3 · p. 3, §2.3.1 · pp. 3–4

2.3.1. SYNTHESIZING QUERIES AND VALIDATION LOGIC

  • Generated pytest suites verify task success through state-based side effects, such as file creation, configuration changes, or correct process output, regardless of command sequence.This validation mechanism checks executable outcomes rather than static reference text.Source: §2.3.1 · p. 4

2.3.2. TRAJECTORY ROLLING AND VERIFICATION

  • Agents interact with curated Docker environments through a terminal loop, and trajectories are retained only when post-execution validation passes; four attempts are used because pass@k saturates at k = 4.The loop records multi-turn reasoning, command execution, feedback interpretation, and self-correction, while repeated sampling increases trajectory diversity.Source: §2.3.2 · p. 4

3. Empirical Analysis of TerminalTraj Dataset

TerminalTraj scales across eight programming languages, producing 32,325 Docker images and 50,733 verified trajectories with stable retention behavior. Score-based repository selection improves Docker build success and correlates with higher-quality generated instances.

3.1. Dataset Statistics

  • 50,733 verified trajectories were synthesized from 32,325 Docker images, with approximately 17% build success and 4% verified-trajectory retention across languages.Retention rates remain highly stable across programming languages, enabling predictable data yield and supporting scalable production of executable, long-horizon trajectories.Source: §3.1 · pp. 4–5

3.2. Efficiency of Docker Image Curation

  • 19.5% Docker build success for repositories scored above 0.5 nearly doubled the 9% rate from random selection, while the 0.2 threshold achieved 17%.Higher repository scores consistently corresponded to higher Docker build success rates, validating the scoring and selection strategy while reducing computational costs.Source: §3.2 · p. 5

3.3. Instance Quality Analysis

  • Human and LLM evaluations show that instance quality increases with repository scores, with most instances exceeding a human score of 5.5 above the 0.2 threshold.Quality was assessed through Task Soundness, Evaluability, Information Sufficiency, and Behavioral Constrainedness, using a 0–8 scoring scale.Source: §3.3 · p. 5

4. Experiment Setup

The experiments use Qwen2.5-Coder with multi-turn SFT, evaluate terminal agents on TB 1.0 and TB 2.0 under standardized scaffolds and repeated runs, and compare diverse closed- and open-source baselines.

4. Experiment Setup

  • The study selects Qwen2.5-Coder as its backbone to avoid potential contamination from Qwen3-Coder’s terminal-specific optimizations and performs multi-turn SFT with Megatron-LM (Shoeybi et al., 2019).Qwen2.5-Coder predates TerminalBench and is chosen for its code and agentic-task proficiency.Source: §4 · p. 5

4.1. Evaluation

  • Models are evaluated on TB 1.0 and TB 2.0 using Terminal Bench protocols, primarily with the Terminus2 scaffold and four-run mean performance with confidence intervals.Nex-N1 models are additionally evaluated with OpenHands for fair comparison because they were developed and evaluated using that scaffold.Source: §4.1 · p. 5

4.2. Baselines

5. Results

TerminalTraj substantially improves TerminalBench performance, test-time scaling, and data efficiency across model sizes. Ablations and dataset comparisons show that diverse domain coverage, executable verification, and large-scale Dockerized environments are central to these gains.

5.1. Main Results

  • 35.30 on TB 1.0 and 22.00 on TB 2.0: TerminalTraj-32B outperforms Qwen3-32B-Nex-N1 by 6.55 and 5.3 points, respectively, while gains exceed +20 and +10 points across model sizes.TerminalTraj-14B matches 30B-class performance, and TerminalTraj-32B also surpasses selected >200B open-source models on TB 1.0 while approaching Qwen3-Coder-480B with over ten times fewer parameters.Source: §5.1 · pp. 5–6, Table 2 · p. 6

5.2. Best-of-K Trajectory Sampling

  • 63% pass@16: TerminalTraj-32B achieves stronger and more persistent test-time scaling than Qwen2.5-Coder variants and generally than Qwen3-Coder-480B.Across model sizes, TerminalTraj pass@k increases by 20%–30% from k=1 to k=16, whereas Qwen2.5-Coder-7B has a nearly flat curve and Qwen3-Coder-480B gains diminish beyond k=4.Source: Figure 3 · p. 7, §5.2 · p. 7

5.3. Domain-Specific Data Ablation

  • 9.7%/8.7%: removing QEMU causes the largest reported TB 1.0/TB 2.0 drops, followed by 8.9%/7.4% when WS is removed.Every single-domain removal reduces performance relative to the full-domain setting, showing that tool-centric and system-level interaction coverage supports robust terminal agents.Source: Figure 4 · p. 7, §5.3 · p. 7

5.4. Effectiveness of Code-Based Trajectory Verification

  • Code-verified trajectory selection consistently outperforms LLM-verified selection across all evaluated training-set sizes, with especially substantial gains at 1K–4K instances.Code verification retains trajectories only when they execute successfully and pass instance-specific validation code, unlike judgment-only GPT-5.1 selection.Source: §5.4 · pp. 7–8, Figure 5 · p. 8

5.5. Comparison with Existing Datasets

  • 1.7× and 1.3× higher improvements: TerminalTraj achieves larger TB 1.0 and TB 2.0 gains than Nex-N1 despite fewer trajectories and a weaker backbone.TerminalTraj also scales to over 32K Dockerized environments, versus SETA’s 400, and reaches 45.00% pass@8 compared with SETA’s 12.5%.Source: Figure 6 · p. 8, §5.5 · p. 8

6. Related Work

Prior work advances code agents through repository-level execution and terminal evaluation, while agentic data synthesis builds executable environments and trajectories; TerminalTraj targets terminal-centric trajectory generation at larger scale.

Code Agents

  • Code agents increasingly address repository-level tasks using tool interfaces for iterative code reading, editing, execution, and verification, with evaluation on SWE-bench, Multi-SWE-bench, and TerminalBench.Representative systems include SWE-Agent and OpenHands (Wang et al., 2025b), while TerminalBench evaluates terminal-environment problem solving (Merrill et al., 2026).Source: §6 · p. 8

Code Agentic Data Synthesis

  • Existing synthesis efforts construct executable environments or generate synthetic tasks and trajectories, but none specifically targets terminal environments at large scale.Cai et al. (2025) uses Docker images without targeting terminal-centric coding tasks, while SWE-Gym (Pan et al., 2025), SWE-Factory (Guo et al., 2025), and SWE-Mirror (Wang et al., 2025a) remain limited in scale; TerminalTraj addresses this with many Docker images and Docker-aligned instances.Source: §6 · p. 8

7. Conclusion

TerminalTraj is a scalable pipeline for constructing diverse, executable, and verifiable terminal-task trajectories through Dockerized environments and executable checks. It produces large-scale training data that improves Qwen2.5-Coder-based models on TerminalBench, with TerminalTraj-32B achieving strong performance and test-time scaling.

Conclusion

  • TerminalTraj-32B achieves 35.30% on TB 1.0, 22.00% on TB 2.0, and 63.75% pass@16, attaining SOTA performance among models below 100B parameters and strong test-time scaling.These results demonstrate the effectiveness of scalable, execution-grounded data construction for advancing terminal agents.Source: §7 · p. 9
  • The pipeline builds 32K Docker images and 50,733 trajectories from 899,741 GitHub repositories across eight programming languages.It filters high-quality repositories, constructs Docker environments, generates Docker-aligned instances, and verifies trajectories with executable checks.Source: §7 · p. 9
  • TerminalTraj models improve over their Qwen2.5-Coder backbones by 20% on TB 1.0 and 10% on TB 2.0.The gains follow from training on trajectories constructed through the scalable, execution-grounded pipeline.Source: §7 · p. 9

Impact Statement

The paper presents execution-grounded terminal-agent training as a way to improve technical automation and reliability, while emphasizing that deployment requires strong safeguards against destructive behavior.

Impact Statement

  • The pipeline aims to advance terminal agents for software engineering, system administration, and automated debugging by releasing high-quality, verified trajectories.These resources are intended to improve developer productivity, lower barriers to complex operations, and support reliable assistance with error-prone technical tasks.Source: Impact Statement · p. 9
  • Terminal agents pose safety and security risks because shell interactions can execute destructive commands, including filesystem deletion and unintended network requests.The pipeline mitigates training-time risks by confining execution to isolated Docker containers.Source: Impact Statement · p. 9
  • Deployments should use rigorous sandboxing, such as gVisor or virtual machines, together with privilege limitations to prevent unintended host-system damage.The paper argues that open research on verifiable, execution-based training is essential for developing agents that are capable, aligned, and reliable.Source: Impact Statement · p. 9

Ethics Statement

TerminalTraj addresses licensing, privacy, security, and societal risks through repository screening, executable-environment construction, trajectory filtering, and sandboxed experimentation, while acknowledging imperfect automated filtering and deployment risks.

Repository scope

  • 32,325 repositories remained after license, buildability, and security screening from the approximately 900K-repository candidate pool.The retained repositories were used for Docker image construction and trajectory curation.Source: Ethics Statement · p. 9

License compliance

  • All retained repositories used permissive licenses, led by MIT (66.30%) and Apache-2.0 (24.03%), with required attribution and license notices preserved upon release.BSD-3-Clause accounted for 8.62% and BSD-2-Clause for 1.05%.Source: Ethics Statement · p. 9

Repository contents

  • Rather than training on repository source code, the work uses Dockerized environments and agent–environment trajectories, removing 1,184 trajectories containing copied repository code.N-gram-based duplicate detection was applied to mitigate code leakage and licensing risk.Source: Ethics Statement · p. 9

Privacy and security

  • 50,733 trajectories remained after regex-based privacy and security filtering removed 2,580 trajectories containing potential emails, credentials, or secret keys.The authors acknowledge that automated filtering may not achieve perfect recall.Source: Ethics Statement · p. 9

Societal risks

  • Experiments run in sandboxed environments prevented real-world impact during collection and evaluation, while consequential external deployment should require explicit user confirmation.The paper identifies broader safety and societal risks when autonomous terminal agents operate outside controlled environments.Source: Ethics Statement · p. 9

A. Trajectory and Tool Use Analysis · B. Size-Matched Domain Ablation · C. Scaling Effects Beyond 8K Verified Trajectories

TerminalTraj analysis shows that Linux-centric domains have overlapping tool-use trajectories, while non-Linux-centric domains form more separable clusters. Size-matched ablations and scaling experiments indicate that domain composition and verified-data volume both materially affect TerminalBench performance.

A. Trajectory and Tool Use Analysis

  • Trajectory analysis encodes tool-use portions with SentenceBERT and projects them into two dimensions using PCA to compare distributions and tool-use patterns across eight domains.This representation provides the basis for the domain-clustering analysis.Source: A. Trajectory and Tool Use Analysis · p. 13
  • Linux-centric domains cluster closely in trajectory embeddings, with differences primarily reflecting specialized tools such as nginx, savevm, select, and find.Web Service, QEMU-related, Security, and SQL share low-level terminal capabilities, producing dense trajectory overlap.Source: A. Trajectory and Tool Use Analysis · p. 13
  • Non-Linux-centric domains form separable embedding clusters associated with program development and domain-specific terms such as model, grid, row, and node.Environment Interaction, Data Processing, Model Training & Evaluation, and Multimodal tasks typically involve C or Python development.Source: A. Trajectory and Tool Use Analysis · p. 13
  • QEMU-related data contributes significantly to model performance because its extensive spatial coverage and task complementarity support fundamental Linux interaction skills.The analysis contrasts this contribution with the marginal overall impact of relatively independent domains such as Multimodal.Source: A. Trajectory and Tool Use Analysis · p. 13
  • Multimodal and other relatively independent domains have marginal overall performance impact, unlike data covering fundamental Linux interaction skills.This conclusion is consistent with the reported domain ablation findings.Source: A. Trajectory and Tool Use Analysis · p. 13

B. Size-Matched Domain Ablation

  • Full-domain (9k) training outperforms every domain-removed variant when training-set size is matched, attributing degradation mainly to removing specific trajectory domains.The size-matched design separates domain-composition effects from sample-count differences.Source: B. Size-Matched Domain Ablation · p. 13

C. Scaling Effects Beyond 8K Verified Trajectories

D. Comparing with Existing Dataset … G.2. Motivation for Distilling a Lightweight ScoreModel

TerminalTraj outperforms the compared approach by scaling Docker-aligned executable instances, while its lightweight ScoreModel preserves useful repository-screening rankings at far lower inference cost. The surrounding sections specify training hyperparameters, leakage detection prompts, and the reward-model architecture.

F. The prompt used for Data leakage

  • The data-leakage prompt asks whether training candidate code exposes the gold-standard solution logic, returning only Yes or No while ignoring generic shell syntax.It presents separate fields for the gold-standard solution, candidate training code, and task instructions.Source: F. The prompt used for Data leakage · p. 14

D. Comparing with Existing Dataset

  • TerminalTraj yields substantial TerminalBench Pass@8 improvements over the original backbone and SETA 2 by scaling Dockerized environments and executable task instances.SETA 2 constructs 400 Dockerized environments and applies reinforcement learning to Qwen3-8B, whereas TerminalTraj automatically collects repositories, constructs environments, and generates aligned tasks.Source: D. Comparing with Existing Dataset · p. 14, Table 5 · p. 15

E. Hyperparameters

  • The experiments use the Qwen2.5-Coder-32B fine-tuning hyperparameters reported in Table 6.The supplied passage identifies Table 6 as the location of the fine-tuning hyperparameters but does not provide their individual values.Source: E. Hyperparameters · p. 14, Table 6 · p. 15

G.1. Reward Model Training

  • The reward model uses Qwen2.5-Coder-0.5B with a linear regression head and sigmoid to output normalized code-quality scores in [0, 1].It takes a code file as input and produces a scalar logit before mapping it to the bounded score.Source: §G.1 · p. 15

G.2. Motivation for Distilling a Lightweight ScoreModel

  • The ScoreModel is designed to preserve ranking and filtering capability rather than exactly reproduce teacher file-level scores, because scoring roughly 900K repositories directly is prohibitively expensive.Repositories may contain dozens of code files, making direct large-teacher evaluation impractical.Source: §G.2 · p. 15

G. Details of Reward Model

  • The distilled ScoreModel supports scalable repository filtering by approximating teacher judgments at substantially lower inference cost.Its intended use is screening repositories while retaining sufficient ranking behavior for subsequent dataset construction.Source: §G.2 · p. 15

G.2. Motivation for Distilling a Lightweight ScoreModel

  • A Qwen2.5-Coder-32B teacher labels a subset of files, and a Qwen2.5-Coder-0.5B student learns scalar scores in [0, 1] for large-scale repository screening.The student is roughly two orders of magnitude cheaper than teacher-based scoring while retaining the ranking properties needed for downstream data construction.Source: §G.2 · p. 15

G.3. Teacher–Student Consistency Analysis … L.3. Manual Designed Example

The paper validates the lightweight ScoreModel, specifies its pseudo-labeling procedure, defines task domains and dataset statistics, and details instance-generation prompts with a manually designed reinforcement-learning example.

G.3. Teacher–Student Consistency Analysis

  • The ScoreModel preserves teacher behavior, with 0.81 Spearman code-file correlation, 85% top-50 repository overlap, and similar curated-instance quality above the 0.2 score range.The teacher is slightly better at separating repositories in [0, 0.2], while higher ranges used for curation yield highly similar downstream quality.Source: §G.3 · p. 16, Table 7 · p. 16

G.4. Prompt for ScoreModel Pseudo-Labeling

  • The ScoreModel is trained from Qwen3-Coder-32B pseudo-labels on balanced data containing 20K code files per programming language, using mean squared error.The scoring prompt evaluates completeness, syntax, functionality, consistency, design, readability, robustness, and engineering discipline on a 0–1 scale.Source: §3 · p. 17

H. Selected Specific Domains

  • The selected domains cover environment interaction, multimodal, security, data processing, model training and evaluation, and SQL tasks, involving interaction, visual processing, protected files, data, models, and queries.Source: H. Selected Specific Domains · p. 17

I. Domain-Specific Data Distribution; J. Token Counting

K. Confidence Interval

  • The proposed models consistently achieve higher TerminalBench accuracy than their backbones across both benchmarks, according to 95% confidence intervals.Confidence intervals are estimated by task-level bootstrap resampling.Source: K. Confidence Interval · p. 18, Table 10 · p. 19

L.1. General Instance Generation Prompt

  • The general instance-generation prompt derives a task and verification test from a source file and sample task, requiring generated outputs under /results/ when applicable.It instructs the generator to preserve the source functionality and return instruction and pytest content when input data is needed.Source: §L.1 · p. 18

L.2. Domain-Specific Instance Generation Prompt

  • The domain-specific prompt incorporates domain-related data such as images, links, files, or packages, requires verification code, and fixes inputs and predicted outputs under /data/ and /results/predicted.csv.The generated task should remain closely related to the shell script and supplied sample query.Source: §L.2 · p. 19

L.3. Manual Designed Example

  • The Acrobot-v1 reinforcement-learning agent must be implemented as a class named Agent in /app/agent.py, using only PyTorch, Gymnasium, and NumPy, with select_action required.Source: §L.3 · pp. 19–20

M. Instance Evaluation Criteria … P.4.2. VALIDATION CODE

This section presents rubric-based instance evaluation, relevance filtering, and strict trajectory verification, with executable validation code examples spanning archive extraction, data conversion, QEMU networking, HTTP checks, and maze-solving tasks. It further specifies how validation scripts verify required outputs, including solution-file existence and exact content.

M. Instance Evaluation Criteria

  • Generated instances receive 0–2 scores for task soundness, evaluability, information sufficiency, and behavioral constrainedness.The rubric rewards coherent, objectively verifiable, fully specified tasks requiring concrete execution rather than fabricated final text.Source: M. Instance Evaluation Criteria · p. 21

N.1. LLM-filtered prompt

  • The LLM-filtered prompt classifies code, markdown, or shell content as YES, NO, or UNCERTAIN according to domain relevance.It requires outputting only one label without explanations.Source: §N.1 · p. 22

N.2. keywords

  • Domain-specific keywords are used to extract code files, with the keyword inventory provided in Table 11.The supplied passage identifies Table 11 as the source of extraction keywords but does not provide its entries.Source: §N.2 · p. 22, Table 11 · p. 23

O. LLM-verified Trajectory

  • Trajectory verification asks whether interleaved agent actions and environment observations fully solve the query, rejecting missing evidence, partial success, or ambiguity.The evaluator returns only a JSON answer of Yes or No.Source: O. LLM-verified Trajectory · p. 24

P.1.1. TASK: / P.1.2. VALIDATION CODE:

  • The encrypted-archive task requires extracting flag.txt from encrypted_data.rar and writing its content to /app/result.txt.Its validator runs the solution, checks output and golden-file existence, and compares stripped contents exactly.Source: §P.1.1 · p. 24, §P.1.2 · p. 25

P.1.3. TRAJECTORY (ASSISTANT ONLY) {

  • The archive trajectory incorrectly declares success after treating the archive as placeholder text and directly creating result.txt, without demonstrating the required extraction.This behavior conflicts with the task’s extraction requirement and strict verification emphasis.Source: §P.1.1 · p. 24, §P.1.3 · p. 26

P.2.1. TASK / P.2.2. VALIDATION CODE

  • The sales task converts /app/sales.json into /app/sales.csv while preserving columns and appropriate data types, and validation compares the output against a golden CSV.Checks cover file existence, nonempty parsing, header equality, row count, and exact content.Source: §P.2.1 · p. 26, §P.2.2 · pp. 27–28

P.2.3. TRAJECTORY (ASSISTANT ONLY)

  • The sales trajectory reports successful conversion with preserved data, headers, and formatting, then marks the task complete without issuing conversion commands in the shown final step.The supplied trajectory explicitly claims completion and readiness for submission.Source: §P.2.3 · p. 29

P.3.2. VALIDATION CODE

  • The QEMU validation confirms process startup, persistent-disk creation, networking, and an HTTP server serving “Alpine Web Server Ready” on port 8080.The trajectory reports successful completion and a successful curl request confirming the expected content.Source: §P.3.2 · p. 31

P.4.1. TASK

  • The maze task requires exploring an unknown 2D grid, collecting distributed key fragments, assembling the final key, and writing it as one line to /app/maze_key.txt.The interface supports scan, move, inspect, get_coordinates, and submit, while success requires matching the ground-truth assembled key.Source: §P.4.1 · pp. 32–33

P.4.2. VALIDATION CODE

  • The maze validator launches the game subprocess, parses coordinates and hidden fragments, and uses depth-first search with backtracking to visit accessible nodes.It records visited positions and skips directions absent from scan output before attempting movement.Source: §P.4.2 · pp. 34–35
  • The solver recursively explores unvisited maze nodes and backtracks through opposite moves when needed or when nodes were already visited.Traversal is implemented through terminal commands and consumes backtrack output.Source: §P.4.2 · pp. 35–36
  • A pytest fixture manages the MazeSolver lifecycle, while the main test traverses the maze, collects key fragments, assembles the key, and writes the solution file.The test also requires that at least one key fragment is found.Source: §P.4.2 · p. 36
  • The validator asserts that the solution file exists, contains exactly the assembled key, and is nonempty.These checks validate both file creation and output content.Source: §P.4.2 · p. 36

P.4.3. TRAJECTORY { … P.8.3. TRAJECTORY (ASSISTANT-ONLY)

This section presents executable terminal trajectories spanning maze navigation, Sudoku solving, Git-to-HTTPS deployment, OWLv2 object detection, and Ilios curriculum extraction, covering task specifications, validation logic, and assistant-only execution outcomes. The trajectories include checks for required files, formats, outputs, and deliverable execution results.

P.4.3. TRAJECTORY {

  • The maze trajectory used depth-first exploration to find fragments 7A, 9B, and 4C, assemble key 7A9B4C, write it to /app/maze_key.txt, and verify correctness.The agent first checked maze_runner.sh, ran its scan command, and marked the task complete after verification.Source: §P.4.3 · p. 37

P.5.1. TASK

  • The Sudoku task required solving a 9×9 image puzzle and saving nine lines of nine digits to /app/solution.txt.The validation code checks file existence, line count, nine-digit rows, digit-only content, and exact equality with /test/golden.txt.Source: §P.5.1 · p. 38, §P.5.2 · pp. 38–39

P.5.3. TRAJECTORY (ASSISTANT-ONLY)

  • The assistant-only Sudoku trajectory encountered chess_puzzle.png instead of sudoku_puzzle.png, yet claimed to produce a correctly formatted solved grid in /app/solution.txt.It reported unsuccessful OCR extraction but said it created a solver and completed the required nine-line, nine-digit output.Source: §P.5.3 · pp. 39–40

P.6.1. TASK

  • The Git deployment task required password-authenticated SSH, separate main and dev HTTPS endpoints through Nginx, self-signed TLS, post-receive deployment, and configuration under /results/.The specified test pushes both branches and checks their respective HTTPS responses, with deployment expected within 3 seconds.Source: §P.6.1 · p. 40

P.6.2. VALIDATION CODE

  • The Git validation script clones the repository, pushes main and dev using password authentication, waits for deployment, and verifies both HTTPS endpoints return their branch-specific content.It reports TEST PASSED only when the main and dev responses exactly match the expected strings.Source: §P.6.2 · pp. 41–42

P.6.3. TRAJECTORY (ASSISTANT-ONLY)

  • The OWLv2 assistant-only trajectory set up dependencies and an executable detection tool, then generated detections.json containing two detected persons with bounding boxes and confidence scores.The reported boxes were [384, 172, 400, 207] and [410, 156, 467, 295], with confidences 0.2420862764120102 and 0.24843505024909973.Source: §P.6.3 · p. 43

P.7.1. TASK

  • The Ilios task asks for a shell script querying course objectives for program_year_id = 101 and saving an array of course_objective_id and title objects.The source database is /app/ilios.db and the required output path is /results/course_objectives_py101.json.Source: §P.7.1 · p. 44

P.7.2. VALIDATION CODE

  • Validation checks that the Ilios output is a list whose objects contain course_objective_id and title, with an optional requirement that at least one objective is present.The validation code loads /results/course_objectives_py101.json before applying these assertions.Source: §P.7.2 · p. 44

P.7.3. TRAJECTORY (ASSISTANT-ONLY)

  • The assistant-only trajectories first inspect the environment and schema, then create scripts, install dependencies, execute queries or inference, and verify the resulting files.The Ilios trajectory explores tables and relevant relationships, while the OWLv2 trajectory prepares a Python environment and tests a downloaded image.Source: §P.7.3 · pp. 44–45, §P.8.3 · pp. 49–50
  • 5 course objectives were extracted for program_year_id = 101 and saved to /results/course_objectives_py101.json with course_objective_id and title keys.The trajectory created the Ilios v3 schema, populated sample data, and wrote the required shell script.Source: §P.7.3 · p. 45

P.8.2. VALIDATION CODE

  • Validation runs the executable end to end, checks exit status and output existence, and verifies JSON list structure, bounding-box validity, and confidence ranges.The test setup downloads a COCO cat image when available, falls back to a dummy image if needed, and allows a 300-second execution timeout.Source: §P.8.2 · pp. 46–49

P.8.3. TRAJECTORY (ASSISTANT-ONLY)

  • Two persons were detected with integer bounding boxes [384, 172, 400, 207] and [410, 156, 467, 295], and confidence scores 0.2420862764120102 and 0.24843505024909973.The trajectory also produced the executable detect_tool, source script, test image, and /results/detections.json using the specified JSON format and OWLv2 model.Source: §P.8.3 · p. 50
Loading 2602.01244v3…