Source-linked AI summary
CodeTracer: Towards Traceable Agent States
Han Li, Yifan Yao, Letian Zhu, Rili Feng, Hongyi Ye, Jiaming Wang, Yancheng He, Pengyu Zou, Lehan Zhang, Xinping Lei, Haoyang Huang, Ken Deng, Ming Sun, Zhaoxiang Zhang, He Ye, Jiaheng Liu
TL;DR
Code-agent executions are difficult to debug because existing evaluations and analyses provide limited visibility into failures across long, heterogeneous trajectories. CodeTracer structures these runs into hierarchical traces, localizes failure onset, and evaluates diagnosis with CodeTraceBench. It substantially outperforms raw-log and lightweight baselines, while the study identifies systematic behavior patterns and iteration-budget saturation.
Problem
Existing evaluations collapse trajectories into end-to-end outcomes, while trajectory analyses rely on coarse judgments or small manual samples that do not scale to realistic workloads.
Method
CodeTracer extracts normalized steps, indexes them into hierarchical trace trees, predicts failure-critical stages and steps, and evaluates localization with step-level supervision in CodeTraceBench.
Results
CodeTracer substantially outperforms raw-log prompting and lightweight baselines in step-level localization, while analyses reveal evidence-to-action gaps, stage-dependent errors, and diminishing gains from extra iterations.
Takeaways & Limitations
Structured tracing supports scalable process-level diagnosis of code-agent failures and exposes how agents search, act, and fail across workflow stages.
Takeaways & Limitations
CodeTraceBench does not cover the full design space of software-engineering agents or real-world repositories, so failure-pattern and replay conclusions may not transfer uniformly.
Abstract
from arXiv · showhide
Code agents are advancing rapidly, but debugging them is becoming increasingly difficult. As frameworks orchestrate parallel tool calls and multi-stage workflows over complex tasks, making the agent's state transitions and error propagation hard to observe. In these runs, an early misstep can trap the agent in unproductive loops or even cascade into fundamental errors, forming hidden error chains that make it hard to tell when the agent goes off track and why. Existing agent tracing analyses either focus on simple interaction or rely on small-scale manual inspection, which limits their scalability and usefulness for real coding workflows. We present CodeTracer, a tracing architecture that parses heterogeneous run artifacts through evolving extractors, reconstructs the full state transition history as a hierarchical trace tree with persistent memory, and performs failure onset localization to pinpoint the failure origin and its downstream chain. To enable systematic evaluation, we construct CodeTraceBench from a large collection of executed trajectories generated by four widely used code agent frameworks on diverse code tasks (e.g., bug fixing, refactoring, and terminal interaction), with supervision at both the stage and step levels for failure localization. Experiments show that CodeTracer substantially outperforms direct prompting and lightweight baselines, and that replaying its diagnostic signals consistently recovers originally failed runs under matched budgets. Our code and data are publicly available.
1 Introduction
CodeTracer addresses limited visibility into complex code-agent failures by structuring heterogeneous trajectories, localizing failure onset, and enabling systematic process-level evaluation. Its benchmark and empirical analysis support scalable diagnosis of agent behavior.
- Code agents execute long, heterogeneous action sequences, making it difficult to identify where failures begin and which decisions matter.
- End-to-end metrics collapse trajectories into success or failure, while existing analyses depend on coarse judgments or small-scale manual inspection.
- CODETRACER converts heterogeneous run directories into structured hierarchical traces and automatically identifies the earliest failure-critical stage.
- CODETRACEBENCH aggregates trajectories from multiple agent frameworks and diverse software-engineering workloads with structured step metadata and failure-critical labels.
- The benchmark enables large-scale analysis of evidence-to-action gaps, action-efficiency variance, and stage-dependent error modes across model backbones and agent frameworks.
- The work contributes a scalable tracing framework, a benchmark of thousands of annotated trajectories, and an empirical analysis of code-agent failure patterns.
2 Trajectory Analysis
The trajectory analysis examines a filtered corpus of code-agent runs, annotates workflow steps and failure patterns, and compares task preferences, orchestration cost, stage-dependent errors, and iteration budgets.
- Data and annotation: The corpus spans five benchmarks, four agent frameworks, and five model backbones, with trajectories normalized into iterations for consistent analysis.Filtering reduced 7,936 raw trajectories to 3,326 runs spanning all benchmark–framework–backbone combinations.
- Data and annotation: Authors annotate each step with workflow stages and identify redundant steps in successful runs or failure-critical chains in failed runs.The stages include environment verification, dependency installation, inspection/debugging, patching, and verification.
- Task preferences and failure modes: Backbones differ in task regimes, but all five solve 66 categories and fail on 65 categories, with universally hard tasks eliciting fabrication, placeholders, or premature stopping.The universally hard categories emphasize external grounding or long-horizon reasoning.
- Overengineering: Additional orchestration raises process cost without proportional success gains: SWE-Agent and OpenHands nearly double MiniSWE-Agent’s token usage for modest improvements.The reported token figures are 44.6k for MiniSWE-Agent, 86.7k for SWE-Agent, and 91.4k for OpenHands, with gains of +2.4–5.5 pp.
- Stage-dependent errors: Failure-critical steps concentrate in workflow phases: setup and dependency errors occur early, while mislocalized edits, incorrect hypotheses, and verification misinterpretation occur later.Unsolved runs also spend disproportionate budget on early setup and recurrent inspection loops after wrong commitments.
- Iteration budget: Success rises rapidly to about 40 iterations before flattening, while extra iterations often amplify redundant exploration and trial-and-error edits after incorrect commitments.The saturation ceiling is mainly backbone dependent rather than substantially later for stronger models.
- Trace structure: CODETRACER’s hierarchical trace tree keeps exploration under the current state and represents state-changing steps as transitions to child states for compressed trace navigation.The structure is built during tree indexing.
3 CodeTracer
CODETRACER localizes failure onset in standardized agent trajectories by combining evolving extraction, hierarchical trace processing, and structured evidence retrieval. CODETRACEBENCH provides annotated trajectories across diverse benchmarks, frameworks, backbones, and difficulty levels for evaluating this diagnosis.
- Failure localization: CODETRACER predicts a failure-responsible stage, error-relevant steps within that stage, and a compact evidence set supporting the diagnosis.Evaluation uses step-level precision, recall, and F1 against gold incorrect-step sets, together with token cost.
- Tracing pipeline: The pipeline separates evolving extraction, tree indexing, and diagnosis to handle heterogeneous, agent-dependent run-directory formats.Existing parsers are reused when possible, while unmatched formats receive synthesized and registered parsers.
- Benchmark construction: CODETRACEBENCH draws from five benchmarks, five backbones, and four frameworks, totaling 4,354 standardized, step-level annotated trajectories.The benchmark includes repository-level bug fixing and long-horizon terminal interaction tasks.
- Benchmark construction: The resulting benchmark contains a 3.32K-instance full split and a 1.06K verified split with stage boundaries, failure-critical stages, and incorrect-step annotations.Metadata also records frameworks, backbones, task information, raw artifact pointers, and difficulty labels.
4 Experiments
Experiments evaluate CODETRACER for step-level failure localization, component contributions, behavioral analysis, reflective replay, and industrial trajectory analysis. Results show improved localization, distinct backbone search strategies, an evidence-to-action gap, and consistently better recovery with diagnostic replay.
- Main Results: CODETRACER improves step-level localization over raw log prompting and lightweight standardization while reducing token usage through compact evidence retrieval.Across frontier backbones, F1 reaches 46–48%; GPT-5 has 45.0% overall precision at 31.1k tokens, while Claude-sonnet-4 reaches 54.9% recall at 56.8k tokens.
- Diagnosis Behaviors: Frontier backbones achieve similar F1 but use different search-depth strategies: GPT-5 favors precision and lower cost, whereas Claude-sonnet-4 favors recall through deeper search.DeepSeek-V3.2 balances both strategies, and localization cost scales with trajectory length rather than model identity.
- Evidence to Action Gap: Ineffective actions nearly double from 22% in solved trajectories to 40% in unsolved trajectories, while correct state changes fall from 30% to 21%.Useful exploration declines only mildly, indicating that agents often gather relevant evidence without converting it into effective state changes.
- Component Ablation: Tree indexing provides the largest component gain, adding 18.3 F1 points, while evolving extraction contributes a further 9.4-point lift.Tree indexing enables compressed hierarchical navigation; evolving extraction standardizes formats through parser reuse.
- Reflective Replay: Reflective replay consistently improves Pass@1 on originally failed runs when CODETRACER’s localized evidence is injected under matched budgets.The diagnosis pass averages 8.4k tokens for Claude-sonnet-4, and replay helps agents revise early wrong commitments.
- Industrial Agent Analysis: Claude Code combines richer tooling and context management with a lower exploration-to-change ratio that correlates with higher trajectory efficiency.The analysis covers more than 40 specialized tools across eight categories, alongside compaction and token budgeting.
- Action Efficiency: Stronger models have higher mean effective action ratios, including 73% for Claude-sonnet-4 and 71% for GPT-5, but every backbone has trajectories below 50%.The broad left tails indicate substantial variation in action efficiency even among strong models.
5 Related Work
Related work situates CodeTracer among software engineering agents, executable benchmarks, and emerging process-level trajectory analyses. The paper emphasizes step-level error localization and reproducible supplementary evaluation materials.
- Code Agents and Software Engineering Benchmarks: Software engineering agents increasingly operate in realistic settings such as repository-level bug fixing and long-horizon terminal interaction.Representative systems include SWE-Agent, OpenHands, CodeAct-style agents, and Agentless, with benchmarks such as SWE-bench and Terminal-Bench.
- Trajectory Error Localization, Debugging, and Replay: Recent trajectory research and benchmarks use process-level quality analysis, step-level supervision, and error identification to expose failures missed by outcome-only evaluation.CodeTracer’s localization objective is positioned within this broader shift toward process-level diagnosis.
- Reproducibility: The appendix provides annotation, framework, metric, prompt, schema, and extended empirical details for reproducibility.Additional breakdowns cover backbones, frameworks, difficulty settings, and iteration budgets.
6 Conclusion
The paper presents CODETRACER and CODETRACEBENCH as tools for tracing and diagnosing code-agent failures at scale. Its study finds recurring evidence-to-action failures and strong localization and replay results under matched budgets.
- Contributions: CODETRACER converts heterogeneous logs into hierarchical traces and localizes failure onset, while CODETRACEBENCH provides thousands of annotated trajectories across four frameworks, five backbones, and diverse tasks.The benchmark includes step-level annotations for process diagnosis.
- Findings: Agents frequently gather relevant diagnostic evidence but fail to translate it into effective actions, while extra orchestration and iteration yield diminishing returns after the backbone reasoning ceiling.Error types also shift predictably across workflow stages.
- Results: CODETRACER reaches up to 48% macro F1 with lower token cost and consistently recovers originally failed runs through reflective replay under matched budgets.These results compare against raw log prompting and lightweight baselines.
Limitations
The study’s conclusions are bounded by benchmark coverage, annotator-dependent labels, and offline replay evaluation. These constraints limit transferability and leave interactive deployment behavior unresolved.
- Scope: CodeTraceBench does not cover the full design space of software engineering agents or real-world repositories, so findings may not transfer uniformly across settings.The stated boundary includes other frameworks, domains, and deployment environments.
- Annotation: Failure-critical labels involve annotator interpretation, especially for long trajectories containing intertwined exploration and state changes.The annotation guidelines aim for consistency but do not eliminate this judgment dependence.
- Evaluation Setting: Tracing and replay are evaluated under matched offline budgets on previously collected runs, which does not fully capture interactive oversight or changing environments.The replay experiments also do not by themselves establish a general transfer claim.
A Annotation Guidelines
The annotation scheme labels every trajectory step by workflow stage and identifies steps relevant to success or failure. Failed runs are analyzed backward from observed errors to locate the earliest responsible stage, with explicit edge-case rules.
- Each trajectory step receives an ordered stage label covering environment verification, dependency installation, inspection/debugging, patching, and verification.
- Successful trajectories: Successful trajectories mark redundant actions and trial-and-error actions that are later reverted or superseded.
- Failed trajectories: Failed trajectories are traced backward from failing test output through upstream decisions until the earliest error cause is reached.
- Edge cases: Ambiguous steps default to incorrect, while externally caused setup failures are labeled incorrect only when the agent chose an avoidable path.
- The appendix supplies annotation tooling interfaces alongside the schema definitions.
B Tracing Framework Details
CODETRACER combines trace scoring with reflective replay to diagnose failure stages and support recovery. Its per-run diagnosis cost scales linearly with the number of stages, while extraction is amortized across runs sharing a format.
- Scoring features: Candidate stages are ranked using verification regressions, diff magnitude, backtrack frequency, and exploration-to-action ratio.
- Replay protocol: Reflective replay presents the diagnosed failure stage and extracted evidence to the backbone, requesting a corrective action under the original token budget.
- Complexity: Per-run diagnosis scales linearly with stage count, typically 3–15 stages per trajectory.
- Complexity: Evolving extraction runs once per unique framework format and is amortized across runs using that format.
C Evaluation Metrics
The evaluation measures evidence retrieval quality with per-instance macro Precision, Recall, and F1, while also specifying a constrained diagnosis protocol and structured outputs. The protocol requires targeted inspection, evidence-based labeling, and coverage across multiple stages.
- Evaluation Metrics: Recall measures coverage of gold failure-relevant steps, while Precision measures the selectivity of predicted evidence.
- Evaluation Metrics: F1 combines Precision and Recall as their harmonic mean.
- Aggregation: Macro metrics average per-instance scores so every trajectory contributes equally, and total LLM tokens measure efficiency.
- Inspection protocol: The diagnosis workflow uses tree navigation, exact stage spans, and targeted step inspection rather than scanning all trajectory steps.
- Evidence constraints: Every labeled step must have been inspected and supported by concrete evidence tied to the task objective.
- Coverage expectations: Before finalization, the workflow seeks multiple incorrect and unuseful steps across at least three stages when possible.
E Additional Empirical Results
Additional results show that CODETRACER’s advantage is largest for long, multistage trajectories and remains consistent across task categories and backbones. Short, single-stage tasks show smaller differences because their responsible stage is often easy to identify.
- Per category breakdown: CODETRACER ranks above MINI-CODETRACER and Bare LLM across the 26 task categories.
- Per category breakdown: The largest gains occur in build-system, dependency-resolution, and multi-file-refactoring categories with long, multistage failure cascades.
- Per category breakdown: Short, single-stage categories show smaller absolute differences because the failure critical stage is often trivially identifiable.
- Difficulty scaling: +1.2 pp, +3.8 pp, and +5.1 pp are CODETRACER’s F1 advantages over MINI-CODETRACER on short, medium, and long trajectories, respectively.
- Remaining backbones: Kimi-K2-Instruct has slightly lower overall F1 across methods, while Qwen3-Coder-480B performs comparably to DeepSeek-V3.2 on localization.
- Step range error distribution: Across backbone–agent pairings, error-critical steps concentrate at different workflow phases.
F Industrial Agent Analysis: Claude Code
The Claude Code analysis identifies structural differences between industrial and academic agents in tooling, context management, action efficiency, parallelism, and training signals, while cautioning against direct numerical comparison.
- Architecture: Claude Code uses 40+ specialized tools across eight categories, compared with 5–10 tools in typical academic agents.Its architecture also includes context compaction, token budgeting, and feature-gated code paths.
- Key findings: Industrial agents invest heavily in specialized tooling and error-recovery infrastructure, whereas academic agents use narrower general-purpose tool sets.
- Key findings: Sophisticated context management, including compaction, budget tracking, and feature gating, enables longer effective trajectories than academic agents typically support.
- Key findings: A lower exploration-to-change ratio in Claude Code correlates with higher task success, indicating more actions per exploration step.
- Key findings: Parallel tool execution reduces wall-clock time but introduces ordering-sensitivity issues absent from sequential academic frameworks.
- Key findings: Per-step deviation labels from CODETRACER can provide dense training signals for narrowing the behavioral gap between industrial and academic agents.