PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent trajectories as programs: fingerprinting and programming coding-agent behavior

Hamidah Oderinwale

arXiv:2606.16988v1cs.SEcs.LG

TL;DR

Traditional program analysis is too static for agent scaffolds, and comparing traces meaningfully requires shared representations. The paper represents traces as procedures and develops tools for analyzing them. Procedural fingerprints distinguish models at 85.7% accuracy versus a random baseline of 11.1%.

  • Problem

    Traditional program analysis is too static for agent scaffolds, and meaningful aggregate comparison of traces requires shared representations.

    Source: §1 · p. 1

  • Method

    The paper represents agent traces as structured procedures, induces action vocabularies, and provides ProcGrep for top-down trace analysis and querying.

    Source: §2.2 · p. 5

  • Results

    85.7% accuracy versus a random baseline of 11.1%: models are distinguishable by their procedural fingerprints alone.

    Source: §2 · p. 3

  • Takeaways & Limitations

    ProcGrep supports comparative study and procedural search, while the authors envision applications including model routing and resource-aware long-horizon training.

    Source: §7 · p. 15

    The reward scores are relatively arbitrary, and dynamic scoring adapted to observed final states remains future work.

    Source: §5.1 · p. 14

Abstract

from arXiv · show

Benchmark scores tell you what an agent got right; they do not tell you how it got there. In this work, we introduce methods for comparing agents procedurally in different contexts, where the model, tasks, and approaches vary. We compare ten agents and find that they are identifiable by their behavioral habits, which we define as fingerprints: a probe over these procedural signatures attributes an unseen trajectory to the correct agent at 85.7% accuracy, controlling for leakage across tasks. We develop procedural representations for agent problem-solving procedures with an emergent vocabulary induction technique that is meant to be maximally compressive to avoid surface-level variation while being expressive enough to unveil the quirks of the models' patterns. We apply our framework to the software engineering evaluation dataset SWE-Bench to study the structural distinctness of agent trajectories and find that behavior is most similar between models from similar release periods and those that are distilled from one another (e.g., a distilled student model and its teacher have a Jensen-Shannon divergence of 0.25, about half the distance between other model pairs). As more models saturate evaluations, we believe that it will be important to probe model behavior along more holistic dimensions than success rates alone. We introduce ProcGrep, a library for auditing and evaluating agents for how they approach tasks at a procedural level given their traces in a top-down fashion. We believe this work has a range of applications to help developers work with and program coding agents, such as task-aware model routing, agent monitoring, and finer-grained cost analysis.

1 Background

Agent developers increasingly shape scaffolds rather than write code directly, but traditional program analysis is too static for this work. The paper motivates shared procedural representations and tools for comparing how agents solve tasks.

1 Background

  • Agent scaffolds wrap models to determine their tools, context acquisition, and reactions to feedback, while traditional program analysis is too static to support this reliably at scale.Meaningful aggregate trace comparison also requires shared representations.Source: §1 · p. 1
  • Chain-of-thought can rationalize behavior post hoc but cannot be treated as a grounded account of action, motivating representations built from traces.The paper proposes turning traces into structured objects for analysis.Source: §1 · p. 2
  • The paper uses procedural analysis to compare trajectories outside controlled settings and extend evaluation beyond task pass-or-fail outcomes.It introduces ProcGrep as a framework for studying procedural variation.Source: §1 · p. 2
  • Procedural representations also make it possible to write queries as procedural specifications and search traces, the inverse of generating plans from formal specifications.The authors identify preference-adapted model routing as a possible avenue for extending their explorations.Source: §1 · p. 2
  • Unlike code-only analysis in siloed environments, the approach extends parsing to multimodal, free-form settings that include natural-language prompts.The authors say this makes a rich set of features observable and interpretable.Source: §1 · p. 2

2 A theory of procedural understanding

The framework represents coding-agent trajectories as procedures that can be compared, searched, and used to distinguish agents. Across ten agents, procedural fingerprints identify the correct agent at 85.7% accuracy, while induced action vocabularies support analyses of procedural diversity and cost.

2 A theory of procedural understanding

  • 85.7% accuracy identifies an agent from its procedural fingerprint, versus an 11.1% random baseline; deterministic scaffolds also have near-perfectly identifiable fingerprints.In pairwise comparisons, Claude-4 and SWE-agent-LM-32B are most identifiable, while GPT-4 and Claude-3 Opus are most confusable (F1 0.20–0.30).Source: §2 · p. 3

2.1 Inducing action vocabularies

  • DARS over-uses search repo→create file by 31.6× and Moatless edit→submit by 15.7×; deterministic agents have the most discriminating action pairs.The analysis also finds semblance of teacher behavior in SWE-agent-LM-32B, distilled from Claude-3.7 Sonnet.Source: §2 · p. 3, Table 2 · p. 4
  • Procedural diversity does not track capability: Claude-3.7 and Claude-4 use 32 and 35 procedures, compared with 42–44 for older Claude-3/3.5 models.The paper describes the lower repertoires of stronger, newer extended-thinking models as counter-intuitive.Source: §2.1 · pp. 3–4

2.2 An information theory of procedures

  • Vocabulary stability is selected by maximizing V-measure, combining homogeneity and completeness; the sweep peaks at K = 192 with V-measure 0.644.V-measure plateaus across K = 128–256 before degrading; actions are extracted from solution patches parsed with an AST and embedded with code context and behavioral descriptions.Source: §2.2 · pp. 4–5
  • The method induces action vocabularies bottom-up, using BPE to chunk actions from generated code rather than tying procedures to task-specific performance.BPE offers better separability than PrefixSpan on the evaluated metrics, though the authors note different methods may favor different goals.Source: §2.1 · p. 3, §2.2 · p. 5
  • Passing trajectories have a higher browsing-to-other-actions ratio, while model and scaffold action patterns vary and inform agent cost forecasts.The analysis reports extended-thinking Claude models spend more time in the shell, Agentless disproportionately browses, and Moatless devotes a large share to testing.Source: §2.2 · p. 6, Figure 3 · p. 7

3 Why procedural representations matter

Procedural representations provide a grounded way to study what agents actually do, including whether their actions match their stated reasoning and how their trajectories can be searched. The section evaluates these representations against natural-language accounts and examines follow-through and descriptive precision.

3 Why procedural representations matter

  • Procedural traces support grounded process analysis by testing what agents actually do against what they say they will do or have done, and by enabling structural search over trajectories.The section frames these as forward and reverse follow-through questions, alongside procedural search.Source: §3 · p. 7
  • GPT-4 and GPT-4o have reverse follow-through rates of 1.0; DARS+R1 and Claude-3 / 3.5 / 3.7 / 4 have rates of 0.875 / 0.857 / 0.833 / 0.800 / 0.750, respectively.Reverse follow-through checks whether actions taken are mentioned in the agent’s reasoning.Source: §3 · p. 7
  • Trace-derived procedural representations beat the random baseline (0.13–0.24) at signaling agent success or failure, while natural-language accounts barely improve on it.The comparison uses F1 on single nearest-neighbor prediction.Source: Table 4 · p. 8
  • Behavioral descriptions are generally verbose and may signal performative chain-of-thought [Boppana et al. 2026]; extended-thinking models resemble the shuffled-rationale baseline.The authors describe extended-thinking models as the most verbose.Source: §3 · pp. 8–9

4 Natural studies with procedural representations

Procedural analyses reveal differences in failure patterns, action diversity, and procedural similarity across models and scaffolds. File-access and edit trajectories also distinguish agent behavior, including patterns associated with task outcomes.

4 Natural studies with procedural representations

  • Compositional problems—familiar subproblems that fail when combined—signal failure more strongly than novel actions, especially for RLHF-trained models and more modestly for classically trained models.Familiar procedures have the lowest failure rates.Source: §4 · p. 9
  • Open-weight agents have about 9% lower failure rates on compositional problems than other model families across the measured points.The reported difference compares open-weight model failure at all points with other model families.Source: §4 · p. 10
  • GPT-4o has the greatest action diversity compared with Claude models, while procedural entropy describes action variance and compression roughly proxies repetitiveness.The paper presents these measures as ways to characterize a model’s procedural space.Source: §4 · p. 10
  • Procedural habits are most strongly associated with scaffolds and model generation; the authors report that a distilled student’s procedural style is largely derived from its teacher.The paper compares divergence between procedural distributions and notes similar trends within model families.Source: §4 · p. 10

4.1 How agents acquire context

  • Claude-3.5 accessed a mean of 2.41 files per task versus an average of 1.66–1.90, but pass rate was not proportional to file access and tended to drop as edits increased.The drop was steeper for older models and milder for newer ones; the authors suggest success is tied to how easy a problem is to pinpoint.Source: §4.1 · p. 11, Figure 7 · p. 11
  • Across 63 tasks Agentless passed that SWE-agent did not, SWE-agent was over twice as likely to make five or more consecutive edits when failing than when succeeding; average steps were 13 for Agentless and 33 for SWE-agent.The authors describe Agentless as localizing the error, patching it, and validating in a clean loop.Source: §4.1 · p. 11

5 Controlled evaluations with procedural controls

Action-level control is underexplored in agent programming and benchmarking. The section demonstrates next-action prediction and procedural rewards as ways to assess and shape agent behavior.

5 Controlled evaluations with procedural controls

  • For Moatless+DeepSeek-V3, 43% of trajectories have edit streaks of at least five, and their fail rate is 80% versus 59% without streaks.For Claude-3.5, pass rate drops from 26% to 11% with a streak of at least five.Source: §5 · p. 12
  • Agentless, Moatless, and open-ended Claude/GPT agents form distinct next-action prediction regimes, with Agentless reaching 82% accuracy (+51 points over baseline).Moatless reaches 73% accuracy (+8 points), while Claude-3.5 reaches 50% accuracy (+33 points); Table 8 reports next-action and next-stage prediction accuracy.Source: Table 8 · p. 12, §5 · p. 12

5.1 Programming procedural rewards

  • Procedural specifications assign partial rewards to milestones such as exploration, implementation, and test verification, and can penalize long edit streaks or missing search.The example specification also rewards completion and running a test before the first edit.Source: §5.1 · pp. 12–13
  • The example gives the Django instance a procedural score of 0.35 despite binary failure, with exploration, implementation, and test verification satisfied and an edit-streak penalty triggered.The authors present this as illustrative; they hand-author the action sequence and feedback rather than closing the loop with a trained model.Source: §5.1 · p. 13
  • Test-driven and patch-driven specifications compare how consistently rollouts follow each regime, while the authors describe their reward weights as relatively arbitrary and use-dependent.They identify dynamic scoring adapted to observed final states in continual-learning contexts as future work.Source: §5.1 · p. 14

6 Do students behave like their teachers? (a distilled model case study)

The distilled student inherits its teacher’s full action vocabulary, and its successful trajectories are more procedurally similar to the teacher’s than its failures. The authors examine this relationship through vocabulary preservation, entropy shifts, and outcome divergence.

6 Do students behave like their teachers? (a distilled model case study)

  • The student inherits the teacher’s full action vocabulary, and successful student trajectories have lower entropy and greater procedural similarity than failures.The authors suggest procedural mimicry could be a means to improve success rates for distilled models.Source: §6 · p. 14
  • At the native level, stratifying by outcome, passing child trajectories share 0.204 of the parent’s tool signatures, versus 0.193 for failing trajectories (∆= 0.01).The analysis uses 498 child trajectories and examines vocabulary preservation, entropy shifts, and outcome divergence.Source: §6 · p. 15
  • The case study compares Claude-3.7 Sonnet as teacher with SWE-agent-LM-32B as child on SWE-bench Verified.Source: Figure 8 · p. 15

7 Conclusion

The conclusion frames agent trajectories as a lens for studying coding-agent behavior beyond task success. It reports that agents can be identified from traces and points to ProcGrep applications and complementary model-diffing analyses.

7 Conclusion

  • The authors frame coding-agent trajectories as a new lens for understanding how agents solve tasks, beyond whether they pass or fail.Source: §7 · p. 15
  • Agents are recoverable from their traces alone at a near 90% rate, compared with a random baseline of 11%.The conclusion also reports that procedural quirks are linked to success and failure, and suggests telemetry could reduce compute spend by inferring outcomes before a full trajectory is complete.Source: §7 · p. 15
  • ProcGrep is proposed for long-horizon training with resource management, and the methods may complement model-diffing approaches to holistic agent analysis [Aranguri and McGrath 2025, Jiralerspong and Bricken 2026].Source: §7 · pp. 15–16

A.1.1 Byte-Pair Encoding

The appendix presents BPE vocabulary induction as a procedure for building a vocabulary from action sequences and returning tokenized sequences.

A.1.1 Byte-Pair Encoding

  • BPE vocabulary induction starts with each action as a token, then repeatedly merges the most frequent adjacent token pair until the target vocabulary size or no pairs remain.The algorithm returns the vocabulary and tokenized sequences.Source: §A.1.1 · p. 19

A.1.2 PrefixSpan

The supplied sections describe PrefixSpan’s sequential-pattern mining and report comparisons involving agent clustering, judge agreement, procedural reward specifications, and trajectory behavior.

A.1.2 PrefixSpan

  • PrefixSpan recursively extends frequent sequential patterns and projects each sequence to the suffix after the extension’s first occurrence.The algorithm returns patterns meeting minimum support σ.Source: §A.1.2 · p. 19

A.2 Classifier prompt

  • Judge agreement is low across models, with mean Cohen scores (κ) of 0.12–0.16.GPT-family models have low agreement with open-weight models, while same-family pairs and open-weight models tend to agree with each other.Source: Table 13 · p. 20

A.3 Procedural reward specifications

  • Both test-driven and patch-driven procedural reward specifications have a maximum attainable score of 0.70 before penalties.The listed specifications reward exploration or localization, implementation, and verification or completion; they also include edit-streak penalties, and one includes a test-thrash penalty.Source: §A.3 · pp. 20–21
  • Figure 9 reports that BPE separates agents better than PrefixSpan by V-measure against agent labels.Its golden set counts clustering as correct when all trajectories from each agent are grouped together.Source: Figure 9 · p. 21
  • The distilled student SWE-agent-LM-32B is closest to teacher Claude-3.7 Sonnet, at JSD = 0.25.Figure 10 compares pairwise divergence across ten agents.Source: Figure 10 · p. 22
  • Pass rate declines non-linearly with trajectory length, from 45% at 1–10 steps to ∼29% beyond 60.Trajectory length is measured by canonical-action count.Source: Figure 11 · p. 22
Loading 2606.16988v1…