PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Grounded Theory for Agent Behavior Analysis at Scale

Zhuoran Lu, Yangyang Yu, Zhuoyan Li, Yibo Meng, Nan Jiang, Chengxi Zang, Jie Gao, Ziang Xiao

arXiv:2608.30391v1cs.CLcs.AI

TL;DR

Agent behavior analysis needs scalable methods that explain failures in long, unfamiliar trajectories beyond pre-defined classifiers. The paper introduces AutoTraceGT, a multi-agent grounded-theory pipeline that iteratively codes trajectories until saturation; across six environments, it recovers most human-taxonomy failure modes, surfaces additional patterns, and supports failure prediction. The authors conclude that grounded theory can scale to thousands of trajectories while retaining grounding and interpretability.

  • Problem

    Agent trajectories can remain brittle and difficult to explain, while quantitative metadata, costly human analysis, and rigid pre-defined classifiers each provide limited scalable coverage.

  • Method

    AutoTraceGT automates open, axial, and theoretical coding with role-specialized agents and a manager that performs sampling, constant comparison, and saturation-based iteration.

  • Results

    Across 7,500+ trajectories in six environments, AutoTraceGT covers 74–91% of failure modes in human taxonomies, surfaces missed patterns, recovers expert accounts, and supports predictive analysis.

  • Takeaways & Limitations

    Grounded theory offers a scalable, inductive, and auditable tool for richer behavioral descriptions and more human-centered agent evaluation.

  • Takeaways & Limitations

    Future work should develop saturation criteria that also account for category-example density and relationship stability.

Abstract

from arXiv · show

Understanding agent behavior requires methods that scale to thousands of trajectories and surface new patterns in long, often unfamiliar tasks where pre-built classifiers fall short. We propose to bring grounded theory into agent trajectory analysis: a six-decade-old qualitative method from the social sciences, with a principled saturation criterion and an auditable trail from data to theory. We propose AutoTraceGT (Automated Trace analysis through Grounded Theory), the first multi-agent pipeline that automates grounded theory on agent trajectories. It iteratively performs open, axial, and theoretical coding until saturation, producing a behavioral taxonomy tailored to each task. Across six trajectory corpora, AutoTraceGT produces codebooks that recover 73-91 percent of the failure modes in human-annotated taxonomies and surface additional patterns that those taxonomies miss. The emergent theoretical narrative aligns with prior expert accounts. Used as a deductive feature space, the codebook outperforms zero-shot and few-shot LLM baselines on downstream failure prediction. These results suggest Grounded Theory offers a scalable analytic tool for ML researchers and agent developers studying what agents actually do.

1. Introduction

Agent capabilities have advanced, but long multi-step trajectories remain brittle and difficult to interpret. AutoTraceGT applies grounded theory to produce scalable, auditable behavioral analyses that iterate until saturation.

  • Long agent trajectories can fail after locally reasonable decisions, obscuring which behaviors support success or lead to failure.
  • Existing quantitative metadata scales but explains little, while human analysis is interpretable yet expensive for hundreds-step trajectories.
  • Pre-defined behavioral classifiers are too rigid for novel tasks and emergent behaviors, leaving ML without a scalable, generalizable behavior-analysis methodology.
  • Grounded theory derives categories from data, iteratively refines them, and stops at theoretical saturation when new sampling yields no new structure.
  • AutoTraceGT combines OpenCode, AxialCode, TheoreticalCode, and Manage to analyze trajectories through layered coding, strategic sampling, comparison, and saturation.
  • Across 7,500+ trajectories, six datasets, and four backbone LLMs, AutoTraceGT reaches saturation, reproduces codebooks, recovers human and expert patterns, and supports failure prediction.

2. Related Work

Grounded theory provides an inductive, auditable framework for deriving agent-behavior explanations from data. AutoTraceGT extends prior LLM-assisted qualitative analysis with end-to-end orchestration and explicit saturation verification.

  • Grounded theory builds theories from data through open, axial, and theoretical coding, preserving links from categories and claims to raw incidents.
  • LLM-assisted qualitative analysis has expanded across thematic, hierarchical, multi-agent, and grounded-theory coding applications.
  • Prior grounded-theory pipelines rarely preserve the full multi-stage structure or clearly specify when analysis should stop.
  • AutoTraceGT addresses these gaps with role-specialized agents, a cross-batch manager, and saturation verification through codebook convergence.
  • Agent benchmarks report task success but reveal little about failure causes, motivating structured failure analyses such as operational taxonomies and manual grounded theory.

3. Multi-Agent Framework

AutoTraceGT distributes grounded-theory analysis across agents operating at trajectory, batch, and corpus scales. Its versioned manager coordinates iterative coding until saturation, after which theoretical coding consolidates the resulting account.

  • Framework overview: AutoTraceGT uses OpenCode, AxialCode, TheoreticalCode, and Manage across single trajectories, batches, the full corpus, and cross-batch analytic state.
  • Saturation: The iterative algorithm samples batches, runs OpenCode and AxialCode, updates the codebook, and stops after sustained rounds with few additions.
  • Open coding: OpenCode labels trajectory incidents with conceptual codes, spans, and short quotes, abstracting recurrent behavior rather than literal tool calls.
  • Axial coding: AxialCode groups batch-level codes into categories and typed relations while retaining definitions, supporting records, and success or failure status.
  • Codebook management: Manage performs constant comparison by reconciling each round against a versioned codebook and logging add, merge, split, or flag revisions.
  • Auditability: Every code, category, and theoretical claim links to supporting evidence, preserving an auditable trail through the coding process.
  • Theoretical coding: After saturation, TheoreticalCode selects a core category and organizes remaining categories into conditions, contexts, strategies, and consequences.

4. Experiments

The experiments evaluate AutoTraceGT’s process reliability and artifact quality across diverse trajectory corpora. It reaches saturation, produces stable codebooks, recovers human-identified failure modes while surfacing additional patterns, and supports behavioral interpretation and prediction.

  • Evaluation setup: AutoTraceGT was evaluated on outcome-labeled and expert-annotated trajectories spanning six environments, with 2,000 trajectories sampled from each outcome-labeled environment.The outcome-labeled corpora are Tau-Bench, Go-Browse, and SWE-Agent; annotated corpora are ALFWorld, GAIA, and WebShop.
  • Process reliability: As coding progresses, add actions taper while merge and confirm actions increase, and the codebook stabilizes in category count and content, indicating theoretical saturation.Termination occurs when the add rate remains below the threshold for consecutive rounds.
  • Process reliability: Codebooks from repeated runs of the same dataset–model configuration are significantly more similar than cross-configuration codebooks, exceeding a data-derived reproducibility threshold.The within-configuration comparison is significant at p<10^-20.
  • Process reliability: Across every dataset, cross-LLM codebook coverage exceeds null baselines, with permutation tests significant at p<0.001, indicating dataset-specific behavioral structure.The comparison includes cross-dataset and permutation-based null distributions.
  • Artifact quality: The codebook covers at least 70% of human failure modes across three datasets, peaking at 90.9% on WebShop, and matches 58–88% of individual trajectory reasonings.Recall can exceed precision because AutoTraceGT also surfaces categories absent from the predefined human taxonomy.
  • Artifact quality: AutoTraceGT identifies distinctions that predefined taxonomies collapse, including inaction, planning–execution decoupling, and mode oscillation without synthesis.These examples span granularity, cross-module structure, and catch-all labeling limitations.
  • Interpretation and prediction: The theoretical account converges with the prior cascade-of-errors analysis, while codebook features provide predictive signal for downstream failure classification.The prediction study represents trajectory prefixes with codebook presence and co-occurrence features and compares AutoTraceGT against few-shot and direct LLM baselines.

5. Conclusion

The paper presents grounded theory as a scalable, inductive approach to analyzing agent behavior and introduces AutoTraceGT as an auditable multi-agent implementation. Across thousands of trajectories, it reports saturation, reproducible codebooks, broad human-taxonomy coverage, additional patterns, theoretical convergence, and utility for interpretive and predictive analysis.

  • Conclusion: AutoTraceGT applies open, axial, and theoretical coding through a multi-agent pipeline with an auditable trail from trajectories to theory.The pipeline is designed to scale grounded theory analysis while preserving grounding and interpretability.
  • Conclusion: On 7,500+ trajectories across six environments, AutoTraceGT reaches saturation and yields stably reproducible codebooks.The reported evaluation also includes coverage of human taxonomies and downstream analytical uses.
  • Conclusion: AutoTraceGT covers 74–91% of failure modes in human taxonomies while surfacing patterns those taxonomies miss and recovering prior expert accounts.The conclusion frames these outputs as supporting both interpretive and predictive trajectory analysis.
  • Limitations and future work: Future work could use richer saturation criteria that account for example density within categories and stability of relationships between categories.The stated goal is to make the algorithmic stopping rule more closely match the methodological idea of saturation.

2 repeat

The stopping condition monitors newly added categories across recent rounds. It terminates only after a minimum number of rounds and sufficiently low add rates for every round in the window.

  • Stopping rule: The add rate a_t counts categories newly added to the codebook in round t.It is defined from the difference between the current and previous codebook category sets.
  • Stopping rule: The algorithm stops when t≥W and a_j<ε for every j in the preceding W rounds.This requires sustained low novelty rather than a single low-add round.

A.1. Detailed algorithm with explicit state

Algorithm 2 makes AutoTraceGT’s state and data flow explicit across iterative trajectory coding and codebook revision. It tracks remaining data, coded records, chunk-level state, axial outputs, and versioned codebooks with revision history.

  • Explicit algorithm state: The explicit state includes the remaining trajectory pool, each round’s batch and coded records, chunk partitions, and a memo carried across chunks.These variables expose how trajectory data and intermediate coding artifacts move through the pipeline.
  • Explicit algorithm state: The algorithm also records each round’s axial output and maintains a versioned codebook with a revision log.The set-builder formulation makes the inter-agent data flow explicit.

A.2. Details of Corpora

The experiments use two trajectory corpora: one with expert-coded failure reasoning and another with only binary success or failure outcomes.

  • The failure-annotated corpus covers ALFWorld, GAIA, and WebShop, with expert-coded failure reasoning for each trajectory.
  • The outcome-labeled corpus covers Tau-Bench, Go-Browse, and SWE-Agent, with only binary success or failure labels.

A.3. Coverage Stability Analysis

Coverage stability is assessed through replicate consistency within dataset–model cells and agreement across backend LLMs, calibrated against cross-cell and null distributions. The results indicate reproducible codebooks and dataset-specific behavioral structure.

  • Within-configuration replicate stability: 12 dataset–backend-model cells each yield three saturated codebooks from disjoint trajectory subsets.The cells combine 3 datasets with 4 backend models.
  • Within-configuration replicate stability: The cross-cell distribution provides a data-derived null for calibrating reproducibility instead of an arbitrary similarity cutoff.
  • Within-configuration replicate stability: 0.929 median cosine similarity occurs among in-cell codebook pairs, versus 0.791 across 594 cross-cell pairs.The cross-cell distribution has a 95th percentile of 0.901, adopted as the high-reproducibility threshold.
  • Within-configuration replicate stability: All 12 cell means exceed the 0.901 high-reproducibility threshold, with in-cell and cross-cell distributions differing at p< 10^-20.
  • Cross-LLM stability on the same dataset: Observed cross-LLM coverage exceeds all three null distributions for every dataset, with permutation tests yielding p< 0.001 for each dataset.The nulls include cross-dataset any-model, cross-dataset same-model, and dataset-label permutation comparisons.

A.4. GLM Fit Diagnositic

The GLM diagnostics assess whether codebook-derived features distinguish successful from failed trajectories and provide an additional check on model fit.

  • A significant likelihood-ratio test at p< 0.001 indicates that codebook-derived features carry information about trajectory success or failure.
  • Table 5 reports model-fit statistics and representative coefficients for the GLM analysis.
  • Table 6 presents examples of emergent behavioral categories used in the analysis.
  • Pearson χ2/df serves as an additional model-fit diagnostic, with values closer to 1 preferred.

A.5. Example emergent categories

AutoTraceGT’s iterative grounded-theory process produces reproducible, human-audited behavioral codebooks that capture known failure modes while surfacing additional categories. Ablations further show benefits from iteration, robustness to prompts and sampling choices, and limits to cross-family generalization.

  • Example emergent categories: Three emergent failure-mode categories were surfaced by open coding but absent from the human taxonomy, one per dataset.These patterns operate at the agent’s decision or action level, beyond the human taxonomy’s module × failure_type schema.
  • Iterative coding: AutoTraceGT achieves higher Recall and Match than a direct one-pass baseline on all three datasets.The comparison uses the same gpt-5-mini backbone and coverage judge, supporting a practical benefit of iterative coding under the tested setup.
  • Human validation: Both annotators judge all 17 categories unmatched to the human taxonomy as valid, with complete agreement on valid-versus-spurious status.The audit distinguishes valid failure-related behaviors from valid failure-neutral behaviors rather than treating all unmatched categories as failures.
  • Sampling robustness: Uniform random sampling reaches saturation in the fewest batches without consistently sacrificing final codebook size, whereas codebook-conditioned sampling converges more slowly.Codebook-conditioned sampling attains approximately 0.99 Chao1 coverage and produces fewer low-frequency categories, but no policy consistently dominates.
  • Scope boundary: Cross-family evidence remains limited: one Gemini-3-Flash model on SWE-Agent cannot establish broad cross-family robustness.The ablation converges after a median of four batches, with mean per-run add rate 0.15 and median final codebook size 11 categories.
  • Robustness: Across-prompt similarity exceeds the 0.901 high-reproducibility threshold and the cross-dataset null, while varying trajectory order produces comparable variation.All six prompt-and-order runs reach saturation after four codebook versions and produce 16–20 categories.
Loading 2608.30391v1…