Source-linked AI summary
Agent ATO: Visualizing Agent Interaction Timelines from Logs
Takuto Kawamoto, Yoshiki Higo, Raula Gaikovina Kula
TL;DR
Console logs record agent events in temporal order but do not clearly present interaction types or their ordering, leaving developers to search long logs for process cues. Agent ATO reconstructs those events into an all-interaction timeline, filtered Discovery, Reading, Writing, and Execution views, and an aligned token-usage graph. In selected case studies, these views exposed differences in edit validation and distinguished high token usage from long LLM output versus large tool results.
Problem
Console logs do not clearly present interaction types or ordering, so developers must read long logs to locate actions and process-level cues.
Source: §I · p. 1Method
Results
Case studies showed that timelines distinguish repeated edit-validation cycles from edits without subsequent test execution, while token views separate high usage from long LLM output versus large tool results.
Source: §V · p. 5Takeaways & Limitations
Trajectory visualization can guide detailed log inspection and support prompt-debugging and agent-design discussions.
Source: §V · p. 5Evidence is limited to one agent, two tasks, selected runs, and unvalidated rule-based tags.
Source: §V · p. 5
Abstract
from arXiv · showhide
AI coding agents are becoming part of developers' workflows, but their behavior is difficult to understand from final code changes alone. During a task, agents interact with software repositories through sequences of actions such as searching for files, reading code, editing programs, and running tests or build commands. These interactions, together with token usage, are often recorded in console logs, but raw logs are difficult for developers to inspect. In this paper, we propose Agent ATO (Agentic Trajectory Observer), a tool for visualizing AI coding agent interaction timelines from console logs. Agent ATO extracts agent interactions, classifies them by command or tool type, and visualizes them as timelines. In addition to an all-interaction timeline, Agent ATO provides filtered timelines that emphasize file discovery, file reading, file editing, and execution while preserving surrounding context. We illustrate how Agent ATO may help developers inspect and compare agent actions using selected runs from two repair tasks. Future work will apply Agent ATO to more agents, tasks, and development environments, and will evaluate whether it reduces the effort needed to compare trajectories.
I. INTRODUCTION
Console logs record agent actions in temporal order but do not clearly show interaction types or their ordering, making process-level cues difficult to locate. Agent ATO turns those logs into classified timelines, and two case studies illustrate how the views support comparisons across repeated runs.
I. INTRODUCTION
- Prior work examined agent trajectories, failure causes, workflow debugging, prompt comparisons, and visual analytics of coding-agent behavior, [3],,, –.The cited work spans trajectory and execution-trace analysis, agent failure analysis, agent-history review, prompt/trial comparison, and coding-agent behavior comparison.Source: §I · p. 1
- Console logs do not clearly present interaction types or ordering, so developers must search long logs for actions and process-level cues such as missing validation after edits.The paper identifies unclear validation after edits and lack of further exploration after failures as cues that are difficult to notice.Source: §I · p. 1
- Agent ATO reconstructs console logs as interaction sequences and provides an all-interaction timeline plus filtered Discovery, Reading, Writing, and Execution timelines.A single interaction can belong to multiple tags, while filtered views preserve temporal context; the prototype was implemented for the Pi coding agent.Source: §I · p. 1
- Two case studies compare repeated runs and show differences in edit-validation behavior and in whether high token usage comes from LLM output or tool results.The runs used the same coding task, prompt, and environment; some token spikes came from long LLM-output segments reconsidering repair policies, while others came from large file-reading results.Source: §I · p. 1
- The paper's third contribution is demonstrating through case studies how the visualization supports run comparison and exposes edit-validation and token-usage differences, without automating diagnosis.The visualization provides a basis for observing and comparing trajectories rather than diagnosing them automatically.Source: §I · p. 2
II. AGENT ATO (AGENTIC TRAJECTORY OBSERVER)
Agent ATO presents agent activity as a sequential visual representation of reconstructed interactions. Its all-interaction timeline preserves execution order so users can inspect when exploration, reading, writing, and execution occur.
II. AGENT ATO (AGENTIC TRAJECTORY OBSERVER)
- The overview combines a complete interaction timeline, filtered perspective timelines, and a token-usage graph aligned to the interaction sequence.Filtered rows highlight interactions for one perspective while leaving others gray for context; the graph separates LLM-output tokens from tool-result tokens.Source: §II · p. 2
A. 1 All-interaction timeline
- The all-interaction timeline reconstructs command and tool-use order from execution event logs while retaining detailed messages and outputs for inspection.Inputs include LLM messages, tool-call events, bash commands, tool results, token usage, and timestamps.Source: A. 1 All-interaction timeline · p. 2
- Left-to-right execution order lets users inspect whether exploration and reading precede writing, whether writing happens early, and whether execution follows it.Each mark represents one interaction.Source: A. 1 All-interaction timeline · p. 2
B. 2 Filtered timeline
Filtered timelines reuse the same interaction sequence while emphasizing Discovery, Reading, Writing, or Execution. Unselected interactions remain visible, preserving temporal context for each perspective.
B. 2 Filtered timeline
- Filtered timelines reuse the same interaction sequence to emphasize four analytical perspectives.The perspectives are Discovery, Reading, Writing, and Execution.Source: B. 2 Filtered timeline · p. 2
- Discovery highlights file- or information-finding operations in blue, Reading highlights content or diff inspection in green, and Writing highlights file or repository changes in red.Examples include ls, find, rg, and grep -l for Discovery; read, cat, grep, and git diff for Reading; and edit, write, and rm for Writing.Source: B. 2 Filtered timeline · p. 2
- Execution highlights tests, builds, linters, type checkers, and scripts in yellow, while other interactions remain gray and multi-tag interactions use blended colors.This lets users focus on one perspective without losing the surrounding temporal context.Source: B. 2 Filtered timeline · p. 2
C. 3 Token-usage View
The token-usage view aligns token counts with the interaction sequence to connect usage peaks to activity. The case captions also contrast repeated edit-validation cycles with edits that lack subsequent execution.
C. 3 Token-usage View
- The aligned token-usage graph separates LLM-output tokens from tool-result tokens, helping users identify what type of activity coincides with high usage.Alignment with the operation sequence allows inspection of both where usage is high and what type of activity occurred.Source: C. 3 Token-usage View · pp. 2–3
- Fig. 2 shows repeated cycles in which Writing interactions are followed by Execution interactions.Source: Figure 2 · p. 3
- Fig. 3 shows several Writing interactions after context gathering without corresponding Execution interactions afterward.Source: Figure 3 · p. 3
III. TECHNICAL DETAILS
Agent ATO’s prototype converts Pi event logs into labeled interactions and tagged timelines. Its static HTML output supports filtered views and can be opened without a separate visualization server.
A. Implementation
- A TypeScript extension records Pi agent events to JSONL, and a Python script generates the visualization.Recorded events include turn boundaries, LLM messages, tool calls and results, bash commands, token usage, and timestamps.Source: A. Implementation · p. 3
- Each interval from turn_start to turn_end is treated as one interaction and labeled by its executed command, pipeline, or tool name.Representative labels include find | grep | head, git diff, and npm test.Source: A. Implementation · p. 3
- Rule-based matching assigns Discovery, Reading, Writing, and Execution tags, then generates static HTML with all-interaction and filtered timelines, token usage, and detail views.The file opens in a browser without a separate visualization server.Source: A. Implementation · p. 3
B. User Interactivity
Agent ATO links timeline overviews to detailed logs and lets users search for recurring interaction patterns. The section then turns to comparing selected runs under the same task, prompt, and environment.
B. User Interactivity
- Hovering reveals an interaction’s representative label and highlights other interactions with the same label; clicking opens details including timing, token usage, results, and outputs.The visualization helps locate log segments for closer inspection rather than replacing the original log.Source: B. User Interactivity · p. 3
- Users can select a consecutive interaction sequence and search elsewhere in the trajectory for the same operation pattern.This supports visual inspection of recurring patterns.Source: B. User Interactivity · p. 3
IV. CASE STUDIES
- The case studies compare selected runs under the same task, prompt, and environment.Source: §IV · p. 3
A. Study Design
The study examines process-level differences rather than whether tasks succeeded, using timelines to locate potential differences and detailed labels or logs to interpret them. It compares repeated Pi-agent runs on two GitHub issues and includes contrasting token-usage patterns.
A. Study Design
- The study does not assess task completion success; it examines edit timing, whether edits are validated, and which interactions use the most tokens.For each case, the timeline first identifies a potential difference, which is then interpreted using labels or logs.Source: A. Study Design · pp. 3–4
- For Case 2, Attempt 1’s largest token-usage spikes mainly came from long LLM output, while Attempt 2’s mainly came from large tool results.Source: Figure 4 · p. 4, Figure 5 · p. 4
- The study ran the same Pi coding agent 10 times on each of two GitHub issues, using the same prompt and execution environment.The tasks were IMMICH issue #28383 and METABASE issue #74530; collected logs included events, commands, results, token usage, and timestamps.Source: A. Study Design · p. 4
B. Case 1: Comparing Trajectories for the Same Issue
Among four IMMICH attempts with similar page-state repair directions, Agent ATO reveals differences in whether edits were validated. The first validation involved extra interactions, while later test executions followed edits immediately.
B. Case 1: Comparing Trajectories for the Same Issue
- The four compared attempts were selected from 10 executions because they shared a page-state repair direction and modified overlapping locations.Source: B. Case 1: Comparing Trajectories for… · p. 4
- Both representative attempts showed Discovery and Reading before Writing, indicating codebase context gathering before edits.Source: B. Case 1: Comparing Trajectories for… · p. 4
- The first test execution followed several additional interactions after the first write, whereas later test executions appeared immediately after writes.The authors suggest the agent may first have needed to identify a relevant test command, then reused it in later cycles.Source: B. Case 1: Comparing Trajectories for… · p. 4
- Among attempts pursuing similar repair directions, one formed repeated edit-validation cycles, while another edited without subsequent validation in the observed trajectory.In Fig. 2, the Execution interactions were test commands; Fig. 3 showed no corresponding validation after several edits.Source: B. Case 1: Comparing Trajectories for… · p. 4
C. Case 2: Token Consumption and Its Causes
The case study shows that high token usage can come from either long LLM-output segments or large tool results, which point to different issues and interventions.
C. Case 2: Token Consumption and Its Causes
- Among two high-token attempts, one had large LLM-output spikes and the other large tool-result spikes, illustrating distinct sources of token use.The comparison was drawn from two selected executions among 10 runs of the same task.Source: C. Case 2: Token Consumption and Its… · p. 5
- For repeated reconsideration of repair policies, prompts could ask the agent to summarize candidate policies, select one, and continue concisely.The paper also suggests that unusually long thinking segments could trigger interruption, summarization, or user confirmation.Source: C. Case 2: Token Consumption and Its… · p. 5
- For large reading results, suggested responses include checking file size, reading partial ranges, searching identifiers first, or summarizing long tool results.Large reading results are not necessarily errors because relevant information may reside in a large file.Source: C. Case 2: Token Consumption and Its… · p. 5
- High token usage can indicate reasoning-control issues when driven by long LLM output, or context-management and partial-reading issues when driven by large tool results.The paper frames these as suggested interpretations, not proof that the agent's final modification was wrong.Source: C. Case 2: Token Consumption and Its… · p. 5
V. CONCLUSION AND FUTURE WORK
Agent ATO visualizes coding-agent trajectories reconstructed from console logs, combining interaction timelines with token usage to help users inspect operation- and token-level differences. Case studies illustrate validation differences and distinct sources of high token usage, while evidence remains limited in scope.
V. CONCLUSION AND FUTURE WORK
- Agent ATO combines an all-interaction timeline, filtered timelines for Discovery, Reading, Writing, and Execution, and an aligned token-usage graph.It aims to show what the agent did, in what order, and where operation- or token-level differences occur, rather than infer hidden intentions or diagnose failures automatically.Source: §V · p. 5
- The case studies distinguished repeated edit-validation cycles from edits without subsequent test execution, and separated high token usage from long LLM output versus large tool results.These observations suggest that trajectory visualization can guide detailed log inspection and support prompt-debugging and agent-design discussions.Source: §V · p. 5
- The evidence is limited to one agent, two tasks, selected runs, and rule-based tags that have not been validated.Future work will evaluate tagging accuracy, compare Agent ATO with raw-log inspection, and study whether observations lead to effective prompt or agent revisions.Source: §V · p. 5