Atlas: Making Sense of Long Horizon Agent Trajectories

1 INTRODUCTION

• After returning to an unobserved run, researchers must recover consequential decisions, relationships among outputs, and connections to their research goal.Long-horizon workflows can accumulate many decisions and heterogeneous outputs over hours or days.Source: §1 · p. 2
• Asking an agent for summaries or explanations only partly helps, because researchers may not know which decisions, abandoned directions, or artifact relationships merit attention.Researchers still have to reconstruct how those elements connect across the run.Source: §1 · p. 2
• In a formative study, researchers described three linked challenges: understanding agent decisions, synthesizing outputs, and connecting both to research goals.The study involved 12 researchers and research engineers.Source: §1 · pp. 2–3
• Atlas links the agent’s decision trajectory and artifacts to the research goal, supporting shifts among artifact representations and traversal across goals, decisions, and artifacts.The workspace combines a trajectory view with multiple coordinated artifact representations.Source: §1 · p. 2
• Atlas participants reported higher decision transparency, artifact sensemaking, and goal connection, together with substantially lower subjective workload than with a standard chat-and-artifact workspace.The within-subjects study involved 14 researchers reviewing two long-running research runs in counterbalanced interface conditions.Source: §1 · pp. 2–3

2.1 Agentic Systems for Scientific Discovery

• Autonomous scientific systems demonstrate agents’ capacity for extended, multi-stage research, while leaving a complementary question of how researchers evaluate work performed in their absence.Examples include end-to-end research agents and domain-specific systems for biomedical hypothesis generation and code-based experimentation [28].Source: §2.1 · p. 3

2.2 Human-Agent Interaction Before and During Long-Horizon Tasks

• Human-agent interaction systems support co-planning, co-execution, goal specification, or oversight while work unfolds, whereas Atlas addresses researchers’ return after autonomous execution.The cited systems include Cocoa, Magentic-UI [32], Compass versus Railway Tracks, and OnGoal.Source: §2.2 · p. 3

2.3 Agent Provenance, Trajectory, and Execution Visualization

• Workflow and agent provenance systems capture or visualize execution traces for inspection, diagnosis, auditing, or reuse; Atlas links provenance to multiple representations and research goals.Related systems include VisTrails, AgentTrails, and Graph of Trace.Source: §2.3 · pp. 3–4

2.4 Research Sensemaking and Multiple Representations

• Research sensemaking tools externalize organization of scholarly materials, but often assume researchers understand provenance and work with relatively homogeneous artifacts.Agent runs can instead combine papers, hypotheses, experiments, analyses, and syntheses, with decisions researchers did not directly observe shaping what gets created.Source: §2.4 · p. 4
• Atlas supports shifts among coordinated representations of the same heterogeneous artifact set while linking artifacts to decisions and research goals through provenance.This enables representational movement within the artifact level and traversal across levels of a run.Source: §2.4 · p. 4, §2.3 · p. 4

3.1 Method

• The formative interview study involved 12 researchers and research engineers using AI coding agents on complex research tasks lasting from under 30 minutes to over 19 hours.Interviews lasted 40–70 minutes and were analyzed through iterative open coding.Source: §3.1 · p. 4

3.2 Findings

• Participants devised tools including task graphs, clickable decision points, research-objective files, and knowledge graphs to check tasks, revisit choices, and preserve hypothesis relationships.Their workarounds included a task graph checked against its originating specification and a knowledge graph for supporting and contradicting hypotheses across sessions.Source: §3.2.1 · p. 4

3.2.1 Challenge 1: Goal Provenance

• Participants struggled to trace how agent decisions and outputs connected to research goals, so they developed ways to preserve those connections and assess relevance.They reported that written plans left execution choices open, requiring post-run reconstruction of how those choices related to their goals.Source: §3.2.1 · pp. 4–5

3.2.2 Challenge 2: Decision Transparency

• All 12 participants described difficulty reconstructing what the agent chose and why, across both research-direction decisions and implementation choices.Written plans did not fully specify which directions agents pursued, which hypotheses they prioritized, or how they resolved ambiguities.Source: §3.2.2 · p. 5, §3.2.1 · p. 5
• Participants reported hidden strategic choices and implementation errors, including an experiment invalidated by an incorrect if-statement and a hardcoded sampling limit of N=20 instead of N=2000.Some errors were caught only when researchers noticed implausible assumptions or suspicious results.Source: §3.2.2 · p. 5
• Participants described delegating more when agent decisions appeared relevant and productive, and pulling back when decision visibility was lacking.One participant followed the agent’s top recommendation 80% of the time when potential decisions were presented with explicit rationale.Source: §3.2.2 · p. 5

3.2.3 Challenge 3: Artifact Sensemaking

• All participants described difficulty making sense of outputs and synthesizing which findings related to hypotheses, agreed or contradicted, and suggested future directions.Participants often or always felt overwhelmed by output volume and built visualizations such as evidence matrices to reveal patterns.Source: §3.2.3 · p. 5

3.2.4 Summary and Design Implications

• The challenges converge on connecting what the run produced, which decisions shaped it, and how the work served the research goal.Participants also struggled to trace connections among those three questions, motivating a representation framework.Source: §3.2.2 · p. 5, §3.2.4 · p. 6, §3.2.3 · p. 6

4 A FRAMEWORK FOR SENSEMAKING OVER LONG-HORIZON AGENT RUNS

• The framework links three levels of a run—goal, decision, and artifact—to persistent anchoring, structural visibility, and representational plurality, respectively.Provenance links support traversal among artifacts, the decisions that shaped them, and the goals those decisions were intended to serve.Source: §4 · p. 6

4.1 Current Approach: Linear Execution Logs as the Default Representation

• Linear execution logs support sequential review but make comparing distant findings, reconnecting outputs to goals, and recovering abandoned directions more demanding.Their sequential structure may not match the sensemaking question, prompting researchers to construct alternative representations.Source: §4.1 · pp. 6–7

4.2 Goal Level: Persistent Anchoring

• Persistent anchoring keeps the research goal accessible and preserves traceable connections to decisions and artifacts, helping researchers evaluate whether the agent’s work served their investigation.Researchers also need to assess whether the work remained relevant to their goal, not merely whether it followed an initial plan.Source: §4.2 · p. 7

4.3 Decision Level: Structural Visibility

• Structural visibility represents consequential choices, alternatives, and their relationships, including what the agent considered, set aside, why, and what artifacts resulted.A branching representation can distinguish pursued, rejected, and deferred directions, while artifact links can expose downstream effects of implementation choices.Source: §4.3 · p. 7

4.4 Artifact Level: Representational Plurality

• Participants’ matrices, dependency graphs, and task graphs illustrate the need for representations suited to different artifact questions.These examples reflect researchers constructing and adapting representations themselves.Source: §4.4 · p. 7
• Coordinated representational plurality lets researchers inspect the same artifact set through multiple question-suited views while retaining context across shifts.Runs can include heterogeneous artifacts, and cross-artifact questions concern patterns, disagreements, and relationships that individual inspection may not reveal.Source: §4.4 · p. 7

4.5 From Representational Requirements to Design Goals

• The three design goals are to anchor research goals and provenance, expose agent decisions structurally, and support cross-artifact sensemaking through coordinated representations.Atlas implements them in a unified canvas linking goals, decision trajectories, and multiple coordinated views of the same artifacts.Source: §4.5 · p. 7

5 SYSTEM: ATLAS

• A unified canvas links artifacts to the agent’s decision trajectory, letting researchers move from outputs to the decisions that shaped them and back.The two modes are artifact projections for cross-artifact sensemaking and a trajectory tree for inspecting decisions.Source: §5 · p. 8

5.1 Artifact Projections (DG3)

• Six projections show persistent artifacts: five organize attributes, while the graph displays typed relationships among them.Selection, focus, and bookmarks persist across projections; paper dimensions can be selected or generated, while hypotheses and findings use specified axes.Source: §5.1 · p. 8

5.2 Trajectory Tree (DG1, DG2)

• The trajectory tree roots the run in its research goal; agent-turn nodes branch into pursued, deferred, and ruled-out paths.Node cards connect steps to their artifacts and intended goal contribution; ruled-out paths record closure reasons, while deferred paths remain open.Source: §5.2 · p. 8

5.3 Usage Scenario

• The scenario links a sparse multi-hop literature cluster to the agent’s decision to rule it out because the available literature left the question underspecified.The researcher flags the direction for possible reconsideration.Source: §5.3 · p. 8
• Paul traces a high-confidence position-bias finding through its hypothesis and experiment, then questions exact-match evaluation and leaves the effect magnitude unresolved.The scenario shows him comparing the experiments’ recorded rationale and inspecting their evaluation methods and results.Source: §5.3 · pp. 8, 10

5.4 Implementation

• The implementation uses React and a 71-tool Claude Agent SDK pipeline; Extended Run repeats five research phases until its goal or time budget is reached.The phases are Explore, Direct, Analyze, Experiment, and Reflect.Source: §5.4 · p. 10

6.1 Procedure

• 14 participants reviewed one run per condition, with system order counterbalanced; the 28 runs averaged 170.4 artifacts and 15.9 trajectory turns.Runs averaged 22.2 deferred options and 6.2 ruled-out options; agent execution averaged 83.9 minutes.Source: §6.1 · pp. 10–11

6.2 Research Questions

• The study asked how Atlas compared with the baseline on trajectory sensemaking, artifact sensemaking, workload, and researchers’ ability to maintain orientation.Review tasks came from formative-study problems rather than Atlas-specific interaction techniques, and participants could use any available functionality.Source: §6.2 · p. 11

6.3 Conditions

• The baseline paired chat with an artifact viewer; Atlas added a canvas with a trajectory tree and coordinated artifact projections linked to trajectory nodes.Baseline users could still prompt the agent for summaries or structured outputs in chat.Source: §6.3 · p. 12

6.4 Participants

• 10 participants reported feeling overwhelmed by output volume, and 9 said completed-run sensemaking was often or always mentally demanding.Five often or always felt uncertain whether outputs addressed their research goal; 13 reported skimming and prioritizing outputs.Source: §6.4 · p. 12

7.1 RQ1: Trajectory Structure Supports Decision, Provenance, and Next-Step Sensemaking

• Atlas made considered-but-abandoned paths and finding provenance more legible; participants used visible deferred options to identify possible next steps.Every participant reported greater confidence identifying next steps with Atlas.Source: §7.1 · p. 14
• Goal connection improved more modestly, with five participants giving identical ratings and every non-tied comparison favoring Atlas.Participants valued seeing how individual steps were intended to contribute to the research goal.Source: §7.1 · p. 14

7.2 RQ2: Artifact Projections Support Cross-Artifact Sensemaking

• 13/14 participants reported better artifact sensemaking with Atlas; projections answered different questions, while substantive scientific judgment remained necessary.Cluster and table views helped with thematic coverage, while the graph helped one participant build a mental model of relationships.Source: §7.2 · p. 14

7 RESULTS

• 13/14 participants reported lower workload in Atlas (baseline median 12.0, Atlas median 8.8, p=.0004), while Atlas rated higher across six post-task measures.The workload reduction was concentrated in mental demand, temporal demand, and frustration; both study blocks had the same time limit.Source: §7 · p. 13, §7.3 · pp. 14–15, Figure 9 · p. 15

7.3 RQ3: Atlas Reduces Subjective Workload but Introduces Learning Costs

• Three participants preferred the baseline for staying oriented, and one participant said Atlas required more time to learn.All three who preferred the baseline for orientation still rated Atlas at or above the baseline on their post-task surveys.Source: §7.3 · pp. 14–15

7.4 Exploratory Deployment Across Sessions

• In the exploratory deployment, all six participants used Atlas for literature search and synthesis, while others also used it for experiments, writing, hypotheses, or dataset analysis.One participant used a synthesis of more than 100 papers, including prevalence estimates, to inform a methodological decision.Source: §7.4 · p. 15
• Participants used provenance to situate outputs in the research process and the trajectory to recover consequential details that chat summaries could miss.One participant used the trajectory to discover that an experiment had failed to retrieve its intended dataset.Source: §7.4 · p. 16
• Different artifact representations served different questions, but participants did not find every representation equally interpretable.One participant found the basis for cluster grouping unclear, while another preferred the table for its interpretability.Source: §7.4 · p. 16

8 DISCUSSION

• Participants reported stronger support for all three challenges and lower subjective workload with Atlas.The three challenges were decision transparency, artifact sensemaking, and goal connection.Source: §8 · p. 16

8.1 Design Implications

• Information availability alone was insufficient: Atlas's structural arrangement helped participants recover consequential alternatives, separating faithful provenance capture from interaction design.The latter determines which relationships become perceptually and cognitively available.Source: §8.1 · p. 16
• Provenance should connect goal, decision, and artifact representations, keeping paths between artifact inspection and trajectory decisions traversable in both directions.Artifacts inspected in tables, clusters, or graphs remain linked to the decision that produced them and the research goal.Source: §8.1 · p. 16
• AI-generated projections need interpretable construction and visible treatment of artifacts that fit poorly or remain unplaced, or flexibility can relocate opacity into the representation.Source: §8.1 · p. 16

8.2 Future Work

• Deployment surfaced needs for chat-to-view routing, easier return to prior trajectory context, and recording why branching decisions fit the research goal.Source: §8.2 · p. 16
• Future work could add persistent artifact annotations and custom objects, and connect post-hoc trajectories to steering actions such as reopening branches or revising decisions.Participants also wanted checkpoints before consequential decisions; editable trajectories could support interventions without restarting a run.Source: §8.2 · p. 17
• The framework's requirements may differ across exploratory, procedural, creative, and multi-agent work, where verification, exceptions, or emergent decisions may matter differently.Future work should test generalization and develop multiscale provenance for parallel or interacting agents.Source: §8.2 · p. 17

8.3 Limitations

• Evidence is bounded to retrospective review, subjective outcomes, brief first-use sessions, variable runs, a small descriptive deployment, and predominantly single-agent scientific workflows.The six-person, approximately one-week deployment is descriptive rather than evidence of long-term effectiveness or stable work practices.Source: §8.3 · p. 17

• In a within-subjects study with 14 researchers, participants reported higher decision transparency, artifact sensemaking, goal connection, and action confidence, with substantially lower subjective workload.Source: §9 · p. 17
• Atlas connects multiple artifact representations through provenance to the trajectory and research goal, addressing challenges in evaluating goal service, recovering decisions, and synthesizing outputs.Source: §9 · p. 17
• The conclusion frames interpretation as an increasingly important bottleneck as agents take on more scientific work, and points toward workspaces that make goals, decisions, and artifacts legible and connected.Source: §9 · p. 17

A.1 Extended Run Execution

• Extended Run gives unattended research tasks a recurring five-phase cycle: Explore, Diverge, Analyze, Experiment, and Reflect.Actions, tools, and directions within each phase depend on the run state and research goal; phase outputs inform later phases and cycles.Source: §A.1 · p. 19
• For the user study, runs used the same phase orchestration and underlying prompts despite differences in research questions and activity or quantity constraints.Source: §A.1 · p. 19

B.1 Research Goals

• Participant research goals retained their substantive questions while being converted into common run specifications fixing required activities and numeric limits.Source: §B.1 · p. 19

• The standing instruction was identical across conditions; the opened thread supplied the concrete research goal.Source: §B.2 · p. 19

B.3 Study Run Statistics

• Each of 14 participants reviewed two independently generated runs, one per condition, for 28 runs using the same generation pipeline, phase prompts, and tools.Total artifacts comprise papers, hypotheses, findings, syntheses, and experiments; non-paper artifacts were also reported separately because literature retrieval varied substantially.Source: §B.3 · p. 19
• 108.5 papers versus 23.3 non-paper artifacts was the within-participant standard deviation of Atlas-minus-baseline volume differences, while findings were absent in 8/14 baseline and 11/14 Atlas runs.Hypothesis and synthesis counts were particularly similar across conditions.Source: §B.3 · p. 19
• Atlas screenshots show the same artifact set shifting from list and PDF views to cluster, timeline, and graph projections as questions change.The artifact set is preserved across representations.Source: Figure 10 · p. 20
• The Atlas artifact viewer includes papers, syntheses, hypotheses, experiments, and findings, which participants browsed in combination during review tasks.Source: Figure 11 · p. 21
• Baseline participants could prompt the agent to produce structured visual outputs in the chat panel.Source: Figure 12 · p. 22

C POST-TASK SURVEY

• Participants completed the survey twice, after each task block and immediately before NASA-TLX, using 7-point Likert items.The five scales instantiate three formative challenges and two predicted outcomes.Source: C POST-TASK SURVEY · p. 22
• Decision transparency, provenance tracing, goal connection, and action confidence contribute to RQ1; artifact sensemaking addresses RQ2; orientation support and NASA-TLX address RQ3.Source: C POST-TASK SURVEY · p. 22
• The study used Extended Run mode; its five phases ran in order, and the study’s scoping questions were left unanswered so the agent chose autonomously.The scoping questions covered duration, research focus, experimental strategy, literature search, and question design.Source: Figure 13 · p. 23, Table 8 · p. 28
• The mean within-participant Atlas-minus-baseline runtime difference was 11.5 minutes (SD=20.9), while the median difference was 1 minute.Ten of 14 paired runs differed by no more than 4 minutes.Source: C POST-TASK SURVEY · p. 29
• The comparative survey asked participants to compare Chat and Chat+Visual based on their experiences reviewing completed agent runs, on a seven-point scale from favoring either interface to no difference.Source: Table 15 · p. 30, Table 16 · p. 31