PaperlayerView exact revision on arXiv

Source-linked AI summary

ForkSCOPE: Charting the Agentic Garden of Forking Paths

Arjun Balaji, Batuhan Duru Yeltekin, Tian Zheng

arXiv:2609.12438v2cs.HCstat.AP

TL;DR

Even with a fixed dataset and question, defensible analytic choices can yield different results, while large agentic analysis corpora create an organization and review challenge. ForkSCOPE induces an auditable fork-option map from code and prose without a fixed taxonomy; applied to 223 analyses, it charted 317 forks and 777 options, with option-layer self-agreement comparable to independent human agreement.

  • Problem

    Scaling agentic multiverses shifts the bottleneck to organizing analyses, while fixed taxonomies can miss genuine choices and understate uncertainty.

    Source: §1 · p. 2

  • Method

    ForkSCOPE uses a human-AI framework to chart agentic analysis corpora bottom-up from code and prose, without a taxonomy fixed before or after generation.

    Source: §1 · p. 2, §4.1 · p. 7, §8 · p. 22

  • Results

    Applied to 223 agent- and human-written analyses, ForkSCOPE induced 317 forks over 777 options, and a three-rater benchmark found option-layer self-agreement comparable to independent human agreement.

    Source: §8 · p. 22

  • Takeaways & Limitations

    The induced fork-option structure matches the axis/level abstraction expected by existing multiverse tools and can be handed to them directly.

    Source: §4.1 · p. 7

    Evidence comes from one in-depth case study, transfer to other domains is untested, and coverage of the full analytic space is unknown.

    Source: §7.2 · p. 21

Abstract

from arXiv · show

Even with a fixed dataset and research question, data analysis involves many defensible decisions. Understanding how these choices influence the results is scientifically important but remains challenging. Crowdsourcing and agentic AI can generate hundreds of end-to-end analyses, but scaling generation alone can create a processing bottleneck and an analytic ``black hole.'' A common workaround is to impose a shared fixed decision taxonomy, which can limit insight and understate uncertainty. We present ForkSCOPE, a human-AI collaboration framework that induces structure bottom-up from the code corpus of end-to-end analyses, without a taxonomy fixed before or after generation, so the organization and evaluation of the garden can scale with the corpus. ForkSCOPE surfaces the charted garden of forking paths through a human-AI collaboration pipeline and an evidence-linked interactive viewer for steering and verification: it spotlights organically identified forks and structures and produces a derived taxonomy and decision map compatible with existing multiverse tools.

1 Introduction

Even with a fixed dataset and question, defensible analytic choices can yield different estimates and narratives. ForkSCOPE charts those choices bottom-up and makes them auditable and inspectable.

1 Introduction

  • Fixed taxonomies can simplify comparison but miss unanticipated choices, while scaling agentic analyses creates a processing bottleneck and risks an analytic black hole.ForkSCOPE addresses the need for bottom-up structure construction and auditability.Source: §1 · p. 2
  • ForkSCOPE extracts raw decision points from code and prose, induces canonical fork-option structure without a prespecified taxonomy, and links decisions to source evidence.The framework keeps human judgment central for assessing forks, assumptions, and further investigation.Source: §1 · pp. 2–3
  • Its interactive viewer supports browsing forks and options, association views, corpus-level studies of pivotal forks and outcome impact, lifecycle summaries, and individual-run inspection.The viewer surfaces where variation concentrates without showing how any single choice would move the reported outcome.Source: §1 · pp. 2–3
  • A case study of an agentic multiverse corpus shows a short-shoulder, long-tail garden, pivotal contested forks, and measurable code-prose gaps.The contribution describes this as an empirical case study of bottom-up charting in practice.Source: §1 · p. 3
  • ForkSCOPE proposes design implications for agentic-multiverse tools: generate from an open-ended space rather than a fixed grid, and audit forks rather than whole runs.It also treats charted gardens as scaffolding for human judgment.Source: §1 · p. 3

2 Related Work

Prior work addresses analytic flexibility, multiverse visualization, agentic analysis, and human oversight. ForkSCOPE positions bottom-up charting and fork-level evidence review as additions across these areas.

2.1 Analytic flexibility and many-analyst studies

  • ForkSCOPE induces options and forks from the analysis corpus without a prewritten list or retrospective codebook, capturing empirical branches beyond a prespecified decision space.Many-analyst studies have instead defined choices before analysis or reconstructed them afterward; uncurated multiverses can also be difficult to justify.Source: §2.1 · p. 4

2.2 Multiverse tooling in HCI and visualization

  • ForkSCOPE works upstream of multiverse visualization tools, producing a canonicalized, option-tagged corpus with a derived taxonomy that can feed Boba-style outcome views or Milliways consonance curves.Existing systems such as explorable multiverse reports and Boba present or compile alternatives when a decision taxonomy is provided [24].Source: §2.2 · p. 4

2.3 AI analysts and agentic multiverses

  • Where agentic-multiverse studies use fixed specifications or taxonomies, ForkSCOPE charts organic decisions and audits evidence and rationale at the option level across runs.Earlier work reported that about a third of 1,120 recovered items fell outside a fixed taxonomy [4]; ForkSCOPE supports selective flagging and cross-run comparison rather than one pass/fail label per script.Source: §2.3 · p. 5

2.4 AI-assisted analysis and human oversight

  • ForkSCOPE keeps humans structurally involved by surfacing unresolved fork-level questions for auditable, asynchronous review, rather than delegating organization to autonomous agents and reviewing only aggregate results.This design draws on interaction patterns for human-AI decision making.Source: §2.4 · p. 5

3 Design Principles for Charting the Agentic Garden of Forking Paths

ForkSCOPE’s design principles pair bottom-up organization with evidence-linked review and human control. The workflow separates mechanical application from judgments that require an owner.

Design principles

  • Induce forks and options from what analyses do instead of fitting them to a fixed taxonomy; the owner determines parsing specifics and auditing rubrics.The workflow uses operational tests, including co-occurrence constraints and a foreclosure test, rather than a prewritten codebook.Source: §3 · p. 6
  • Link summaries to code and report spans so claims are falsifiable, and use numerical and visual comparisons to identify shared structure across analyses.Runs are represented with aligned code and prose spans, while audits produce concrete exhibits under pinned prompts and schema checks.Source: §3 · p. 6
  • Keep the garden owner in control by routing uncertain merges and splits to review rounds, then applying owner decisions deterministically and replayably.Unresolved ambiguities become reviewable questions rather than final verdicts.Source: §3 · p. 6
  • The owner resolves judgments and locks study decisions; a visitor filters, conditions, and compares analyses within the shared fork-option space.The viewer supports systematic inspection of forks and options, corpus-level summaries, and individual-run browsing.Source: §3 · p. 6

4 ForkSCOPE: System Design

ForkSCOPE turns code and reports into a charted garden for systematic review. Its linked data model preserves traceability from individual decisions to recurring options and analytic forks.

4 ForkSCOPE: System Design

  • ForkSCOPE is an auditable instrument for charting decisions from analyses and making them inspectable through concrete actions, pinned prompts, named artifacts, and review gates.The system-design section covers the data model, architecture, staged workflow, collaboration skills, and viewer.Source: §4 · pp. 6–7

4.1 Data model: raw decision points, options, and forks

  • A raw decision point is a concrete action in one run, an option canonicalizes that action across runs, and a fork groups mutually exclusive and equivalent options addressing one analytic question.This option/fork structure matches the axis/level abstraction expected by existing multiverse tools [24].Source: §4.1 · p. 7
  • Records retain run IDs, actions, labels, alternatives, rationales, and source spans through canonicalization, enabling end-to-end traceability and stratification by provenance tags.Scripts are divided into decision-related exons and remaining introns; full line accounting also identifies silent decisions with no corresponding prose.Source: §4.1 · p. 7

4.2 Architecture: scripts, prompts, skills, and agents

ForkSCOPE organizes charting as a reproducible, auditable workflow whose artifacts feed downstream stages, with recurring checks and human review built into its operation.

4.2 Architecture: scripts, prompts, skills, and agents

  • Pinned prompts, response validation, caching, and fixed study context make the charting instrument and its outputs reproducible and verifiable.Changing a prompt invalidates prior answers; rerunning a stage against the cache yields byte-identical output.Source: §4.2 · p. 7
  • Parallel audits produce verdicts, metrics, worst-first exhibits, and source handles; reporting rejects verdicts without exhibits.An auditor attempts to refute findings, while an interpreter turns verdicts into prose without changing the numbers.Source: §4.2 · p. 7
  • The SCOPE workflow distills raw decisions and code-prose links, canonicalizes them into options, organizes options into repaired forks, and plots the results.Evaluation audits run throughout charting rather than only at the end.Source: Figure 2 · p. 8
  • Versioned skills specify which scripts to run, what to check or refuse, and when to request human decisions, supporting consistent attended and autonomous runs.Skills package recurring procedures; scripts and pinned prompts support reproducibility and cost control.Source: §4.2 · p. 8

4.3 Charting the garden: steps, scripts, and prompts

The charting pipeline transforms analysis scripts and write-ups into an auditable fork-option map through segmentation, canonicalization, organization, analysis, and plotting.

4.3.1 Segment: Distillation into Raw Decision Points (step 1)

  • Distillation converts each run’s script and write-up into raw decision points anchored to minimal code spans and any prose spans that describe them.The viewer distinguishes decision-bearing code exons from non-decision introns and colors silent decisions.Source: §4.3.1 · pp. 8–9, Figure 3 · p. 9
  • Code segmentation assigns every line exclusively to a decision span or typed non-decision category, while prose segmentation exhaustively divides text into claims and typed non-claims.Incomplete segmentation is rejected and re-elicited rather than accepted with gaps.Source: §4.3.1 · p. 9
  • Linking code choices to prose claims reveals silent decisions and misalignments as quantifiable complements of required alignment, displayed at corpus and individual-analysis levels.A silent decision is unreferenced by any claim; a misalignment links code and prose that characterize the same action differently.Source: §4.3.1 · p. 9, Figure 4 · p. 10

4.3.2 Canonicalize: From Raw Decision Points to Options (steps 2 to 4c)

  • Canonicalization clusters raw decisions into shared options without a predefined schema, grouping by operational behavior and splitting uncertain cases to avoid fabricating consensus.The detailed viewer aligns extracted decisions with code and prose evidence and highlights silent decisions.Source: §4.3.2 · p. 10
  • Options are induced through three passes—within batches, across duplicate labels, and across underlying questions—then targeted repairs normalize expressions and collapse within-fork paraphrases.The within-fork paraphrase pass is repeated after fork merges.Source: §4.3.2 · p. 10

4.3.3 Organize: From Options to Forks (steps 4b to 4f)

  • Forks group mutually exclusive options answering one analytic question; the foreclosure test separates choices one analyst could make together from rival choices, exempting declared robustness refits.Within-run co-occurrence supplies structural evidence and triggers deterministic splitting when options co-occur in two or more runs.Source: §4.3.3 · p. 10
  • Duplicate forks merge edge by edge only with no pairwise veto and at least 60% judged-same links; repair rounds are logged and stop at a 2% movement threshold or a round cap.Verdicts are persisted by audit scripts and applied by merge scripts without model calls; a final deterministic option split yields the headline vocabulary.Source: §4.3.3 · p. 11

4.3.4 Finalize, Analyze, Audit, Plot, and Gate (steps 5 to 9)

  • Finalization recomputes fork metadata and assigns life-cycle stages; analysis derives corpus summaries, audits enforce evidence requirements, plotting renders stored artifacts, and Gate measures code-to-prose recovery.Stage assignment uses only the fork label, allowing later comparison with script position without circularity.Source: §4.3.4 · p. 11

4.4 LLM-based ForkSCOPE Skills

ForkSCOPE packages charting, review, and map-reading procedures as versioned skills that guide human and command-line agents through checks and decisions.

4.4 LLM-based ForkSCOPE Skills

  • Fourteen skills specify their steps, required checks, decisions deferred to people, and potential model-call costs; they cannot edit prompts or report unaudited numbers.Skills are free when answers are cached.Source: §4.4 · p. 11
  • The /intake interview records corpus facts, domain conventions, research questions, and charting choices as decided, defaulted, or unknown in policy.json.Decision cards present method choices and their consequences before results exist.Source: §4.4 · p. 11
  • Charting skills run the pipeline, audit failures with exhibits, test stability under an empty cache, and lock settled checkpoints by content hash and owner decision.The /distill skill first checks two runs, then spot-checks a stratified sample.Source: §4.4 · p. 11
  • Review skills rank evidence-backed questions, apply the owner’s answers one at a time, rerun audits, and re-derive reported numbers from artifacts.Questions can concern whether forks are the same question or an option bundles multiple decisions.Source: §4.4 · pp. 11, 13
  • Map-reading skills support exploratory triage, tracing claims to source spans, stress-testing audits, recording itineraries, and checking a run against a path specification.The Highway view reveals constituent forks and traversing runs for a selected common route.Source: §4.4 · p. 13, Figure 5 · p. 13

4.5 The viewer

The evidence-linked viewer presents the induced fork-option map through coordinated views for navigating the garden, inspecting pivotal forks and lifecycle patterns, and auditing individual runs.

4.5 The viewer

  • The viewer coordinates five views—Garden, Pivotal Forks, Lifecycle, Corpus Viewer, and About—over the induced fork-option map.Garden arranges forks by data science life-cycle stage and supports sorting, filtering by corpus tags, and search.Source: §4.5 · p. 13
  • Highways summarize common routes through the garden, and selecting one reveals its constituent forks and the runs that traverse it.A fork close-up shows its induced options and the runs realizing each option.Source: §4.5 · p. 13, Figure 6 · p. 14
  • Pivotal Forks displays options, their shares, and aggregate contribution to outcome variation for the analyzed subset, without showing how a specific option or path moves an individual result.The view is intended to focus review on fork and option composition.Source: §4.5 · pp. 13–14
  • Lifecycle summarizes fork distributions across analysis stages and returns to earlier stages, while Corpus Viewer supports audits by browsing runs and their fork-option coverage.About reports the artifacts and vocabulary tags behind each view; Figure 6 illustrates a fork close-up with supporting runs.Source: §4.5 · p. 14

5 Case Study: Charting an Agentic Multiverse from Bertran et al. [4]

Using a pooled case study of agent and human analyses, ForkSCOPE charts an organically induced decision vocabulary. The case study shows that prose omits some code-level decisions and that the garden has a skewed coverage profile with contested forks.

5.1 Corpus and Setting

  • The case study pools 204 AI runs and 19 human teams with scripts, yielding 223 analyses; 33% of 1,120 extracted items fell outside Bertran et al.’s fixed 21-slot schema.The agent corpus is one slice: the soccer dataset, claude-sonnet-4-5, and all five personas; the prior schema files were used only as an external check.Source: §5.1 · pp. 14–15

5.2 Prose loses decisions and leads to silent decisions.

  • Prose captures only ≈0.64 of decisions in AI reports and ≈0.53 in human reports; reported agreement with code is 0.921 versus 0.847, and 7–9% of decisions are silent.The silent decisions are performed in code but never stated in prose.Source: §5.2 · p. 15

5.4 The shape of the garden: a short shoulder and a long tail

  • Most induced forks are reached by too few runs for reliable stability statistics, producing a short shoulder of rarely reached forks and a long tail of widely visited ones.Figure 8 ranks forks by coverage on a log axis.Source: Figure 8 · p. 16, §5.4 · p. 16

5.3 How human and agentic analyses compare

  • Adding 19 human runs shifts option identity only slightly (ARI 0.865) and fork identity more noticeably (ARI 0.756), without substantially reconfiguring the map.The authors caution that this comparison is limited by the 19 human runs.Source: §5.3 · p. 16

5.4 The shape of the garden: a short shoulder and a long tail

  • Analyses frequently iterate among data cleaning, exploratory data analysis, and modeling, and between reporting and modeling.Figure 10 encodes transitions with from-stages in rows and to-stages in columns; below-diagonal counts indicate returns to earlier stages.Source: Figure 10 · p. 17, §5.4 · p. 17

5.5 Pivotal forks

  • A fork is pivotal when at least 25% of analyses reach it and it has more than 1.5 effective options; 17 forks meet both thresholds.Effective options are the inverse Herfindahl-Hirschman index (HHI) over option shares, including a null “never visited” option.Source: §5.5 · p. 17
  • The 36-fork contrast set changes cross-validated R2 only modestly, while the 17 pivotal forks dominate predictive signal; the viewer attributes four outcomes to forks using Shapley decomposition.The contrast set also includes rare, low-diversity forks that can still separate analyses that do and do not reach them.Source: Table 4 · p. 18, §5.5 · p. 18, Figure 12 · p. 18

6 Technical Evaluation

The evaluation tests reproducibility, stability across independent builds, and agreement with human judgments. It finds a stable high-coverage core alongside variability concentrated in rare forks, while also identifying composite forks and reporting computation costs.

6.1 Reproducibility (same build, same input)

  • With caching enabled, rerunning an unchanged stage returns byte-identical output, and versioned vocabularies keep reported numbers traceable to the vocabulary that produced them.The evaluation records exact inputs, models, and prompts for provenance.Source: §6.1 · p. 19, §6 · p. 19

6.2 Stability under replication (independent builds)

  • Across ten identical-input replicates, fork induction achieved ARI = 0.62; the induced vocabulary ranged from roughly 281–317 forks, with most variation in the rarely visited tail.Fork induction contributed the larger share of the shortfall from perfect agreement (58%).Source: §6.2 · pp. 19–20
  • Every fork reached by more than 10 runs reproduced across builds (40 of 40), as did every fork meeting the ≥25% coverage bar (18 of 18); the high-coverage trunk converged while the rare tail did not.Run-level geometry reproduced strongly (Mantel r = 0.766).Source: §6.2 · p. 19, Table 5 · p. 20

6.3 A human benchmark for “natural” variation

  • Two pipeline builds agreed on option judgments 0.849 of the time, within the 0.818 human-rater agreement; raters also flagged composite forks that conflate variable inclusion with transformation.When both builds judged an item to be the same option, human majority agreed on all 19 of 19 pairs.Source: §6.3 · p. 20

6.4 Computation

  • Charting the pooled 223-analysis vocabulary cost $804 in logged API spend across 2,273 model calls.Distillation accounted for $447 across 680 calls; the ten frozen-option fork-formation replicates cost $356 total.Source: §6.4 · p. 20

7 Discussion

The discussion weighs open-ended charting, option-level review, and itinerary use against their costs, then identifies scope, validation, and comparison limits that leave rigorous end-to-end multiverse inference as future work.

7.1 Design implications for AI-analyst tooling

  • 91% of Bertran et al. [4]’s eleven curated decision points were recovered, while 60% of ForkSCOPE’s most-walked decision points lacked a codebook counterpart.The authors note that open-ended generation can surface choices outside delivered code, such as specification search or globally suppressed warnings.Source: §7.1 · p. 21
  • Option-level auditing preserves uncontested parts of a run and yields exhibits tied to a specific fork and option.Whole-run verdicts are more efficient but coarse because one flawed step can fail an otherwise sound run.Source: §7.1 · p. 21
  • Itinerary-walking lets visitors choose defensible options at contested forks, checks internal compatibility, and compares the itinerary with nearby analyses.Because the viewer hides which option would move an individual outcome, visitors must assess options on their own merits rather than seek a preferred conclusion.Source: §7.1 · p. 21
  • Open-ended generation raises charting effort and option-level auditing raises per-decision review, while itinerary use has not yet been evaluated with real visitors.The authors argue these costs are worth paying once analysis generation is cheap enough to shift the bottleneck to interpretation and verification.Source: §7.1 · p. 21

7.2 Limitations and future work

  • Evidence comes from one in-depth case study with untested transfer, and fork-to-outcome attribution is observational and subject to confounding and surrogate error.The corpus reflects one generation design, so coverage of the full analytic space is unknown.Source: §7.2 · p. 21
  • The chart is sensitive to the Claude models used, and validation for the task is limited to annotator agreement.Switching the reader model from Opus to Sonnet roughly doubled the observed silent-decision rate, and some induced forks bundle multiple independent choices.Source: §7.2 · pp. 21–22
  • Some forks bundle independent choices, garden-owner correction preferences remain unstudied, and the human subset cannot support claims about which decisions each corpus favors.The cross-corpus evidence supports only rough compatibility between induced vocabularies.Source: §7.2 · p. 22
  • Conservative canonicalization makes spurious consensus less likely but likely overestimates the garden’s complexity in the reported counts.The authors identify the stable core as a more reliable indicator.Source: §7.2 · p. 22

8 Conclusion

ForkSCOPE charts agentic analysis corpora bottom-up without a fixed taxonomy. In a pooled corpus of agent- and human-written analyses, it induced a structured garden and showed comparable option-layer agreement with human raters, while providing a viewer and artifacts for downstream inspection.

8 Conclusion

  • ForkSCOPE charts and supports reading an agentic analysis corpus without a taxonomy fixed before or after generation.Source: §8 · p. 22
  • Across 223 pooled analyses, ForkSCOPE induced 317 forks over 777 options, identified pivotal forks, and found option-layer self-agreement comparable to independent human agreement.A modest subset of forks carried much of the out-of-sample predictive signal for reported outcomes.Source: §8 · p. 22
  • ForkSCOPE combines reproducible artifacts and an interactive viewer for systematic inspection, and its fork-option map can seed downstream multiverse visualization and analysis.The viewer supports tags, associations, pivotal-fork impact views, lifecycle dynamics, and run-level browsing.Source: §8 · p. 22
Loading 2609.12438v2…