PaperlayerView exact revision on arXiv

Source-linked AI summary

Searching for What Matters: Open-Ended Discovery of Multiple Hypotheses with Agentic Data Exploration

Sanchit Kabra, Nurendra Choudhary, Jiangshu Du, Eddie Huang, Danai Koutra, Chandan K Reddy

arXiv:2610.17874v1cs.LG

TL;DR

Existing hypothesis-generation settings often prespecify a question or context, whereas open-ended discovery starts with only a dataset and outcome. The paper introduces OpenHypBench and HypAgent for this task; across the benchmark, HypAgent achieves 0.49 F1 versus 0.35 for the strongest baseline.

  • Problem

    Existing hypothesis-generation settings often narrow discovery by supplying a research question, target, or curated context, unlike discovery from only a dataset and outcome.

    Source: §1 · p. 1

  • Method

    OpenHypBench evaluates discovery in controlled causal worlds, while HypAgent iteratively proposes and refines hypotheses, removing explained outcome variation to redirect later search.

    Source: §1 · pp. 2–3, §4 · p. 4

  • Results

    Across OpenHypBench, HypAgent achieves 0.49 F1 versus 0.35 for the strongest baseline.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Many missed recoveries occur after a relevant variable surfaces, indicating that characterizing a mechanism beyond variable identification remains challenging.

    Source: §1 · p. 3

    The evaluation covers binary outcomes and seven mechanism families in synthetic causal worlds grounded in real-world schemas, not verified relationships in source domains.

    Source: A LIMITATIONS · p. 14

Abstract

from arXiv · show

Discovering what drives an outcome from data is central to science and often requires deciding what is worth investigating before a specific hypothesis is known. Yet most existing hypothesis generation settings substantially narrow this search by providing a research question, target relationship, or curated context. We introduce open-ended multiple hypothesis discovery, where a system receives only a large tabular dataset and an outcome and must discover multiple data-supported hypotheses without knowing in advance the relevant variables, functional forms, conditions, or number of underlying mechanisms. To study this problem, we introduce a benchmark built from structural causal models grounded in real-world data schemas, containing 351 planted hypotheses across 15 tasks and five domains, with 53--65 variables and 1M--10M rows per task. The causal model is hidden from the discovery method and retained only by the evaluator, allowing proposed hypotheses to be tested through interventions and separating observational discovery from causal validation. We further propose HypAgent, which treats the dataset as an interactive environment and performs iterative search by proposing candidate directions, triaging and refining promising hypotheses, and removing explained signal so that each accepted discovery redirects subsequent search. HypAgent achieves 0.47 precision, 0.51 recall, and 0.49 F1, compared with 0.35 F1 for the strongest baseline. Our analysis shows that many failures occur after a relevant variable has already been surfaced, suggesting that correctly characterizing a mechanism's functional form and conditions, rather than merely identifying relevant variables, remains a key challenge in open-ended discovery.

1 INTRODUCTION

The paper frames open-ended discovery as finding multiple data-supported hypotheses from only a tabular dataset and outcome, and introduces an agent and benchmark for iterative search and interventional evaluation.

1 INTRODUCTION

  • Open-ended discovery asks a system to find multiple data-supported hypotheses from only a tabular dataset and outcome, without guidance on variables, functional forms, conditions, or number of mechanisms.The discovery target is unspecified even though the input remains structured tabular data.Source: §1 · pp. 1–2
  • The search space grows with variables, nonlinear relationships, interactions, thresholds, and subgroup conditions, while multiple mechanisms may coexist in the same data.Finding one strong relationship therefore does not establish that discovery is complete.Source: §1 · p. 2
  • OpenHypBench contains 351 planted hypotheses across 15 tasks and five domains, with 53–65 variables and 1M–10M rows per task, and evaluates proposals through interventions on a hidden SCM.The discovery method uses observational data to generate and support candidates, while the evaluator tests interventional validity.Source: §1 · p. 2
  • HypAgent proposes candidate directions, triages and refines promising ones using sampled observations and the full dataset, then removes explained outcome variation to redirect subsequent search.Each accepted discovery changes what is searched next, in a procedure inspired by matching pursuit.Source: §1 · pp. 2–3
  • HypAgent achieves an F1 of 0.49, compared with 0.35 for the strongest baseline, and its recall advantage persists under matched proposal budgets.Most failures occur after a relevant variable has surfaced, pointing to hypothesis refinement as a remaining challenge.Source: §1 · p. 3

2 RELATED WORK

Prior hypothesis-discovery work constrains either the hypothesis space or the objective, whereas this paper targets discovery without predefined targets or a fixed research question. Existing methods also fall short of returning complete hypotheses with conditions, directions, and magnitudes.

Benchmarks for hypothesis discovery

Methods for hypothesis generation

3 PROBLEM FORMULATION

Open-ended hypothesis discovery must produce multiple testable causal claims from an observational dataset and designated outcome without being told what mechanisms or claim components to search for. A hypothesis specifies the variables, relationship, direction, conditions, and effect magnitude, while mechanism identity is distinct from claim correctness.

3 PROBLEM FORMULATION

  • A hypothesis specifies participating and intervened-on variables, their structural relationship to the outcome, effect direction, applicable condition or population, and claimed absolute probability change.The relationship may be monotone, thresholded, context-dependent, or mediated; direction may be mixed across regions or conditions.Source: §3 · p. 4
  • Open-ended Hypothesis Discovery takes an observational dataset and designated outcome and produces testable causal hypotheses about distinct mechanisms without being given the mechanisms or their claim components.The search space includes an unknown number of mechanisms, variables, relationships, conditions, directions, and effect magnitudes.Source: §3 · p. 4
  • Mechanisms are distinct when their variable sets, structural relationships, or conditions differ; wording, direction, and effect magnitude alone do not establish a new mechanism.Direction and magnitude instead determine whether a claim about a mechanism is correct.Source: §3 · p. 4

4 OPENHYPBENCH

OpenHypBench evaluates open-ended hypothesis discovery in controlled data-generating worlds, where causal mechanisms are known to the evaluator but hidden from discovery methods. It preserves realistic variable semantics and distributions while enabling intervention-based assessment of proposed claims.

4 OPENHYPBENCH

  • The benchmark addresses why fixed-answer matching is insufficient: valid claims may differ from planted hypotheses, and correct variables can still have incorrect conditions, directions, or magnitudes.Observational datasets also lack interventional ground truth for arbitrary claims, so OpenHypBench uses controlled worlds with hidden causal mechanisms.Source: §4 · p. 4

4.1 CONSTRUCTING MULTI-MECHANISM DATA-GENERATING WORLDS

  • OpenHypBench comprises 15 tasks across five domains, with 53–65 features, 1M–10M generated observations per task, and 351 planted mechanisms.Tasks retain source datasets’ variable vocabulary, data types, semantics, and empirical marginal distributions while using newly constructed causal mechanisms; Table 2 summarizes the domains and source schemas.Source: §4.1 · p. 4, Table 2 · p. 5
  • Seven hypothesis families define heterogeneous mechanisms, which are composed into shared SCMs with roles such as decoys, moderators, and mediators.The families are monotone threshold, non-monotone, derived-feature, interaction-reversal, subpopulation-specific, Simpson’s-reversal, and mediation. Mechanisms can operate only under specific conditions or feature-space regions, and decoys can correlate with causal variables without directly affecting the outcome; planted mechanisms participate in data generation rather than being post hoc annotations.Source: §4.1 · pp. 4–5

4.2 INTERVENTIONAL EVALUATION OF PROPOSED HYPOTHESES

  • The evaluator tests proposed claims through interventions on the hidden SCM, using 60,000 paired samples per hypothesis to assess form, direction, conditions, and magnitude.Precision measures the intervention-supported share of distinct proposals, while recall measures recovery of planted mechanisms; magnitude is correct by default within 10% of the interventional effect. Discovery methods use observational data, with causal support assessed separately by the evaluator.Source: §4.2 · pp. 5–6

5 HYPAGENT: RESIDUAL-DRIVEN HYPOTHESIS SEARCH

HypAgent searches an outcome iteratively: it removes variation explained by accepted hypotheses, screens proposed directions for residual signal, and refines promising directions into conditional, adjusted hypotheses using observational data alone.

5 HYPAGENT: RESIDUAL-DRIVEN HYPOTHESIS SEARCH

  • To handle datasets too large for an LLM context window, HypAgent uses sampled observations for proposals and triage but runs residualization and refinement over the full dataset.The LLM accesses data through code and receives only resulting statistics or summaries, not the complete table.Source: §5 · p. 6
  • Each search round residualizes the outcome, triages proposed candidates, and refines the strongest candidate, so accepted discoveries redirect subsequent search.The procedure is inspired by matching pursuit and uses observational data without access to the hidden SCM or interventional evaluator.Source: §5 · p. 6
  • HypAgent fits a linear predictor from accepted hypotheses’ variable representations and searches the resulting outcome residual, reducing their fitted signal without assuming a unique causal decomposition.Residualization removes fitted observational variation, not a known causal mechanism.Source: §5 · p. 6
  • Because the candidate space is too large to search systematically, HypAgent uses LLM world knowledge to propose directions and cheaply triages them by how much residual outcome variation they explain.Candidates scoring at least a tunable threshold are retained, and the highest-scoring candidate is selected.Source: §5 · p. 6
  • After triage, HypAgent examines the selected direction across distribution regions, subpopulations, and potential conditioning variables to refine it into a more specific explanation of residual variation.This shift from broad search to exploitation addresses relationships that depend on conditions, subpopulations, or interactions.Source: §5 · p. 7
  • Refinement tests candidate moderators for interactions and adjusts for potential confounders, then specifies the final hypothesis’s variables, conditions, direction, and magnitude.If the relationship disappears after confounder adjustment, the initial evidence is insufficient to support the claim.Source: §5 · p. 7

6 EXPERIMENTS

The experiments evaluate discovery accuracy and efficiency, how hypothesis structure and benchmark dimensions affect recovery, and what HypAgent’s components contribute.

6 EXPERIMENTS

  • The comparison covers eight statistical, causal, LLM-based, and agent baselines, with all LLM-based methods using Claude Sonnet 4.6 and a shared evaluator.The evaluator uses 60,000 post-intervention samples and a 10% effect-magnitude tolerance; confidence intervals use 2,000 cluster-bootstrap resamples over tasks.Source: §6 · p. 7

6.1 RQ1: OVERALL PERFORMANCE ON OPEN-ENDED HYPOTHESIS DISCOVERY

  • HypAgent achieves precision 0.47, recall 0.51, and F1 0.49, versus Claude Code’s strongest-baseline F1 of 0.35.Claude Code has the strongest baseline precision at 0.31, while XGBoost+LLM has the strongest baseline recall at 0.40.Source: Figure 3 · p. 7, §6.1 · pp. 7–8
  • At matched budgets, HypAgent has the highest recall from 1–25 proposals, yet recovers only 51% of planted hypotheses.Statistical and causal methods perform near zero; LLM-based methods trade higher recall for lower precision, and agentic baselines vary. The results indicate that discovery needs expressive hypothesis generation and data-grounded validation.Source: Figure 3 · p. 7, §6.1 · p. 8

6.2 RQ2: EFFECT OF HYPOTHESIS STRUCTURE

  • Compositional hypotheses are easier to recover (0.58–0.84 among stronger methods), while thresholds, Simpson’s reversals, and subgroup-specific hypotheses are difficult; 119 of 172 misses were characterization errors.Wrong-form errors account for 87 misses (50.6%), missed variables for 53 (30.8%), and wrong effect magnitudes for 32 (18.6%).Source: §6.2 · pp. 8–9, Figure 4 · p. 8

6.3 RQ3: FACTORS AFFECTING OPEN-ENDED HYPOTHESIS DISCOVERY

  • HypAgent’s F1 rises from 0.44 at 10^4 rows to 0.61 at 10^7, falls from 0.61 at 50 features to 0.45 at 100, and rises from 0.47 with 5 hypotheses to approximately 0.63 with 50.The analysis attributes greater difficulty to the search space than to available evidence: discovery requires choosing explanations to test without a prespecified target.Source: Figure 5 · p. 9, §6.3 · p. 9

6.4 RQ4: CONTRIBUTION OF HYPAGENT COMPONENTS

  • Residualization promotes diverse searches, triage allocates refinement to stronger residual evidence, and refinement specifies hypotheses while filtering non-robust observational patterns.Across all 15 tasks, refinement has the largest contribution; together, the components support complementary stages of discovery.Source: §6.4 · p. 9

7 CONCLUSION

Open-ended Hypothesis Discovery shifts from explaining a prespecified relationship to finding multiple data-supported hypotheses. OpenHypBench and HypAgent provide a foundation for this setting, while the conclusion emphasizes targeted search, precise mechanism characterization, and independent validation.

7 CONCLUSION

  • HypAgent outperforms all baselines, but missed recoveries often follow the surfacing of a relevant variable, highlighting search and precise relationship characterization as priorities.The authors position OpenHypBench as a controlled benchmark separating observational discovery from interventional validation, and present both it and HypAgent as a foundation for further study.Source: §7 · pp. 9–10

ETHICS STATEMENT

  • OpenHypBench uses synthetic data grounded in public dataset schemas and marginals, whose mechanisms are not established real-world causal relationships or a formal privacy guarantee.HypAgent outputs candidate hypotheses, not actionable scientific or policy recommendations; real-world use requires independent, domain-appropriate validation.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

  • Reproducibility materials cover benchmark construction, interventional evaluation, recall matching, agent configurations, and baseline implementations; future targets require new mechanisms and parameters.Merely resampling observations from a published structural causal model does not create fresh discovery targets.Source: REPRODUCIBILITY STATEMENT · p. 11

A LIMITATIONS · B FUNCTIONAL FORM DEFINITIONS

The paper scopes its evaluation to observational discovery tested through interventions in benchmark causal models, and defines functional forms using empirical-quantile thresholds and subgroup boundaries. It also identifies extensions and broader scientific validation as future work.

A LIMITATIONS

  • OpenHypBench enables precise interventional evaluation with synthetic causal mechanisms grounded in real-world schemas, not verified relationships in their source domains.The evaluation focuses on controlled discovery in large tabular datasets.Source: A LIMITATIONS · p. 14
  • The benchmark covers binary outcomes and seven mechanism families; other outcomes, modalities, and forms of scientific explanation are outside its present scope.Source: A LIMITATIONS · p. 14
  • All LLM-based methods use a common backbone to isolate discovery-procedure differences, while cross-family evaluation is left to future work.Source: A LIMITATIONS · p. 14
  • HypAgent performs observational discovery and is evaluated through interventions in the benchmark SCM, rather than claiming real-world causal identification.Source: A LIMITATIONS · p. 14
  • Evaluation measures interventional support and mechanism recovery under a finite search budget.Source: A LIMITATIONS · p. 14
  • Future evaluation should assess downstream scientific utility with domain experts and use continually refreshed hidden mechanisms.Source: A LIMITATIONS · p. 14

B FUNCTIONAL FORM DEFINITIONS

  • Functional forms use treatment T, moderator or subgroup M, and binary outcome Y, with thresholds and subgroup boundaries fixed from task-variable empirical quantiles.Table 4 summarizes the seven structural forms represented in OpenHypBench.Source: B FUNCTIONAL FORM DEFINITIONS · p. 14

C BENCHMARK DETAILS · C.1 HYPOTHESIS AND SCM CONSTRUCTION

OpenHypBench uses real-world data schemas to define 15 large, binary-outcome tasks, while its task-specific causal graphs, planted mechanisms, and generated rows are constructed for the benchmark. Its hypotheses are curated as distinct causal mechanisms, assembled into task-level structural causal models, and checked for valid generation.

C BENCHMARK DETAILS

  • OpenHypBench spans 15 tasks across census, e-commerce, education, finance, and health, using public datasets for variable vocabulary, types, semantics, and empirical marginal distributions.The causal graph, structural equations, planted mechanisms, and generated rows are constructed specifically for the benchmark.Source: C BENCHMARK DETAILS · pp. 14–15
  • Each task has 53–65 visible features and 1M–10M rows, and the three tasks per domain use distinct source schemas rather than different samples of one schema.Every outcome is binary, and its description specifies the event represented by Y = 1.Source: C BENCHMARK DETAILS · p. 15

C.1 HYPOTHESIS AND SCM CONSTRUCTION

  • Claude Opus 4.8 proposed candidate mechanisms from each task’s domain, schema, outcome, and four few-shot examples, without access to generated observations or the final SCM.Mechanisms were retained when they could be cleanly represented as a tuple h = (Xh, ψ, d, c, mh).Source: §C.1 · p. 15
  • A hypothesis tuple specifies participating variables, structural form, direction, optional population condition, and absolute interventional change in outcome probability; distinct mechanisms differ in variables, form, or non-equivalent conditions.Differences in wording or effect-magnitude estimates alone do not make mechanisms distinct; curated hypotheses become DAGs composed into a task-level graph defining the SCM.Source: §C.1 · p. 15
  • Task variables can be background causes, confounders, moderators, mediators, causes, decoys, or nulls, with causes corresponding to answer-key mechanisms and decoys correlated with causes but lacking outcome terms.Background variables are upstream without outcome terms; confounders affect treatment and outcome, moderators alter mechanisms through interactions or gates, mediators lie on treatment–outcome paths, and nulls are independent.Source: §C.1 · p. 15
  • Before materialization, specifications with undefined parents, directed cycles, missing moderators, or numerical primitives applied to categorical variables are rejected to ensure valid topological generation.Source: §C.1 · p. 16

D EVALUATION PROTOCOL … D.2 HYPAGENT CONFIGURATION

The protocol evaluates standardized hypotheses for intervention support and matches supported claims against planted mechanisms. HypAgent iteratively searches residual signal, refining selected directions into claims that must pass validation before acceptance.

D EVALUATION PROTOCOL

  • Proposals specify a natural-language claim, variables, direction, magnitude, and optional condition; precision measures intervention support, while recall counts matches to planted mechanisms.Proposals are deduplicated by mechanism identity; supported claims matching no planted mechanism count toward precision but not recall.Source: D EVALUATION PROTOCOL · p. 16

D.1.1 RESIDUALIZATION

  • Residualization jointly refits the original binary outcome using five-bin indicators for continuous variables and one-hot encoding for categorical variables named by accepted hypotheses.The fit uses the full dataset and is recomputed after each acceptance; rejected hypotheses remain in memory but do not enter the fit.Source: §D.1.1 · p. 16

D.1.2 PROPOSAL AND TRIAGE

  • Each round proposes six directions from a randomly queried data slice; triage scores each using 1,000 observations and sends the highest-scoring candidate above 0.02 to refinement.If no candidate clears the threshold, search terminates; residual fitting and refinement use the full dataset.Source: §D.1.2 · p. 16

D.1.3 REFINEMENT

  • Refinement allows up to 10 code turns to analyze the full observational data and choose functional forms, conditions, thresholds, moderators, or derived features.The final claim states its variables, direction, magnitude, and any condition; magnitude is estimated from the original binary outcome, and an absent condition means a full-population contrast.Source: §D.1.3 · p. 17
  • Acceptance requires the claimed direction to agree with the recomputed direction and the adjusted effect to exceed the 0.02 rejection threshold.Mixed-direction agreement is checked within each stated region or condition; adjustment uses 12 covariates, and the threshold corresponds to a two-percentage-point outcome difference over the treatment interdecile range.Source: §D.1.3 · p. 17

D.2 HYPAGENT CONFIGURATION

  • Runs terminated after 20–36 refinement attempts, averaging 26.3, when no candidate in the current batch cleared the triage threshold.The refinement budget was 50 attempts and did not bind in reported runs.Source: §D.1.3 · p. 17
  • HypAgent uses Claude Sonnet 4.6, a 0.02 triage threshold, five continuous-variable quantile bins, and 12 observed covariates for confounding checks.Proposal, moderator-scan, and validation temperatures are 0.7, 0.6, and 0.5, respectively.Source: §D.2 · p. 17

D.1 HYPAGENT IMPLEMENTATION

  • HypAgent repeats residualization, six-direction proposal, triage, refinement, and acceptance checks; accepted claims update the search, while rejected complete claims are stored rather than retried.Each refinement attempt consumes budget whether accepted or rejected.Source: §D.1.3 · p. 17, §D.2 · p. 18

D.3 BASELINE IMPLEMENTATIONS · D.4 FAILURE ANALYSIS · E PROMPT TEMPLATES

The appendix specifies baseline implementations, the scope of HypAgent’s missed-hypothesis analysis, and the LLM prompt stages used in benchmark construction and discovery.

D.3 BASELINE IMPLEMENTATIONS

  • All methods’ hypotheses pass through a shared parser and interventional evaluator, and all LLM-based methods use Claude Sonnet 4.6.This provides a common evaluation pipeline across the baselines.Source: §D.3 · p. 18
  • Correlation Mining ranks features by point-biserial correlation and returns monotone hypotheses, while Subgroup Analysis ranks single-feature subgroups by outcome lift.Source: §D.3 · p. 18
  • PC uses conditional-independence tests and returns outcome-adjacent edges as candidate causal relationships.Source: §D.3 · p. 18
  • XGBoost+LLM provides feature importances, distribution summaries, and 1,000 sampled rows to an LLM that formulates complete hypotheses.Source: §D.3 · p. 18
  • LLM (no code) receives domain, schema, outcome, distribution summaries, and 1,000 sampled rows, while ReAct interleaves reasoning with code execution.Claude Code can execute analysis code over the complete table; AutoDiscovery uses its original iterative proposal-and-testing procedure.Source: §D.3 · p. 18
  • Figure 6 reports precision, recall, and F1 at each method’s reported operating point, with 95% cluster-bootstrap confidence intervals over tasks.Source: Figure 6 · p. 19

D.4 FAILURE ANALYSIS

  • Figure 7 covers failure modes among the 172 planted hypotheses missed by HypAgent.Source: Figure 7 · p. 19

E PROMPT TEMPLATES

  • The prompt appendix covers benchmark-mechanism construction and HypAgent’s two LLM-driven stages; triage, confounder adjustment, and final validity checks are programmatic.Source: E PROMPT TEMPLATES · p. 19

E.1 BENCHMARK CANDIDATE CONSTRUCTION · Benchmark Candidate Construction

Benchmark candidates are generated separately for each target functional form, with at least three proposed per form and only structurally checked mechanisms retained. The construction prompt specifies how candidates must be represented and constrained as causal hypotheses.

E.1 BENCHMARK CANDIDATE CONSTRUCTION

  • For each target functional form, the benchmark requests at least three candidates and retains only mechanisms that pass structural checks.Source: §E.1 · p. 19

Benchmark Candidate Construction

Related papers

Loading 2610.17874v1…