Source-linked AI summary
AutoKD: Autonomous Knowledge Discovery
Qinwen Ge, Bo Ni, Haowei Fu, Ngoc N. Tran, Erik Blasch, Tyler Derr
TL;DR
Scientific discovery is constrained by human bandwidth, while prior LLM-based systems are largely one-shot and focus on settings where validation is not fully automatable. AutoKD combines iterative computational discovery with a persistent insight graph that accumulates validated findings and guides subsequent inquiry. Across three datasets and complementary evaluation perspectives, it covers known findings and produces substantive discoveries that extend human-driven research.
Problem
Prior LLM-based research systems mainly generate hypotheses in settings where validation is not fully automatable and do not carry findings across one-shot runs.
Method
AutoKD couples an iterative agentic discovery loop with a persistent insight graph that stores validated findings, filters duplicates, and steers later exploration.
Results
Across three diverse datasets, AutoKD covers known findings and produces substantive discoveries under open-ended and conditioned quality evaluations.
Takeaways & Limitations
The framework provides a cumulative form of autonomous knowledge discovery in which accepted findings remain available to guide future inquiry.
Takeaways & Limitations
The current implementation restricts execution to four deterministic statistical templates rather than fully general code generation.
Abstract
from arXiv · showhide
Scientific discovery in data-rich domains is currently constrained by human bandwidth: the growth in the volume and complexity of real-world data far outpaces the rate at which researchers can read, reason, and synthesize. Recent LLM-based multi-agent systems have begun to automate portions of the research cycle, but they target hypothesis generation in settings where validation cannot itself be automated, and each run is one-shot, with no mechanism for findings to accumulate or steer subsequent inquiry. This paper introduces AutoKD, a multi-agent framework for autonomous knowledge discovery that is both computational and cumulative, allowing validated findings to persist and inform subsequent inquiry. Six coordinated LLM agents collaborate in an open-ended discovery loop, where accepted findings are stored in a persistent insight graph that serves as both long-term memory and an exploration-steering mechanism. We evaluate AutoKD on three diverse datasets from two perspectives: Open-ended Quality against published findings, and Conditioned Quality via literature-derived queries. Across both evaluation perspectives, AutoKD covers known findings and surfaces substantive discoveries that complement human-driven research. Our code is available at https://github.com/GeQinwen/AutoKD.
1 Introduction.
AutoKD targets autonomous knowledge discovery as a cumulative, computational process rather than one-shot hypothesis generation. Its persistent insight graph stores validated findings, reduces redundancy, and steers later inquiry, while evaluation compares discoveries with published research.
- AutoKD addresses the gap between rapidly growing data and limited human capacity to read, reason, explore, and synthesize scientific findings.
- Unlike mostly one-shot AI scientist systems, AutoKD performs computational discovery whose validated findings persist and influence subsequent inquiry.
- The framework combines an iterative agentic loop for generating, testing, and validating findings with a persistent insight graph for cross-round accumulation.
- AutoKD evaluates discovery using published findings through open-ended quality of individual insights and conditioned quality of cumulative coverage.
- The insight graph filters duplicate candidates and directs exploration toward under-explored regions so newly accepted findings add substantive knowledge.
- The framework builds on AutoML, classical knowledge discovery, AI scientist systems, and LLM agent capabilities to make discovery iterative, data-driven, and less dependent on human judgment.
3 Problem Definition
AutoKD defines autonomous knowledge discovery as producing validated empirical insights from data through iterative, cumulative inquiry. Its persistent insight graph stores findings, organizes their relations, and guides subsequent exploration and knowledge extraction.
- Autonomous knowledge discovery produces directional, testable findings supported by evidence from a dataset, optionally guided by a natural-language objective.
- The process couples per-round candidate generation, testing, and validation with cross-round retention, organization, and exploration of under-covered regions.
- AutoKD combines iterative knowledge creation with knowledge extraction, which retrieves and synthesizes accumulated insights to answer research queries.
- Each discovery round plans investigations, generates hypotheses, tests them against data, evaluates validity and novelty, and stores accepted findings in the insight graph.
- The updated graph steers later rounds by shifting exploration modes and tightening duplicate thresholds as semantically adjacent insights accumulate.
- The insight graph is persistent memory whose nodes contain validated findings and whose typed edges capture epistemic relations between them.
5 Proposed Literature-Grounded Evaluation for Autonomous Knowledge Discovery
The evaluation uses published findings from the same datasets as reference points for two complementary questions: whether AutoKD produces individually comparable insights and whether accumulated insights cover literature findings. Open-ended comparisons use matched gold claims and blinded human or LLM judgments, while conditioned evaluation turns gold claims into queries answered from the insight graph.
- Published findings on the same datasets provide reference claims for evaluating individual insight quality and cumulative coverage.
- Open-ended quality compares AutoKD’s top-ranked insights with matched gold claims from published research.
- Eight blinded raters—four graduate-trained humans and four LLM judges—evaluate paired findings.
- Raters compare findings on interest, surprise, usefulness, and overall scientific strength, with ties excluded from preference percentages.
- Conditioned quality samples five diverse gold claims per dataset, reformulates them as research questions, and retrieves and synthesizes relevant insights into grounded answers.
- Coverage is scored as More-than-covered, Covered, Partially covered, or Not covered according to how the retrieved graph evidence matches each claim.
6 Experiments
AutoKD is evaluated across three datasets using open-ended comparisons with published claims, conditioned literature-coverage queries, and analyses of graph diversity and cumulative discovery mechanisms. Results show parity or better against matched gold claims, broad thematic coverage, and complementary benefits from persistence, graph structure, and graph-guided steering.
- Experimental setup: AutoKD is evaluated on e-commerce, cross-national social science, and bibliometrics datasets using human and LLM-rater comparisons.The datasets are Amazon Reviews, WVS Wave 7, and SciSciNet; Table 1 aggregates human and LLM results.
- Open-ended quality: AutoKD achieves at- or above-parity with matched gold claims on every evaluated quality dimension.The largest human-evaluation margin is on Surprising; representative cases include pairs unanimously preferred by all four human evaluators.
- Open-ended quality: Overall win rates increase from Amazon to SciSciNet to WVS, tracking dataset complexity and distance from raters’ prior intuitions.Amazon’s gold claims provide a stronger baseline because examples such as longer reviews attracting more helpful votes are well-known folk results.
- Conditioned quality: 33.3% of evaluated queries are covered or more-than-covered, while 66.7% are partially covered and none are rated not covered.Partial coverage often captures the main relationship or theme without matching every variable or condition.
- Conditioned quality: 20.0% of queries are more-than-covered, indicating that the accumulated graph can add details or interaction evidence beyond published findings.The reported coverage analysis therefore includes elaboration of known findings, not only reproduction.
- Insight graph analysis: Diversity reranking reduces concentration on frequent variables and extends top-10 coverage toward less-used variables relative to score-only selection.The comparison uses variable-usage distributions for the full graph, score-only top-10, and reranked top-10.
- Ablation: Removing persistence causes 16.7% exact rediscovery and the largest deep-test drop, −56.7%; flat memory and global-only exploration also reduce depth.The ablation shows complementary benefits from persistence, graph structure, and graph-guided steering under a matched 1,000-round budget.
7 Conclusion.
AutoKD formalizes autonomous agentic knowledge discovery as cumulative, validated empirical inquiry and introduces a framework that maintains and steers an insight graph. Across three datasets, its graph is preferred over matched gold claims on all four open-ended dimensions and retrieves relevant evidence for every literature-grounded query.
- AutoKD defines autonomous agentic knowledge discovery as producing a cumulative, validated body of empirical findings from a dataset.
- The framework couples an iterative discovery loop with a persistent insight graph that accumulates findings, steers exploration toward under-covered regions, and filters duplicates.
- Across three diverse datasets, AutoKD’s insight graph is preferred over matched gold claims on all four open-ended dimensions.
- AutoKD returns relevant evidence for every literature-grounded query.
A AutoKD’s Agent Descriptions
AutoKD coordinates specialized agents to select exploration modes, generate and filter hypotheses, execute statistical tests, evaluate results, and update the insight graph. The graph supports cumulative exploration by retaining accepted findings and organizing their relations.
- Orchestrator: The Orchestrator selects exploration modes and anchor insights to balance deepening existing findings with exploring new territory.
- Orchestrator: Exploration modes extend the frontier locally, deepen findings, resolve contradictions, or explore freely from the schema.
- Generator: The Generator selects a thematic direction, then refines it into concrete testable hypotheses using variable-level metadata and prior rejection information.
- Coder/Runner: The Coder/Runner translates surviving hypotheses into executable Python statistical tests, extracts structured results, and retries failed executions up to k attempts.
- Critic: The Critic filters redundant or ill-formed candidates before execution and routes borderline post-execution results to refinement rather than graph insertion.
- Historian: The Historian retrieves contextual graph neighborhoods and inserts accepted findings with typed relations into the persistent graph.
B Extraction Pipeline: Worked Example
The worked example shows AutoKD’s literature-grounded extraction pipeline answering a SciSciNet query. It presents the query, grounded answer, retrieved insights, and verifier result.
- Figure 5 shows a SciSciNet extraction example containing the query, grounded answer, top-5 retrieved insights, and verifier result.
C Human Comparison with Published Gold Claims
The human comparison uses randomized, blinded A/B presentation of AutoKD insights and published gold claims across three datasets. The hypothesis-space analysis shows why selecting candidates, rather than exhaustively enumerating them, is necessary.
- Human comparison: Thirty insight pairs, ten per dataset, are presented in fixed dataset groups with randomized within-group order and randomized A/B assignment.
- Human comparison: All raters see identical presentations, and neither section headings nor findings reveal which side is system-generated.
- Human comparison: The complete human-evaluation set contains 30 pairs, with 10 pairs each from WVS, Amazon Books, and SciSciNet.
- Hypothesis space: AutoKD supports association, group-difference, interaction, and heterogeneity hypothesis templates over dataset variables and controls.
- Hypothesis space: The two pair-based templates contribute N(N−1) assignments each, while the two triple-based templates contribute N(N−1)(N−2) each before controls.
- Hypothesis space: Adding controls expands the template counts further, and the resulting totals are lower bounds because richer templates would enlarge the space.
- Hypothesis space: At k=0 the WVS space reaches 10^7, while five controls push it beyond 10^16, making exhaustive testing infeasible.
- Implications: The resulting bottleneck is hypothesis selection, which AutoKD addresses through its insight graph and Critic agent.
E.1 Orchestrator modes
The system combines adaptive round selection with deterministic statistical execution and persistent insight linking. Its exploratory analyses remain intended as evidence for discovery and ranking, not causal estimates.
- Orchestrator modes: The Orchestrator samples exploration, refinement, and conflict-resolution modes using graph structure rather than a fixed round schedule.Mode weights adapt to insight count, cluster structure, chain density, and conflict presence.
- Historian edge types: Accepted insights receive deterministic relation labels including Contradicts, Deepens, Extends, and Narrows.The labels encode directional disagreement, depth progression, cross-module bridging, or the fallback relation.
- Statistical execution: Hypotheses are parsed into typed specifications and compiled through a fixed statistical pipeline without an LLM call.Unsupported variables are rejected, and estimators are selected deterministically from the template and data conditions.
- Statistical execution: The pipeline handles binary outcomes with logistic regression and maps the target log-odds coefficient to an odds ratio and standardized effect.For multi-level categorical terms, omnibus effects are unsigned and direction is restricted to individual contrasts.
- Statistical execution: Preprocessing uses complete-case analysis, treatment coding, and recorded coverage without standardization or other data-dependent transformations.Inference uses procedure-specific tests and model-based regression standard errors.
- Statistical execution: Minimum-sample, variance, group-size, and cardinality checks can reject specifications or invalid fits instead of silently changing the hypothesis.The resulting estimates support exploratory discovery and ranking rather than causal inference.
S2.1 Generator.
The Generator first selects high-level themes, then grounds them in retrieved dataset variables to produce concrete DSL hypotheses. Its prompts prioritize cross-module novelty, surprise, and compliance with strict output constraints.
- Theme Selection: Phase 1 selects thematic directions without concrete variables, using exploration state and prior feedback to shape the prompt.The output includes themes, a research question, constructs, relations, confounders, retrieval anchors, and a surprise angle.
- Theme Selection: Thematic generation is instructed to avoid variable codes, remain high-level, and favor surprising connections or subgroup reversals.The prompt explicitly prioritizes ideas that challenge common assumptions or connect distant themes.
- Variable-Level Refiner: Phase 2 uses TF–IDF retrieval to supply variable metadata before refining the general idea into concrete, testable hypotheses.Retrieved cards include names, labels, scales, types, questions, and semantic descriptions, alongside recent insight and cooldown context when applicable.
- Variable-Level Refiner: The refiner outputs exactly 10 hypotheses using only exact dataset columns and one of four DSL templates: assoc, diff, interact, or heterogeneity.Controls are optional, weights use a dedicated syntax, and explanations are prohibited after DSL lines.
- Variable-Level Refiner: Hypothesis priority favors cross-module links and novelty over already represented knowledge, while allowing simple associations early in exploration.The generator can retry when the returned list size does not match the requested batch size.
- Critic interface: The Critic is integrated into the generation workflow through pre-execution novelty estimation and post-execution epistemic-surprise scoring.The pre-execution stage can short-circuit computation before statistical execution.
S2.2 Critic.
The Critic evaluates candidates against retrieved neighboring insights before and after execution. Its post-execution rubric combines mechanism, novelty, and surprise to determine acceptance strength.
- Pre-Execution Novelty Estimate: Before execution, the Critic receives up to ten neighboring insights to estimate candidate novelty relative to local knowledge.It returns a novelty value between 0.0 and 1.0 with a short reason.
- Pre-Execution Novelty Estimate: The pre-execution judge is configured for low-temperature, strict-JSON responses with a 256-token budget.Its task is limited to novelty relative to existing insights and common-sense expectations.
- Post-Execution Epistemic-Surprise Judge: After execution, the Critic receives the empirical finding, dataset domain, and neighboring insights to assess coherence, novelty, and epistemic surprise.This stage evaluates the finding rather than only the hypothesis wording.
- Post-Execution Epistemic-Surprise Judge: A plausible explanatory mechanism is required; findings without one are classified as incoherent and rejected regardless of statistical significance.The rubric labels such cases 3–4 and assigns the action REJECT.
- Post-Execution Epistemic-Surprise Judge: Scores 9–10 denote surprising findings that contradict expectations or reveal subgroup reversals or amplifications, receiving strong acceptance.Lower scores separately identify tautological or arbitrary relationships as trivial.
- Post-Execution Epistemic-Surprise Judge: Scores 5–6 denote foundational findings accepted for quantifying known phenomena, while scores 7–8 denote novel findings eligible for strong acceptance.Novel findings may bridge research areas, reveal moderators, or quantify assumed effects.
RESPONSE FORMAT (JSON ONLY)
The appendix documents the evaluation comparison set and its response-format material. It reports 30 randomized AutoKD–gold claim pairs across three datasets and identifies the complete comparison tables.
- RESPONSE FORMAT (JSON ONLY): The response format requires a mechanism, reasoning, classification, score, and critique in a JSON-only structure.The classification options are TRIVIAL, FOUNDATIONAL, NOVEL, and SURPRISING.
- S3. Complete AutoKD–Gold Claim Comparison Pairs: Tables 10–12 contain all 30 pairs used in the open-ended human evaluation, with 10 pairs per dataset.Each pair matches an AutoKD-discovered insight with a published gold sub-claim.
- S3. Complete AutoKD–Gold Claim Comparison Pairs: The displayed A/B order was randomized to preserve blinding in the evaluation form.
- S3. Complete AutoKD–Gold Claim Comparison Pairs: Table 10 lists complete SciSciNet CS comparison pairs between AutoKD insights and matched published gold claims.
- S3. Complete AutoKD–Gold Claim Comparison Pairs: Table 11 lists complete Amazon Books comparison pairs between AutoKD insights and matched published gold claims.
- S3. Complete AutoKD–Gold Claim Comparison Pairs: Table 12 lists complete WVS comparison pairs between AutoKD insights and matched published gold claims.