Source-linked AI summary
HypoForge: A Self-Improving Multi-Agent Framework for Automated Hypothesis Generation and Testing via Scientific Skill Learning
Ziqing Qian, Jiaying Lei, Yifang Wang, Nan Cao
TL;DR
Existing AI scientist systems often rely on static workflows and struggle to accumulate reusable experience across scientific discovery tasks. HypoForge learns stage-specific skills through adversarial critique for hypothesis generation and empirical feedback for hypothesis testing, and experiments show consistent improvements over existing frameworks and skill-level variants without fine-tuning foundation models.
Problem
Existing systems often use static prompting or fixed workflows and cannot effectively accumulate and reuse experience across hypothesis generation and testing.
Method
HypoForge is an experience-guided multi-agent framework that uses adversarial comparative critique for generation and execution outcomes with ground-truth feedback for testing.
Results
HypoForge consistently outperforms existing AI scientist frameworks and skill-level variants in hypothesis quality and testing performance.
Takeaways & Limitations
Stage-specific skill learning improves both hypothesis generation and testing while enabling continual capability improvement without fine-tuning foundation models.
Takeaways & Limitations
Skill refinement depends on the quantity and reliability of feedback, and the framework mainly targets data-driven discovery with executable computational experiments.
Abstract
from arXiv · showhide
Large language models (LLMs) have enabled AI scientist systems to automate scientific discovery, yet existing approaches most rely on static prompting or fixed workflows and fail to accumulate experience for continual improvement. We propose HypoForge, an experience-guided multi-agent framework that learns reusable scientific skills for automated hypothesis generation and hypothesis testing. HypoForge is built on the observation that these two stages involve different supervision signals. For hypothesis generation, where explicit feedback is unavailable, HypoForge adopts an adversarial generator--discriminator mechanism to improve reasoning through comparative critique. For hypothesis testing, where empirical feedback is available, HypoForge learns testing skills from execution outcomes and ground-truth results. By matching skill learning strategies with stage-specific supervision, HypoForge enables continual improvement without fine-tuning foundation models. Experiments on hypothesis generation and testing benchmarks show that HypoForge consistently outperforms existing AI scientist frameworks and skill-level variants. Further analysis demonstrates the effectiveness of the proposed stage-specific skill learning paradigms.
Introduction
Scientific discovery requires both hypothesis generation and hypothesis testing, but existing AI scientist systems often rely on static capabilities and fail to accumulate experience. HypoForge addresses this gap with stage-specific skill learning tailored to the supervision available in each stage.
- Hypothesis generation and testing form the core stages of hypothesis-driven scientific research.Scientists formulate candidate explanations from observations and validate them through statistical analyses and experimental evidence.
- Existing AI scientist systems often focus on one discovery stage or use static prompting and predefined workflows, limiting experience accumulation and reuse.These systems repeatedly expend similar reasoning effort across tasks, reducing discovery efficiency.
- Hypothesis generation lacks explicit supervision, whereas testing provides empirical supervision through executable experiments and ground-truth outcomes.The differing feedback signals make direct quality assessment difficult for generation but enable evaluation of testing design and implementation.
- HypoForge distills accumulated discovery experience into reusable skills through adversarial refinement for generation and feedback-based refinement for testing.The framework aligns each learning strategy with its stage’s supervision characteristics without fine-tuning foundation models.
- HypoForge consistently improves hypothesis generation and testing over existing AI scientist frameworks and skill-level variants.Extensive ablations further verify the effectiveness of its stage-specific skill learning paradigms.
Related Work
Prior work has advanced autonomous scientific discovery, memory, reflection, and iterative refinement, but commonly treats discovery stages and skill learning uniformly. HypoForge instead integrates generation and testing while targeting stage-specific procedural skills.
- Autonomous scientific discovery aims to generate and validate novel hypotheses from empirical evidence through an iterative generation-testing loop.The loop transforms observations into testable explanations and evaluates their evidential support.
- Existing scientific discovery systems typically address only hypothesis generation or testing and rely on fixed pretrained-model reasoning.This limits their ability to accumulate experience and progressively improve scientific reasoning.
- HypoForge integrates hypothesis generation and testing into a unified discovery process while distilling accumulated experiences into reusable reasoning capabilities.
- Memory-augmented, reflective, and self-refining LLM agents support long-term information storage, learning from failures, or iterative feedback without parameter updates.
- Unified skill-learning approaches aggregate feedback across stages, making stage-specific refinement and feedback attribution difficult in complex workflows.HypoForge addresses this problem by explicitly modeling distinct capabilities across stages.
Problem Formulation
HypoForge models scientific discovery as iterative generation of hypotheses and testing trajectories, then updates reusable procedural skills from accumulated experience and feedback. The formulation distinguishes generation feedback from testing feedback.
- Given a research problem P and data source D, the system generates hypotheses H = {h_i} and corresponding testing trajectories V = {τ_i}.
- Each testing trajectory τ_i includes a hypothesis h_i, experimental design e_i, executable code c_i, and execution outcome r_i.
- At iteration t, hypotheses and testing trajectories are generated using current skills and accumulated discovery experiences.
- The framework learns reusable procedural skills for hypothesis generation, experimental design, and experiment execution.These skills are iteratively updated from newly acquired experiences and feedback.
- Hypothesis-generation skill learning uses evaluation feedback F_h, while hypothesis-testing skill learning uses the testing process Φ_t.
Method
HypoForge decomposes scientific discovery into hypothesis generation and testing, learning reusable skills through stage-specific feedback. It uses adversarial, distribution-level critique for generation and experience-based refinement of experiment design and execution for testing.
- Stage-wise framework: HypoForge separates hypothesis generation from testing and learns dedicated skills according to each stage’s reasoning process and supervision signals.The framework uses adversarial learning for generation and empirical outcomes for testing.
- Hypothesis generation: The hypothesis generator produces batches of candidate hypotheses from a research problem and data, using skills accumulated from prior adversarial interactions.Generated sets approximate distributions of plausible discoveries and support joint evaluation of quality, diversity, and discovery coverage.
- Hypothesis generation: A multi-dimensional discriminator evaluates hypothesis sets against quality dimensions including empirical grounding, causal plausibility, falsifiability, theoretical consistency, and novelty.Its measurements combine continuous scores with hard constraints for falsifiability, theoretical consistency, and empirical verifiability.
- Adversarial skill learning: Unlike conventional GANs, HypoForge distills distribution-level discriminator feedback into reusable procedural knowledge rather than using binary classification or gradient back-propagation.The resulting skills include variable selection, causal reasoning, hypothesis construction, and avoidance of common failures, supporting generalization across tasks.
- Hypothesis testing: Testing skills are refined from trajectories and ground-truth outcomes, with the test pass rate guiding iterative improvement of experiment design and execution reliability.The distiller attributes successes and failures to stages of the process and extracts transferable practices rather than memorizing individual cases.
- Hypothesis testing: For hypothesis testing, experiment designers create protocols and execution agents convert them into executable programs whose outcomes include statistical results, effect sizes, and conclusions.Protocols cover operationalization, variable selection, preprocessing, confounder control, statistical methods, significance criteria, and evaluation metrics.
Experiments
Experiments evaluate HypoForge on hypothesis generation and testing using HypoBench tasks, system-level baselines, skill-level variants, held-out splits, and stage-specific metrics. Results show strong performance, iterative skill improvement, and benefits from matching refinement signals to each stage.
- Experimental Setup: HypoBench provides 13 diverse tasks, each containing a research question, associated dataset, and reference hypotheses.Tasks are divided into training and testing sets at a 1:1 ratio for skill learning and evaluation.
- Experimental Setup: HypoForge is compared with scientific discovery systems and skill-level variants lacking, generating, or manually specifying refined skills.The evaluation includes system-level baselines for generation and testing, plus No-Skill, AI-Generated-Skill, and Human-Designed-Skill variants.
- Hypothesis Generation Evaluation: 0.785 Q(Ht) and 0.648 Hit Rate are achieved for hypothesis generation, the best results among compared methods.HypoForge improves Hit Rate over HypoGeniC and HypotheSAEs by 17.6% and 44.0%, respectively, and removing skill learning reduces Hit Rate from 0.648 to 0.409.
- Hypothesis Testing Evaluation: 0.659 test pass rate and 0.966 execution success rate are obtained for hypothesis testing, with the highest test pass rate among compared methods.HypoForge outperforms system-level and skill-level baselines; POPPER’s higher execution success rate is accompanied by a substantially lower test pass rate.
- Skill Learning Analysis: Generation metrics rise from Q(Ht) = 0.612 to 0.808 and Hit@K = 0.208 to 0.679 across refinement, despite intermediate fluctuations.The reported trend attributes refinement progress to discriminator feedback refining hypothesis-construction strategies.
- Skill Learning Analysis: Testing improves from Th = 0.395 to 0.669 and Eh = 0.516 to 1.000 across refinement iterations.The results indicate progressive capture of more effective experiment design and execution procedures.
- Ablation Study: Removing discriminator feedback lowers generation performance, while removing execution outcomes lowers Th from 0.659 to 0.565 with comparable Eh.The ablation results support stage-specific refinement signals for generation and testing skill learning.
Discussion
The framework’s benefits depend on reliable feedback, remain focused on executable data-driven discovery, and face open challenges in transferring and organizing skills.
- Skill refinement depends on the quantity and reliability of feedback information.Errors in hypothesis evaluation or insufficient ground-truth validation can produce suboptimal skill updates.
- The framework mainly targets data-driven scientific discovery with executable computational experiments.Extending skill learning to theoretical reasoning, wet-lab research, and interdisciplinary knowledge integration remains future work.
- Automatically determining scientific-skill granularity, composition, and organization remains an open challenge.This challenge concerns effective task performance in complex discovery pipelines, even when learned skills may transfer across tasks.
Conclusion
HypoForge is an experience-guided multi-agent framework that learns reusable procedural skills for hypothesis generation and testing. Experiments show consistent improvements across both stages, with learned skills generalizing across diverse scientific tasks.
- HypoForge is an experience-guided multi-agent framework for hypothesis generation and hypothesis testing.
- The framework learns procedural skills from previous scientific-discovery experiences instead of relying only on static model capabilities.
- Adversarial refinement uses discriminator-guided feedback to distill hypothesis-construction strategies without explicit supervision.
- Experience-guided refinement uses empirical validation outcomes to improve experiment-design and execution skills.
- HypoForge consistently improves hypothesis-generation and testing performance, while learned skills generalize across diverse scientific tasks.The reported findings support continually improving AI agents that accumulate experience and strengthen their capabilities.