Source-linked AI summary
SGHA: Evidence-Grounded Research Problem Discovery with Local Language Models
Sarvesh Gharat, Junpei Komiyama
TL;DR
Research-problem generation often starts with open-ended ideation, making it difficult to identify problems already implied by unresolved patterns in the literature. SGHA instead builds an evidence structure, verifies cross-paper gaps, and formulates traceable problem families locally. Across five machine-learning domains, it achieved the highest overall score in a formulation-only evaluation and emphasized source grounding, formalizability, and ambiguity handling.
Problem
Existing automated research-idea systems commonly begin with open-ended generation, leaving unresolved whether literature patterns can directly suggest research problems.
Method
SGHA builds a structured evidence base, detects and verifies cross-paper gaps, and converts supported candidates into traceable semi-formal research-problem families using a local LLM.
Results
Across five machine-learning domains, SGHA achieved the highest overall score in the formulation-only evaluation and was strongest on source-grounded specificity, assumption-boundary clarity, formalizability, and ambiguity.
Takeaways & Limitations
SGHA produces evidence-linked problem artifacts with provenance, assumptions, objectives, risks, and ambiguity flags rather than proposal-style ideas alone.
Takeaways & Limitations
SGHA is complementary to open-ended ideation and is intended for problems suggested by the literature rather than freely speculative directions.
Abstract
from arXiv · showhide
Recent efforts toward fully automated AI scientists have demonstrated that language-model agents can generate hypotheses, execute experiments, and draft scientific manuscripts. However, during the early stages of research, when research problems are formulated, these AI scientists often rely heavily on proprietary frontier models. Their proposals are shaped by opaque parametric knowledge and by literature searches conditioned on the proposals themselves. Such knowledge is effectively a black box, and this dependence makes the evidential basis and validity of generated research problems difficult to audit and leaves the process vulnerable to model-specific hallucinations and biases. Furthermore, if proprietary research materials are transmitted to external APIs, the use of these models creates confidentiality, privacy, and data-governance concerns. We introduce the Structural Gap Hypothesis Agent (SGHA), a fully automated, corpus-first research-problem discovery system that runs entirely on a local LLM. SGHA structures a scientific literature corpus into evidence-linked paper objects and a typed evidence graph, detects unresolved structural patterns across papers, screens candidate gaps before formulation, and produces traceable research-problem families. In particular, it is able to output assumptions, objectives, success criteria, and remaining ambiguities. All LLM-based components of SGHA are executed using a locally served open-weight 9B language model, without requiring proprietary frontier-model APIs. We compare SGHA with the AI Scientist-v2 idea formulation module in five machine-learning domains. Our results suggest that explicit corpus structure and evidence-constrained reasoning can support promising, inspectable research-problem formulation without relying on frontier models during generation or verification.
C-MInDS, IIT Bombay
The section provides the SGHA project’s GitHub repository link.
- The project repository is available at github.com/SarveshVGharat/structural-gap-hypothesis-agent.
1 Introduction
SGHA is a corpus-first, local-LLM system that discovers cross-paper structural gaps, verifies them before formulation, and produces traceable research-problem families. Across five machine-learning domains, it structures evidence from scientific papers and generates problem specifications with explicit assumptions, objectives, success criteria, and ambiguities.
- Corpus-first discovery: SGHA identifies research problems whose evidence is distributed across multiple papers by extracting evidence-backed tuples and paper-level objects into a typed evidence graph.The system searches the graph for structural gaps rather than treating problem generation as unconstrained prompt completion.
- Verification before formulation: SGHA verifies candidate gaps for evidential support and prior coverage before formulation, then uses a separate critic to reject unsupported ambition-expanded variants.Only candidates passing the hard verification gate proceed to direct formulation, broader variants, and project-family grouping.
- Traceable formulations: SGHA produces traceable semi-formal problem specifications containing assumptions, objectives, success criteria, passage-level evidence, counterevidence links, and explicit ambiguity annotations.Each specification also retains its source motif type and describes variables and feedback or observation models.
- Evaluation scope: Across 1,250 selected papers, SGHA extracts structured content from 1,044 papers, retains 39 candidate gaps after verification, and formalizes 15 project families.The evaluation covers bandits, in-context learning, reasoning and test-time computation, offline reinforcement learning, and uncertainty estimation.
- Local-model operation: All generation and verification stages run on a locally served open-weight LLM, using a frozen Qwen/Qwen3.5-9B model without task-specific training or fine-tuning or proprietary frontier-model APIs.The primary comparison is with the AI-Scientist-v2 ideation workflow configured with the same local model and matched output counts.
2 Structural Gap Hypothesis Agent
SGHA discovers research problems by building a provenance-linked evidence corpus and typed graph, detecting structural motifs, and verifying candidate gaps before formulation. Its final problem objects expose objectives, assumptions, success criteria, source grounding, and unresolved ambiguities for inspection.
- Evidence construction: SGHA builds a topic-specific corpus, extracts paper-level scientific objects and relation-level evidence tuples, and validates each tuple against allowed relations and source text.Paper objects include methods, tasks, datasets, metrics, assumptions, results, limitations, failure conditions, claims, contradictions or tensions, and future work; tuples retain provenance and evidence metadata.
- Evidence graph: The validated evidence is organized into a typed multigraph whose nodes represent papers and scientific objects, with edges derived from extracted relations.Node types include methods, assumptions, limitations, failure conditions, claims, tasks, datasets, metrics, and results.
- Structural gap discovery: SGHA searches the graph for structural motifs, such as assumption–failure, shared-failure, repeated-unaddressed-limitation, conflicting-claim, sparse-evaluation, and theory–practice patterns, to create candidate gaps.A candidate gap is a structured pattern rather than a verified research problem; in the Bandits example, dTS is linked to linear assumptions and nonlinear score-function failure conditions.
- Gap verification: Candidate gaps undergo a corpus-level screen and role-based verification before becoming formulation-ready gaps.For the Bandits example, support, skeptic, critic, feasibility, and mechanism agents refine the interpretation and keep the assumption–failure pattern grounded without overstating it.
- Problem formulation: Final semi-formal problem objects record entities, variables, observations or feedback, decisions, objectives, constraints, success criteria, assumptions, result types, source grounding, and ambiguity flags.The Bandits artifact flags non-convexity, identification failure, and the approximation-error threshold as definitions requiring further sharpening.
3 Experimental Setup
SGHA is evaluated on five machine-learning corpora using a frozen local Qwen/Qwen3.5-9B model, structured extraction, verification-gated project-family generation, and output-count-matched comparisons. Candidates are assessed with blinded formulation-quality judgments and structural artifact properties, alongside profile-conditioned and other sensitivity settings.
- Main evaluation: The main experiments use a frozen local Qwen/Qwen3.5-9B model without task-specific training or fine-tuning, and evaluate retained candidates with the same blinded formulation-only judge.Structural reporting additionally covers source grounding, assumptions, formal problem statements, risks, and ambiguity flags.
- Corpora: 1,250 selected papers span five machine-learning domains, with each domain contributing a budget of 250 papers.The domains are bandits, in-context learning, reasoning and test-time computation, offline reinforcement learning, and uncertainty estimation.
- Corpora: 1,049 papers are parsed, 1,044 yield valid structured extractions, and the extractions produce 8,634 evidence tuples for typed evidence graphs and candidate structural gaps.Parsed means usable text is obtained; extracted means a valid structured extraction is produced.
- Generation pipeline: SGHA’s evaluated candidate is the final project-family artifact produced after gap detection, corpus screening, hard verification, formulation, expansion, independent-critic filtering, and consolidation.Intermediate motif hits, direct formulations, and rejected variants are not the evaluated candidates.
- Baselines: The primary comparison uses native AI-Scientist-v2 ideation with the same local model, while Claude Opus and MOOSE-Star provide stronger-generator and public trained-hypothesis reference points.All methods are matched to 15 retained candidates across the five domains.
- Evaluation: Candidates are converted to a common packet and scored with a ten-criterion, 0–10 formulation-only rubric by method-label-masked judges from five providers.The judges assess the provided text and context, while external literature novelty and correctness are outside their task.
4 Results
Across five machine-learning domains, SGHA reduces a literature-derived evidence base to 15 formalized project families and achieves the highest overall formulation-quality score. Its main advantages are source grounding, explicit assumptions and ambiguity handling, formal structure, and inspectable research-problem artifacts, while baselines can produce more polished or compact proposals.
- Pipeline yield: Across 1,250 selected papers, SGHA extracts content from 1,044 papers, produces 8,634 evidence tuples, accepts 39 gaps, and consolidates 19 variants into 15 project families.The pipeline progressively refines literature-derived evidence through extraction, gap discovery, verification, formulation, criticism, consolidation, and formalization.
- Pipeline yield: All five domains reach the project-family stage, with uncertainty estimation producing the largest set and reasoning/test-time computation plus offline reinforcement learning producing more focused outputs.The differing yields reflect that some corpora contain several separable directions while others produce smaller, tighter family sets.
- Formulation quality: SGHA obtains the highest overall formulation-quality score across five LLM judges, with strongest gains in source-grounded specificity, ambiguity hygiene, assumption-boundary clarity, well-posedness, and formalizability.The comparison evaluates how clear, grounded, formalizable, and explicit about ambiguity the generated artifacts are.
- Formulation quality: Using the same local Qwen model as AI-Scientist-v2, SGHA scores higher on nearly all formulation criteria, whereas Claude Opus produces clearer and more technically detailed surface formulations.SGHA’s evidence graph, candidate-gap verification, and formal problem objects particularly improve ambiguity hygiene and assumption-boundary clarity.
- Artifact structure: SGHA includes problem formulation, notation, objectives, assumption statuses, success criteria, risks, falsification conditions, ambiguity flags, and source grounding for all 15 final project families.In qualitative examples, AI-Scientist-v2 produces proposal-style ideas and MOOSE-Star produces compact source-grounded hypotheses, with fewer SGHA-style inspection fields.
5 Discussion
SGHA shifts research-problem generation from open-ended ideation toward literature-grounded problem families that retain evidence, verified gaps, challenged assumptions, and unresolved ambiguities. Its extensions and sensitivity studies show context-specific and broader exploration, while verification and consolidation shape outputs beyond simple scaling.
- Corpus-grounded formulation: SGHA produces evidence-linked problem families that track the literature evidence, verified gap, questioned assumptions, and remaining ambiguities.This differs from polished proposal generation by preserving the formulation’s evidential structure.
- Inspectability: A useful research problem must identify its origin, the assumption being challenged, what counts as progress, and which elements remain underspecified.The qualitative examples contrast proposal sketches with a more compact hypothesis-oriented artifact.
- Extensions: Profile conditioning shifts the evidence base toward researcher-specific contexts, whereas evolutionary exploration searches more broadly around literature-grounded seeds.Both uses retain source grounding while serving different exploration purposes.
- Sensitivity and limitations: Larger models and corpora can alter the evidence graph and candidate gaps, but verification, criticism, and consolidation shape the final problem families.The discussion cautions against interpreting SGHA as a simple scaling pipeline or forcing a fixed number of ideas.
6 Conclusion
SGHA is a training-free, corpus-first system that converts scientific-literature evidence into verified research-problem families. Across five machine-learning domains, it produced formalized project families and achieved the highest overall formulation-only evaluation score among compared methods.
- System contribution: SGHA uses assumptions, limitations, empirical patterns, and open directions from papers to build an evidence graph, identify candidate gaps, verify them, and formulate surviving candidates as project families.The system treats the literature itself as the starting point for research-problem discovery.
- Empirical results: 15 formalized project families were produced from 1,250 selected papers and 8,634 extracted evidence tuples across five machine-learning domains.These results summarize SGHA’s corpus scale and output across the evaluated domains.
- Empirical results: SGHA achieved the highest overall score among compared methods and was strongest on source-grounded specificity, assumption-boundary clarity, formaliability, and ambiguity.Qualitative examples likewise contrasted SGHA’s grounded formulations with polished proposal-style ideas from strong ideation baselines.
- Broader role: By making the path from literature evidence to problem statement explicit, SGHA offers researchers a more grounded starting point for reading, refining, and developing research directions.The conclusion positions scientific assistants as tools for organizing evidence across papers into clearer research-problem formulations.
7 Scope and Intended Use
SGHA targets research problems whose evidence is already partly visible in scientific corpora, converting recurring assumptions, failure modes, limitations, and narrow evaluations into structured formulations. It complements rather than replaces open-ended ideation, producing source-grounded starting points with explicit assumptions and ambiguity flags rather than claims of external novelty or correctness.
- Scope and intended use: SGHA collects recurring assumptions, failure modes, unresolved limitations, and narrow evaluation conditions from papers to formulate clearer evidence-linked research problems.Its intended inputs are problems whose supporting evidence is already partly visible in a scientific corpus.
- Scope and intended use: SGHA complements open-ended ideation by asking which problems are suggested by the literature’s assumptions, gaps, and tensions.Open-ended ideation remains useful for free speculation, distant connections, and directions not yet reflected in papers.
- Scope and intended use: The evaluation measures clear, source-grounded, formalizable problem artifacts with explicit assumptions and ambiguity flags, not externally novel or broadly correct problems.The intended output is a structured starting point that preserves the problem’s evidential origins and what still needs sharpening.
Generative AI Disclosure
The authors used several AI tools for language editing, formatting, preliminary feedback, code organization, and README preparation, while retaining sole responsibility for the paper’s technical content and conclusions.
- ChatGPT supported editing and formatting; Stanford Agentic Reviewer1 and CMU Paper Reviewer provided preliminary feedback; Codex organized code and prepared README files.The authors state that technical content, experimental design, analyses, and conclusions remained their sole responsibility.
A SGHA Prompts, Output Schemas, and Validation Rules
SGHA combines bounded LLM extraction, review, and formulation with deterministic graph construction, motif detection, verification gating, and report auditing. Its schemas and validation rules preserve source traceability while filtering and verifying corpus-grounded candidate gaps before final outputs.
- A SGHA Prompts, Output Schemas, and Validation Rules: Runtime prompts use fixed instructions with placeholders for paper text, candidate-gap evidence, verification summaries, and project-family records.The appendix documents prompt templates, output contracts, and validation rules while leaving instance-specific content represented by placeholders.
- A SGHA Prompts, Output Schemas, and Validation Rules: SGHA uses language models for bounded extraction, review, and formulation, while deterministic code handles graph construction, motif detection, hard gating, consolidation, and auditing.Main outputs are generated only after candidate gaps pass screening and verification.
- A.1 Evidence Extraction: Evidence extraction returns a structured paper object and typed evidence tuples containing claims, limitations, methods, assumptions, evaluations, results, and related fields.The prompt requests real scientific content from the parsed paper rather than a summary or template.
- A.1 Evidence Extraction: Extraction rules require paper-grounded entities and evidence, forbid invented content, and distinguish assumptions from failures.Evidence text and source spans must come from the provided parsed paper.
- A.1 Evidence Extraction: Before graph insertion, deterministic validation checks fixed relations, required fields, source-span matches, specificity, evaluation context, and normalized relation polarity.Generic subjects and under-specified evaluation tuples are removed; fails_under and limited_by become negative, while assumes becomes neutral.
- A.2 Graph Construction and Motif Detection: Deterministic motif detectors create candidate structural gaps from typed evidence-graph patterns, but a motif hit alone does not establish validity or novelty.Graph construction preserves provenance, removes junk labels, deduplicates parallel edges, and avoids creating gaps from arbitrary missing edges.
- A.3 Novelty and Corpus-Coverage Filtering: Corpus-grounded screening removes trivial, solved, or nearby-covered candidates, retaining novel, known-open, and partially corpus-addressed gaps before stricter verification.This screen is not an external literature novelty proof, and corpus counterevidence can bypass LLM classification when it already marks a candidate solved.
- A.4 Verification Agents and Hard Gate: Role-specific verification agents assess support, counterevidence, feasibility, mechanism, and conservative final judgment using a shared output schema.Novelty survivors are reviewed by support, skeptic, feasibility, mechanism, and verification-critic roles.
B LLM-Judge Rubric and Candidate Normalization
SGHA candidates are normalized into blinded packets that preserve each method’s naturally produced fields, then evaluated by formulation-only LLM judges using a fixed, evidence-constrained rubric. The protocol includes criterion caps for missing structure, panel blinding and calibration, and a separate profile-conditioned extension.
- B.1 Candidate Packet Format: Outputs are converted into a common candidate-packet format while preserving original fields and marking unproduced fields as “not provided.”The packet includes identifiers, domain, title, problem statement, motivation, direction, contribution, evaluation plan, caveats, grounding, assumptions, formalization, and ambiguity fields.
- B.1 Candidate Packet Format: Normalization enables cross-system comparison while keeping differences between SGHA’s formal, provenance-rich outputs and ideation baselines’ planning-oriented fields visible.SGHA families often include formal problem statements, assumptions, source provenance, and ambiguity flags, whereas baselines often include motivation, directions, evaluation plans, and caveats.
- B.2 Formulation-Only Judge Prompt: The formulation-only judge assesses the problem statement itself, not implementation plans, experiments, software details, actionability, or external novelty.Judges use only the blinded candidate text, and missing fields marked “not provided” are treated as missing.
- B.3 Formulation Rubric: The rubric covers ten criteria, including clarity, technical specificity, well-posedness, assumption boundaries, formalizability, nontriviality, scope control, grounding, ambiguity hygiene, and overall quality.Each criterion is scored from 0 to 10, with score anchors included to make the scale consistent across models.
- B.6 Judge Response Schema: Each judge returns strict JSON containing criterion scores, a recommendation, confidence, strengths, weaknesses, rationale, and a novelty caveat.The overall_formulation_quality_10 value is holistic and is not computed as a weighted average of the other criteria; novelty is judged only from the provided text.
- B.7 Judge Panel and Blinding: Five judges from different providers score frozen candidates using identical blinded packets, rubrics, anchors, cap rules, and response schemas, with method labels revealed only after scoring.Before real-candidate scoring, calibration examples test treatment of vague, underformalized, over-broad, and stronger source-grounded formulations without tuning judge-specific prompts.
- B.9 Personalized Judge: Profile-conditioned candidates retain the ten formulation criteria and add profile-specific criteria, using only supplied profile context and candidate text for personalization judgments.The personalized rubric penalizes generic problems and name-dropping while assessing artifact-supported profile themes, source or corpus evidence, and profile-relevant technical style.
C Additional Result Tables
This appendix provides detailed count and score tables supporting the main results, including full SGHA stage counts and selected per-domain formulation scores.
- C Additional Result Tables: The appendix reports full SGHA stage counts and selected per-domain formulation scores to support the main text’s central trends.It provides detailed count and score tables rather than introducing additional results.
C.1 Full SGHA Pipeline Counts
The full SGHA pipeline reports separate paper-, graph-, gap-, and formulation-level counts across five domains. Its stages progress from deterministic motif matching through corpus screening, verification, and critic-approved ambition expansion, with quantities that are not one-to-one.
- Full SGHA Pipeline Counts: Table 21 separates SGHA counts into paper, graph, gap, and formulation levels across five main domains.The reported quantities are organized as a count funnel rather than a single aggregate count.
- Full SGHA Pipeline Counts: The pipeline tracks motif hits, novelty survivors, reviewed gaps, verified gaps, and accepted variants as successive stages.Motif hits are deterministic graph-pattern matches; later stages apply corpus screening, verification, and independent criticism.
- Full SGHA Pipeline Counts: Pipeline quantities are not one-to-one: papers can yield multiple evidence tuples, while accepted variants can consolidate into one project family.Several evidence tuples may instantiate one motif before formulations are consolidated downstream.
C.2 Selected Per-Domain Formulation Scores
Per-domain formulation scores report overall quality alongside source grounding, formalizability, and ambiguity hygiene. Across domains, SGHA leads in source grounding and ambiguity hygiene, while the strongest ideation baseline is often competitive overall.
- Per-domain formulation scores: Table 22 reports overall formulation scores and the criteria of source grounding, formalizability, and ambiguity hygiene for each domain.The main text instead reports method-level averages.
- Per-domain formulation scores: SGHA is strongest across domains on source grounding and ambiguity hygiene, whereas the strongest ideation baseline is often competitive on overall formulation quality.These per-domain patterns are consistent with the aggregate results reported in the main text.