Source-linked AI summary
Toward Auto-Research: Mining Falsifiable Research Ideas from Paper Knowledge Graphs with Categorical Structure
Yuchen Wang, Zhongzhi Luan
TL;DR
Automated research-idea systems often flatten papers into text, vectors, or untyped pairings, losing the typed relation chains needed to check cross-domain analogies. This paper models papers as small categories and bridges as partial functors, then evaluates a three-layer front end whose judge filters candidates at roughly 17:1 while accepted ideas retain a quantitative-falsifier rate above 83%.
Problem
Existing LLM-only, random-pairing, and embedding-similarity approaches flatten paper structure and cannot reliably check typed cross-domain analogies.
Method
The paper models each paper as a small category and implements categorical signature clustering, a functor-preservation gate, and a six-axis LLM plausibility judge.
Results
The judge reduces 163 mined cross-domain opportunities to 9, while the structural front end preserves a quantitative-falsifier rate of ≥83% in every mode.
Takeaways & Limitations
The front end provides an advisory, logged structural screen for cross-paper idea generation rather than a silent filter.
Takeaways & Limitations
The judge is fail-open, and its false-negative rate has not been measured against a held-out human-graded set.
Abstract
from arXiv · showhide
Automated research-idea generation systems built on large language models (LLMs) share a structural weakness: they reduce ideation to free-text recombination, random paper pairing, or embedding-similarity retrieval. The three approaches fail in the same way: each treats a paper as a flat object, a string or a vector, and so quotients away the typed problem-method-metric-claim arrows a researcher actually uses when reasoning about a cross-domain analogy. We recover the missing structure with the minimal piece of category theory that a typed graph alone does not provide: composition, together with identity arrows, which makes it possible to ask whether a proposed analogy preserves relation chains. Concretely, each paper $p$ is modelled as a small category $C_p$ whose objects are extracted typed research entities and whose morphisms are the relations the paper asserts; a cross-paper bridge from $p$ to $q$ is then a partial functor candidate $F: C_p -> C_q$ that preserves object kinds and covered relation classes. We instantiate the model as a three-layer algorithm: categorical signature clustering, a functor-preservation gate, and a six-axis LLM plausibility judge. Evaluated on a corpus of tens of thousands of full-text-parsed papers under four ablation conditions, the categorical gate filters cross-domain candidates at roughly a 17:1 ratio while the quantitative-falsifier rate of accepted ideas stays above 83% throughout; every rejected candidate is retained with its per-axis rationale, so the gate doubles as a logging layer rather than a silent filter.
1 Introduction
The paper frames automated research-idea generation as a representational problem: flat text, vector, and untyped-node approaches lose the typed arrows needed for checkable analogies. It proposes a categorical front end with explicit structural constraints and a three-layer algorithm.
- LLM-only ideation, random pairing, and embedding retrieval treat papers as flat objects, losing typed problem-method-metric-claim relations.
- A valid cross-paper bridge must preserve typed units, typed relations, and relation chains that remain checkable.
- The algorithm combines categorical signatures, a sparse graph clustered into 216 intra-discipline communities, a functor-preservation gate, and a six-axis plausibility judge.
- The evaluation uses four experimental conditions—within×cross × dump×complete—to ablate the algorithmic layers.
- The paper builds on PARNESS for parsing, typed-entity extraction, and knowledge-graph storage; its contribution is the algorithm running on that substrate.
2 Why Existing Approaches Are Insufficient
The three baseline approaches fail because each removes the internal typed arrows that make a cross-domain analogy structurally checkable. Their specific failures are hallucinated or untyped recombination, combinatorial random pairing, and homonymy-prone embedding similarity.
- LLM-only recombination produces fluent novelty but can hallucinate citations, substitute mechanisms without type-checking, and collapse toward fashionable techniques.
- ≈1.56×10^8 paper pairs are available in the random-pairing space, whose overwhelming majority share no plausible mechanism.
- Embedding similarity often matches shared vocabulary without shared structure, because mapping a paper to one vector quotients out its morphisms.
- The baselines differ in implementation but share the same structural loss: papers become strings, vectors, or KG nodes without typed internal arrows.
3 A Categorical Model of Research-Paper Structure
This section formalizes a valid cross-paper bridge as more than nearby text or shared vocabulary: it must preserve typed units, typed relations, and coherent composed relation chains. The categorical formulation supplies the language for these requirements.
- A typed knowledge graph expresses units and relations, but categorical structure adds composition, identities, and preservation needed to assess whether a bridge exists.
- A valid bridge must map a source method to a method-like target unit rather than a dataset or generic theme.
- Source relations such as method-solves-problem must map to analogous typed target relations, not merely shared words.
- Composed relation chains must remain coherent so the transferred mechanism can still be checked on the target side.
2. Are the relations preserved? If p asserts Mp
The model represents each paper as a small category and each cross-paper bridge as a partial structure-preserving map. The implementation scales this idea through typed signatures, local preservation checks, and an LLM judge for mechanism plausibility.
- A functor preserves identities and composition, expressed by F(g ◦f) = F(g) ◦F(f) and F(idA) = idF(A).
- Each paper category contains extracted entities such as Problem, Method, Metric, Dataset, and Claim, connected by asserted typed relations.
- Categorical signatures compact paper-internal morphisms into typed object–metric atoms, while later gates distinguish real bridges from homonymy artefacts.
- A partial functor maps only a transferred mechanism’s subcategory while preserving object kinds, corresponding relation classes, and local commutativity.
- The scalable implementation checks object-kind agreement, morphism-class agreement, preservation rate, and six-axis plausibility rather than enumerating all paths or using a theorem prover.
- The categorical vocabulary functions as an engineering specification for the schema, candidate miner, and preservation checks.
4 Related Work
The paper builds on PARNESS and situates its structured idea-generation approach among AI scientists, KG-grounded systems, categorical methods, and evaluation benchmarks.
- PARNESS supplies full-text parsing, typed-entity extraction, and knowledge-graph storage, while this paper contributes the algorithm operating on that substrate.
- End-to-end AI scientists: Contemporary AI-scientist systems use agentic search, literature retrieval, hypothesis generation, or collaborative review, but often rely on free-text judgment or user-provided seeds.
- Structured / KG-grounded idea generation: Structured idea-generation approaches organize literature through inspirations, research chains, heterogeneous agents, extracted facets, recombination instances, or external knowledge graphs.
- Categorical and sheaf-theoretic gates: Categorical and sheaf-theoretic work provides related formal languages for graph structure, embeddings, neural architectures, and obstruction detection.
- Evaluation: Evaluation benchmarks assess idea quality, research agents, and first-gate soundness, motivating an advisory fail-open plausibility judge rather than a blocking grader.
5 Data and KG Construction
The algorithm runs on a PARNESS-produced corpus and typed knowledge graph whose schema preserves the structural, provenance, and completeness information required for categorical mining.
- The substrate contains 17,650 full-text-parsed papers and 9,081 abstract-only papers; abstract-only records contribute metadata but no method-step signatures.
- PARNESS extracts typed entities from paper bodies through a structured LLM pipeline and stores them as records consumed by the algorithm.
- Typed object–metric pairs define the smallest signature units, preventing categorical morphism families from collapsing into keyword bags.
- Section-level provenance links every typed entity to its source paragraph and section, making generated-idea evidence pointers human-verifiable.
- Signatures with document frequency ≥80% are flagged as boilerplate but retained for completeness while clustering can ignore them.
6 Method: The Categorical Idea-Mining Algorithm
The method converts typed paper relations into signatures, clusters papers by shared non-boilerplate structure, proposes partial-functor bridges, and screens them before hypothesis generation.
- The algorithm has three layers: categorical signature projection, corpus clustering, and mining with preservation and commutativity-style checks.
- Schema-to-algorithm mapping: Typed object×metric pairs form signature atoms, while provenance and retained boilerplate support evidence verification, completeness, and IDF-based clustering.
- Categorical signatures and clustering: Each paper receives a compact signature family, typically containing 1–6 de-boilerplated elements representing internal arrow structure.
- Clustering: The sparse paper graph weights edges by IDF-weighted shared signature classes and excludes universally frequent signatures before intersection.
- Clustering: Leiden / CPM clustering yields 216 clusters, including 76 participating in cross-cluster edges surviving the bridgeability prior.
- Mining: The cross-domain miner uses same-kind kNN anchors and bridgeability thresholds to propose partial-functor candidates and compute structural preservation.
- Gates: The preservation gate demotes opportunities below 0.30 rather than blocking them, allowing downstream judging and audit visibility.
- Gates: The reasonability gate combines method overlap, formalism overlap, institutional bridge, and distance penalty into tiered decisions.
7 Experimental Conditions
The evaluation is a 2×2 ablation varying the miner and gate configuration while holding the generation and upstream corpus settings constant.
- The four conditions vary within versus cross-domain mining and dump versus complete gate configurations.
- All conditions share the proposer prompt, family expander, generation cap, sampling parameters, upstream KG corpus, LLM provider, and entity embedder.
8 Results
The results show that categorical filtering sharply narrows cross-domain opportunities while preserving diverse, quantitatively falsifiable ideas. Across conditions, structural gates reduce downstream cost, and surviving bridges exhibit interpretable variation in structural depth.
- Funnel: 13,951 within-domain opportunities versus 163 cross-domain pairs yield an 86× opportunity gap before judging.The cross-domain funnel then rejects 154 of 163 candidates, leaving 9 survivors in C4.
- Funnel: The C4 judge threshold of 0.30 converts 163 cross-domain candidates into 9 survivors, then budgeted expansion produces 50 idea members.Two samples per survivor yield 18 proposal-level ideas, and five falsifier forms per expanded proposal yield 50 members.
- Judge survivors: Survivors form high-confidence, worth-investigating, and speculative tiers whose scores track progressively weaker structural correspondence.The strongest bridge scores 0.753, while rows 2–3 score 0.42–0.57 and rows 4–9 score 0.30–0.42.
- Coverage: Cross-domain opportunities touch 76 of 216 clusters, with Gini coefficients of 0.568 and 0.514 respectively.The small +0.054 cross-versus-within Gini difference is presented as evidence against concentration in a few interdisciplinary corridors.
- Diversity: Branch-frame uniqueness is ≥0.993 in every condition, while the unique-hypothesis-first-sentence ratio is 100%.Conditioning proposals on signature atoms rather than only cluster labels preserves distinct framings across structurally similar opportunities.
- Quality: Quantitative-falsifier rates range from 83.4% to 88.9% across conditions, and the judge separates survivors from rejections most strongly on type agreement.The reported type-agreement means are 0.49 for survivors versus 0.10 for rejected candidates, a 4.9× gap.
- Cost: C4 costs ∼700 tokens per final idea member versus ∼800 for C1, while C3 reaches ∼2,500 under quota contention.The C4 saving is attributed to the upstream collapse from 13,951 to 163 candidates rather than to a smarter LLM.
- Sample ideas: The sample ideas specify target metrics, baselines, and numerical thresholds, making them falsifier-form hypotheses rather than vague improvement claims.One example tests a ≥15% normalized-RMSE reduction, while another requires effect size ≥0.4 Cohen’s d.
9 Discussion
The discussion frames cross-domain bridges as rare, structurally filtered opportunities and argues that the judge should remain advisory, replayable, and fully archived. It also records important evaluation and deployment limitations.
- Within vs cross: 163 cross-domain opportunities were reduced to 9 survivors by the judge, a roughly 17:1 funnel consistent with sparse structure-preserving analogies.The miner also found 13,951 within-domain opportunities, reflecting the rarity of cross-cluster bridges after specific-term and parent-domain filtering.
- Why the judge remains advisory: False negatives are especially costly because they silently discard potentially novel ideas before those ideas are recorded as hypotheses.The upstream structural constraints make the false-positive base rate small, strengthening the asymmetric-cost argument.
- Why the judge remains advisory: Rejected opportunities are archived with per-axis scores and rationales, while null, unparseable, or timed-out judgments pass downstream with explicit null-score markers.This fail-open design makes the gate replayable and preserves rejected candidates for later inspection.
- Positioning: The sheaf-obstruction literature supplies a gate-level intuition for coherence checking, but this work applies that pattern to cross-paper idea generation rather than introducing new mathematical machinery.The cited comparison concerns local beliefs that may fail to glue into a coherent global belief.
- Limitations: The study estimates token costs, has not measured fail-open false-negative rates against held-out human judgments, and uses only one LLM and embedding model.It also evaluates only the front end; downstream experiment execution and paper writing are separate components.
10 Conclusion
The paper presents a structured front end for automated research-idea generation that combines categorical knowledge-graph mining, advisory plausibility judging, and falsifier-family expansion. Across four modes, quantitative-falsifier rates remain at least 83%, while future work targets judge false negatives and richer non-numeric evaluators.
- The system combines a categorical paper knowledge graph, two complementary miners, an advisory six-axis plausibility judge, and a five-form hypothesis-family expander.
- Across four ablation modes, the structural front end preserves a quantitative-falsifier rate of ≥83%.
- The contribution is applying categorical structure to cross-paper idea generation on a real paper knowledge graph with a fail-open operational design.
- Future work will measure the judge’s false-negative rate against a human-graded subset and support richer evaluator specifications for non-numeric falsifiers.