PaperlayerView exact revision on arXiv

Source-linked AI summary

ShadowMiner v1 - An Experience Report on Implementing and Measuring a Problem-and-Hypothesis Discovery Engine

Jinhyuk Choi

arXiv:2610.04339v1cs.AIcs.CL

TL;DR

Finding unanswered research questions by hand is becoming harder as papers proliferate. ShadowMiner implements and measures previously published ideas for generating hypotheses from AI papers, evaluating them against later publications; 40 of 103 hypotheses appeared in papers published after the freeze date.

  • Problem

    Finding unanswered research questions by hand is becoming harder as thousands of papers appear every month.

    Source: §1 · p. 1

  • Method

    The report implements and operates previously published ideas in ShadowMiner, then tests hypotheses generated from literature up to a freeze date against papers published afterward.

    Source: §1 · p. 1

  • Results

    40 of 103 hypotheses (39%) appeared in papers published after the freeze date.

    Source: §7 · p. 17

  • Takeaways & Limitations

    Hypotheses labeled “exists” by the critic panel appeared in later papers more often than those labeled “new.”

    Source: §7 · p. 17

    The LLM judge’s absolute accuracy is unknown because its judgments have not been verified by human scoring.

    Source: §7 · p. 18

Abstract

from arXiv · show

ShadowMiner v1 is a system that automatically discovers research problems and generates hypotheses from AI papers. It is a nine-stage pipeline. It structures documents into a knowledge graph and finds graph gaps in it - structural blind spots in research. These graph gaps are included in the LLM generation prompt. Each generated hypothesis is then verified by checking whether it is already covered by existing research, scoring its quality, and checking that the facts it relies on are accurately drawn from its sources. This report does not propose a new generation or evaluation technique. It describes our experience of implementing and applying ideas from prior work, and measuring whether each one actually contributed.

1 Introduction

ShadowMiner automates research-problem extraction and hypothesis generation from AI papers, addressing the growing difficulty of manually identifying unanswered questions. This experience report implements prior ideas and measures their contributions, while positioning ShadowMiner among related systems.

1 Introduction

  • ShadowMiner automates extracting research problems and generating hypotheses from AI papers as literature searches become harder to carry out by hand.Source: §1 · p. 1
  • The report implements and operates previously published ideas, then measures what contributed and what did not; it proposes no new generation or evaluation technique.Source: §1 · p. 1

2 Background and Related Work

  • The report compares systems using eight axes, while its table covers six and excludes evaluation method and verifiability because papers describe them differently.Source: §2 · p. 2
  • ShadowMiner is a training-free, prompt-only system with generation but no execution, feedback, or dedicated diversity mechanism in its baseline configuration.Source: §2 · p. 2
  • ShadowMiner adopted parallel multi-candidate generation with diversity metrics and multi-judge scoring; attaching sources to graph gaps was tested but not adopted.Source: §2 · p. 4

3 System

ShadowMiner turns collected AI papers into grounded problems and hypotheses, then checks novelty, quality, and citation faithfulness. Its knowledge graph supports both generation and detection of missing relations across papers.

3.1 Overview

  • Given a topic and collected AI papers, ShadowMiner outputs research problems and hypotheses with predictions, expected results, verification methods, citations, novelty verdicts, and four-axis quality scores.Source: §3.1 · p. 4

3.3 Pipeline: Nine Stages

  • After preparing graph-based inputs, the pipeline generates hypotheses in Stage 6 and follows each with novelty search, multi-judge scoring, and citation-faithfulness checking.Source: §3.1 · p. 4

3.1 Overview

  • ShadowMiner does not run experiments on generated hypotheses; it covers hypothesis generation and evaluation only.Source: §3.1 · p. 5

3.2 Knowledge Graph (KG)

  • The knowledge graph stores extracted triples with their source sentences, enabling cross-paper queries and graph-gap detection.Source: §3.2 · p. 5
  • The narrow graph includes one-hop triples for the generation prompt, while the two-hop wide graph supports graph-gap detection.Source: §3.2 · p. 5

1. Document-importance re-ranking

The pipeline ranks documents using relevance and an importance score, then extracts and persistently accumulates normalized knowledge-graph triples. This preparation supports comparisons across papers.

1. Document-importance re-ranking

  • Documents are ranked by search relevance plus 0.15 times an importance score derived from citation count and the disruption index.Source: §1 · p. 5
  • Only approximately relevance-tied documents are reordered; citation count supplies most of the importance score, and positive disruption adds up to 0.3 without a penalty for negative disruption.Source: §1 · p. 6

2. Knowledge-graph extraction

  • The LLM extracts entities and triples from each document’s first 12,000 body characters using six entity types and eight relation types.Source: §2 · p. 6

3. Graph storage and retrieval

  • Concept names are normalized through candidate matching, string or embedding similarity, and LLM review, then triples are persistently accumulated for graph-gap detection.Source: §3 · p. 6

4. Problem normalization

Problem normalization merges differently worded problems, while graph-gap detection identifies missing relations or unresolved limitations. These gaps and a limitation summary inform problem and hypothesis generation, followed by novelty search.

4. Problem normalization

  • Problem normalization merges differently worded Problem entities linked by limited-by or addresses-problem relations, using the same three-step procedure as Stage 3.Source: §4 · p. 6

5. Graph-gap detection and description

  • Graph-gap detection finds missing relations among similar methods or tasks, plus limitations recorded for multiple methods or tasks that no method is recorded as solving.Source: §5 · p. 7

6. Problem and hypothesis generation (core stage)

  • The generation prompt combines the narrow graph, described graph gaps, and the top 20 author-identified limitations, including limitations not represented by gaps.Source: §6 · p. 7

7. Novelty search

  • Novelty search retrieves the top three similar existing works using keyword and vector search combined with Reciprocal Rank Fusion.Source: §7 · p. 8

8. Critic multi-judge scoring

ShadowMiner uses LLM judges to assess hypotheses and check whether their claims are supported by cited sources. Its design choices reflect the need to link concepts across papers and the lack of human evaluation so far.

8. Critic multi-judge scoring

  • Multiple judges combine inclusion and novelty decisions by majority vote and four-axis quality scores by median.Inclusion votes split evenly means inclusion; novelty votes split evenly default to glm-5.1.Source: §8 · pp. 8–9

9. Citation-faithfulness and factuality check

  • Citation faithfulness checks source support for generated problems or predictions, while factuality checks support for each atomic fact.For example, a performance claim is split into the task addressed and the reported accuracy.Source: §9 · p. 9

3.4 Design Rationale

  • ShadowMiner uses threshold-based rules to find missing combinations in a knowledge graph, rather than a learned link-prediction model.The authors do not directly compare this design with MOOSE-Chem, noting results could vary with the LLM version.Source: §3.4 · p. 9
  • LLMs substitute for human evaluators in novelty judgments, critic scoring, and citation-faithfulness checks; their trustworthiness remains to be checked separately.The authors had not yet had human evaluators review these judgments.Source: §3.4 · p. 9

4 Techniques and Their Sources

Table A1 catalogs 13 ShadowMiner techniques, their sources, implementations, measured results, and adoption decisions.

4 Techniques and Their Sources

  • Table A1 lists 13 techniques and records each technique’s source, implementation, measured results, and whether it was adopted.Techniques from other work retain their source regardless of adoption, and the authors mark their own improvements.Source: §4 · p. 10

5 Evaluation

The evaluation tests whether generated hypotheses match later research, whether critic scores predict those matches, and whether the judging tools are reliable. The reported 0.388 appearance rate varies by topic, while source-date ambiguity and model disagreement qualify its interpretation.

5.1 Setup

  • The public benchmark was not adopted because its contamination risk, input constraints, and different base model prevent a fair side-by-side comparison with ShadowMiner.ShadowMiner is designed to gather more than the benchmark’s five cited papers; limiting it to five would understate its ability, while expanding its input would advantage it.Source: §5.1 · p. 10
  • The evaluation covers nine AI/CS topics, with all runs completing and no errors.The median runtime was 91 minutes per topic (range 65–118 minutes).Source: §5.1 · p. 10
  • The study searches post-freeze papers for matches, using o3’s three-vote majority across each hypothesis–paper pair.The freeze date is 2026-01-01; the date boundary is the study’s only contamination control.Source: §5.1 · p. 10, §5.2 · p. 11

5.2 Main Results — Actual Appearance Rate

  • 0.388 (40/103) of hypotheses matched later research; topic rates ranged from 0.167 to 0.667, making the overall rate a rough guide.Resampling whole topics gives a 95% CI of [0.282, 0.500], 18% wider than the independence-based interval; Table 5 compares the rate with two other values.Source: §5.2 · p. 11, Table 5 · p. 13
  • The 0.388 rate may include matches to papers made public before the freeze date, because some sources record publication rather than first-publication dates.Counting only matches to arXiv papers yields a lower-bound rate of 0.262 (27/103); the study did not check whether the other 13 matches also appeared on arXiv.Source: §5.2 · pp. 11–12
  • The critic panel’s overall quality score did not consistently distinguish hypotheses that appeared from those that did not across topics.The per-topic difference between the two groups’ mean scores ranged from −0.175 to +0.377.Source: §5.2 · p. 13

5.3 Reliability of the Judging Tool

  • The appearance judgments were imperfectly stable: 7 of 40 o3 matches had split votes, and gpt-5 did not confirm 22 of the 40 matches.Without human judgments, the report cannot determine which model is correct when they disagree.Source: §5.3 · p. 14
  • Hypotheses labeled “exists” appeared more often than those labeled “new,” but the difference was not statistically significant when o3 relabeled them.The panel–o3 agreement was uneven: they agreed on only 2 of the panel’s 26 “exists” labels, so the contrast may depend on the judge.Source: §5.3 · p. 15

6 Negative Results

Section 6 tests whether selected pipeline choices improve outcomes; comparisons found no measurable benefit for some additions and favored parallel generation over sequential generation.

6 Negative Results

  • Small differences in these comparisons warrant caution because each covers about a dozen hypotheses per topic and LLM outputs can vary across runs.The comparisons held each topic’s documents and knowledge graph fixed while changing only the variable under test.Source: §6 · p. 15

Generation mode — sequential dropped

  • Sequential generation had lower actual appearance (5/24 = 0.208) than parallel generation (12/31 = 0.387), alongside higher waste and longer runtime.The difference was not statistically significant (p = 0.155), and the sample was small.Source: §6 · p. 16

Attaching sources to graph gaps

  • DGG-SP adds up to six source-paper numbers for the triples establishing a gap and includes a linked unresolved limitation when present.The default described graph gap contains the gap kind, a sentence stating what is missing, and a direction phrase shared by gaps of that kind.Source: §6 · p. 16
  • DGG-SP raised actual appearance to 17/36 = 0.472 from 14/34 = 0.412 with DGG, but the difference was not significant and prompts cost more.Mean groundedness was similar overall (3.24 for DGG against 3.22 for DGG-SP); the authors kept DGG as default.Source: §6 · p. 16

Critic-feedback regeneration loop

  • The critic-feedback regeneration loop replaced 1 of 12 selected hypotheses, and that replacement did not appear in a later paper.The loop was kept off by default based on the low replacement rate and nonappearance.Source: §6 · pp. 16–17

Self-assessment has no predictive power

  • Across nine topics, self-assessment score differences ranged from −0.175 to +0.377, with direction varying by topic, so the score did not predict later appearance.With nine topics, the authors could not distinguish random variation from a real pattern.Source: §6 · p. 17

7 Discussion and Limitations

The study found later-paper matches for a portion of ShadowMiner’s hypotheses, but its evidence is qualified by measurement and contamination limits. The authors also report a corrected appearance rate and several limits on what their evaluation can establish.

Results

  • “Exists” labels tracked higher later-paper appearance under both the critic panel and o3, but neither difference was statistically significant (p = 0.054; p = 0.130).The authors therefore cannot rule out random variation as an explanation for the observed direction.Source: §7 · p. 17
  • Mean overall quality scores were higher for later-appearing hypotheses in some topics and lower in others.Source: §7 · p. 17

Contamination boundary

  • The freeze-date control cannot exclude idea-level or preprint contamination, and evaluation covered only about nine months after the freeze date.The authors also note that training data collection and model training can occur at different times; the model's training cutoff was not publicly known.Source: §7 · pp. 17–18

Results

  • After excluding one duplicate-index match, 39/103 hypotheses (0.379) appeared in later papers, 0.9 percentage points below the uncorrected rate.The uncorrected rate was 0.388 (40/103).Source: §7 · pp. 17–18

Data defect

  • The duplicate-date defect was found once among the 40 main matches and not in negative-control pools, but Section 6 and A1 matches lacked dates for equivalent checks.Source: §7 · p. 18

Measurement limits

  • Without human scoring, the LLM judge's absolute accuracy remains unknown, and the critic panel may have used post-freeze knowledge when judging hypotheses.The authors say glm-5.1 may have been trained on papers up to early 2026; the training cutoffs of two other panel members were unknown.Source: §7 · p. 18
  • A hypothesis recorded as not appearing may have had a matching paper missed by retrieval, so the 63 non-appearing hypotheses are not a count of prediction failures.The appearance judgment retrieves at most six candidates and evaluates only the top five.Source: §7 · p. 18

Scope and operational limits

  • The study covers one AI/CS domain, and the reliability of the citation-faithfulness check was not investigated.For 11 of 103 hypotheses, only two judges returned results; the authors say this did not change the direction of the reported results.Source: §7 · pp. 18–19

A Techniques and Their Sources

Appendix A collects the techniques, sources, implementations, measured results, and adoption decisions. The supplied passages identify graph-gap terminology and a few reported outcomes.

Technique inventory

Measured results

B Robustness Check: Hypotheses Judged by Two Judges

The robustness check found that removing hypotheses judged by only two judges did not change the appearance-rate pattern. The overall quality-score comparison remained mixed across topics and too weak to establish a reliable relationship.

Hypotheses Judged by Two Judges

  • Across all four cases, hypotheses labeled “exists” appeared in later papers more often than those labeled “new,” even after excluding the 11 two-judge cases.Table B1 reports the exists/new appearance-rate comparison across the four cases.Source: Table B1 · p. 28, B Robustness Check: Hypotheses Judged… · p. 28
  • Among three-judge hypotheses across eight topics, appearing hypotheses did not consistently score higher; per-topic mean differences ranged from −0.175 to +0.377.The agents topic was excluded because all 11 of its hypotheses were judged by two judges.Source: B Robustness Check: Hypotheses Judged… · p. 28
  • The quality-score correlation was weak and nonsignificant (r = 0.191, p = 0.650), so the check could not distinguish random variation from a real pattern.The earlier comparison had r = 0.049 (p = 0.901); the authors report that the conclusions do not change.Source: B Robustness Check: Hypotheses Judged… · p. 28

C Reproducibility

The report documents settings for reproducing the nine-topic actual-appearance experiment, along with its runtime, cost, and availability constraints. The code and data are not released.

Reproducibility

  • All nine topics completed with zero failures, taking a median of 91 minutes per topic (range 65–118 minutes).The completion rate was 9/9.Source: C Reproducibility · p. 28
  • Settings include a 0.15 document-importance weight cap, a 180-item hard cap for hypothesis evaluation, novelty-search top K of 3, and up to six papers with top 5 checked.The 180-item cap is 60 per type; the passages also specify narrow/wide graph and 1-hop/2-hop variants.Source: C Reproducibility · p. 29
  • The experiment used a 2026-01-01 freeze date, up to 40 input documents per topic, graph extraction on the top 30, and up to six problems and 12 hypotheses per topic.The freeze date was based on the generation model’s release date.Source: C Reproducibility · p. 29
  • Table 7 provides the hyperparameters used for the actual-appearance experiment across nine topics.The table is identified as the settings applied in Section 5.2.Source: C Reproducibility · p. 29, Table 7 · p. 29
  • The actual-appearance evaluation cost $0.52–0.70 per topic, and the code and data are not released.These are the report’s stated cost and availability conditions.Source: C Reproducibility · p. 29
  • Parts of the report text were drafted and revised with Claude Code, while the authors remained responsible for all content.Source: C Reproducibility · p. 29
Loading 2610.04339v1…