Source-linked AI summary
Rebuilding a Field's Open-Problem Catalog from the Literature: A Two-Layer Knowledge Graph, and Why Status Must Stay Proposed
Madhulatha Mandarapu, Sandeep Kunkunuru
TL;DR
Open-problem catalogs are written in prose and cannot directly answer who works on a problem or whether literature has settled it. The paper builds a two-layer graph and evaluates it on two fields; its audit finds weak status-verdict accuracy, so status changes remain proposals with evidence rather than being applied.
Problem
Status changes can be hidden in existing literature, while current trackers lack an underlying bibliographic graph and query language over problems and evidence.
Source: §I · p. 1, §III · p. 2Method
The system preserves a curated problem catalog beneath rebuilt bibliographic evidence, connects the layers with typed provenance-carrying links, and constrains model stages to offered candidates.
Source: §III · p. 2, §IV · p. 3Results
Of 41 hand-checked solves and refutes verdicts, 9 were correct, 30 partly correct, and 2 wrong.
Source: §I · p. 2Takeaways & Limitations
A status change should remain a proposal with its evidence attached rather than being applied to the catalog.
Source: §XI · p. 8Without a labelled set of problems solved over time, status evaluation can measure precision but not recall, so the number of missed genuine solutions is unknown.
Source: §VIII · p. 7
Abstract
from arXiv · showhide
A research field records its open problems in surveys, keynotes and community reports. Those are prose, written every few years, and nobody can ask them who works on a problem now or whether it was settled last year. We argue the problems should be data, and we build the system that makes them so: a hand-curated problem layer carried in unchanged, under an evidence layer rebuilt from a bibliographic index, joined by typed, provenance-carrying links in one graph database and queryable from one language. A language model does four jobs -- per-paper extraction, catalog grounding, per-paper status verdict, and idea generation -- each constrained to a candidate set so out-of-set answers are refused rather than stored. We report a study on two curated catalogs, database research (1,053 problems) and AI research (1,000 problems), each with a prior catalog-only graph. The rebuilt graph answers every question the catalog-only graph answered -- 47 of 47 on databases, 34 of 35 on AI -- plus questions the catalog-only graph refused: recent workers, untouched problems, methods tried, and outside papers bearing on a problem (336 problems, 770 works on databases). The capability we most wanted does not survive its own audit. Of 41 solves and refutes verdicts hand-checked against the papers, 9 are correct, 30 partly correct and 2 wrong; no refutes verdict (0 of 34) disproves an open problem, and only 1 of 7 solves closes the central question. We report that failure in full and draw the design conclusion: a status change must be proposed with its evidence and never applied, making the human reviewer part of the architecture. We close with the data-engineering problems this raises: entity resolution across curated and harvested layers, verifying model status claims, freshness under source quotas, and evaluation without ground truth.
I. Introduction
Open-problem reports are prose that quickly age, leaving questions about current workers and status to literature searches. The paper proposes a queryable graph with a curated problem layer and a rebuilt evidence layer, governed by requirements designed to prevent defects encountered during construction.
I. Introduction
- Status changes can hide in existing literature, while reports leave current workers, settlements, and untouched problems to manual searches.The authors cite the rapid growth of machine-learning literature as an added survey difficulty.Source: §I · p. 1
- A field’s open problems should be first-class data with identity, status, and provenance beneath an evidence layer rebuilt from literature.The authors distinguish this from scholarly graphs that model papers, authors, and contributions, and trackers without an underlying bibliographic graph and query language.Source: §I · p. 1
- The proposed system combines a curated problem layer, a rebuilt evidence layer, typed links, constrained model stages, and refusal of out-of-set answers.Its study compares two fields, and its hand-check of 41 status verdicts found 9 correct, 30 partly correct, and 2 wrong.Source: §I · pp. 1–2
II. Requirements
- Stable, separate identities for each layer prevent silent label collisions that can produce plausible but incorrect query counts.The first rebuild returned 8,769 catalog-cited papers instead of 2,679 when harvested records shared catalog labels.Source: §II · p. 2
- Provenance on every node and edge lets readers distinguish bibliographic facts, extracted claims, web findings, and synthesis.The authors say provenance also made their status-verdict audit possible.Source: §II · p. 2
- Rebuilding derived evidence while versioning the curated catalog makes evidence added since the last catalog edit queryable.For AI, the system found 71 works from 2026, 66 absent from the catalog.Source: §II · p. 2
- Status changes remain proposals with evidence and reasoning; the system does not alter the catalog.The authors call this the design’s most important requirement after the audit.Source: §II · p. 2
- One graph database and one query language let the two graphs be compared using the same query text.Source: §II · p. 2
III. Architecture
The architecture keeps the curated catalog intact and separate from harvested evidence, joining them through explicit, typed links. The links preserve how matches were made and what evidence supports a problem relation.
III. Architecture
- The curated catalog enters unchanged, preserving its ids, statuses, and text for comparison with the catalog-only graph.Nothing in the pipeline writes to the catalog.Source: §III · p. 2
- The build rejects any label shared between catalog and evidence layers to prevent queries from silently spanning both populations.Source: §III · p. 2
- SAME_AS edges record the matching rule, while EVIDENCE_FOR links a work to a problem with a verdict, supporting quote, and confidence.Papers are matched by DOI, then arXiv identifier, then title and year; people are matched through authored papers only when exactly one candidate matches.Source: §III · p. 2
- Catalog name variants remain separate nodes linked to one Author, so queries can traverse SAME_AS without replacing catalog text.The three variants “Neumann,” “T. Neumann,” and “Thomas Neumann” resolve to one Author.Source: §III · pp. 2–3
IV. The Pipeline
The pipeline combines deterministic graph-building stages with four language-model stages constrained to offered candidate sets. The model extracts and organizes information, grounds extracted problems against the catalog, proposes status changes, and generates ideas only for catalog problems.
IV. The Pipeline
- The pipeline has nine stages: five deterministic and four language-model stages, with the latter constrained by design.Source: §IV · p. 3
- Extraction reads titles and abstracts to produce structured entities and claims; it does not judge whether a problem is solved.Each extracted node is marked with source=extraction.Source: §IV · p. 3
- Canonicalisation maps extracted names to synonyms and broader parents using only names already extracted, preventing invention of new concepts.Source: §IV · p. 3
- Grounding matches each extracted problem to a catalog problem as same or narrower, or declines; status checks then classify up to eight evidence papers and may write a suggested status, not change the catalog.Verdicts include advances, restates, solves, refutes, or unrelated, with a quote and confidence.Source: §IV · p. 3
- Idea generation is limited to offered catalog problems, and ideas naming anything outside that set are rejected at parse time.Across two runs, 104 ideas were generated and none were rejected under this rule.Source: §IV · p. 3
V. Study Design
The study compares a catalog-only graph with a rebuilt graph using the same catalog commit, testing preservation of prior answers, questions newly answerable with evidence, and model judgments. The catalog-only benchmarks show no regressions on databases and near-identical query results on AI.
V. Study Design
- The comparison uses V1 and V3 built from the same catalog commit, isolating the addition of an evidence layer beneath an unchanged catalog.Source: §V · p. 3
- The study tests non-regression, whether previously refused questions gain answers, and the quality of system judgments.The audit covers every solves and refutes verdict from both runs, 41 in total.Source: §V · p. 3
A. Non-regression
- 47 database queries return identical answers on the catalog-only and rebuilt graphs; 34 of 35 AI queries return the same rows.On AI, 26 are row-for-row identical, eight differ only in tie ordering, and one differs by a tied row under LIMIT 10.Source: A. Non-regression · p. 4
- The non-regression test is limited because its original queries read only the catalog layer and cannot show what the evidence layer adds.Source: A. Non-regression · p. 4
B. What the evidence layer adds
The evidence layer improves answers to catalog questions and exposes outside literature, although abstract grounding remains limited. Bridging draws on non-DOI identifiers, while conservative person matching leaves some researchers unlinked.
B. What the evidence layer adds
- Of 11 attempted database questions, 8 improved; one was independently corroborated, one unchanged, and one solved-status answer remained unverified.Twelve of the 23 partially answered or refused questions were not attempted because they required catalog content the evidence layer does not supply.Source: B. What the evidence layer adds · p. 4
- Among 16 attempted AI questions, 5 improved, 4 moved from refused to answered, 3 to partly answered, and 4 were unchanged or mixed.The mixed researcher-ranking result improved name matching but dropped a researcher whose catalog node had no matched author.Source: B. What the evidence layer adds · p. 4
- 336 database problems link to 770 outside papers; abstract grounding is 19.0% on AI and 25.0% on databases, with catalog scope and extraction granularity inseparable as causes.The outside papers were not cited by the catalog curators; four fifths of abstract extractions do not correspond to a curated concept.Source: B. What the evidence layer adds · p. 4
C. The bridge, and what it costs
- On AI, arXiv ids produced 3,026 matches and title-year matching 1,604, while only one of 4,932 catalog papers carried a DOI.A DOI-only bridge would have matched one paper.Source: C. The bridge, and what it costs · p. 4
- Exact-one-candidate person matching leaves 3,445 of 15,102 catalog people unlinked, including a seed researcher split across upstream author ids.The rule preserves precision but can omit a researcher from a headline query.Source: C. The bridge, and what it costs · p. 4
- In AI, 2,732 authors link to multiple catalog nodes, covering 6,582 nodes that remain separate rather than being merged.Keeping nodes intact avoids silently rewriting counts asserted by the catalog's benchmark.Source: C. The bridge, and what it costs · p. 4
D. Status verdicts: the component that failed its audit
The audit found that most status verdicts did not close the questions they addressed: none of 34 refutes disproved an open problem, and only one of seven solves closed the central question. The failures were systematic, while status suggestions remained unapplied.
D. Status verdicts: the component that failed its audit
- The hand-check covered all 41 solves and refutes verdicts using titles, abstracts, rationales, and quotes; full texts were not read, and no graph status was changed.Four status-change suggestions were written, but none was applied.Source: D. Status verdicts: the component that… · pp. 4–5
- 8 of 32 AI refutes were correct when read as contradicting a claim on the problem page, but none showed that the open problem was closed negatively.Five of the eight correct AI refutes concerned the minority of statements that named the claim under examination.Source: D. Status verdicts: the component that… · pp. 4, 6
- 9 of 41 verdicts made a logic error: evidence that current methods fail does not refute an open question about whether a task is possible.The paper identifies this as the dominant failure pattern, rather than a reading error.Source: D. Status verdicts: the component that… · p. 5
- The formal-mathematics comparison likewise distinguishes proving a restricted sub-case or refuting an intermediate proof step from closing the full problem.A candidate proof under additional unproved assumptions left a milestone open; a counterexample to a proof step did not refute the theorem.Source: D. Status verdicts: the component that… · p. 5
- 9 of 41 verdicts were correct, 30 partly correct, and 2 wrong; none of 34 refutes disproved an open problem, and 1 of 7 solves closed the central question.The other six solves settled sub-cases rather than the central question.Source: D. Status verdicts: the component that… · pp. 4–5, Table VII · p. 6
- Confidence was mildly informative, but with n = 41 the authors considered it insufficient to set a threshold.Mean confidence was 0.73, 0.6, and 0.4 for correct, partly correct, and wrong AI verdicts; database means were 0.8, 0.61, and 0.4.Source: D. Status verdicts: the component that… · p. 6
E. Grounding to a shared vocabulary
The AI run tested grounding against a merged computer-science vocabulary of 13,551 classes. It placed a quarter of current AI-paper concepts, with methods especially poorly represented in the vocabulary.
E. Grounding to a shared vocabulary
- The AI run grounded extracted concepts against a merged computer-science vocabulary of 13,551 classes from CSO and the ACM classification.The test examined whether extracted concepts could be placed in a vocabulary not written by the project.Source: E. Grounding to a shared vocabulary · p. 6
- 25.4% of methods were placed in the shared vocabulary, which the authors describe as a vocabulary gap rather than an extraction gap.The authors say named entities from 2024–2026 machine learning are largely absent from shared field vocabularies.Source: E. Grounding to a shared vocabulary · p. 6
- 5.9% of datasets and tools were placed, and 9 answers were refused because they named a class not offered.The table covers concepts placed at the same or narrower level in the vocabulary.Source: E. Grounding to a shared vocabulary · p. 6, Table IX · p. 6
F. Cost
A full rebuild is modest in model-call and storage terms, but an upstream index quota can stop it. The reported defects show how shared labels, incorrect abstracts, and precision-first linking can distort or omit graph data.
VII. Three Defects, Reported in Full
- A shared Paper label made a query count 8,769 catalog-cited papers instead of 2,679, without an error or warning.The earlier build could not be recomputed; the authors found the problem because an older benchmark asserted the correct count.Source: §VII · p. 6
- An upstream wrong abstract yielded six unrelated methods that ranked fifth to tenth among methods used for the most problems.A frequency floor was a mitigation, not a fix; the known-bad paper happened not to carry a solves or refutes verdict in this run.Source: §VII · pp. 6–7
F. Cost
- A full build costs a few thousand model calls and about 30M input tokens, produces a 17 MB snapshot, and can be stopped by the index’s free-tier daily quota.The deep model accounts for 54 of 3,698 AI calls and 7.0M input tokens; no dollar cost is computed.Source: F. Cost · p. 6, Table X · p. 7
VII. Three Defects, Reported in Full
- The precision rule dropped a seed researcher, but the alternative could have added a wrong co-authorship to a graph built around provenance.Source: §VII · p. 7
VIII. Limitations
The authors identify limits in status evaluation, study scope, data sources, and manual assessment. These boundaries constrain which conclusions the reported benchmarks and audits support.
VIII. Limitations
- With no labelled set of problems solved between 2015 and 2026, status evaluation can measure precision but not recall, leaving missed genuine solutions unknown.Source: §VIII · p. 7
- Both catalogs were built by the authors, so the schema may reflect their own curation habits.Source: §VIII · p. 7
- Every extracted fact relies on a title, abstract, and metadata rather than full text.Source: §VIII · p. 7
- The improvement grades are the authors’ readings of returned rows, and no automatic checker covers them.The AI question set was also assembled from the catalog’s own scope notes and stated limits.Source: §VIII · p. 7
- Per-call logs are unpublished because they are large and contain third-party abstracts; reported numbers instead come from committed summary files.Source: §VIII · p. 7
- Non-regression uses one benchmark per field, written by the catalog’s authors, and therefore inherits whatever those benchmarks do not ask.Source: §VIII · p. 7
IX. A Research Agenda
The study motivates a research agenda around linking curated and harvested data, checking status claims, and maintaining freshness and accountability. It also identifies the need for evaluation and governance when system outputs are proposals about knowledge.
IX. A Research Agenda
- Entity resolution must link a small authoritative curated layer to a larger noisy harvested one: the DOI rule fired zero times for one catalog, and precision rules left 3,445 of 15,102 people unlinked.A label collision also inflated a count by 3.3× without an error.Source: §IX · p. 7
- Status verification must distinguish settling a whole question from settling a sub-case; the authors propose explicit claims and sub-case decompositions, as in the 108 milestones and about 380 tasks in.Six of 41 verdicts failed at this distinction, and nine more confused evidence about a question with an answer to it.Source: §IX · p. 7
- Freshness depends on rebuilds rate-limited by an external index’s per-IP daily budget; incremental rebuilding and prioritizing rechecks remain open problems.Source: §IX · p. 7
- Provenance alone does not let researchers contest a wrong SAME_AS link or refutes verdict, and the system has no mechanism for disputes to persist across rebuilds.Source: §IX · p. 7
- Evaluation without ground truth remains open: non-regression works because a prior graph had its own benchmark, a historical accident rather than a general method.The field needs an evaluation design for outputs that propose a state of knowledge.Source: §IX · p. 7
- Curation and governance remain unresolved: this architecture treats the catalog as authoritative, while mathematics trackers use community status updates by pull request, a design the authors want to copy.Source: §IX · pp. 7–8
X. Related Work
Related systems address open-problem tracking, scholarly graphs, text mining, idea generation, and living reviews, but leave a gap this paper targets: a bibliographic evidence graph linked to curated problem statuses. The conclusion keeps status changes as evidence-backed proposals; the artifact is a frozen, reproducible release.
X. Related Work
- Open-problem trackers such as Erdős Problems and ProofAtlas maintain statuses, while formal proof projects such as can check sub-tasks; these works motivate, but do not supply, a bibliographic graph with typed links and queries.Source: §X · p. 8
- ORKG models papers, contributions, and research problems but not statuses that evidence can change; Semantic Scholar and OpenAlex provide bibliographic data without problem statuses.Source: §X · p. 8
- Problem-mining work extracts challenges and future directions, but its outputs lack persistent entities, status, and links to a curated catalog.Source: §X · p. 8
- Idea-generation systems with novelty checks assess ideas against literature at generation time and retain nothing; this paper’s constrained generation produced 104 ideas against offered problems.Source: §X · p. 8
- Living systematic reviews update one clinical question with human reviewers and a protocol; the paper envisions that loop across a thousand questions using a proposal queue.Source: §X · p. 8
XI. Conclusion
- The conclusion reports that the rebuilt graph preserves catalog-only answers, while its status component gets 9 of 41 judgments right; status changes therefore remain proposals with evidence attached.The authors argue that refutation requires a problem statement that names its claim.Source: §XI · p. 8
Artifact
- The repository is a frozen v1.0-paper29 artifact under Apache-2.0 for code and CC BY 4.0 for measurements; per-call logs are not published because they contain third-party abstracts.Source: Artifact · p. 8