PaperlayerView exact revision on arXiv

Source-linked AI summary

A Domain-Aware Pipeline for Structured Extraction and Retrieval-Augmented Question Answering over arXiv Superconductivity Literature

Ivan Iorsh, Aleksandr Kapitonov, Ivan Filianin, Matvey Lesonen, Nikita Chizhov

arXiv:2610.34391v1cs.IRcs.CL

TL;DR

Superconductivity literature is challenging to search and compare because PDFs, chemical notation, and multiple kinds of temperature complicate extraction and evidence retrieval. This paper builds an integrated pipeline for structured extraction and retrieval-augmented question answering, with extraction gains but no aggregate source-recall improvement from adding extracted metadata to hybrid retrieval.

  • Problem

    PDF layout and notation complicate literature review, while identifying superconducting Tc requires distinguishing it from other temperatures and linking it to the correct material.

    Source: §I · pp. 1–2

  • Method

    The paper builds a thematic arXiv superconductivity corpus and combines domain-aware material and Tc extraction, structured metadata, hybrid retrieval, and grounded question answering.

    Source: §I · p. 2, §VIII · p. 9

  • Results

    45.5% versus 9.1% material–Tc pair accuracy on the primary positive set; hybrid retrieval with extracted fields does not improve aggregate source recall over plain hybrid.

    Source: §VIII · p. 9

  • Takeaways & Limitations

    The pipeline can assemble literature-derived experimental Tc data for comparison with independent theoretical data and expert inspection of deviations.

    Source: A. APPLICATION: SCREENING OF ANOMALOUS · p. 9

    The work is specialized to one domain, and downstream results depend on PDF processing and extraction quality; extracted metadata is noisy in hybrid field retrieval.

    Source: B. LIMITATIONS · p. 9

Abstract

from arXiv · show

Scientific literature in materials physics is growing faster than researchers can manually inspect it, yet most retrieval tools rely on keyword matching or generic question answering that ignores domain-specific structure in PDF files. This paper presents an end-to-end pipeline that transforms arXiv superconductivity publications into a searchable corpus with structured metadata and retrieval-augmented question answering. The system combines thematic corpus construction, domain-aware extraction of materials and superconducting transition temperatures, and hybrid retrieval that merges dense chunk embeddings with structured publication fields. We test whether normalized domain attributes improve evidence retrieval and answer groundedness relative to lexical-only and dense text-only methods. Extraction is evaluated on a manually verified gold set of 59 superconducting materials, 46 of which have confirmed experimental critical temperatures. On the 33-material primary positive subset, domain-aware pair accuracy reaches 45.5% versus 9.1% for a regex baseline. Retrieval and generation are compared under a fixed language model (qwen2.5:7b-instruct) on a 100-question benchmark, varying only the retrieval mechanism: BM25, dense, hybrid, and hybrid with domain metadata. BM25 gives the highest source recall (Recall@5 of 47%), and adding extracted metadata to hybrid retrieval does not improve it (39% versus 41%), which we attribute to noise in the extracted fields; the main bottleneck is locating the supporting span inside a correctly retrieved paper. As a materials-informatics application, literature-derived temperatures are matched to an independent theoretical reference set: 200 materials appear in both sources, and 20 have a median experimental critical temperature above the theoretical value.

I. INTRODUCTION

Superconductivity literature is difficult to search reliably because PDF conversion, chemical notation, and multiple physical transitions complicate the identification and linking of materials and temperatures. The paper integrates domain-aware extraction with retrieval and question answering, addressing a research gap between structured extraction and text-based search.

I. INTRODUCTION

  • Manual review is increasingly inefficient, while keyword search and PDF conversion struggle with aliases, formulas, layout, and distinguishing superconducting Tc from other temperatures.Correct assignment requires contextual analysis and linking each temperature to the corresponding material.Source: §I · pp. 1–2
  • The paper tests whether normalized domain features improve evidence retrieval and answer groundedness, integrating structured material and physical-property data with semantic text-fragment representations.Its research question compares these features with lexical-only or dense text-only methods.Source: §I · p. 2
  • The contributions span an end-to-end pipeline, context-filtered material–Tc extraction, hybrid retrieval, evaluation on verified materials and domain questions, and a theoretical-reference screening application.The evaluation includes a gold set of 59 materials and a 100-question benchmark.Source: §I · p. 2

II. RELATED WORK

  • Prior extraction work builds structured chemical records and improves materials entity recognition, but the cited systems do not test whether extracted fields help downstream retrieval.Rule-based quantity handling and neural semantic matching are presented as complementary.Source: A. STRUCTURED INFORMATION EXTRACTION… · p. 2

B. SUPERCONDUCTIVITY LITERATURE EXTRACTION

The paper situates material–Tc extraction within prior superconductivity literature tools, then connects domain fields to retrieval and answering in an integrated arXiv pipeline. Its research gap is whether normalized fields improve evidence ranking and grounded answers.

B. SUPERCONDUCTIVITY LITERATURE EXTRACTION

  • The paper’s extraction contribution is not material–Tc extraction itself, but using extracted fields as retrieval and answering signals in an end-to-end evaluation.Context checks address ambiguity between identical symbols denoting different physical quantities.Source: B. SUPERCONDUCTIVITY LITERATURE… · p. 3

D. SCIENTIFIC DOCUMENT RETRIEVAL

  • BM25 can match exact formulas but struggles with paraphrases, while dense retrieval lacks built-in chemical constraints and document-level scientific encoders do not locate fine-grained claims.The paper extends hybrid sparse–dense retrieval with structured material and Tc metadata.Source: D. SCIENTIFIC DOCUMENT RETRIEVAL · p. 3

E. RETRIEVAL-AUGMENTED GENERATION FOR SCIENCE

  • Scientific RAG systems can return thematically related but physically wrong spans, and the cited systems do not expose normalized superconductivity fields as filters or reranking features.Prior systems handle domain structure implicitly through text.Source: E. RETRIEVAL-AUGMENTED GENERATION FOR… · p. 3

F. RESEARCH GAP

  • The paper measures extraction, hybrid retrieval with and without domain metadata, and answer groundedness together because normalized material–Tc fields have received little measurement for evidence ranking.The stated hypothesis is that fields reduce physically irrelevant evidence when extraction is accurate enough.Source: F. RESEARCH GAP · p. 3

III. SYSTEM OVERVIEW

  • The pipeline turns thematically selected arXiv superconductivity papers into a searchable corpus with structured metadata and RAG, linking extraction, storage, hybrid retrieval, and grounded answer generation.Publications with reliable material extraction are chunked and embedded for retrieval.Source: §III · p. 3

IV. METHODS

Corpus construction harvests and filters arXiv metadata, locates selected PDFs in bulk TAR archives, and extracts only target papers. A further subject filter excludes papers outside the current superconductivity analysis scope.

A. CORPUS CONSTRUCTION

  • The corpus workflow harvests OAI-PMH metadata, filters for cond-mat.supr-con, indexes PDF identifiers to TAR locations, and selectively extracts the needed PDFs.The index avoids scanning all archives for every document request.Source: A. CORPUS CONSTRUCTION · p. 4
  • 46,177 superconductivity-related PDFs were extracted from the bulk store, out of 46,746 category-filtered publications; 569 corresponding PDFs were absent from the source store.The index covered 2,921,746 PDF identifiers across 10,847 TAR archives.Source: A. CORPUS CONSTRUCTION · p. 4

B. THEMATIC FILTERING OF CANDIDATE PUBLICATIONS

  • The subject filter excludes papers with specified out-of-scope topics or explicit pressure values, and documents receive a pass, exclusion, or processing-error outcome.The current rules exclude high-pressure experiments because their temperature and material behavior fall outside the extraction rules.Source: B. THEMATIC FILTERING OF CANDIDATE… · p. 4

C. PDF PROCESSING AND DOMAIN-AWARE EXTRACTION

PDF processing preserves reading order and chemical notation before domain-oriented extraction. Material and Tc are extracted jointly, with contextual evidence and diagnostics retained in a normalized publication payload.

C. PDF PROCESSING AND DOMAIN-AWARE EXTRACTION

  • PDF text processing accounts for multi-column layout, rejoins hyphenated words, and preserves superscripts and subscripts needed for chemical formulas.For two-column documents, it restores reading order among text blocks.Source: C. PDF PROCESSING AND DOMAIN-AWARE… · p. 5
  • The extractors target studied materials, superconducting Tc, system dimensionality, unconventional-superconductivity cues, and reported Debye frequencies.The output includes a normalized subject payload with these attributes and Tc extraction diagnostics.Source: C. PDF PROCESSING AND DOMAIN-AWARE… · p. 5
  • Material and Tc extraction are coupled: candidate materials are ranked, Tc is extracted, the material choice is refined using temperature evidence, and Tc is extracted again.The system stores supporting sentence and context, evidence type, confidence, and the final numeric Tc value.Source: C. PDF PROCESSING AND DOMAIN-AWARE… · p. 5

D. STRUCTURED STORAGE

The system stores normalized publication attributes alongside source documents and links them to searchable text fragments. Structured fields support deterministic queries and metadata-aware retrieval, while extraction and retrieval remain coupled through shared processing and indexing decisions.

D. STRUCTURED STORAGE

  • A normalized publication record links bibliographic fields, subject attributes, processing statuses, and the source PDF through an arXiv identifier.Bibliographic data use relational columns; evolving subject attributes are stored as JSONB.Source: D. STRUCTURED STORAGE · p. 5

E. CHUNKING AND VECTOR INDEX CONSTRUCTION

  • Eligible publications are converted into overlapping text fragments, embedded, and indexed with metadata copied from their publication.Fragments are approximately 1600 characters long with 250-character overlap; boundaries may be adjusted to preserve complete units.Source: E. CHUNKING AND VECTOR INDEX… · p. 5

F. HYBRID RETRIEVAL AND ANSWER GENERATION

  • Structured requests use deterministic SQL, while open questions use semantic search, domain constraints, reranking, and publication-level source diversity.The reranker down-weights bibliographies, captions, tables, corrupted text, and dense formulas, and up-weights explicit material mentions or Tc confirmation.Source: F. HYBRID RETRIEVAL AND ANSWER… · pp. 5–6

G. COUPLING OF EXTRACTION AND RETRIEVAL

  • Extraction determines index membership and supplies fragment metadata for filtering and reranking, while shared PDF conversion keeps extraction and indexing text consistent.PDF conversion quality affects both material extraction and the coherence of search chunks.Source: G. COUPLING OF EXTRACTION AND RETRIEVAL · p. 6

V. EXPERIMENTAL SETUP

The evaluation specifies the corpus, manually checked extraction data, scoring criteria, and comparison configurations. It distinguishes material and temperature extraction from correct material–Tc linkage and includes both expected-positive cases and labeled failure cases.

A. CORPUS

  • The evaluation uses the arXiv superconductivity corpus, excluding 569 metadata records whose PDFs were unavailable in the arXiv bulk store.Corpus statistics are summarized in Table 4.Source: A. CORPUS · p. 6

B. GOLD SET FOR EXTRACTION EVALUATION

  • The gold set contains 59 superconducting materials; experimental Tc is confirmed for 46, and 33 have a publication illustrating correct automatic extraction.Each material record checks material identification, superconducting Tc extraction, and the material–Tc link.Source: B. GOLD SET FOR EXTRACTION EVALUATION · p. 6
  • Gold labels require the publication’s primary studied material and its superconducting transition temperature, excluding other transitions and processing temperatures.Aliases are treated as equivalent only when the publication context makes the correspondence unambiguous.Source: B. GOLD SET FOR EXTRACTION EVALUATION · p. 6

C. EXTRACTION EVALUATION PROTOCOL

  • Scoring uses an 8 K Tc tolerance and evaluates material identity, Tc, and their linkage on 33 positive cases, alongside 22 labeled failure cases.The extractor is compared with regex, no-filter, and no-material–Tc-binding configurations; errors include wrong transitions, wrong compounds, and article-type misclassification.Source: C. EXTRACTION EVALUATION PROTOCOL · pp. 6–7

D. QUESTION ANSWERING BENCHMARK

The question-answering evaluation uses questions with corpus evidence and compares retrieval configurations under controlled generation settings. It separately measures finding a source, finding a supporting span, and producing a grounded answer.

D. QUESTION ANSWERING BENCHMARK

  • The benchmark contains 100 questions with reference answers, supporting arXiv sources, and one to three evidence spans; questions lacking direct corpus evidence are excluded.This separates evaluation of source retrieval, evidence-span retrieval, and grounded answer generation.Source: D. QUESTION ANSWERING BENCHMARK · p. 7

E. RETRIEVAL AND RAG EVALUATION

  • Four retrieval configurations are compared using Recall@1, Recall@5, MRR, and Evidence Recall@10, which requires retrieving an annotated supporting span.All runs use qwen2.5:7b-instruct with fixed generation parameters and prompt; only the retrieval mechanism changes.Source: E. RETRIEVAL AND RAG EVALUATION · p. 7
  • Generation evaluation measures answer accuracy, groundedness among non-abstaining answers, abstention rate, and accuracy when gold evidence is present in context.The model is instructed to answer only from retrieved context and abstain when evidence is insufficient.Source: E. RETRIEVAL AND RAG EVALUATION · p. 7

VI. RESULTS

The results show gains from domain-aware extraction but no aggregate retrieval improvement from adding extracted fields to hybrid search. Retrieval performance varies by question type, and locating the exact supporting span remains a major bottleneck.

A. CORPUS AND PROCESSING

  • PDF files were obtained for 98.8% of thematically selected publications; all 569 missing files were absent from the arXiv bulk store.Source: A. CORPUS AND PROCESSING · p. 7

B. EXTRACTION RESULTS

  • 45.5% pair accuracy versus 9.1% for regex on 33 expert-linked materials shows that domain-aware extraction improves material–Tc pairing.Material accuracy reaches 81.8% for all non-regex configurations, leaving temperature selection as the main area for further gains.Source: B. EXTRACTION RESULTS · p. 7
  • 60.6%/54.5% Tc/pair accuracy without filters versus 54.5%/45.5% for the full pipeline shows that filtering heuristics can over-reject or misselect candidates.The heuristics trade recall for precision, sometimes suppressing correct temperatures or selecting incorrect ones.Source: B. EXTRACTION RESULTS · pp. 7–8

C. RETRIEVAL AND QUESTION ANSWERING RESULTS

  • 47.0% Recall@5 and 0.366 MRR make BM25 the strongest overall source-retrieval method on the 100-question benchmark.Dense retrieval performs better on open-evidence questions, while hybrid retrieval improves comparison and threshold queries.Source: C. RETRIEVAL AND QUESTION ANSWERING… · p. 8
  • 39% versus 41% Recall@5 shows that adding extracted material/Tc fields does not improve plain hybrid retrieval overall.Only 4 of 16 non-abstaining answers with fields match the reference, although all 16 are grounded in retrieved context.Source: C. RETRIEVAL AND QUESTION ANSWERING… · p. 8
  • 33.3% conditional answer accuracy for hybrid with fields exceeds BM25 (18.2%), hybrid (16.7%), and dense retrieval (12.5%) when gold evidence is already in context.The evaluated subsets are small, so this difference is indicative rather than conclusive.Source: C. RETRIEVAL AND QUESTION ANSWERING… · p. 8
  • 47% maximum source Recall@5 versus 23% maximum Evidence Recall@10 identifies finding the supporting span, rather than the publication, as the main bottleneck.A retrieved paper may still fail to provide the required evidence in the language-model context.Source: C. RETRIEVAL AND QUESTION ANSWERING… · p. 8

D. QUALITATIVE ANSWER CASES

The qualitative cases show that correct answers depend on placing the supporting span—not merely the paper—in context. The discussion connects extraction fidelity to retrieval performance, while the screening application uses expert-verified matches to identify candidates for further study.

D. QUALITATIVE ANSWER CASES

  • Hybrid retrieval with material/Tc fields answered Li2IrSi3 correctly at 3.7 K, while BM25 recovered ZrNi2Ga’s 2.9 K.In both cases, the supporting span reached the model context.Source: D. QUALITATIVE ANSWER CASES · p. 8
  • For NbB2, dense retrieval found the correct paper but missed the supporting span, so the model answered 39.0 K instead of the gold 9.5 K.Evidence Recall@10 was zero; the error arose from intra-document evidence choice, not a document miss.Source: D. QUALITATIVE ANSWER CASES · p. 8

VII. DISCUSSION

  • Extracted fields did not improve aggregate source recall over plain hybrid retrieval, consistent with noisy metadata; conditional answer accuracy was highest with fields when gold evidence was already in context.The conditional results came from small subsets, and document-level recall alone did not prevent a wrong answer when the within-paper evidence was incorrect.Source: §VII · p. 8
  • Unlike specialized extraction systems such as Grobid-superconductors, this pipeline uses structured material and Tc fields for evidence retrieval and grounded question answering.Their value is limited by extraction fidelity.Source: §VII · pp. 8–9

A. APPLICATION: SCREENING OF ANOMALOUS

  • The screening compares a literature-derived experimental dataset with an independent theoretical reference set, using median experimental Tc when a material appears in multiple publications.The median is used to reduce the influence of outliers, incorrect extractions, and sample-to-sample differences.Source: A. APPLICATION: SCREENING OF ANOMALOUS · p. 9
  • After expert alignment and verification, 200 materials appeared in both datasets, and 20 had median literature Tc values above their theoretical references.The discrepancies identify candidates for further physical analysis, not confirmed unconventional superconductors.Source: A. APPLICATION: SCREENING OF ANOMALOUS · p. 9

B. LIMITATIONS

The paper identifies extraction quality and within-paper evidence ranking as important constraints on its pipeline. Its evaluations show both gains and unresolved limits, while the screening results remain candidates for investigation rather than confirmed unconventional superconductors.

B. LIMITATIONS

  • The main limitations include single-domain specialization, a modest manually verified gold set, dependence on PDF processing and extraction quality, noisy metadata, and weak intra-document evidence ranking.Some per-type and conditional metrics use small subsets, and the screening application depends on its theoretical reference dataset.Source: B. LIMITATIONS · p. 9

VIII. CONCLUSION

  • Material–Tc pair accuracy rose from 9.1% for regex to 45.5% for the full extractor, but current filters still trade recall for precision.The conclusion also reports that BM25 is strongest for formula-centered Tc lookup, dense retrieval helps open questions, and hybrid retrieval helps multi-document comparison and threshold queries.Source: §VIII · p. 9
  • For 20 of 200 matched materials, median literature Tc exceeded the theoretical reference; these are candidates for further study, not proven unconventional superconductors.The discrepancy serves only as a selection criterion.Source: §VIII · p. 9
  • Future work includes tighter temperature filters, more precise intra-document evidence ranking, a larger gold set and benchmark, additional physical parameters, and applications to other materials-informatics tasks.Source: §VIII · p. 9

APPENDIX A ANNOTATED DATASET DESCRIPTION

The appendix defines a manually verified gold set and its material-record evaluation criteria for superconducting material and Tc extraction.

APPENDIX A ANNOTATED DATASET DESCRIPTION

  • The gold set contains 59 manually verified superconducting materials; 46 have confirmed experimental Tc, and 33 have an arXiv example of correct automatic extraction.Evaluation uses material records, with comparison-only and review-only mentions excluded from primary material.Source: Appendix A · p. 10
  • A numeric temperature match counts only when it belongs to the annotated material and describes the superconducting transition.Debye, structural, magnetic, and synthesis temperatures are excluded as Tc.Source: Appendix A · p. 10
  • The record schema uses arxiv_id, material_gold, tc_gold, link_ok, and notes, with aliases and formula variants normalized before matching.Incorrect extractions retained in the gold set are used for error analysis.Source: Appendix A · p. 10

Related papers

Loading 2610.34391v1…