PaperlayerView exact revision on arXiv

Source-linked AI summary

URL Extraction from Scholarly Documents: A Cross-Format Comparative Analysis

Rochana R. Obadage, Lamia Salsabil, Sawood Alam, Bipasha Banarjee, William A. Ingram, Edward A. Fox, Jian Wu

arXiv:2609.08019v1cs.DL

TL;DR

URLs in scholarly papers support downstream research, but extraction commonly uses one representation and the effect of document format is not well characterized. This study benchmarks six formats and their combinations using manually annotated arXiv URLs. TEXTWAL+LaTeX performs best overall, and a large-scale analysis confirms substantial differences in extracted URLs across formats.

  • Problem

    URL extraction studies often use a single representation, while format effects and extraction coverage remain insufficiently characterized despite their relevance to downstream research.

    Source: §1 · pp. 1–2

  • Method

    The study benchmarks six document formats and format combinations using manually annotated URLs from arXiv papers, then applies format-specific pipelines to a longitudinal corpus sample.

    Source: §1 · p. 3, §6.1 · p. 12

  • Results

    TEXTWAL+LaTeX provides the best balance of precision, recall, and cost effectiveness (F1=0.69; R=0.88), while large-scale analysis confirms discrepancies in URLs extracted across formats.

    Source: §8 · p. 15

  • Takeaways & Limitations

    The benchmark and longitudinal results support selecting formats deliberately and using robust multi-format pipelines for reliable URL recovery at corpus scale.

    Source: §1 · p. 2, §8 · p. 15

    The findings may not generalize beyond arXiv or to PDFs converted from non-LaTeX sources, and LLM results depend on model versions.

    Source: §7.4 · p. 15

Abstract

from arXiv · show

URLs in scholarly documents link to rich external resources such as datasets, software, publications, and websites. Extracting these URLs is crucial in the data preparation stage of many downstream tasks, such as link rot analysis, web crawling, and building knowledge graphs. However, existing studies often downplay this phase, simply extracting URLs from a single format, usually text directly converted from PDFs. We present a systematic study evaluating URL extraction across six input formats (text with annotation layer, LaTeX, HTML, XML, Markdown, and PNG converted from PDF). To support the evaluation, we compiled a benchmark dataset consisting of 2,338 manually annotated URLs from 200 arXiv papers spanning a wide range of domains over a 33-year period. In addition to evaluating individual file formats, we also compared 63 composite input-format combinations. Our extensive evaluations indicate that TEXTWAL achieves the best performance among single-format inputs, while TEXTWAL+LaTeX achieves the best overall URL extraction performance. The same trend is observed for URLs linking to open-access datasets and software. To further validate these findings, we apply our format-specific URL extraction pipelines to a longitudinal random sample of 364,744 arXiv papers spanning 33 years. We observe a sharp increase in URL density after 2015, along with remarkable differences in URL extraction across file formats over time. Overall, our study highlights the importance of selecting an appropriate format for URL extraction from scholarly documents. The dataset and code are publicly available at: https://github.com/lamps-lab/arxiv-url-bench .

1 Introduction

URLs in scholarly papers support reproducibility and downstream work, yet extraction commonly relies on a single representation, often PDF-derived plain text, despite format-specific information loss. The paper addresses gaps in multi-format benchmarking and URL-complexity diagnosis with a benchmark, taxonomy, broad evaluation, and longitudinal analysis.

1 Introduction

  • URLs support reproducibility and downstream tasks including link-rot analysis, web crawling, knowledge-graph construction, resource discovery, and OADS identification.At repository scale, invalid or missed URLs can propagate into incorrect or incomplete resource collections and analyses.Source: §1 · pp. 1–2
  • Single-format extraction can miss URLs because conversion pipelines lose information, while PDF text extraction can fragment links in multi-column layouts.LaTeX, HTML, XML, and Markdown also have format-specific artifacts or conversion dependencies; VLMs and LLMs bring cost, hallucination, or reproducibility concerns.Source: §1 · p. 2
  • arXiv provides multiple representations of many papers, enabling format-specific benchmarking and assessment of extraction discrepancies and performance-cost trade-offs.The study frames this benchmark as a component of broader infrastructure work on scholarly-document availability, discoverability, and long-term accessibility.Source: §1 · p. 2
  • Existing studies lack a large-scale, statistically robust comparison across formats and a formal taxonomy for URL complexity and extraction-failure diagnosis.The paper notes that interactions between URL complexity and format-specific artifacts have not been systematically characterized.Source: §1 · pp. 2–3
  • 2,338 URLs from 200 arXiv papers anchor a multi-format benchmark, alongside a three-dimensional URL-complexity taxonomy and evaluations of extraction methods and format combinations.The study also analyzes a large longitudinal arXiv sample and offers format and method selection guidance.Source: §1 · p. 3

2 Related Work

Prior work covers URL and reference extraction, OADS discovery, and language-model extraction, but generally leaves format choice and cross-paradigm comparisons unresolved. This study addresses those gaps with format-stratified evaluation.

URL and Reference Extraction from Scholarly Documents

  • Prior URL and reference extraction work evaluates tools largely in isolation, leaving limited guidance on document-format selection and combinations.Regex-based extraction offers interpretability but limited robustness; layout-aware systems and PDF-to-text tools make different trade-offs.Source: §2 · p. 3

Open-Access Datasets and Software Discovery

  • Prior OADS work defines categories and detects dataset references, but has not systematically studied how document-format choice affects OADS extraction.The cited literature also describes distinctive OADS URL structure and accessibility concerns.Source: §2 · p. 3

Vision Language Models and Large Language Models

  • Prior work has not systematically compared VLMs and LLMs against structure-based pipelines for URL extraction, despite their different cost, hallucination, and reproducibility concerns.Open-weight VLMs operate on rendered pages, while proprietary LLMs can extract directly from PDFs.Source: §2 · p. 3

Link Rot and Temporal Analysis of Scholarly URLs

  • Earlier studies document URL decay, but have not performed longitudinal comparisons of URLs extracted across different document formats.Source: §2 · p. 3

3 Dataset Construction

The study builds a six-format arXiv benchmark through stratified sampling, format-specific conversion and extraction, complexity labeling, and manual URL annotation. It characterizes the benchmark's URL locations and includes a separate OADS categorization.

3.1 Stratified Sample Selection

  • The benchmark samples papers across time and document types, with all six formats available for each paper and at least three embedded URLs required.The dataset composition covers two phases and comprises 200 papers, 2,420 URL instances, and 2,338 unique URLs.Source: §3.1 · p. 4, Table 1 · p. 4
  • Phase A retained 182 papers after verification and randomly selected 100, while Phase B retained 285 and randomly selected 100, with additional sampling for scanned papers and early submissions.Phase A's final sample included single-column, double-column, and ETD papers; Phase B expanded sampling to scanned documents and submissions without LaTeX.Source: §3.1.1 · p. 4, §3.1.2 · p. 4

3.2 File Format Conversion and URL Extraction

  • The pipeline collects native PDF and LaTeX sources and derives text, TEXTWAL, HTML, XML, Markdown, and PNG representations for format-specific extraction.The listed conversions use PDF-to-text tools, LaTeXML, GROBID, Marker, and PyMuPDF rendering.Source: §3.2 · p. 4, §3.2.1 · pp. 4–5, Table 2 · p. 5

3.2.2 URL Extraction Methods

  • Extraction combines RegEx with structured-format parsing, while image-based VLM extraction and TEXTWAL-based LLM extraction provide additional approaches.The benchmark's methods parse links and attributes in LaTeX, HTML, XML, and Markdown; VLMs process page images and the evaluated LLM receives TEXTWAL.Source: §3.2.2 · pp. 5–6

3.3 URL Complexity Taxonomy

  • The proposed taxonomy separates URL extraction difficulty into lexicographical, spatial, and representational complexity.These dimensions concern URL character structure, document placement or layout, and how URLs are encoded or concealed.Source: §3.3 · p. 6, §3.3.1 · p. 6, §3.3.2 · p. 6

3.4 Manual URL Annotation and Ground Truth

  • Annotators recorded URL strings, page and paper identifiers, section categories, complexity labels, and OADS categories; a second review found 99.2% label agreement before resolving disagreements.The final ground truth contains 2,338 unique URLs.Source: §3.4 · p. 6

3.5 Sample Characteristics

  • Most URLs appear in references, and line-split URLs account for 430 instances (about 18%); page-spanning URLs are rare at 0.3%.Footnotes account for 11.8% of instances and figure captions for 0.4%.Source: §3.5 · p. 7, Figure 2 · p. 7

3.6 OADS URL Categorization

  • The OADS analysis separately annotates URL categories and evaluates extraction against 488 OADS URLs, about 21% of unique URLs.Source: §3.6 · p. 7

4 Evaluation

Evaluation uses strict URL matching and standard precision, recall, and F1 metrics, with OADS assessed separately. Format combinations are consolidated from representative inputs, and uncertainty and format differences are evaluated statistically.

4.1 Evaluation Metrics

  • Strict matching counts an extracted URL as valid only if it appears in the PDF and matches manual annotation; other outputs, including hallucinated or malformed strings, are invalid.The evaluation uses precision, recall, and F1; OADS URLs are evaluated separately using recall only because true and false positives cannot be separated for that subset.Source: §4.1 · pp. 7–8

4.2 Format Combinations

  • The combination analysis uses six core formats and the best-performing representative from each family, reducing the combination space from 212 to 26 and deduplicating unioned outputs.Qwen2-VL-7B is the selected open-weight VLM representative.Source: §4.2 · p. 8

4.3.1 Bootstrap Confidence Intervals

  • 95% confidence intervals for precision, recall, and F1 are estimated by document-level paired bootstrap resampling, with 10,000 iterations sampling 200 documents with replacement.Source: §4.3.1 · p. 8

4.3.2 Friedman Test for Global Format Comparison

  • A Friedman test on per-paper recall compares the six individual formats separately for All-URL Recall and OADS Recall.Source: §4.3.2 · p. 8

4.3.3 Wilcoxon Signed-Rank Tests for Pairwise Comparisons

  • After a significant Friedman result, pairwise Wilcoxon signed-rank tests compare the remaining 62 combinations with the best single-format recall baseline, using Holm-corrected p-values.Source: §4.3.3 · p. 8

5 Results

The results show that URL extraction depends strongly on the document format and extraction method. TEXTWAL is the strongest single-format option, while combining it with LaTeX gives the best overall balance of recall and F1.

5.1 PDF to Text Extractor Comparison

  • PyMuPDF TEXTWAL reaches 0.81 recall and 0.59 precision at 0.19 s/PDF; text-only extraction finds 558 URLs versus 1,887 for TEXTWAL.PyPDF reaches 0.83 recall but is about 10× slower; PDFMiner.six matches PyMuPDF’s precision and recall but is about 23× slower.Source: §5.1 · p. 8, Table 3 · p. 8

5.2 LLM and VLM Comparison for URL Extraction

  • Neither VLM nor LLM approaches outperform TEXTWAL; Claude Haiku 4.5 reaches 0.53 precision and 0.38 recall at an estimated $0.0285 per paper.Qwen2-VL-7B has the best precision and recall among the evaluated models.Source: §5.2 · p. 9, Table 4 · p. 9

5.3.1 Single-Format Performance

  • TEXTWAL achieves the highest single-format F1 (0.69) and recall (0.81); LaTeX has the highest precision among text-based representations, while IMAGE has the lowest F1 (0.17).Source: §5.3.1 · p. 9

5.3.2 Multi-Format Combination Analysis

  • TEXTWAL+Markdown reaches 0.90 recall; TEXTWAL+LaTeX+XML has the highest three-format F1 (0.68), while the full six-format union has the highest recall but lowest F1.For OADS URLs, TEXTWAL+LaTeX+Markdown reaches 95% recall, and larger combinations reach up to 98%.Source: §5.3.2 · pp. 9–10, Table 5 · p. 10
  • TEXTWAL+LaTeX is the best overall choice, raising recall from 0.81 to 0.88 without reducing F1.Table 5 reports performance across representative format combinations.Source: §5.3.2 · p. 9, Table 5 · p. 10

5.3.3 Statistical Significance of Format Differences

  • All-URL formats differ significantly (χ2(5) = 232.49, p < 0.001); all 31 TEXTWAL combinations beat TEXTWAL alone, and 43 of 62 combinations exceed Markdown for OADS recall.Source: §5.3.3 · pp. 10–11, Figure 3 · p. 11

5.4 Analysis of URL extraction performance

  • TEXTWAL has highest recall for arxiv.org (0.99) and near-perfect DOI-domain coverage, but completely fails on aws.amazon.com, where LaTeX, HTML, and IMAGE achieve perfect recall; LaTeX, HTML, and Markdown also achieve perfect ftp recall (1.00).TEXTWAL recovers 87.0% of annotation-only URLs, versus 44.6% for Markdown and below 37% for all other formats.Source: §5.4 · pp. 11–12

6 Temporal Trends in arXiv URLs

Across a common pool of 364,744 arXiv papers, the study tracks URL extraction yield by format from 1992 to 2024. URL density rises sharply after 2015, while TEXTWAL yields the most URLs in every year.

6.1 Analysis Pipeline

  • The longitudinal comparison uses a common pool of 364,744 papers with successful output in all five formats, sampled 1,000 papers per year across 100 bootstrap replicates.The pipeline began with 467,767 PDFs; the common pool supports comparisons on identical papers.Source: §6.1 · pp. 12–13, Figure 6 · p. 13

6.2 Overall URL Usage Trends

  • URL usage rises sharply after 2015; TEXTWAL yields the most URLs each year, followed by Markdown and HTML, while LaTeX and especially XML yield fewer.In 2022, the formats yield approximately 28, 24, 26, 16, and 3 URLs per paper, respectively; the analysis measures extraction yield, not accuracy.Source: §6.2 · p. 13

7 Discussion

The discussion recommends choosing extraction formats according to source availability, computational budget, and target objective. It also describes format-specific failure patterns, large-scale OADS implications, and limits on generalizing the findings.

7.1 Format and Approach Recommendations

  • TEXTWAL-centered pipelines are a strong low-cost choice when only PDFs are available, while multi-format combinations consistently provide higher recall and OADS coverage.The recommendation framework is based on LaTeX-native arXiv papers with source files available.Source: §7.1 · pp. 13–14, Table 6 · p. 14

7.2 Format-Specific Failure Modes

  • Hidden annotation layers dominate failures in HTML, XML, LaTeX, Markdown, and VLM extraction, reaching 97.8% for XML; references are error-prone across formats.Reference misses reach 70.6% for VLM but are 14.1% for TEXTWAL.Source: §7.2 · p. 14, Table 7 · p. 15

7.3 Implications for Large-Scale OADS Preservation

  • The authors estimate 4M–6M OADS URLs across arXiv, with roughly 0.8M–1.2M potentially inaccessible within 3–5 years due to link rot.The study reports that OADS platforms have driven arXiv’s URL-density growth since 2015 and estimates full-corpus extraction would take roughly 1–2 months.Source: §7.3 · pp. 14–15

7.4 Limitations

  • Findings may not generalize beyond arXiv or to PDFs derived from sources such as Word; LLM results also depend on model versions that may improve.Source: §7.4 · p. 15

8 Conclusion and Future Work

The study finds that URL recovery varies by format, with TEXTWAL+LaTeX offering the best balance of precision, recall, and cost effectiveness; it also outlines future extensions beyond arXiv and LaTeX-derived PDFs.

8 Conclusion and Future Work

  • TEXTWAL+LaTeX provides the best balance of precision, recall, and cost effectiveness (F1=0.69; R=0.88), while TEXTWAL is the strongest PDF-only baseline (F1=0.69; R=0.81).Combining TEXTWAL, LaTeX, HTML, and Markdown recovers 96% of OADS URLs; large-scale analysis also finds substantial extraction discrepancies across formats.Source: §8 · p. 15
  • The authors plan to extend OADS URL datasets beyond arXiv and study PDFs converted from non-LaTeX sources such as Word.They note that layout and encoding properties may differ substantially in those documents.Source: §8 · p. 15

A RegEx Pattern for URL Extraction

  • The URL-extraction regular expression targets diverse forms, including protocol URLs, domain-only references, authenticated URLs, IP addresses, Unicode domains, ports, and resource paths.It excludes private and local network ranges to reduce false positives from non-public addresses.Source: A RegEx Pattern for URL Extraction · p. 18

8 Conclusion and Future Work

  • The complexity taxonomy groups URL challenges into lexicographical, spatial, and representational dimensions for systematic characterization.The appendix presents the framework as support for fine-grained diagnosis of extraction failures.Source: B Three-Dimensional URL Complexity… · p. 18

B.1 Lexicographical Complexity

  • Lexicographical complexity concerns URL structure, including schemes, subdomains, ports, Unicode characters, paths, query strings, fragments, escapes, and omitted schemes.The examples also include trailing slashes.Source: §B.1 · pp. 18–19

B.2 Spatial Complexity

  • Spatial complexity concerns document placement, including URLs split across lines or pages and URLs embedded in footnotes, captions, or references.The appendix gives a caption URL as an example of an embedded layout element.Source: §B.2 · p. 19

B.3 Representational Complexity

The appendix defines representational complexity by how URLs are stored, displayed, or concealed. It also specifies zero-shot image prompts for VLMs and structured text prompts for LLM extraction.

B.3 Representational Complexity

  • Representational complexity covers URLs hidden in annotations or metadata, embedded in visual media, or shortened to mask their destinations.A clickable arXiv identifier is given as an example of visible text linking to a different URL.Source: §B.3 · p. 19

C.1 Vision-Language Model (VLM) Prompt Design

  • For page images, open-source VLMs receive a zero-shot prompt to return URLs only, one per line, including URLs in page regions and reconstructing line breaks.The prompt instructs models to ignore other text and not explain or summarize.Source: §C.1 · p. 20

C.2 Large Language Model (LLM) Prompt Design

  • For compiled TEXTWAL content, LLM extraction uses structured API calls at temperature zero and requests unique URLs and DOI links in a JSON array.The rules cover reconstruction, DOI formatting, completeness checks, and removal of trailing punctuation.Source: §C.2 · p. 20

D OADS URL Categorization Taxonomy

The appendix defines OADS URL categories by provenance, ownership, and function, then describes paired document-level testing and uncertainty procedures for comparing extraction formats.

D OADS URL Categorization Taxonomy

  • The OADS taxonomy classifies ground-truth URLs by provenance, ownership, and functional role, adapting a six-class scheme from Salsabil et al.It includes a general URL baseline category alongside specialized data, software, and project links.Source: D OADS URL Categorization Taxonomy · pp. 20–21
  • Third-party and author-provided dataset and software categories distinguish resources made available independently from those hosted or developed by the paper's authors.Third-party categories cover resources used by authors; author-provided categories cover data or tools shared by them.Source: D OADS URL Categorization Taxonomy · pp. 20–21
  • The Project category covers integrated repositories or landing hubs that combine datasets and software or support experimental frameworks.A GitHub repository containing source code, datasets, and documents illustrates the category.Source: D OADS URL Categorization Taxonomy · p. 21

E.1 Statistical Unit and Paired Design

  • Statistical comparisons use the individual paper as the unit and preserve pairing by evaluating each format on the same documents.Per-document precision, recall, and F1 are computed for each format variant and combination.Source: §E.1 · p. 22

E.2 Bootstrap Confidence Intervals

  • Paired bootstrap resampling over the 200-paper set for 10,000 iterations estimates 95% confidence intervals for precision, recall, and F1.The resampling preserves within-paper correlations across formats.Source: §E.2 · p. 22

E.3 Friedman Test for Global Format Comparison

  • A Friedman test compares recall across the six single-format variants, separately for All-URL Recall and OADS Recall.A significant result at p < 0.05 establishes that at least one format differs systematically and motivates pairwise tests.Source: §E.3 · p. 22

E.4 Wilcoxon Signed-Rank Tests for Pairwise Comparisons

  • After a significant Friedman result, paired Wilcoxon signed-rank tests compare the 62 other combinations with the best single-format baseline for each metric.Holm–Bonferroni correction controls family-wise error across comparisons at α = 0.05.Source: §E.4 · p. 22

F Comprehensive Format-Combination Evaluation Results

The appendix reports exhaustive results for 63 format combinations against total-URL and OADS ground truth, with performance metrics, uncertainty intervals, and selected failure-mode breakdowns.

F Comprehensive Format-Combination Evaluation Results

F.1 Evaluation Parameters and Metrics

  • Results are benchmarked against 2,338 valid total URLs and 488 OADS URLs, reporting precision, recall, and F1 with paired-bootstrap 95% confidence intervals.Intervals use 10,000 document-level resampling iterations.Source: §F.1 · p. 23
  • Figure 8 uses format combinations on the x-axis, with single formats named and multi-format combinations abbreviated; error bars show 95% confidence intervals.“Available in PDF” is the ground-truth category.Source: Figure 8 · p. 23
  • Table 9 defines failure-mode values as misses divided by ground truth and lists counts for document locations, layout positions, annotation layers, and URL structures.The caption supplies the ground-truth counts for those categories.Source: Table 9 · p. 23
  • Reported misses include 18.9% of scheme URLs and 13.4% of path/query URLs.The listed fractions are 433/2,285 and 278/2,080, respectively.Source: §F.1 · p. 23, Table 9 · p. 23
  • TEXTWAL misses 14.0% of embedded-reference URLs, 42.2% of embedded-footnote URLs, and 13.0% of URLs hidden in annotations.The reported fractions are 241/1,727, 116/275, and 168/1,295, respectively.Source: §F.1 · p. 23, Table 9 · p. 23
  • Reported misses include 64.2% of scheme URLs and 65.0% of path/query URLs.The listed fractions are 1,466/2,285 and 1,351/2,080, respectively.Source: §F.1 · p. 23, Table 9 · p. 23
  • HTML misses 63.1% of URLs hidden in annotations, 60.9% in embedded references, and 55.1% of URLs split across lines.The reported fractions are 817/1,295, 1,052/1,727, and 231/419, respectively.Source: §F.1 · p. 23, Table 9 · p. 23

G Error Analysis: Format-Specific Failure Modes

Failure modes differ across formats, and this complementarity is the stated mechanism behind multi-format combinations achieving substantially higher recall than any single format. Table 10 compares URL extraction performance across 63 format combinations.

G Error Analysis: Format-Specific Failure Modes

  • Different format-specific failure modes make multi-format combinations complementary, enabling substantially higher recall than any single format.Plain text and TEXTWAL fail mainly on spatial and representational complexity; XML on document location; LaTeX on macro conventions; Markdown on layout edge cases; and VLMs on visually unresolved content.Source: G Error Analysis: Format-Specific… · p. 24
  • Practitioners should audit target corpora for likely failure modes and select format combinations to address them.The passage names complex layouts, annotation-heavy workflows, and scanned content as examples to audit for.Source: G Error Analysis: Format-Specific… · p. 24
  • Table 10 compares URL extraction across 63 format combinations using valid URLs extracted, precision, recall, and F1, with bootstrap 95% confidence intervals.The caption specifies 10,000 paired document-level resamples for the intervals.Source: Table 10 · p. 25

Related papers

Loading 2609.08019v1…