PaperlayerView exact revision on arXiv

Source-linked AI summary

Revolutionizing Retrieval-Augmented Generation with Enhanced PDF Structure Recognition

Demiao Lin

arXiv:2401.12599v1cs.AI

TL;DR

Professional QA requires private domain knowledge absent from LLM pretraining, while PDF extraction and table structure pose challenges for retrieval. The paper evaluates whether parsing and chunking quality affects RAG; ChatDOC performed better than baseline more often than it was inferior on both question types.

  • Problem

    Professional tasks require private domain knowledge that is not part of LLM pretraining, while PDF extraction and table row-column relationships challenge retrieval.

    Source: §1 · p. 1

  • Method

    The paper examines rule-based and deep-learning PDF parsing methods and empirically evaluates their efficacy through practical examples.

    Source: §1 · p. 2

  • Results

    Across 86 extractive questions, ChatDOC performed better than baseline on 42, tied on 36, and was inferior on 8; across 216 comprehensive analysis questions, it was better on 101, tied on 79, and inferior on 36.

    Source: Results of Extractive Questions · p. 8, Results of Comprehensive Analysis… · p. 9

  • Takeaways & Limitations

    Information loss across conversion steps can significantly affect the effectiveness of RAG responses.

    Source: §1 · p. 2

    When a large irrelevant table ranks first, it can consume the context window and prevent access to relevant information.

    Source: §3.3 · p. 14

Abstract

from arXiv · show

With the rapid development of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) has become a predominant method in the field of professional knowledge-based question answering. Presently, major foundation model companies have opened up Embedding and Chat API interfaces, and frameworks like LangChain have already integrated the RAG process. It appears that the key models and steps in RAG have been resolved, leading to the question: are professional knowledge QA systems now approaching perfection? This article discovers that current primary methods depend on the premise of accessing high-quality text corpora. However, since professional documents are mainly stored in PDFs, the low accuracy of PDF parsing significantly impacts the effectiveness of professional knowledge-based QA. We conducted an empirical RAG experiment across hundreds of questions from the corresponding real-world professional documents. The results show that, ChatDOC, a RAG system equipped with a panoptic and pinpoint PDF parser, retrieves more accurate and complete segments, and thus better answers. Empirical experiments show that ChatDOC is superior to baseline on nearly 47% of questions, ties for 38% of cases, and falls short on only 15% of cases. It shows that we may revolutionize RAG with enhanced PDF structure recognition.

1 Introduction

Professional QA requires private domain knowledge absent from LLM pretraining, making RAG a popular approach. This paper asks whether PDF parsing and chunking quality affects RAG outcomes, given extraction and table-structure challenges and compounded information loss.

1 Introduction

  • RAG equips LLMs with domain knowledge by retrieving relevant private-base content, combining it with the query as context, and generating an answer.Accurate retrieval is pivotal to RAG efficacy.Source: §1 · p. 1
  • PDF retrieval can suffer from inaccurate text extraction and disordered table row-column relationships, so documents must be converted into retrievable content before RAG.The conversion includes parsing and chunking extracted content, followed by embedding chunks as vectors in a database.Source: §1 · pp. 1–2
  • Information loss at each conversion step can compound and significantly affect the effectiveness of RAG responses.Source: §1 · p. 2
  • The paper asks whether PDF parsing and chunking quality affects RAG outcomes, examining rule-based and deep-learning methods through empirical evaluations.It frames the issue as a question about RAG quality and document parsing quality.Source: §1 · p. 2

2 PDF Parsing & Chunking

PDFs lack the structural tags that make text and tables machine-readable, so parsing and chunking must reconstruct document structure. The section contrasts PyPDF’s rule-based workflow with ChatDOC’s deep-learning parser and their handling of complex pages.

2.1 Challenges and Methods Overview

  • Unlike tagged documents, PDFs encode character placement without paragraph or table structure, motivating parsers that recognize content blocks and handle complex layouts.The stated target layouts include multi-column pages, borderless tables, and tables with merged cells.Source: §2.1 · p. 3

2.2 Rule-based Method: PyPDF

  • PyPDF serializes PDF characters without structure, then applies separator-based chunking and merges adjacent chunks up to a preset limit of 300 tokens.The described implementation uses RecursiveCharacterTextSplitter and may split content according to separators such as newline.Source: §2.2 · p. 3
  • PyPDF misses table and paragraph boundaries, destroys table-cell relationships, and can output content in storage order rather than reading order.Its line-based serialization and separator heuristic can split a table and merge part of it with the following paragraph.Source: §2.2 · pp. 4–5

2.3 Deep Learning-based Method: ChatDOC PDF Parser

  • ChatDOC is a deep-learning parser trained on over ten million document pages, using OCR, object detection, reading-order determination, and table and logical-structure recognition.After parsing, it treats paragraphs and tables as basic blocks and merges adjacent blocks up to a token limit; it can output structured JSON or HTML.Source: §2.3 · p. 5
  • ChatDOC addresses PyPDF’s three Case 1 shortcomings by separating mixed layouts, preserving table structure including merged cells, and restoring reading order.Its parsed chunks represent tables in Markdown, repeating merged-cell text across the original cells.Source: §2.3 · p. 5, Figure 5 · p. 6

3 Experiments on the Impact of PDF Recognition on RAG

The section introduces a systematic experiment addressing whether document parsing and chunking affect RAG answer quality.

  • The authors conducted a systematic experiment to assess whether document parsing and chunking affect the quality of RAG answers.Source: §3 · p. 6

3.1 Quantitative Evaluation of RAG Answer Quality

The evaluation compares two RAG systems that differ in PDF parsing and chunking, using 302 screened questions from a collection of 188 real-world documents.

3.1.1 Settings

  • ChatDOC uses its PDF Parser and document structure for chunking, while the Baseline uses PyPDF and RecursiveCharacterTextSplitter.Embedding, retrieval, and QA components are the same for both systems.Source: §3.1.1 · p. 6

3.1.2 Data Preparation

  • The dataset contains 188 documents: 100 academic papers, 28 financial reports, and 60 documents from other categories.The collection includes textbooks, courseware, and legislative materials among the other categories.Source: §3.1.2 · p. 6
  • After screening 800 crowdsourced questions, the authors retained 302 for evaluation and divided them into extractive and comprehensive-analysis categories.Extractive questions seek pinpoint answers from direct excerpts; comprehensive-analysis questions require synthesis across sources and aspects.Source: §3.1.2 · pp. 6–7
  • Extractive answers were rated by humans on a 0–10 scale, while GPT-4 scored comprehensive-analysis results from 1–10.For comprehensive-analysis questions, retrieved content was evaluated without the lengthy answer; each pair was scored four times with order flipped and averaged.Source: §3.1.2 · pp. 7–8

Results of Extractive Questions

Across 86 extractive questions, ChatDOC was rated better than the Baseline more often than it tied or fell short.

Results of Extractive Questions

  • 42 of 86 extractive questions favored ChatDOC, 36 tied, and 8 favored the Baseline.The comparison is reported in Table 3.Source: Results of Extractive Questions · p. 8
  • Most cases with a clear winner fall in the region where ChatDOC scores higher than the Baseline.Figure 7’s distribution table places ChatDOC wins in the lower-left half and Baseline wins in the upper-right.Source: Results of Extractive Questions · p. 8
  • ChatDOC received full marks (10) in 40 cases, nearly half of the cases described in the score distribution.The passage reports the count as 40.Source: Results of Extractive Questions · p. 8

Results of Comprehensive Analysis Questions

For comprehensive-analysis questions, ChatDOC outperformed the Baseline more often than it tied or scored lower, and its retrieval scores were concentrated in the 8.5–9.5 range.

Results of Comprehensive Analysis Questions

3.2 Case Studies of RAG

The case studies contrast how ChatDOC and the Baseline retrieve content from a Tesla manual and a research paper, illustrating the role of document structure in the returned segments.

3.2 Case Studies of RAG

  • The paper presents these examples as cases where ChatDOC demonstrates superiority in practical RAG retrieval.Source: §3.2 · p. 9

3.2.1 Case A - Find Specific information in the Tesla manual

  • In the Tesla manual case, ChatDOC preserves table structure in Markdown, while the Baseline merges the target table with the one above, leaving the chunk unintelligible to the LLM.The Baseline’s answer says the information is not explicitly mentioned; both systems located the table, but supplied different text to the LLM.Source: Figure 9 · p. 10, Figure 10 · p. 10, §3.2.1 · pp. 11–12

3.2.2 Case B - Research paper

  • For the research-paper query, ChatDOC retrieves the complete Table 8, including its title and content, enabling an accurate answer about its dependent variables.The answer identifies estimates of returns to cognitive and noncognitive ability and factor loadings of unobserved heterogeneity in high school and college.Source: §3.2 · p. 11, Figure 11 · p. 11, §3.2.2 · p. 12
  • The Baseline misses the true Table 8 because it is embedded in a large mixed-content chunk whose low similarity score keeps it out of retrieval results.The comparison is illustrated alongside ChatDOC’s result in Figures 11 and 12.Source: Figure 11 · p. 11, Figure 12 · p. 12, §3.2.2 · p. 13

3.3 Discussion on Limitations

ChatDOC has retrieval-quality failures in some cases, described through two patterns: ranking and token limits, and fine segmentation.

3.3 Discussion on Limitations

  • ChatDOC’s retrieval quality is not better than the baseline in every instance.The discussion identifies two patterns among these cases.Source: §3.3 · p. 14

Ranking and Token Limit Issue

  • Large irrelevant tables can consume ChatDOC’s context window when the embedding model ranks them first, blocking access to relevant information.The authors suggest a better embedding model or retaining only relevant parts of large tables or paragraphs as possible remedies.Source: §3.3 · p. 14

Fine Segmentation Drawback

  • Separating a table title from its table can cause retrieval to return the title and footnotes without the table’s key content.The authors identify improved table-title recognition as a possible remedy.Source: §3.3 · p. 14

4 Applications in ChatDOC

ChatDOC is presented as an AI file-reading assistant with features for exploring documents and finding information. The section lists product capabilities and notes that the PDF Parser API is not yet published.

4 Applications in ChatDOC

  • ChatDOC is presented as an AI file-reading assistant for summarizing long documents, explaining complex concepts, and finding key information.The section describes it as reliable and accurate, and claims it is top among ChatPDF products.Source: §4 · p. 14
  • Users can select tables or text for closer inspection and converse across multiple files without a page-count constraint.Source: §4 · p. 14
  • ChatDOC says its answers have direct-quote citations and that it supports scanned files, ePub, HTML, and docx.Source: §4 · p. 14
  • The ChatDOC PDF Parser API is still being prepared for publication, with a wait list available.Source: §4 · p. 14

5 Conclusion

The conclusion argues that PDF parsers that extract and integrate structured information can improve the relevance of LLM inputs and output quality. It calls for broader comparisons of deep-learning parsing methods.

5 Conclusion

  • A PDF parser that extracts and integrates structured document information can improve the quality and relevance of LLM inputs and their outputs.Source: §5 · p. 14
  • The authors plan to compare more deep-learning document parsers to better understand the relationship between parsing quality and RAG quality.Source: §5 · p. 14
  • The authors report initial experiments suggesting some open-source PDF parsers do not meet the bar for high-quality RAG.Source: §5 · p. 14

A More Cases on PDF Parsing & Chunking

The Case 2 figures contrast PyPDF’s fragmented representation of a large borderless table spanning two pages with ChatDOC Parser’s structure-preserving chunk. ChatDOC keeps the table and its title together in a self-contained chunk.

A More Cases on PDF Parsing & Chunking

Loading 2401.12599v1…