PaperlayerView exact revision on arXiv

Source-linked AI summary

PDFTriage: Question Answering over Long, Structured Documents

Jon Saad-Falcon, Joe Barrow, Alexa Siu, Ani Nenkova, David Seunghyun Yoon, Ryan A. Rossi, Franck Dernoncourt

arXiv:2309.08872v2cs.CLcs.AIcs.LG

TL;DR

Plain-text retrieval can miss the structure users reference when asking questions about PDFs and other structured documents. PDFTriage combines structural metadata with model-callable retrieval functions, and the paper reports superior performance to existing approaches. Its described scope is born-digital PDFs.

  • Problem

    Plain-text representations can be incongruous with structured documents, while users may ask questions that refer to pages, tables, or other document structure.

    Source: §1 · p. 1

  • Method

    PDFTriage gives models structural metadata and retrieval functions so they can fetch context based on document structure or content.

    Source: §1 · p. 1

  • Results

    PDFTriage offers superior performance to existing approaches and is effective across various document lengths and retrieval contexts.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The authors provide a tagged question-type dataset to facilitate further research on question answering over structured documents.

    Source: §1 · p. 2

    The approach treats born-digital PDF documents as the structured documents users will interact with.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Large Language Models (LLMs) have issues with document question answering (QA) in situations where the document is unable to fit in the small context length of an LLM. To overcome this issue, most existing works focus on retrieving the relevant context from the document, representing them as plain text. However, documents such as PDFs, web pages, and presentations are naturally structured with different pages, tables, sections, and so on. Representing such structured documents as plain text is incongruous with the user's mental model of these documents with rich structure. When a system has to query the document for context, this incongruity is brought to the fore, and seemingly trivial questions can trip up the QA system. To bridge this fundamental gap in handling structured documents, we propose an approach called PDFTriage that enables models to retrieve the context based on either structure or content. Our experiments demonstrate the effectiveness of the proposed PDFTriage-augmented models across several classes of questions where existing retrieval-augmented LLMs fail. To facilitate further research on this fundamental problem, we release our benchmark dataset consisting of 900+ human-generated questions over 80 structured documents from 10 different categories of question types for document QA. Our code and datasets will be released soon on Github.

1 Introduction

Plain-text retrieval can mismatch the structure of PDFs and other documents, leaving page- or table-referenced questions without the structural context needed to answer them. PDFTriage retrieves context by structure or content, and the paper also contributes a dataset of tagged question types.

1 Introduction

  • Questions referencing page ranges or tables need document structure to identify salient context; plain-text representations discard that structure.The introduction contrasts explicit page references with implicit table references.Source: §1 · p. 1
  • PDFTriage lets models retrieve context by document structure or content, using structural metadata and retrieval functions to answer questions that plain-text retrieval may miss.The paper identifies treating structured documents as plain text as a gap and says its approach reliably answers several question classes that plain retrieval-augmented LLMs could not.Source: §1 · pp. 1–2
  • The authors contribute a dataset of tagged question types and a prompting method intended to improve LLM answers over structured documents.They also state that they will release questions, documents, model answers, annotator preferences, code, and prompts.Source: §1 · p. 2

2 Related Works

Prior work covers tool- and retrieval-augmented LLMs, while document QA datasets vary in their document sources and question formats. PDFTriage’s evaluation dataset aims to include questions grounded in document structure as well as content.

2.1 Tool and Retrieval Augmented LLMs

  • Tool-augmented LLM work includes API-using prompting and tool-learning approaches, while retrieval-augmented models retrieve related documents from external knowledge sources.Examples include ReAct, Self-ask, Toolformer, TALM, HyDE, and InteR.Source: §2.1 · p. 2

2.2 Question Answering

  • DocVQA and DUDE use document scans or PDFs and short-form answers, while QASPER targets information-seeking questions about research-paper content.PDFTriage’s evaluation dataset seeks questions that reference structure or content, can be extractive or abstractive, and can require long-form answers or rewrites.Source: §2.2 · p. 3

3 PDFTriage: Structured Retrieval from Document Metadata

PDFTriage represents PDF structure as metadata, lets an LLM select and retrieve relevant document content, and then generates an answer from the retrieved context. The described setting uses born-digital PDFs.

3 PDFTriage: Structured Retrieval from Document Metadata

  • PDFTriage answers a question through three stages: generate document metadata, use LLM-based triage to select precise content, and answer from the retrieved content.The overview describes retrieval from pages, sections, figures, or tables.Source: §3 · p. 3

3.1 Document Representation

  • The representation starts from born-digital PDFs converted into an HTML-like tree, from which sections, titles, page information, tables, figures, and nearby text can be extracted.The structured information is mapped to JSON for the LLM prompt, and content is converted to Markdown.Source: §3.1 · p. 4

3.2 LLM Querying of Document

  • Five functions—fetch_pages, fetch_sections, fetch_table, fetch_figure, and retrieve—gather precise document information and can be used across separate queries to synthesize an answer.The functions are called in separate chat turns through the function-calling API.Source: §3.2 · p. 4

3.3 Question Answering

  • For each question, PDFTriage calls one function per turn to gather needed information, then uses the retrieved context to produce the final answer with gpt-35-turbo-0613.The prompt includes document metadata and the user's query.Source: §3.3 · p. 4

4 Dataset Construction

The authors constructed a human-written document QA dataset spanning 82 documents and 908 questions. Its ten-category taxonomy covers document content, structure, and multi-step tasks.

4 Dataset Construction

  • The authors sampled 1,000 Common Crawl documents and subsampled 100 based on reading level for question collection.Questions were collected using Mechanical Turk.Source: §4 · p. 5
  • The broad set of document-oriented questions was intended to provide tasks across industries for testing PDFTriage.The tasks evaluate reasoning across text, tables, and figures, from single-page questions to multi-step reasoning across a document.Source: §4 · pp. 4–5
  • Questions were collected from a stratified sample across ten categories after annotators read documents and wrote salient questions in assigned categories.The categories include figure, text, table reasoning, structure, summarization, extraction, rewrite, outside, cross-page, and classification questions.Source: §4 · p. 5
  • The taxonomy also includes summarization, extraction, rewrite, outside, cross-page, and classification tasks.Cross-page tasks require multiple parts of a document; outside questions cannot be answered using only the document.Source: §4 · p. 5
  • 908 questions across 82 documents form the dataset, with an average of 4,257 text tokens per document.The text is connected to headers, subheaders, section paragraphs, captions, and more.Source: §4 · p. 5

5 Experiments

The experiments compare PDFTriage with page- and chunk-based retrieval using GPT-3.5. Human annotators rank and score the systems’ answers across question-document pairs.

5 Experiments

  • The experiments describe the models and strategies used, and the authors state that code and datasets will be released on GitHub.The supplied passage does not specify the models or strategies in further detail.Source: §5 · p. 5

5.1 PDFTriage

  • The primary experiment used PDFTriage with GPT-3.5 to answer questions in the PDF dataset.The approach leverages PDF structure and interactive system functions.Source: §5.1 · p. 5

5.2 Retrieval Baselines

  • Page Retrieval indexes document pages with text-embedding-ada-002 and retrieves pages by cosine similarity until the model context window is reached.Retrieved page text is supplied as answering context.Source: §5.2 · p. 5
  • PDFTriage answers received 50.8% of users’ top-ranked answers overall and beat both retrieval baselines in head-to-head rankings across question categories.It ranked higher on structure and table reasoning tasks, but lower on classification and text tasks.Source: Figure 3 · p. 6
  • Chunk Retrieval concatenates document text, divides it into 100-word pieces, and retrieves similar chunks by cosine similarity until the context window is reached.It uses text-embedding-ada-002 to index the chunks.Source: §5.2 · p. 6

5.3 Human Evaluation

  • Twelve experienced English-speaking annotators judged answers generated by the systems.The study asked about question attributes and answer rankings, accuracy, informativeness, readability/understandability, and clarity.Source: §5.3 · pp. 6–7

6 Results and Analysis

Human annotators favored PDFTriage over retrieval-based alternatives, while answer-quality scores were highest for overall quality and accuracy. Retrieved context averaged 1,568 tokens, and performance was reported to hold across document lengths.

6 Results and Analysis

  • The annotated sample’s largest difficulty group was Easy at 43.3%, while roughly one-third of questions were categorized as Hard.The annotation framework covered question types and difficulties across text, tables, figures, headings, and single- and multi-page queries.Source: §6 · p. 7

6.1 PDFTriage yields better answers than retrieval-based approaches.

  • 50.7% of annotators favored PDFTriage answers, which ranked higher than the alternatives across all question types.The comparison was against Page Retrieval and Chunk Retrieval.Source: §6.1 · p. 7

6.2 PDFTriage improves answer quality, accuracy, readability, and informativeness

  • PDFTriage scored higher than both retrieval baselines on all answer qualities except clarity, with the highest scores for Overall Quality and Answer Accuracy.Average annotator agreement was Cohen’s kappa 0.584.Source: §6.2 · p. 7, Table 3 · p. 7
  • PDFTriage was stronger on summarization, table reasoning, extraction, and figure questions, and performed similarly to the baselines on text questions and classification.The stronger categories require multi-step reasoning across parts of a document.Source: §6.2 · p. 7

6.3 PDFTriage requires fewer retrieved tokens to produce better answers

  • PDFTriage retrieved an average of 1,568 tokens, compared with 3,611 for Page Retrieval and 3,934 for Chunk Retrieval.The retrieved PDFTriage sections were non-consecutive; the authors report that baseline retrieval sections were often insufficient for answering questions.Source: §6.3 · p. 7

6.4 PDFTriage performs consistently across document lengths

  • The authors report that document length had no effect on overall performance and that querying specific sections supports connecting disparate parts while minimizing irrelevant context.This enables multi-page questions involving table reasoning, cross-page tasks, figures, and document structure while reducing computational and financial resources for document QA.Source: §6.4 · p. 8

7 Future Work & Conclusions

The authors conclude that PDFTriage offers superior performance to existing QA approaches across varied document lengths and retrieval contexts. They identify multimodal inputs and question-type conditioning as future directions.

7 Future Work & Conclusions

  • The authors conclude that PDFTriage offers superior performance to existing approaches and is effective across varied document lengths and retrieval contexts.They present it as a technique specialized for document-oriented question answering.Source: §7 · p. 8
  • Future work includes incorporating table and figure information into GPT-4 document question answering.Source: §7 · p. 8
  • The authors propose incorporating question type into PDFTriage to improve efficiency and efficacy.Source: §7 · p. 8

A.1 Question Categories and Examples

  • The appendix describes question categories as document QA tasks relevant across professional fields.It presents descriptions and positive and negative examples for each category.Source: §A.1 · p. 11

A.2 Annotator Demographic Information

The annotator-demographics material describes recruitment and payment and lists participant locations; the appendix also labels positive and negative examples for question categories.

A.2 Annotator Demographic Information

  • 12 English-speaking annotators were recruited to judge answers generated by PDFTriage and the baseline approaches.Source: §A.2 · p. 11
  • Annotators were paid the same standard rate used for US annotators.Source: §A.2 · p. 11
  • The listed locations are India, Pakistan, South Africa, Australia, the Philippines, and the United States.Source: §A.2 · p. 11
  • Tables 6 and 7 are labeled as positive and negative examples for question categories, respectively.Source: Table 6 · p. 12, Table 7 · p. 13

B.1 Human Evaluation Interface

The interface appendix presents annotation-question figures numbered 1 through 8; the supplied captions identify the questions by number but do not state their contents.

B.1 Human Evaluation Interface

B.2 GPT Evaluation and Discussion

GPT-4 evaluation scores aligned negligibly with human judgments, and retrieved-context length was also negligibly connected to PDFTriage efficacy. The discussion suggests precise, question-specific retrieval matters more than longer context.

B.2 GPT Evaluation and Discussion

  • Cohen’s kappa of 0.067 and Pearson’s correlation coefficient of 0.19 indicate negligible alignment between GPT-4 evaluation scores and human Overall Quality annotations.Source: §B.2 · p. 16
  • The authors say automated GPT-4 evaluation needs further instructions or fine-tuning to better align with human preferences in document QA.Source: §B.2 · p. 16

B.3 Performance vs. Context Window Trade-off

  • A Pearson’s correlation coefficient of 0.062 indicates a negligible connection between retrieved-context length and PDFTriage’s overall efficacy.Source: §B.3 · p. 16
  • The authors argue that PDFTriage needs precise information, particularly for cross-page and structure questions requiring multiple query stages, rather than longer context.They note that full-document concatenation would not necessarily improve performance because additional text did not correlate with improved accuracy or overall quality.Source: §B.3 · p. 16

B.4 Evaluation Breakdown by Question Category

The figures report annotation scores by question category for accuracy, overall quality, informativeness, readability, and clarity; the supplied captions provide no score values or category comparisons.

B.4 Evaluation Breakdown by Question Category

  • Figure 13 presents accuracy annotation scores by question category.Source: Figure 13 · p. 16
  • Figure 14 presents overall quality annotation scores by question category.Source: Figure 14 · p. 16
  • Figure 15 presents informativeness annotation scores by question category.Source: Figure 15 · p. 17
  • Figure 17 presents readability annotation scores by question category.Source: Figure 17 · p. 17
  • Figure 16 presents clarity annotation scores by question category.Source: Figure 16 · p. 17
Loading 2309.08872v2…