PaperlayerView exact revision on arXiv

Source-linked AI summary

WebSRC: A Dataset for Web-Based Structural Reading Comprehension

Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, Kai Yu

arXiv:2101.09465v2cs.CL

TL;DR

Web-page question answering is difficult because layout and structure matter alongside text, while web pages vary substantially. WebSRC introduces a dataset and evaluates baselines, finding that layout features improve text-based systems and that current pretrained models still find the task challenging.

  • Problem

    Web-page QA remains difficult because diverse layouts require systems to understand both text and page structure, which plain extracted text can lose.

    Source: §1 · p. 1

  • Method

    WebSRC defines web-based structural reading comprehension and provides 400K question-answer pairs from 6.4K pages with HTML, screenshots, and metadata.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Layout features improve text-only WebSRC baselines, and explicit visual features outperform H-PLM in almost all metrics, while pretrained language models remain challenged by the task.

    Source: §6.4 · p. 8

  • Takeaways & Limitations

    The authors conclude that incorporating layout features with textual content is crucial to web understanding, while effective use of structural information needs further investigation.

    Source: §7 · p. 9

    The dataset uses selected page segments rather than full pages and leaves locating the relevant segment for future work.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Web search is an essential way for humans to obtain information, but it's still a great challenge for machines to understand the contents of web pages. In this paper, we introduce the task of structural reading comprehension (SRC) on web. Given a web page and a question about it, the task is to find the answer from the web page. This task requires a system not only to understand the semantics of texts but also the structure of the web page. Moreover, we proposed WebSRC, a novel Web-based Structural Reading Comprehension dataset. WebSRC consists of 400K question-answer pairs, which are collected from 6.4K web pages. Along with the QA pairs, corresponding HTML source code, screenshots, and metadata are also provided in our dataset. Each question in WebSRC requires a certain structural understanding of a web page to answer, and the answer is either a text span on the web page or yes/no. We evaluate various baselines on our dataset to show the difficulty of our task. We also investigate the usefulness of structural information and visual features. Our dataset and baselines have been publicly available at https://x-lance.github.io/WebSRC/.

1 Introduction

WebSRC frames web-page question answering as structural reading comprehension because answering requires understanding both page content and its spatial and logical organization. The paper introduces a dataset pairing questions with HTML, screenshots, and metadata, and reports that the task challenges existing language models.

1 Introduction

  • WebSRC is designed for questions that require understanding page structure as well as text, since web layouts vary and plain extracted text loses layout context.HTML alone can also miss semantic roles that visual organization helps reveal.Source: §1 · pp. 1–2
  • WebSRC contains 400K question-answer pairs from 6.4K web pages, with HTML, screenshots, and metadata supplied for answering text-span or yes/no questions.The paper describes the task as requiring understanding of both spatial and logical page structure.Source: §1 · p. 2
  • The paper introduces structural reading comprehension on the web as a multimodal task involving text and web-page screenshots.The task asks a model to answer questions about a web page using its content and structural information.Source: §1 · p. 2
  • Baseline evaluations find WebSRC highly different from existing textual QA datasets and challenging even for leading pretrained language models.The introduction also notes that the dataset was evaluated with several baselines.Source: §1 · pp. 2–3

2 Related Work

Prior work addresses text-based question answering, web information extraction, and document layout understanding, but the paper distinguishes web SRC through its need to reason over complex web structure and content.

2 Related Work

  • Traditional MRC datasets use plain-text passages, while multimodal MRC and Text VQA combine images and text in different task settings.Text VQA questions concern text in images, whereas web pages also expose text and layout descriptions.Source: §2 · p. 3
  • Web information extraction has used hand-built DOM templates, visual features, and program synthesis, while table studies model physical or logical table structure.The authors note that template- and feature-based approaches require human labor and may generalize poorly to unseen websites.Source: §2 · p. 3
  • Existing web QA datasets use extracted text snippets or HTML for predefined entity-attribute extraction, while document-layout datasets address less complex information organization.The paper contrasts those settings with web pages, whose information and terms are more varied.Source: §2 · p. 3

3 Data Collection

The authors define web SRC over a page context and question, then construct a dataset from selected web-page segments through collection, question annotation, augmentation, and review.

3 Data Collection

  • The collection process comprises page selection, page collection, question labeling, data augmentation, and final review.The five stages are described in the data-collection section.Source: §3 · p. 3

3.1 Task Definition

  • The task takes context C and question q as input and predicts answer a, with context potentially comprising HTML, screenshots, and corresponding metadata.The context-and-question mapping is expressed as F(C, q) = a.Source: §3.1 · p. 3

3.2 Locating text in HTML

  • To locate text in HTML, the authors define a node’s text as the depth-first concatenation of text in its descendant nodes, then locate an answer by its tag and starting position.Figure 2 illustrates the different page types used in the dataset.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Web page selection

  • The selected pages emphasize complex, information-rich structures; the dataset uses key-value, comparison, and table segments, while leaving full-page segment locating for future work.Key-value pages show attributes and values, comparison pages repeat attributes across entities, and table pages use a shared header with values in rows.Source: §3.3 · p. 4

3.4 Web page collection

  • Six computer science students collected page segments with Selenium, saving HTML, screenshots, and metadata; the resulting collection contains 6,447 pages.The saved metadata includes tag positions and sizes, plus text color and font; some oversized comparison objects or table rows were removed.Source: §3.4 · pp. 4–5

3.5 Question Labeling

  • Annotators wrote questions answerable from segment screenshots, labeled answers in HTML, and expanded question wording through rewriting; 2,735 questions remained after review.About 10% of questions were dropped for grammar errors or inconsistent meaning.Source: §3.5 · pp. 5–6

3.6 Data Augmentation

  • 400,498 question-answer pairs were obtained after automatically applying questions to homogeneous web pages and replacing answers with values extracted from each page.When a question named a specific entity, the entity name was also replaced with the corresponding page’s entity.Source: §3.6 · p. 6

3.7 Final review

  • Final review checked label correctness, saved-file completeness, and dataset format, then had four experienced annotators double-check 100 QA pairs per website.Errors were returned to annotators for another labeling round.Source: §3.7 · p. 6

4 Dataset Analysis

The analysis profiles WebSRC by website type, question form, and answer format, including cases where complete answers are distributed across HTML tags.

4.1 Dataset statistics

  • Type KV accounts for about half of websites, comparison sites 17%, and table questions about 40% of QA pairs.The share of table QA pairs exceeds the share of table websites.Source: §4.1 · p. 6, Table 2 · p. 6

4.2 QAs in WebSRC

  • “What is the” questions account for 29.3% of the dataset, and the average question length is 8.26 words.WebSRC includes both wh-questions and yes-no questions.Source: §4.2 · p. 6
  • 86.78% of answers are within 3 words and 55.21% are one word, yet 2.35% span multiple tags and 13.21% are subspans of HTML-node text.A visually whole answer can be split across tags, while a tag can contain text beyond the answer.Source: §4.2 · p. 6

5 Baseline Models

The paper presents three baselines that progressively use flattened text, HTML markup, and visual page information.

5.1 Pre-trained Language Model with Text (T-PLM)

  • T-PLM removes HTML tags and uses a pre-trained language model to predict answer spans from the concatenated plain-text context and question.It appends “yes” and “no” for yes-no prediction, then maps predicted spans back to the tightest containing HTML tag; token start/end distributions use cross-entropy.Source: §5.1 · pp. 6–7

5.2 Pre-trained Language Model with HTML (H-PLM)

  • H-PLM keeps HTML tag names and brackets as special tokens in the context, removing tag attributes and randomly initializing the tag embeddings.Its architecture otherwise matches T-PLM.Source: §5.2 · p. 7

5.3 Visual Information Enhanced Pre-trained Language Model (V-PLM)

  • V-PLM adds screenshot-derived visual embeddings for HTML tags to H-PLM representations, then applies repeated self-attention to predict start and end distributions.Tag bounding boxes locate visual features extracted from screenshots with Faster R-CNN; tokens within a tag share an embedding.Source: §5.3 · p. 7

6 Experiments

Results show that adding HTML and visual context improves baseline performance, while WebSRC remains difficult for pretrained language models. The experiments also test dataset splits, evaluation metrics, and model behavior across website types and case studies.

6.1 Dataset Splits

  • The 50 training websites and 10 websites each for development and test are split at the website level, with similar website-type distributions in development and test.Both held-out sets include all three website types.Source: §6.1 · p. 7

6.2 Evaluate Metrics

  • EM checks exact answer matches, F1 measures token overlap, and POS evaluates overlap between the DOM paths of predicted and gold answers.POS addresses cases where the answer text is correct but comes from the wrong tag.Source: §6.2 · p. 7

6.3 Experiment Setup

  • The baselines use uncased BERT-Base or ELECTRA-Large, with the best model selected on development EM; training uses a 1e-5 learning rate and batch size 32.V-PLM uses three self-attention blocks.Source: §6.3 · pp. 7–8

6.4 Results & Discussion

  • Across backbones, adding HTML tags and screenshots to text improves performance, and ELECTRA-based models consistently outperform BERT-based models.The authors suggest HTML tags implicitly model some visual structure, while explicit visual features add structural information.Source: Table 4 · p. 8, §6.4 · p. 8
  • ELECTRA achieves about 80 EM on SQuAD2.0 but about 60 EM on WebSRC, which the authors say remains challenging for current pretrained language models.The comparison concerns the same model's performance on a textual QA benchmark and WebSRC.Source: §6.4 · p. 8
  • On table and comparison websites, structural baselines outperform plain-text T-PLM on several examples; all three baselines perform worst on table pages.The cited examples are game10 and sport09 (table) and auto08 (comparison).Source: §6.4 · p. 8, Figure 4 · p. 8
  • The SQuAD model scores 29.68 EM on WebSRC without fine-tuning; fine-tuned models almost always outperform it, but remain far below their textual-QA performance.The authors interpret this gap as evidence that HTML text differs from ordinary textual passages and that more advanced HTML-structure modeling is needed.Source: Table 5 · p. 8, §6.4 · pp. 8–9

6.5 Case Study

  • In the university case, all baselines misidentify tuition despite H-PLM and V-PLM locating Net Price; in the movie case, only V-PLM identifies runtime.The authors use these cases to illustrate the need to understand visual layout and group information according to spatial structure.Source: §6.5 · p. 9

7 Conclusion

The paper introduces WebSRC, a multimodal dataset for web-based structural reading comprehension using HTML documents and screenshots. Baseline results show the importance of layout features alongside text, while effective use of structural information remains an open challenge.

7 Conclusion

  • WebSRC is a multimodal dataset for answering questions about web pages using HTML documents and screenshots.The paper evaluates several baselines on the dataset.Source: §7 · p. 9
  • Baseline results indicate that incorporating layout features with text is crucial to web understanding.The paper leaves how to use structural information effectively for further investigation.Source: §7 · p. 9
  • The authors hope the work will advance web-based structural reading comprehension and identify structural reading comprehension beyond web pages as future work.Source: §7 · p. 9

A.1 Dataset distribution

The dataset spans website types and domains unevenly. KV sites appear across nearly all domains, while comparison and table sites are concentrated in particular domains.

A.1 Dataset distribution

  • Not all domains contain all three website types, while KV websites appear in almost every domain.Figure 6 presents the distribution of website types across domains.Source: §A.1 · p. 13, Figure 6 · p. 13
  • Comparison websites are concentrated in goods domains such as auto and book, while most table websites belong to sports.The paper describes comparison pages as item comparisons and sports tables as containing player score data.Source: §A.1 · p. 13
  • Auto, university, and sports account for more than half of the data; hotel, camera, and restaurant have the least.The authors attribute the distribution to the amount of information on websites and the information in each domain that interests people.Source: §A.1 · p. 13

A.2 HTML statistics

WebSRC pages average 177 HTML tags, and tag type and DOM depth provide structural clues about where answers appear.

A.2 HTML statistics

  • <td> is much more likely than <div> to contain an answer, although <div>, <td>, and <span> are the most common tags overall.<div> often frames a page, while <td> commonly presents a value.Source: §A.2 · p. 13, Figure 8 · p. 13
  • WebSRC pages contain an average of 177 HTML tags.Source: §A.2 · p. 13
  • The mean DOM-tree depth is 9.8, compared with 7.1 for tags containing answers.The authors interpret this as upper nodes providing more structural information and lower nodes more specific information.Source: §A.2 · p. 13
Loading 2101.09465v2…