PaperlayerView exact revision on arXiv

Source-linked AI summary

Web Data Extraction, Applications and Techniques: A Survey

Emilio Ferrara, Pasquale De Meo, Giacomo Fiumara, Robert Baumgartner

arXiv:1207.0246v4cs.IR

TL;DR

Web Data Extraction must handle diverse applications and large, continuously changing online information, while manual maintenance becomes unsuitable for large collections of frequently updated pages. This survey organizes extraction techniques and applications, finding enterprise and Social Web uses alongside opportunities for cross-fertilization and large-scale data analysis.

  • Problem

    Web Data Extraction spans ad-hoc applications and large, continuously produced online information, while manually maintaining wrappers becomes unsuitable for thousands of frequently updated pages.

  • Method

    The survey classifies extraction techniques and systems, then organizes real-world applications into Enterprise and Social Web classes and discusses cross-fertilization.

  • Results

    The survey summarizes 14 enterprise application scenarios and reviews linking independent Web systems and cloud services as application directions.

  • Takeaways & Limitations

    Web Data Extraction supports data analysis across enterprise and Social Web domains, including analysis of human interactions and opportunities to reuse techniques across domains.

  • Takeaways & Limitations

    Social Web extraction must gather data quickly enough to keep pace with rapid changes in users’ social-network structures, or analyses may fail to capture interaction structure and evolution.

Abstract

from arXiv · show

Web Data Extraction is an important problem that has been studied by means of different scientific tools and in a broad range of applications. Many approaches to extracting data from the Web have been designed to solve specific problems and operate in ad-hoc domains. Other approaches, instead, heavily reuse techniques and algorithms developed in the field of Information Extraction. This survey aims at providing a structured and comprehensive overview of the literature in the field of Web Data Extraction. We provided a simple classification framework in which existing Web Data Extraction applications are grouped into two main classes, namely applications at the Enterprise level and at the Social Web level. At the Enterprise level, Web Data Extraction techniques emerge as a key tool to perform data analysis in Business and Competitive Intelligence systems as well as for business process re-engineering. At the Social Web level, Web Data Extraction techniques allow to gather a large amount of structured data continuously generated and disseminated by Web 2.0, Social Media and Online Social Network users and this offers unprecedented opportunities to analyze human behavior at a very large scale. We discuss also the potential of cross-fertilization, i.e., on the possibility of re-using Web Data Extraction techniques originally designed to work in a given domain, in other domains.

1. Introduction

Web Data Extraction systems collect and structure information from Web sources for enterprise, social, scientific, and other applications. This survey organizes the field by techniques and application domains while examining automation, scale, privacy, maintenance, and cross-domain reuse.

  • Web Data Extraction systems collect information from Web sources, optionally post-process it into structured formats, and store it for further use.
  • These systems support document analysis, Business and Competitive Intelligence, Social Web crawling, BioInformatics, and other applications.
  • Key design challenges include balancing automation with accuracy, processing large data volumes quickly, protecting privacy, and operating without extensive labeled training data.
  • Web sources evolve unpredictably, creating a need for extraction systems that can detect structural changes and maintain correct operation over time.
  • The survey classifies extraction techniques, system characteristics, and applications, emphasizing enterprise and Social Web domains and potential cross-fertilization.
  • Its application-oriented perspective differs from surveys centered mainly on tool lists, feature-based classifications, or experimental comparisons.

2. Techniques

This section surveys Web Data Extraction techniques, focusing on HTML's semi-structured tree representation, wrapper induction and maintenance, and tree-matching algorithms. It also highlights the central challenge of extracting reliable structured data as Web sources evolve.

  • Scope and representation: The survey focuses on HTML pages, represented as labeled ordered rooted trees whose nesting structure supports extraction.This scope excludes extraction approaches targeting non-HTML semi-structured sources such as e-mails, software documentation and system logs.
  • Scope and representation: XPath addresses either a single element or multiple occurrences in an HTML document tree, but its expressions remain tied to page structure.Relative paths and wrapper adaptation partially mitigate this lack of flexibility.
  • Tree matching: Simple tree matching runs in O(nodes(A) · nodes(B)) and is efficient for large HTML trees, but it disallows node replacement and level crossing.Despite these limitations, variants have been used in multiple Web Data Extraction scenarios, including restricted top-down matching.
  • Tree matching: Weighted tree matching produces a similarity measure, whereas simple tree matching returns a mapping value; its similarity becomes more accurate for complex, similar trees.In the example, weighted matching returns 3⁄8 (0.375), compared with 7 from simple tree matching.
  • Wrapper maintenance: Wrapper maintenance is critical because manually updating wrappers becomes infeasible for thousands of dynamically changing Web pages.Automatic adaptation searches modified pages for structurally similar candidate elements and matches the most similar one to the original.

3. Web Data Extraction Systems

Web Data Extraction systems access Web sources, extract and transform information, and deliver structured data for downstream use. Their capabilities span interaction, automation, wrapper creation, navigation, extraction, and increasingly layered technological solutions.

  • Web Data Extraction systems implement procedures that extract information from Web sources and may transform and store it for later use.They can process HTML elements or full text and convert results into convenient structured formats.
  • A typical extraction process begins with interaction with Web pages or feeds, including access to static, dynamic, and deeply navigated sources.Advanced systems simulate clicks, fill forms, and support dynamically generated pages.
  • Technological evolution: OxPath extends XPath with user-action steps, dynamic CSS selection, visible-field identification, repeated navigation, and extraction predicates.It is designed for deep Web navigation and extraction from interactive sites.
  • Wrapper lifecycle: Wrapper creation and execution are core system requirements, while wrapper maintenance adapts extraction procedures when source structures change.Wrapper induction automates wrapper definition from examples, but high-level automation remains challenging.
  • Automation and Extraction: Automation features include user-action simulation, form and menu handling, AJAX support, and scheduled periodic extraction.These capabilities reduce manual access and localization work.
  • Data transformation and use: Extracted data can be cleaned, reconciled across sources, transformed into homogeneous structures, and delivered to databases or analytical systems.Targets include XML databases, relational databases, data warehouses, and content-management systems.

4. Applications

The survey classifies Web Data Extraction applications primarily into Enterprise and Social Web domains, while also distinguishing their sources, formats, and purposes. It emphasizes broad application coverage and the combination of heterogeneous data for understanding business and social phenomena.

  • Survey contribution: The survey presents its application classification as an initial attempt to organize techniques originally developed for specific or apparently unrelated domains.This framing motivates examining connections across application areas.
  • The survey’s main application classes are Enterprise and Social Web, selected because both can substantially benefit from Web Data Extraction technologies.Enterprise uses emphasize business efficiency and strategy, while Social Web uses emphasize large-scale analysis of user-generated data.
  • Applications are further classified by single versus multiple sources, homogeneous versus heterogeneous formats, and single versus multiple purposes.These criteria describe how data are collected and combined across platforms and representations.
  • Data characteristics: Enterprise applications may collect numerical, textual, structured, and unstructured information, including data from relational databases and other sources.Business Intelligence applications are presented as a relevant example.
  • Application criteria: Multi-purpose applications link different kinds of data to support more advanced interpretations of business processes or social phenomena.Examples include combining flight and hotel data into travel packages and linking social-platform data.

4.1. Enterprise Applications

Enterprise applications use Web Data Extraction for commercial and business-oriented tasks, including contextual advertising. These systems analyze page content to select thematically relevant advertisements and support user and advertiser interests.

  • Enterprise applications cover software and procedures related to Web Data Extraction with a direct or subsequent commercial scope.
  • Context-aware advertising: Context-aware advertising presents thematically related commercial advertisements alongside the Web content a user is reading.Its stated goal is to increase page value for visitors and interest in the advertisement.
  • Context-aware advertising: Implementing contextual advertising requires analyzing page semantics and extracting relevant structural and content information before contextualizing advertisements.The approach combines page analysis with ad selection.
  • Context-aware advertising: Contextual advertising is described as providing more useful thematized ads and a better income source for advertisers than older Web advertising.

4.1.2. Customer care

Enterprise Web Data Extraction applications also process large collections of unstructured organizational documents and support database-building activities from Web data. These uses target concept extraction and structured reuse of collected information.

  • Customer-care information: Medium- and large-sized companies handle extensive unstructured text, including emails, support discussions, documentation, shipment information, reports, and conversation transcripts.
  • Customer-care information: Analyzing these documents and extracting their main concepts provides concrete advantages for organizational information handling.The supplied passage introduces document classification as one such use.
  • Database Building: Web Data Extraction can support Database Building by reverse-engineering Web data into a DBMS representation and automatically populating it.The survey describes this as applicable across numerous fields.

4.1.4. Software Engineering

Web Data Extraction supports Software Engineering activities, including extracting information for reverse engineering and acquiring market intelligence. These applications use Web sources to support analysis and decision-making.

  • Software Engineering: Web Data Extraction can support Software Engineering by extracting information from Web applications for reverse engineering.Amalfitano et al. developed an approach that abstracts finite-state machines representing client-side behavior.
  • Business and Competitive Intelligence: Market-information acquisition uses Web Data Extraction to gather unstructured and semi-structured information for business analysis.The Lixto Suite supports accessing, extracting, cleaning, and delivering such data.
  • Business and Competitive Intelligence: Competitive Intelligence gathers and analyzes information about products, customers, and competitors to assist managers’ decisions.The activity is closely related to data mining and can use public-domain information from the Web.

4.1.6. Web process integration and channel management

Web Data Extraction integrates semi-structured Web data into enterprise applications and supports Web-based business processes. Its uses include converting sites into services, channel management, and automated Web application testing.

  • Web process integration and channel management: Because much Web data remains in semi-structured HTML, extraction is needed to provide structured, unified access to heterogeneous sources.This includes automatically turning Web applications and sites into Web Services and understanding their application logic.
  • Web process integration and channel management: Web Data Extraction supports business processes conducted through Web portals, including purchasing, sales, quality management, and travel-industry channel management.One example is automating the regular offering of hotel rooms on portals with bidirectional workflows.
  • Functional Web application testing: Web application testing can operate through APIs or GUI-level simulations of user steps and possible application states.The cited testing levels include black-box API testing and GUI testing based on user actions or state models.
  • Functional Web application testing: Testing tools must handle dynamic HTML, robust script maintenance, parametrized execution, meaningful reports, and multiple browser versions.The passage identifies these as requirements for Web application testing tools in Web-oriented architectures.

4.1.8. Comparison shopping

Comparison shopping relies on Web Data Extraction to collect product and service information from Web sites. Mashup platforms extend this model by integrating heterogeneous Web content, including data from the deep Web.

  • Comparison shopping: Comparison-shopping services use Web sites as data sources and internal extraction engines to compare products or services.Comparisons may involve prices, features, technical sheets, or user experiences.
  • Mashup scenarios: Mashups combine multiple Web sites into an integrated view and can use wrappers to expose legacy applications as lightweight REST-like APIs.This allows mashup solutions to extend beyond provider-offered APIs to the whole Web.
  • Mashup scenarios: Web Data Extraction can make deep-Web content accessible by encapsulating complex form queries and application-logic steps in Web Service methods.End users can create their own Web views and embed extracted data into other applications.

4.1.10. Opinion mining

Web Data Extraction supports opinion mining by collecting user-generated reviews and related content, while citation databases apply it to extracting references and building searchable scholarly collections. Maintaining such collections can require repeated extraction as source corpora change.

  • Opinion mining: Opinion sharing commonly appears in blogs containing reviews, comments, tags, polls, and charts about products, experiences, and services.MOpiS generates summaries of commercial-product reviews using both review content and associated information.
  • Opinion mining: Semi-structured Web portals also allow users to publish unmoderated opinions on varied topics.
  • Citation databases: Citation databases extract publications, references, and citations to build structured collections supporting search, comparison, citation counts, and cross-references.CiteSeer, Google Scholar, and DBLP are cited as examples.
  • Citation databases: Rapidly changing publication corpora can make repeated extraction from the same sources excessively time-consuming.An incremental solution is proposed as an attempt to address this maintenance challenge.

4.1.12. Web accessibility

Web Data Extraction supports accessibility by automatically extracting and understanding Web-page data for blind and partially sighted users. Existing screen readers and accessibility standards have difficulty handling Web 2.0 presentation, content, and interactions, while ABBA models pages across multiple semantic axes.

  • Automatic data extraction and document understanding can improve Web accessibility for blind and partially sighted users.
  • Screen readers struggle with Web 2.0 reading order, controls such as date pickers, and navigation between forum posts.
  • ABBA transforms Web pages into a multi-axial semantic model spanning topological, layout, functional, content, genre, and saliency properties.
  • Users can navigate or serialize pages along semantic axes, including visual-saliency ordering of information.

4.1.13. Main content extraction

Main content extraction removes irrelevant Web-page fragments to isolate article content for uses such as offline archiving. The section connects this task with broader Web archiving, including selective replay of interaction paths through rich applications.

  • Main content extraction distinguishes relevant article content from navigation menus, advertisements, and templates.
  • Approaches to main content extraction range from visual Web-page analysis to methods based on text density.
  • Web archiving preserves information no longer available online and records how the Web was used over time.
  • Event-based archiving selectively stores interaction sequences through rich Web applications so users can later restore or replay them.

4.1.15. Summary

The survey summarizes 14 enterprise application scenarios and describes the main ingredients of each scenario’s value chain.

  • 14 enterprise application scenarios are summarized, with each scenario’s main value-chain ingredients described.

4.2. Social Web Applications

Social Web applications use Web Data Extraction to collect large-scale data about users, relationships, activities, and shared resources. The survey covers single- and multi-platform collection, technical constraints, sampling limitations, and privacy risks.

  • Social Web platforms let users create links, share content, and produce data at large scale for studying human behavior.
  • Extracted interaction data supports grouping users around shared interests for selective marketing and identifying influential users.
  • Correlating accounts across platforms helps relate platform design features to user socialization, content production, engagement, and loyalty.
  • Social Web collection techniques address data from single platforms, multiple platforms, relationships, activities, and privacy risks.
  • Single-platform collection uses platform APIs or HTML scraping, with APIs retrieving social connections, user content, and tags.
  • Twitter crawling collected 41.7 million user profiles and 1.47 billion social relations, alongside tweets and semantic topic information.
  • API access and crawling face platform-specific constraints, while breadth-first search can trap crawlers and bias structural estimates.
  • Privacy settings create black holes that reduce crawler coverage by 7% to 9% on average relative to configurations without them.

4.3. Opportunities for cross-fertilization

Web Data Extraction techniques can be reused across application domains, but suitable methods depend on the structural and semantic information available in each domain.

  • Cross-fertilization examines which Web Data Extraction techniques transfer across domains and which require domain-specific information.The discussion distinguishes broadly applicable techniques from those relying on information available only in some application domains.
  • Regular expressions can operate without assumptions about the structure or content of Web-page collections by treating pages as text documents.
  • Wrappers and XPath exploit known HTML structure to locate extracted elements and can work across different application domains.HTML tags provide hierarchical information that supports extraction, while XPath queries locate elements within that structure.
  • A Facebook crawler illustrates cross-fertilization by adapting Lixto techniques originally developed for Business and Competitive Intelligence.The wrapper used XPath queries, with human experts specifying extraction elements during configuration.
  • When Web pages have rich semantic structure, Information Extraction and Natural Language Processing techniques can support extraction in enterprise and Social Web applications.

5. Conclusions

The survey organizes Web Data Extraction research by algorithmic techniques and real-world applications, covering Enterprise and Social Web domains. It also highlights data linking, cloud scalability, Semantic Web conversion, bioinformatics, and Web harvesting as application directions and open challenges.

  • Web Data Extraction addresses unstructured Web data through strategies for automatically extracting structured information across commercial and Social Web applications.
  • It classifies extraction algorithms from basic techniques through variants, then compares systems by perspectives including ease of use and Deep Web extraction.
  • The survey classifies existing Web Data Extraction applications into Enterprise and Social Web classes and discusses cross-fertilization across application domains.
  • Future applications include extracting biomedical relationships from sources such as PubMed and converting Web content into Semantic Web formats such as RDF.
  • Web harvesting remains open because crawling enough heterogeneous sources, integrating relevant information, and determining suitable human involvement are difficult at Web scale.Open questions concern automation, acceptable precision, and where humans should interact; DIADEM addresses automatic wrapper generation for restricted domains.
  • Linking data from independent Web systems can improve recommendations by combining complementary information such as Netflix ratings and IMDb descriptions.The cited study compared ratings-only, tags-only, and combined strategies, with experiments showing improved recommendation accuracy for combined data.
  • Cross-domain recommender systems transfer information such as movie preferences to other domains, using methods including co-clustering, probabilistic models, and shared latent spaces.
  • Cloud architectures distribute computational loads across processing units to improve resource management, reliability, and scalability in Web Data Extraction.Lixto is cited as an early cloud-based platform in which clients receive wrappers and extraction parameters and return retrieved data.
Loading 1207.0246v4…