Source-linked AI summary
Truth Finding on the Deep Web: Is the Problem Solved?
Xian Li, Xin Luna Dong, Kenneth Lyons, Weiyi Meng, Divesh Srivastava
TL;DR
The paper asks how truthful Deep Web data are and how effectively data fusion resolves conflicts among sources. It studies Stock and Flight data, measures inconsistency and source quality, and evaluates state-of-the-art fusion methods. The data contain substantial conflicts, while fusion performs well on average but remains unstable and lacks a consistently superior method.
Problem
The paper addresses limited understanding of Deep Web data truthfulness and whether existing methods can reliably resolve conflicts across sources.
Method
The study analyzes Stock and Flight data and empirically evaluates state-of-the-art data fusion methods for resolving conflicts and finding truth.
Results
Fusion methods find correct values for 96% of data items on average, but show substantial instability and no method is consistently best.
Takeaways & Limitations
Deep Web data can be highly inconsistent even in domains expected to be clean, and current fusion methods show good potential while leaving clear room for improvement.
Takeaways & Limitations
The study assumes one true value per data item, although semantic ambiguity may imply multiple truths.
Abstract
from arXiv · showhide
The amount of useful information available on the Web has been growing at a dramatic pace in recent years and people rely more and more on the Web to fulfill their information needs. In this paper, we study truthfulness of Deep Web data in two domains where we believed data are fairly clean and data quality is important to people's lives: {\em Stock} and {\em Flight}. To our surprise, we observed a large amount of inconsistency on data from different sources and also some sources with quite low accuracy. We further applied on these two data sets state-of-the-art {\em data fusion} methods that aim at resolving conflicts and finding the truth, analyzed their strengths and limitations, and suggested promising research directions. We wish our study can increase awareness of the seriousness of conflicting data on the Web and in turn inspire more research in our community to tackle this problem.
1. INTRODUCTION
The paper examines whether Deep Web data are truthful and whether existing fusion methods can resolve conflicts across sources. In Stock and Flight data, it finds substantial inconsistency and imperfect fusion despite generally strong average performance.
- Web users increasingly rely on rapidly growing online information despite fewer guarantees of quality and credibility than traditional media.
- The study asks whether Deep Web data are consistent, whether majority values and sources are reliable, and whether sources share or copy data.
- 70% of data items have multiple provided values; nearly 50% of conflicts reflect ambiguity, 20% outdated data, and 30% apparent mistakes.
- Authoritative sources are often accurate but imperfect and incomplete, while other sources can have low and unstable quality.
- Fusion methods find correct values for 96% of data items on average, but are unstable and lack a consistently best method.
- The paper empirically compares existing fusion methods and identifies research areas for improving conflict resolution and truth finding.
2. PROBLEMDEFINITIONANDDATASETS
The paper models Deep Web data as attributes and values describing real-world objects, then constructs Stock and Flight collections with normalized records and source-based gold standards. The datasets expose heterogeneous coverage across many sources, while gold-standard generation remains challenging.
- 2.1 Data model: A data item is an attribute of an object, and the model assumes each data item has one true value reflecting the real world.
- 2.1 Data model: Sources differ in schema structure, attribute naming, and object representation, creating heterogeneity that must be resolved before comparison.
- 2.2 Data collections: The Stock collection contains 55 sources covering 1000 stocks sampled from major indexes and Russell 3000, collected daily during July 2011.
- 2.2 Data collections: Stock sources provide 3–71 attributes among 333 observed attributes, reduced to 153 after manual matching; analysis focuses on 16 stable global attributes.
- 2.2 Data collections: Gold standards use voting across selected authoritative financial or airline websites, but generating them is difficult because every source can make mistakes.
- 2.2 Data collections: The Flight collection contains 38 sources covering 1200 flights associated with three airlines and hub airports, collected daily in December 2011.
3. WEB DATA QUALITY
Deep Web data quality is substantially weaker and less consistent than expected, even in Stock and Flight domains considered important and relatively clean. Inconsistency arises from ambiguity, stale data, and errors, while source accuracy and majority-based truth vary considerably across domains and sources.
- Data redundancy: 64% of Stock data items have redundancy above 50%, compared with 29% of Flight data items; average redundancy is 66% and 32%, respectively.Flight redundancy is lower because airline and airport sites cover only related flights.
- Value inconsistency: Stock data items contain 1–13 different values with an average of 3.7, and only 17% have a single value even after bucketing.Inconsistency affects 83% of Stock items, although one value is often dominant.
- Value inconsistency: 64% of numerical Stock data items have deviation within .1, but 14% exceed .5, while Flight actual departure and arrival times average deviations as large as 15 minutes.Different attributes exhibit different inconsistency levels, with scheduled times and gates lowest and actual times highest in Flight.
- Reasons for inconsistency: 56% of Flight inconsistency is attributed to pure errors, while Stock inconsistency includes semantics ambiguity, instance ambiguity, stale data, unit errors, and unexplained errors.The Stock analysis reports 46% semantics ambiguity and 11% unexplained apparent errors; Flight also includes 33% semantics ambiguity and 11% out-of-date data.
- Majority truth: Dominant values achieve precision 0.908 for Stock and 0.864 for Flight, but precision declines as dominance weakens and copying can make wrong Flight values dominant.For Flight, 82% of items have dominance above .5, yet only 88% of those dominant values match the gold standard.
4. DATA FUSION
Data fusion methods resolve conflicting Deep Web values through voting and source-trustworthiness estimation, but their performance varies substantially across Stock and Flight data. The best methods can achieve high precision, yet copying patterns, trustworthiness errors, and added low-quality sources materially affect results.
- Dominant-value voting can fail when the dominant value is supplied by only a small fraction of sources.Its effectiveness depends on a high dominance factor.
- All advanced fusion methods more or less use iterative voting that jointly estimates value votes and source trustworthiness.Each round updates value votes and source trustworthiness until convergence.
- Flight data: Flight fusion favored ACCUCOPY, improving over VOTE by 9%, or about 550 data items, without requiring input trustworthiness.Copy detection helped because Flight attributes are non-numerical, avoiding a potential similar-value problem.
- Cross-domain comparison: Fusion results were highly dataset-dependent: copied low-accuracy sources could bias trustworthiness estimates, while modeling copying significantly improved Flight results.With sampled trustworthiness, all methods improved on Flight, but source-trustworthiness reasoning could otherwise favor copying groups over true-value providers.
- Stock data: Stock fusion favored ACCUFORMATATTR and ACCUSIMATTR in precision, while ACCUCOPY was slow and low-precision despite modeling copying.ACCUFORMATATTR and ACCUSIMATTR finished in 115 and 235 seconds, whereas ACCUCOPY took 855 seconds.
- Summary and comparison: Across the evaluated data, best fusion improved over the highest-recall single source on average, reaching precision .941 for Stock and .987 for Flight.Adding sources beyond the best-performing set reduced recall by 4% for Stock and .4% for Flight on the snapshot.
5. FUTURE RESEARCH DIRECTIONS
The paper identifies several directions for improving data fusion, integration, and evaluation based on observed instability, ambiguity, source-quality variation, and recall deterioration.
- Improving fusion: Precise source trustworthiness could fix nearly half of fusion mistakes, but current iterative estimates may be imprecise.Trustworthiness is promising when copiers do not introduce bias, yet it is often unknown beforehand.
- Improving fusion: Fusion methods are unstable, so combining different models may improve results because no method consistently dominates.The paper reports no universal winner among methods or between basic methods and their proposed improvements.
- Improving integration: Copy detection remains limited for numerical values and Web-scale data because existing methods ignore value similarity and granularity and can be time-consuming.Source copying is promising for fusion, but robust and scalable detection remains open.
- Improving integration: 50% of value conflicts remained attributable to ambiguity despite manual schema- and instance-level heterogeneity resolution.The paper proposes jointly combining schema mapping, record linkage, and data fusion.
- Improving integration: Fusion over all sources can lower recall compared with using a few high-recall sources, motivating automatic source selection.The paper also links quality deterioration to possible instance-deduplication and schema-mapping mistakes.
- Improving evaluation: Evaluation needs better gold standards because trusting selected sources can introduce wrong or coarse-grained values.The paper also calls for uncertainty representation and automated explanations of mistakes and inconsistencies.
6. CONCLUSIONS
This paper studies Deep Web data correctness in two supposedly clean domains and finds substantial inconsistency and low-quality sources. State-of-the-art fusion methods show good potential, but their limitations leave clear room for improvement and motivate future research.
- Contributions: The study is the first to investigate Deep Web data correctness using data from Stock and Flight, domains expected to be fairly clean.The authors were surprised to find high inconsistency and many low-quality sources.
- Findings: State-of-the-art data-fusion methods can resolve value conflicts and find truth with good potential, but substantial improvement remains possible.The paper therefore suggests several directions for future work.