Source-linked AI summary
A Survey on Truth Discovery
Yaliang Li, Jing Gao, Chuishi Meng, Qi Li, Lu Su, Bo Zhao, Wei Fan, Jiawei Han
TL;DR
Truth discovery addresses conflicts in noisy multi-source information, where source reliability is unknown. This survey organizes and compares methods across application-relevant aspects, reviews their principle and uses, and identifies future challenges. It offers guidelines for selecting approaches while highlighting limits involving object features, efficiency, and evaluation without ground truth.
Problem
Conflicting multi-source information requires identifying trustworthy values when source reliability is unknown and labeled truths are difficult to obtain.
Method
The survey reviews truth discovery methods, their general principle, five component aspects, applications, and future research directions.
Results
The survey compares approaches with different assumptions in tables and provides guidelines for choosing truth discovery methods for particular tasks.
Takeaways & Limitations
Truth discovery supports multi-source information integration across domains, including knowledge graph construction and web-page quality assessment.
Takeaways & Limitations
Existing methods need more work on advanced object features, general-case efficiency, and evaluation when ground-truth information is missing.
Abstract
from arXiv · showhide
Thanks to information explosion, data for the objects of interest can be collected from increasingly more sources. However, for the same object, there usually exist conflicts among the collected multi-source information. To tackle this challenge, truth discovery, which integrates multi-source noisy information by estimating the reliability of each source, has emerged as a hot topic. Several truth discovery methods have been proposed for various scenarios, and they have been successfully applied in diverse application domains. In this survey, we focus on providing a comprehensive overview of truth discovery methods, and summarizing them from different aspects. We also discuss some future directions of truth discovery research. We hope that this survey will promote a better understanding of the current progress on truth discovery, and offer some guidelines on how to apply these approaches in application domains.
1. INTRODUCTION
Truth discovery addresses conflicts in multi-source information by inferring source reliabilities rather than treating all sources equally. The survey reviews these methods, compares them across aspects, and discusses applications and future directions.
- Multi-source data about the same object can conflict because information is collected from varied channels and may contain errors.
- Voting and averaging can fail because they assume all sources are equally reliable, even though source quality varies.
- Truth discovery infers source reliability from the data and uses it to identify trustworthy information without supervision.
- The survey provides an overview of truth discovery methods, compares their components, discusses future directions, and introduces applications.
2. OVERVIEW
Truth discovery resolves conflicts by jointly estimating source reliability and selecting trustworthy values. The survey presents iterative, optimization-based, and probabilistic graphical-model formulations of this principle.
- Truth discovery estimates source reliability degrees and selects trustworthy values from conflicting multi-source information.
- Task Definition: An observation consists of an object, a source, and the value provided by that source.
- Task Definition: The identified truth is the most trustworthy candidate value for each object, while source weight reflects the probability that a source provides trustworthy information.
- General Principle of Truth Discovery: The general principle assigns higher reliability to sources that frequently provide trustworthy information and favors values supported by reliable sources.
- Iterative Methods: Iterative methods alternate truth computation and source-weight estimation until convergence, using weighted aggregation to favor high-weight sources.
- Optimization-Based Methods: Optimization formulations minimize weighted distances between source-provided information and identified truths, lowering weights for sources far from aggregated results.
- Probabilistic Graphical Model Based Methods: Probabilistic graphical models generate claimed values from truths and source weights; inference can use EM, with hyperparameters incorporating prior knowledge.
- Comparison: Iterative approaches are easier to interpret, while optimization and graphical-model approaches require more explanation; prior knowledge can be encoded as constraints or hyperparameters.
3. ASPECTS OF TRUTH DISCOVERY
Truth discovery methods are diverse because applications differ in their input data, source and object relations, assumptions, and desired outputs. The survey organizes these methods into general aspects for comparison.
- Truth discovery approaches make different assumptions about input data, relations among sources and objects, and identified truths.
- The survey summarizes truth discovery techniques across aspects that reflect diverse application characteristics.
3.1 Input Data
Truth discovery preprocessing must account for input structure, repeated observations, formatting, uncertainty, data organization, streaming arrival, and available labels. These choices can affect reliability estimation and efficiency.
- Objects without conflict: Preprocessing may remove records where all sources claim the same value, but those records still affect source-reliability estimation.
- Formatting: Claimed values from different formats should be normalized when they represent the same information, such as feet versus meters or alternate name orderings.
- Uncertainty: Truth discovery can incorporate multidimensional uncertainty when observations come from textual extraction or include confidence indicators.
- Input Data: Unstructured inputs provide corpus evidence, URLs, confidence, and question text for reliability estimation, but also introduce more noise and uncertainty.
- Streaming Data: Most existing methods are batch algorithms for static data, motivating specialized approaches for streaming data that arrive over time.
- Supervision: Most methods are unsupervised, although some semi-supervised approaches use a small set of labeled truths to guide source-reliability estimation.
3.2 Source Reliability
Source reliability estimation is central to truth discovery, but methods vary in their assumptions, granularity, meaning, and treatment of source selection and cost.
- Most methods assume a source provides true information with the same probability across all objects.
- Source independence assumes sources observe independently rather than copying, while correlation-aware methods detect copying and adjust source weights.
- Reliability can be estimated at different granularities, including object clusters or separate attributes within a website.
- Some methods enrich reliability beyond truth assertion probability by modeling recall, silent behavior, confidence intervals, or subjective truths.
- Source selection methods account for economic and computational costs because adding bad sources can hurt performance.
3.3 Object
Truth discovery can account for object difficulty and dependencies among objects instead of treating every object as independent.
- 3-Estimates jointly models truth, source reliability, and each object’s difficulty of obtaining the truth.Errors are distributed between source reliability and object difficulty.
- Object relations such as birth year–age or father–son relationships can provide prior knowledge that improves truth discovery results.
- Propositional constraints represent related facts as [0, 1] variables and correct truth probabilities during each iteration.An augmented unknown answer relaxes potentially conflicting constraints.
3.4 Claimed value
Truth discovery methods must account for assumptions about claimed values, including exclusivity, implication, data type, and hierarchical relationships.
- The complementary vote technique assumes one and only one true value and treats a provided value as voting against alternatives.
- Implication models correlations among values for one object, so trust in one continuous value can imply differing trust in nearby values.
- Different data types require different notions of distance, and methods may support categorical, continuous, or string values differently.
- A value’s data type depends on the application context, as the same symbol can represent a number, category, or string.
- Hierarchical value spaces represent related values such as Honolulu, Hawaii, and the USA without treating them as mutually exclusive.
3.5 Output
Truth discovery outputs involve choices about truth assumptions, representation, evaluation, and explanation, with labeling and scoring offering different trade-offs.
- Most methods assume one truth per object, while an unknown output can preserve feasibility when relational constraints conflict.
- Outputs are typically labels or scores: labels are ready to use, whereas scores retain probabilities and reduce information loss.
- Evaluation uses accuracy or error rate for categorical data and MAE or RMSE for continuous data when ground truths exist.Usefulness, memory cost, and running time are additional evaluation considerations.
- Output interpretation matters to users and developers, but many methods produce opaque reliability or support values.Some models express reliability through probabilities with more interpretable meanings.
4. TRUTH DISCOVERY METHODS
The survey compares representative truth discovery methods across data types, source reliability models, object and value assumptions, and output settings. It provides comparison guidelines for selecting methods in specific application scenarios.
- Method comparison: Methods differ in supported input types, reliability assumptions, object modeling, claimed-value treatment, and output scenarios; the survey compares these features to guide selection.
- Representative methods: TruthFinder iteratively estimates source reliabilities and identifies truths using Bayesian analysis, source consistency, and implication.
- Representative methods: AccuSim models similarity among claimed values with an implication function, while AccuCopy additionally reduces copier-source weights.
- Object and value modeling: 3-Estimates extends 2-Estimates by modeling the difficulty of obtaining the truth for each object.
- Representative methods: Investment redistributes source reliability among claimed values and updates confidence through invested reliability and recovered credits.
- Representative methods: SSTF uses labeled truths, mutual exclusivity, and mutual support, whereas LTM models false-positive and false-negative errors separately.
5. FUTURE DIRECTIONS
Future truth discovery research must address limitations in evaluation, data structure, object modeling, initialization, model selection, theory, and efficiency. These challenges become especially important as applications scale and lack labeled ground truth.
- Unstructured data: Unstructured inputs introduce noise and uncertainty from corpora and extractors, while object difficulty can add a third factor to reliability estimation.
- Object relations: Most methods assume independent objects, although applications increasingly require modeling relations that may be difficult to discover automatically at scale.
- Source reliability initialization: Uniform reliability initialization can reproduce voting or averaging at the start and yield different source weights when ties are broken differently.
- Model selection: Selecting among many methods remains difficult, and combining their outputs is proposed as a possible ensemble solution.
- Theoretical analysis: Open theoretical questions include convergence, convergence rates, error bounds, and equivalence among truth discovery approaches.
- Efficiency: Iterative estimation can be inefficient on large-scale datasets, motivating further work beyond existing streaming-data efforts.
- Performance evaluation: Evaluating methods without ground truth is difficult because large applications make labeled validation expensive or impossible.
6. APPLICATIONS
Truth discovery has been applied across healthcare, crowd and social sensing, crowdsourcing, information extraction, knowledge bases, and other big-data settings. These applications use source reliability estimation to identify trustworthy information from conflicting or noisy inputs.
- Healthcare: Online health communities use truth discovery to identify reliable users and trustworthy medical statements from drug reviews.
- Crowd/social sensing: Crowd and social sensing applies truth discovery to improve aggregation of unreliable user observations about physical-world events.
- Crowdsourcing aggregation: Crowdsourcing aggregation infers true labels or answers from workers whose quality varies across labeling efforts.
- Information extraction: Information extraction uses truth discovery to resolve conflicts among outputs from multiple corpora and extractors.
- Knowledge bases: Knowledge-base construction uses truth discovery to identify trustworthy Internet information, and estimated source reliabilities can assess webpage quality.
- Other applications: The survey also identifies potential uses in traffic control, social-media news distillation, and automated homework grading.
7. RELATED AREAS
Truth discovery overlaps with several aggregation and fusion topics, but differs from them in problem settings. Related areas include multi-view learning or clustering, rank aggregation, sensor data fusion, meta-analysis, and ensemble learning.
- Related research topics include multi-view learning or clustering, rank aggregation, sensor data fusion, meta-analysis, and ensemble learning.
- These areas are relevant to truth discovery but use different problem settings, including inputs organized as multiple feature views in multi-view learning or clustering.
8. CONCLUSIONS
The survey synthesizes truth discovery methods and compares their assumptions to guide method selection, while identifying unresolved challenges for real-world deployment and validation.
- The survey examines truth discovery components under five general aspects and compares approaches by input data, constraints, and output assumptions.
- Tables 2 and 3 provide guidelines for choosing a truth discovery approach for a particular task.
- Relations among objects remain an important future direction, with potential benefits for applications such as knowledge graph construction.
- Efficiency is a bottleneck for deploying truth discovery on large-scale data.
- Evaluating performance and validating identified truths remain challenging because limited ground truth is available.