A Survey on Truth Discovery

1. INTRODUCTION

• Multi-source data can conflict, and majority voting or averaging may fail because source reliability varies and is usually unknown beforehand.The Mount Everest example contrasts majority voting’s 29,035 feet with the true value, 29,029 feet, from Wikipedia.Source: §1 · p. 1
• Truth discovery jointly estimates source reliability and finds truths, assigning higher reliability to sources that more often provide true information and favoring information supported by reliable sources.The survey notes that truth discovery methods usually work without supervision and make differing assumptions about input data, source relations, and truths.Source: §1 · p. 1
• The survey reviews truth discovery’s task definition, general principle, components, representative approaches, future directions, applications, and related areas.It presents a comprehensive overview intended to help readers compare approaches and choose one for real-world tasks.Source: §1 · pp. 1–2

2.1 Task Definition

• Truth discovery selects a truth for each object from conflicting source-provided values while estimating source weights that indicate reliability.An observation is an object-source-value tuple; the truth is the most trustworthy candidate value.Source: §2.1 · p. 2

2.2 General Principle of Truth Discovery

• By estimating reliability, truth discovery can break voting ties and select minority values when more reliable sources support them.The overview illustrates these capabilities with a tie for Mahatma Gandhi and a minority value for Barack Obama.Source: §2.2 · p. 2

2.2.1 Iterative methods

• Iterative methods alternate truth computation by weighted aggregation with source-weight estimation based on the currently identified truths.In Investment, sources distribute reliability among claimed values and receive credits back from identified truths; sources that more often provide trustworthy information gain higher weights.Source: §2.2.1 · pp. 2–3

2.2.2 Optimization based methods

• Optimization-based methods minimize weighted distances between source-provided information and truths, jointly estimating both through coordinate descent.The distance can use 0-1 loss for categorical data or L2-norm for continuous data; coordinate descent alternates updates until convergence.Source: §2.2.2 · p. 3

2.2.3 Probabilistic graphical model based methods

• Probabilistic graphical models represent claimed values as generated from truths and source weights, with high weights implying lower variance and claims closer to truth.Expectation Maximization can infer latent truths and weights, while hyperparameters α and β can encode prior knowledge about truth or source-weight distributions.Source: §2.2.3 · pp. 3–4

2.2.4 Comparison and summary

• The survey does not rank the three approaches; iterative methods are easier to interpret, while optimization and PGM methods need more explanation.Prior knowledge can be encoded as constraints in optimization or hyperparameters in PGMs; methods typically initialize weights, alternate truth and weight updates, and stop when changes fall below a threshold.Source: §2.2.4 · p. 4

• Different assumptions about data, source and object relations, and identified truths, together with domain-specific application characteristics, motivate diverse truth discovery techniques.Source: §3 · p. 4
• The survey summarizes truth discovery methods by five aspects: input data, source reliability, object, claimed value, and output.Source: §3 · p. 4

3.1.1 Duplicate input data

• Most methods assume one observation per source-object pair; with timestamps, select the freshest observation, or otherwise use predefined rules to choose one.Source: §3.1.1 · p. 4

3.1.2 Objects without conflict

• Agreement-only records may improve effectiveness when removed, but still affect source-reliability estimates, so remove them cautiously; methods allowing “unknown” outputs are an exception.Source: §3.1.2 · p. 4

3.1.3 Input data format

• Information collected from various sources may appear in different formats; Mount Everest heights listed as “29, 029 feet” and “8848 meters” represent the same value and should be formatted identically.Source: §3.1.3 · p. 4

3.1.4 Input uncertainty

• Text-extracted observations [18] and source-provided confidence indicators require uncertainty modeling; one generalization accounts for multidimensional uncertainty, including extractor uncertainty.Source: §3.1.4 · pp. 4–5

3.1.5 Structured v.s. unstructured data

• Unstructured inputs add useful corpus evidence, URLs, confidence, and question text, but also introduce noise and uncertainty compared with structured-data inputs.Source: §3.1.5 · p. 5

3.1.6 Streaming data

• Because batch methods must rerun when new data arrive, they are inefficient for streaming data; algorithms have been designed for different streaming types [32, 64, 79].Source: §3.1.6 · p. 5

3.1.7 Labeled truths

• Most methods are unsupervised, but semi-supervised approaches use a small set of labeled truths to guide reliability estimation and truth computation, improving performance.Source: §3.1.7 · p. 5

3.2.1 Source consistency assumption

• Most truth discovery methods assume each source has the same probability of providing true information for every object.The survey describes this source consistency assumption as reasonable in many applications and as one of the most important assumptions for estimating source reliability.Source: §3.2.1 · p. 5

3.2.2 Source independence assumption

• Source independence assumes observations are not copied between sources, so true information is more likely to agree than false information.The survey notes that false information from different sources is more likely to differ.Source: §3.2.2 · p. 5

3.2.3 Source dependency analysis

• Methods relax independence by modeling copying or broader source correlations, with approaches for static, dynamic, and externally informed settings.Copy detection uses shared mistakes, but can fail when sources copy a good source; uses Bayesian inference for probabilistic direct copying and iteratively updates copying relations and truths, while aggregates at the latent-group level. uses HMMs on update histories and considers coverage, exactness, and freshness; infers global copying from direct relations, models positive and negative correlations with joint precision and recall, and uses Twitter follower-followee and retweet information.Source: §3.2.3 · pp. 5–6

3.2.4 Fine-grained source reliability

• Fine-grained reliability assigns multiple reliability degrees to a source when its quality varies across object sets, attributes, or topics.The survey also notes that source consistency may fail when observations become very numerous.Source: §3.2.4 · p. 6

3.2.5 Enriched meaning of source reliability

• Other methods enrich source reliability to represent different quality dimensions or application-specific meanings.Examples include true- and false-positive rates, precision and recall in LTM, and silent, false-spoken, and true-spoken rates when objects may have no truth. For subjective truths, separates truthfulness, completeness, and bias; adds freshness for dynamic source selection, while gives reliability different meanings by setting.Source: §3.2.5 · p. 6

3.2.6 Source reliability initialization

• Because uniform initialization can make truth discovery rely on a majority that may be unreliable, methods use prior knowledge to initialize source reliability.The survey lists labeled data, an external trusted source, and source similarity as initialization aids; reports that “62% of the true responses are produced only by 1 or 2 of the 18 systems (sources)”.Source: §3.2.6 · p. 6

3.2.7 Source selection

• Source selection methods balance integration cost and output quality, while negative source weights can let even poor sources contribute.Methods select cost-constrained subsets for static or dynamic sources; IDS formulates selection as an optimization problem and provides polynomial-time approximation algorithms with provable quality guarantees. Work argues a source is useless only if it guesses randomly, since negative weights can make its information signal likely falsehood.Source: §3.2.7 · p. 7

3.3.1 Object difficulty

• The 3-Estimates algorithm jointly estimates truths, source reliability, and object difficulty, separating source errors from object difficulty to improve reliability estimates.It captures difficulty through the trustworthiness of claimed values, distributing the penalty for an error across the source-reliability and object-difficulty factors.Source: §3.3.1 · p. 7

3.3.2 Relations among objects

• Object relations can improve truth discovery; methods encode prior constraints, propagate slot dependencies, and model temporal and spatial dependencies.In, propositional constraints iteratively correct fact probabilities, with an augmented unknown answer relaxing conflicts; propagates credibility across systems, corpora, and extracted information, initialized using a knowledge graph of slot dependencies. Methods and model temporal relations for categorical and continuous data, while and handle temporal and spatial relations; demonstrates improved performance when objects have insufficient observations.Source: §3.3.2 · p. 7

3.4.1 Complementary vote

• Under the single-truth assumption, complementary vote treats a source’s claim as a vote against other candidate values, unlike LCWA [18], which rejects values using trusted external knowledge.Complementary vote uses one candidate value to reject others; LCWA uses external knowledge to reject candidates.Source: §3.4.1 · p. 7

3.4.2 Implication of values

• Claimed values for one object can be correlated: implication, also considered in, makes trustworthy 175cm imply 176cm with high probability, while 185cm is less likely to be trustworthy.Source: §3.4.2 · p. 8

3.4.3 Data type of claimed values

• Methods model data-type distance differently: focus on categorical values, GTM on continuous values, and uses distance functions for heterogeneous data.[75] models mutual exclusivity for categorical observations and mutual support for continuous ones, but these relations can be difficult to establish in practice.Source: §3.4.3 · p. 8

3.4.4 Ambiguity of data type

• A value’s data type depends on the application—“2” may be a number, class label, or address string—so the type assumption should be checked before applying a truth discovery approach.Source: §3.4.4 · p. 8

3.4.5 Hierarchical structure of claimed values

• Hierarchical value spaces can link claims such as Honolulu, Hawaii, and USA, making them non-exclusive and unsuitable for complementary vote [18].These linked true values differ from multiple-truth scenarios because they are connected through a hierarchy.Source: §3.4.5 · p. 8

3.5.1 Single truth v.s. multiple truths

• Most methods assume one truth per object, but LTM and support multiple truths by accounting for source precision and recall.LTM considers false-positive and false-negative claims; make the single-truth assumption stronger by treating a source’s claimed value as a vote against alternatives.Source: §3.5.1 · p. 8

3.5.2 “Unknown” truths

• An “unknown” option can ensure feasibility under conflicting constraints and correctly represent objects that have no truth.[46] includes “unknown” as a candidate output; models whether a truth exists and can assign “unknown” when none does.Source: §3.5.2 · p. 8

3.5.3 Labeling v.s. scoring

• Labeling yields ready-to-use truths but loses information and confidence, whereas scoring retains probabilities and requires threshold- or rank-based post-processing.GTM and the framework can identify truths not claimed by any source.Source: §3.5.3 · pp. 8–9

3.5.4 Performance measurement

• Evaluation uses accuracy or error rate for categorical data, MAE and RMSE for continuous data, usefulness, and memory cost or running time for efficiency.Accuracy, error rate, MAE, and RMSE suit settings with ground truths; usefulness focuses on results worth evaluating.Source: §3.5.4 · p. 9

3.5.5 Output explanation

• Interpretability can improve through probabilistic source-reliability models and generated snapshot or DAG-based comprehensive explanations.SimpleLCA, GuessLCA, MistakeLCA, and LieLCA define source reliability in terms of asserting, knowing, or intending to tell truths. Snapshot explanations gather positive and negative evidence, aggregate similar evidence, and shorten lists; comprehensive explanations trace iterative decisions in a DAG and shorten it by removing rounds without decisions.Source: §3.5.5 · p. 9

4. TRUTH DISCOVERY METHODS

• Tables 2 and 3 compare truth discovery methods across features, following brief descriptions of representative algorithms intended to guide method selection for specific application scenarios.The survey notes that space limits its method descriptions and directs readers to the cited papers for more details.Source: §4 · pp. 9–10, Table 2 · p. 10, Table 3 · p. 10
• Bayesian TruthFinder iteratively estimates reliability and truths, introducing source consistency and implication; AccuSim uses implication for value similarity, and AccuCopy discounts detected copiers.AccuSim also uses Bayesian analysis; AccuCopy improves on AccuSim by accounting for copying relations among sources.Source: §4 · p. 9
• 2-Estimates assumes one true value per object and uses complementary vote; 3-Estimates adds object difficulty, while Investment allocates reliability across claims and returns credits by claim confidence.Investment's claim confidence grows non-linearly with the sum of invested reliabilities from providers.Source: §4 · p. 9
• SSTF uses labeled truths, mutual exclusivity, and mutual support; LTM models false-positive and false-negative errors separately in multiple-truth settings, while GTM targets continuous data.LTM thereby separates source reliability into false-positive and false-negative parameters.Source: §4 · p. 9
• Regular EM uses EM for binary human-observation maximum likelihood, LCA latent reliability parameters, and Apollo-social models social-media source dependencies.CRH handles heterogeneous data by jointly estimating reliability across types with pluggable distance functions; CATD estimates source-reliability confidence intervals when sources have few observations.Source: §4 · p. 10
• Most methods handle categorical data; GTM, CRH, and CATD target continuous data, CRH addresses heterogeneous input, and SSTF handles categorical and continuous data together.TruthFinder, AccuSim, and AccuCopy can handle continuous data with implication, while SSTF uses mutual support; SSTF alone is designed for semi-supervised use, though other methods can be modified to use labeled truths.Source: §4 · pp. 10–11
• AccuCopy and Apollo-social illustrate source-dependency work, while LTM, Regular EM, LCA, and CATD enrich reliability; only 3-Estimates models object difficulty, and complementary vote reinforces single-truth belief.Investment and PooledInvestment can incorporate object relations, but the survey calls for more methods that account for advanced object features.Source: §4 · p. 11

• Output:

• LTM handles multiple truths; Investment and PooledInvestment add unknown truths, and methods such as LTM can be modified by adding an unknown candidate for each object.The survey describes adding an unknown candidate as one possible way to account for unknown truths.Source: • Output · p. 11

5. FUTURE DIRECTIONS

• Truth discovery on unstructured inputs must account for noise and uncertainty, evidence, and reliability contributions from corpora, extractors, and objects [18].When object difficulty is considered, wrong-information penalties must be distributed across all three factors.Source: §5 · p. 11
• Capturing object relations and automatically discovering them at scale need further exploration, beyond existing efforts.Knowledge graphs can relate attributes such as a person’s birth year and age, while manually encoding relations among large numbers of objects is impractical.Source: §5 · p. 11

• Source reliability initialization.

• Uniform reliability initialization can make tie-breaking alter estimated source weights and can leave results reliant on inaccurate majorities, motivating new initialization methods.The majority issue is especially concerning for tasks such as slot filling, where most sources may provide inaccurate information.Source: • Source reliability initialization · p. 11

Model selection

• Combining outputs from multiple truth discovery methods is proposed as a possible ensemble approach to the difficult model-selection problem.The survey asks how to choose a suitable method for a specific task and whether an ensemble can address that challenge.Source: • Source reliability initialization · p. 11

Theoretical analysis

• Convergence, error bounds, and the equivalence or relationships among truth discovery methods remain open questions for theoretical analysis.The survey notes evidence that different ways of capturing the general principle can be equivalent, warranting further exploration.Source: • Source reliability initialization · p. 11

Efficiency

• Iterative truth discovery can be inefficient on large datasets, so efficiency improvements are needed beyond streaming-data scenarios.The survey notes that some work has addressed efficiency for streaming data, but calls for further work in general cases.Source: • Source reliability initialization · p. 11

Performance evaluation

• Groundtruth-dependent evaluation is often impractical, especially for large knowledge graphs, making evaluation without groundtruth a major challenge.Obtaining even a small set of labeled groundtruths can require expensive human effort.Source: • Source reliability initialization · pp. 11–12

Healthcare

• Truth discovery identifies reliable users and trustworthy user-generated medical statements in online health communities.Drug reviews from these communities can be valuable to patients and physicians, but their quality is a concern.Source: §6 · p. 12

Crowd/social sensing

• Truth discovery improves aggregation of unreliable crowd/social sensing observations, including disaster gas-shortage reports and evolving-event summaries.Potential applications also include aggregating crowd GPS data for traffic control, distilling social-media news, and grading MOOC homework automatically.Source: §6 · p. 12

Crowdsourcing aggregation

• Crowdsourcing aggregation infers true labels from diverse worker judgments, unlike passive truth discovery, because platforms can control what and how much data workers generate.Source: §6 · p. 12

Information extraction

• Truth discovery resolves conflicts among outputs from multiple extractors applied to corpora for tasks such as slot filling and entity profiling.Source: §6 · p. 12

Knowledge base

• Google Knowledge Graph, Freebase, and YAGO remain incomplete; truth discovery identifies trustworthy Internet information [18] and estimated source reliability can assess webpage quality [19].Source: §6 · p. 12

RELATED AREAS

• Unlike multi-view clustering [73], rank aggregation, sensor fusion, meta-analysis, and ensemble learning, truth discovery resolves conflicts across sources on shared objects and features through unsupervised reliability estimation.The neighboring fields use different inputs or goals, treat sources indistinguishably, derive weights from study sample size, or learn model weights through supervised training.Source: §7 · p. 12
• Trustworthiness scoring evaluates users, accounts, nodes, or sensors using source content or links, while truth-enhanced recommendations derive trust and recommendations separately.Recommendation combines other users’ ratings for an item using trust weights; trust comes from supplied trust values or recommendation history, while recommendations use current ratings and derived trust.Source: §7 · p. 12
• Truth discovery tightly combines source-reliability estimation and truth computation, discovering both from multi-source data; source reliability is usually the probability that a source gives true claims.Source: §7 · p. 12

8. CONCLUSIONS

• The survey examines truth discovery components under five general aspects and compares methods with different assumptions about input data, constraints, and outputs in Tables 2 and 3.The comparison is intended to guide users and developers choosing an approach for a particular task.Source: §8 · p. 13
• Exploring relations among objects is a future direction that could benefit real-world applications such as knowledge graph construction.Source: §8 · p. 13
• Efficiency is a bottleneck to deploying truth discovery on large-scale data.Source: §8 · p. 13
• Evaluating performance or validating identified truths remains challenging because ground truth is limited in practice.Source: §8 · p. 13