Source-linked AI summary
Beyond "Made with AI": Visualizing Provenance Density to Mitigate the Transparency Penalty
Qing Zhang, Yifei Huang, Juyoung Lee, Thad Starner, Jun Rekimoto
TL;DR
Generative AI makes fluent language an unreliable truth cue, while binary AI disclosures reveal authorship without showing claim support. This paper proposes Provenance Density, an interface and metric for visualizing verified-claim density, and evaluates it in technical and user studies. Under correct signaling, the interface produced a large truth-discernment effect, while the paper’s limitations include dependence on digital citation networks and risks from false or biased provenance signals.
Problem
Generative AI decouples polished prose from veracity, while binary “Made with AI” labels do not provide claim-level evidence and can discount accurate content.
Method
The paper defines Provenance Density as a costly signal combining verified claims, source reputation, contextual relevance, and cross-sample consistency, then evaluates it with technical and Oracle user-study protocols.
Results
+4.15 points and d = 1.82 showed a large truth–fabrication discernment gap under correct Provenance Density signaling, whereas the no-signal condition showed no detectable discrimination (p = .43).
Takeaways & Limitations
Effective transparency should visualize evidence supporting claims rather than rely solely on authorship disclosure.
Takeaways & Limitations
Provenance Density can favor established consensus over novel information and may conflate missing digital evidence with falsehood, especially for non-digitized knowledge systems.
Abstract
from arXiv · showhide
As generative AI makes polished prose cheap to produce, users can no longer rely on fluency as a proxy for truth. We call this failure mode the Fluency Trap: users trust fluent hallucinations while also discounting accurate content once it is disclosed as AI-generated. Binary ``Made with AI'' labels respond with authorship disclosure, but they do not show what supports a claim. We propose Provenance Density, an evidence-visualization interface that shows the density of verified claims in a text. In a user study with 81 participants, an idealized Provenance Density interface produced a large discernment gap between truth and fabrication ($+4.15$ points, $d=1.82$), whereas participants given no signal showed no detectable discrimination. A technical audit with 200 samples shows that retrieval density alone is insufficient; unexpectedly, the Consistency Veto carries most of the discriminative signal on dynamic queries. As AI-generated content becomes indistinguishable from human writing, effective transparency must move from authorship disclosure toward evidence visualization.
1 Introduction
Generative AI separates linguistic fluency from truth, weakening fluency-based judgments and making binary authorship labels an inadequate transparency response. The paper proposes visualizing verified-claim density and evaluates this approach through technical and user studies.
- Problem: Generative AI reduces fluency’s production cost, creating a Fluency Trap in which polished text can remain trusted despite being ungrounded.The paper links this vulnerability to hallucinated plausibility: syntactically perfect but semantically ungrounded text.
- Problem: Binary “Made with AI” labels disclose identity rather than claim-level support and can discount accurate content alongside fabrication.The paper calls this accuracy-independent discounting a Transparency Penalty.
- Design: Provenance Density visualizes the density of verified claims, shifting attention from prose fluency toward extrinsic evidence.The interface is intended to offload part of the verification burden from working memory.
- Results: M = 6.28 versus M = 5.78 showed no detectable control-condition discrimination (p = .43), while correct Provenance Density signaling restored discernment (d = 1.82, p < .001).The comparison is between high-fluency hallucinations and ground truth in the absence of signals.
- Evidence: The paper evaluates Provenance Density through a technical audit of 200 samples and a within-subjects truth-discernment study with 81 participants.The audit tests robustness against adversarial misconceptions and dynamic ambiguity, while the user experiment measures discernment.
2 Related Works
Related work explains the Fluency Trap as a structural decoupling of polished language from veracity, reinforced by LLM alignment and resistant to binary warnings or authorship detection. These limitations motivate claim-level provenance signals.
- Structural account: The paper frames fluency–veracity decoupling as a structural result of cheap linguistic signaling and objectives that prioritize plausibility over truth.This combines an economic shift from costly signaling to cheap talk with a technical objective function favoring plausibility.
- Governance: Binary warnings may reduce global trust without reducing reliance on specific misleading articles, revealing a failure of inoculation.The discrepancy separates theoretical acknowledgment of AI bias from practical acceptance of fluent content.
- LLM alignment: RLHF can incentivize sycophancy, producing outputs optimized for rhetorical persuasion rather than factual accuracy.The cited work describes agreement with illogical premises and adaptation to user views as mechanisms of this pattern.
- LLM alignment: Recursive training on high-fluency, low-entropy data can produce Model Collapse, reducing human variation to a homogenized mean.This is presented as a self-amplifying consequence of the fluency–veracity decoupling.
- Governance: Binary disclosures can impose an epistemic stigma, lowering perceived trustworthiness without enabling claim-level verification.Related advertising experiments found identical ads received more critical evaluations when labeled AI-generated.
- Detection limits: Automated detectors can misclassify lower-perplexity lexical patterns from non-native speakers as AI-generated, creating hermeneutical access injustice.The limitation is tied to detectors that rely on perplexity heuristics.
3 Defining Provenance Density
Provenance Density is a theoretically costly signal that aggregates verified claims, source reputation, relevance, and cross-sample consistency. Its implementation combines claim segmentation, retrieval, relevance scoring, and aggregation, with an Oracle protocol isolating interface effects.
- Metric definition: Provenance Density D(T) is defined as a function of verifiable claims weighted by source reputation, contextual relevance, and internal semantic consistency.The design aims to make high-density signals computationally hard to fake.
- Metric definition: The metric combines retrieval-based external verification with an NLI-based penalty for disagreement among stochastic generations.The penalty term Pint down-weights generations whose independent samples disagree.
- Metric definition: The Consistency Veto uses K = 5 samples and reduces the score as pairwise inconsistency rises, but cannot detect consistently repeated false answers.Internally stable generations have Pint ≈ 0, while mutually inconsistent generations approach Pint → 1.
- Metric definition: A cubic MatchRatio weighting suppresses weak conceptual matches and keyword stuffing, while β = 5.0 scales the hyperbolic-tangent aggregation.The cubic term is motivated by preliminary tests in which quadratic weighting remained too permissive.
- Implementation: The implementation performs claim segmentation, evidence retrieval, source-relevance scoring, and aggregation for each remaining factual claim.Claims shorter than five tokens are discarded, and query construction depends on claim length.
- Evaluation design: The user study uses a Wizard-of-Oz Oracle protocol that pairs grounded summaries with high-density indicators and fabricated summaries with null indicators.This separates interface interaction effects from live retriever noise.
4 Technical Validation
The technical audit evaluates Provenance Density across static and dynamic knowledge, finding that it reflects epistemic maturity but relies chiefly on consistency rather than retrieval density for hallucination discrimination.
- Ecological Sensitivity: 150 static TruthfulQA items averaged M = 0.79, while 50 dynamic FreshQA items averaged M = 0.64 with a broader spread.Static items clustered above the high-trust threshold D(T) ≥0.7; dynamic items were more variable.
- Hallucination Detection: Evidence density alone was near chance overall (AUC = 0.47) and anti-discriminative on FreshQA (AUC = 0.28).High retrieval density can still support false claims when misconceptions are popular or topics are changing.
- Hallucination Detection: The Consistency Veto Pint was the principal discriminator on dynamic queries, achieving overall AUC (0.60) and dynamic-query AUC = 0.92.These FreshQA estimates are suggestive rather than definitive because the split contained only three hallucinated items.
- Hallucination Detection: The combined D(T) score reached AUC = 0.72 on FreshQA while retaining sensitivity to epistemic maturity on static knowledge.It remained weaker than the Consistency Veto alone because density was noisy on adversarial misconceptions.
- Hallucination Detection: The Indonesia-capital transition case shows retrieval finding competing time-sensitive answers while the consistency gate down-weights internally unstable generations.This illustrates why consistency contributes discriminative signal when dynamic evidence conflicts.
5 Empirical Evaluation: Interaction Paradigm
The study tested whether interface signals improve trust calibration and truth discernment when fluent summaries include grounded or hallucinated content. In a within-subjects experiment with 81 participants, Provenance Density produced the strongest separation between truth and fabrication.
- Experimental Design: The study used matched high-fluency fabrications so interface effects were not attributable to obvious textual defects.Hallucinated summaries were matched to grounded summaries in length, tone, and scholarly style.
- Results: χ2(2) = 31.32, p < .001 for the Interface × Veracity interaction, showing that interface condition changed users’ epistemic responses.A linear mixed model included Interface × Veracity and a random intercept for participant.
- Results: M = 6.28 versus M = 5.78, p = .43, d = −0.21: without signals, participants did not detectably distinguish high-fluency hallucinations from ground truth.The Control condition was designed to represent the status quo, where fluency was the only visible signal.
- Results: PDI created the largest separation between truth and fabrication, while Binary Disclosure reduced trust in hallucinations without comparably restoring confidence in true content.The figure and results report PDI’s largest discernment gap with d = 1.82; the Binary Disclosure condition acted as a static AI-generated-content warning.
6 Discussion
The discussion interprets the results as evidence that fluent presentation can substitute for veracity judgments when provenance signals are absent. It presents Provenance Density as a more calibrated alternative while emphasizing that its verification metric has important limits.
- Core Findings: PDI significantly outperformed Control and Binary Disclosure in restoring truth discernment.Table 4 identifies d = 1.82 as the largest separation between truth and fabrication.
- Mechanism of the Fluency Trap: In the Control condition, participants’ failure to distinguish truth from hallucination was consistent with heuristic substitution from veracity judgments to fluency judgments.Participant comments described relying on wording, flow, and believability when evaluating accuracy.
- Binary Labels and Calibration: Binary Disclosure operated as a blunt warning or epistemic stigma, whereas PDI shifted users toward evaluating extrinsic evidence.The discussion contrasts risk signaling from binary labels with evidence evaluation under Provenance Density.
- Limits of Cognitive Offloading: The Consistency Veto provides a partial safety rail for cognitive offloading, not a guarantee, particularly for ambiguous queries.The discussion cites suppression of scores in ambiguous queries such as the Indonesia capital transition.
7 Limitations and Future Work
The paper identifies deployment, ecological, latency, and adversarial limitations for Provenance Density, motivating more robust verification and broader evaluation. Its signals can favor consensus, remain costly to compute, and be vulnerable to gaming or automation bias.
- From Oracle to Deployment: PDI’s oracle-based user-study efficacy is an upper bound because live retrieval can reinforce misconceptions and consistency checks can miss repeatedly recurring false answers.The paper prioritizes near-miss cases, especially false-positive high-density signals.
- Ecological Conservatism: PDI favors established consensus over emerging novelty, risking classification of legitimate new information as unverified.Consensus scored M = 0.79 versus emerging novelty at M = 0.64; temporal metadata is proposed as a remedy.
- Latency Constraints: Inference latency ranged from 17.0s for Dynamic queries to 26.4s for Static queries, making D(T) unsuitable for synchronous turn-by-turn chat.The paper proposes asynchronous checking and notes that latency was not experimentally manipulated.
- Adversarial Gaming and Automation Bias: PDI remains vulnerable to domain spoofing, empty-keyword exploitation, SEO-style inflation, and repeated misconceptions inherited from open-web consensus.Suggested defenses include stricter domain parsing, conservative vague-claim handling, per-domain contribution caps, and duplicate-evidence checks.
- Adversarial Gaming and Automation Bias: Automation bias could shift users from reading the text to scanning for the green bar, motivating design friction that periodically forces manual verification.This risk is presented as a second-order consequence of PDI’s interface.
8 Conclusion
The paper argues that cheap fluency breaks the link between style and substance, leaving users unable to distinguish grounded truth from high-fluency hallucinations without external scaffolding. It presents Provenance Density as a calibrated alternative to binary authorship labels, while acknowledging that dense evidence can still support misconceptions.
- 8 Conclusion: Cheap fluency makes style an unreliable signal of substance, and users without external scaffolding could not distinguish grounded truth from high-fluency hallucination.The conclusion frames AI-generated misinformation as a signaling problem as well as a generation problem.
- 8 Conclusion: +4.15 points: Provenance Density restored users’ ability to separate truth from fabrication under correct signaling.The technical audit qualifies this result by showing that high-density evidence can still support misconceptions.
- 8 Conclusion: AI transparency should shift from identifying authorship toward visualizing the evidence supporting claims.The proposed direction emphasizes verification that is visible, intuitive, and contestable.
Ethical Statement
The study followed institutional ethical review, informed-consent, privacy, and compensation procedures. The paper also warns that provenance-based credibility standards may exclude non-digitized knowledge systems and should not serve as universal arbiters of truth.
- Procedural Ethics: The user study recruited 81 Prolific participants who provided informed consent, with submission IDs stored separately and no other personally identifying information collected.The paper states that submission IDs were used solely for payment processing and are not retained in released datasets.
- Sociotechnical Implications: PDI may privilege digitized consensus over indigenous, oral, and low-resource knowledge systems lacking dense citation networks.The paper recommends deploying PDI strictly as a verification tool for digitized consensus rather than as a universal arbiter of truth.