Source-linked AI summary
The Generative AI Paradox: GenAI and the Erosion of Trust, the Corrosion of Information Verification, and the Demise of Truth
Emilio Ferrara
TL;DR
The paper addresses how GenAI may erode shared epistemic ground as synthetic content, identities, and interactions become cheap, scalable, and difficult to audit. It formalizes synthetic reality as a layered stack, expands a harm taxonomy, synthesizes recent risk realizations, and proposes mitigation and measurement approaches. Its central conclusion is the Generative AI Paradox: widespread synthetic content may lead societies to discount digital evidence altogether.
Problem
The paper examines how GenAI can move beyond isolated fake artifacts toward coherent synthetic realities that weaken shared evidence and increase institutional verification burdens.
Method
The paper formalizes a content–identity–interaction–institutions stack, expands a GenAI harm taxonomy, synthesizes 2023–2025 risk realizations, and proposes mitigation and research agendas.
Results
As synthetic content becomes ubiquitous and difficult to distinguish, societies may rationally discount digital evidence, producing contested evidence and escalating verification load.
Takeaways & Limitations
The paper calls for defense-in-depth combining provenance infrastructure, platform governance, institutional workflow redesign, public resilience, and epistemic-security measurement.
Takeaways & Limitations
Detection remains imperfect under compression, re-encoding, adversarial perturbations, and cross-model remixing, while watermarking is removable or bypassable in open ecosystems.
Abstract
from arXiv · showhide
Generative AI (GenAI) now produces text, images, audio, and video that can be perceptually convincing at scale and at negligible marginal cost. While public debate often frames the associated harms as "deepfakes" or incremental extensions of misinformation and fraud, this view misses a broader socio-technical shift: GenAI enables synthetic realities; coherent, interactive, and potentially personalized information environments in which content, identity, and social interaction are jointly manufactured and mutually reinforcing. We argue that the most consequential risk is not merely the production of isolated synthetic artifacts, but the progressive erosion of shared epistemic ground and institutional verification practices as synthetic content, synthetic identity, and synthetic interaction become easy to generate and hard to audit. This paper (i) formalizes synthetic reality as a layered stack (content, identity, interaction, institutions), (ii) expands a taxonomy of GenAI harms spanning personal, economic, informational, and socio-technical risks, (iii) articulates the qualitative shifts introduced by GenAI (cost collapse, throughput, customization, micro-segmentation, provenance gaps, and trust erosion), and (iv) synthesizes recent risk realizations (2023-2025) into a compact case bank illustrating how these mechanisms manifest in fraud, elections, harassment, documentation, and supply-chain compromise. We then propose a mitigation stack that treats provenance infrastructure, platform governance, institutional workflow redesign, and public resilience as complementary rather than substitutable, and outline a research agenda focused on measuring epistemic security. We conclude with the Generative AI Paradox: as synthetic media becomes ubiquitous, societies may rationally discount digital evidence altogether.
1 Introduction
GenAI amplifies familiar harms while enabling personalized synthetic realities that can erode shared evidence and raise verification burdens. The paper organizes these risks across personal, economic, informational, socio-technical, and epistemic domains.
- GenAI may reshape trust and deception by lowering the cost, increasing the speed, and expanding the personalization and reach of harmful acts.
- Personalized synthetic realities can embed falsehoods in coherent, interactive, and socially reinforced narratives that are difficult to falsify from within.
- The taxonomy covers personal harms including impersonation, targeted harassment, defamation, privacy breaches, and psychological distress.
- GenAI creates financial and economic damage by scaling fraud, increasing verification and compliance costs, and reducing the default credibility of digital evidence.
- Information manipulation combines fabricated narratives and evidence with interactive persuasion that adapts to people’s beliefs and emotional states.
- Socio-technical harms include institutional delegitimation, polarization, governance paralysis, supply-chain compromise, unsafe deployment, and misuse in critical workflows.
- Epistemic and institutional integrity is threatened by both credulity toward fabrications and cynicism toward authentic evidence.
2 From synthetic media to synthetic reality
The paper distinguishes synthetic reality from isolated synthetic media by modeling a layered system in which content, identity, and interaction reinforce one another and burden institutions with verification.
- Synthetic reality is defined as a layered stack of content, identity, interaction, and institutions, mapping to distinct attack surfaces and defensive levers.
- Institutions: At the institutional layer, synthetic systems corrode verification workflows in elections, courts, finance, and journalism.
- Interaction: Synthetic interaction uses chatbots, social engineering, and persuasion loops to create socially validated narratives.
- Identity: Synthetic identity manufactures personas, voice clones, face swaps, and fake documents that can function as credible witnesses.
- Content: Synthetic content includes text, images, audio, and video produced at low cost with high realism and many plausible variants.
2.1 Layer 1: Synthetic content
Synthetic content provides the baseline capability: realistic text, images, audio, and video can be produced cheaply, rapidly, and in many targeted variations. Its systemic potency increases when combined with identity and interaction.
- Synthetic content includes generated or edited text, images, audio, and video with high realism, rapid production, and low marginal cost.
- Iterative variation enables many plausible alternatives, while compositional editing changes specific details without sacrificing global plausibility.
- The content layer supports familiar harms such as forgery, defamation, and propaganda, but becomes more potent when coupled with identity and interaction.
2.3 Layer 3: Synthetic interaction
Synthetic interaction turns belief manipulation into an adaptive social process. Interactive systems can probe targets, generate social proof, and sustain relationships while institutions absorb the resulting verification burden.
- Interactive systems simulate dialogue and social presence, enabling persuasion that adapts to targets and sustains manipulative relationships.
- Compared with static misinformation, interactive agents use conversation, feedback, and social reinforcement to shape belief formation.
- Synthetic content and identity interfere with evidence-dependent workflows in elections, courts, finance, and journalism.
- Content realism is only the entry point; identity and interaction provide credibility and momentum while institutions verify under time pressure.
3 What you can’t tell apart can harm you: why GenAI changes the game
GenAI changes deception by making synthetic content, identities, and interactions cheap, scalable, tailored, and difficult to verify. These mechanisms can shift societies from isolated fake artifacts toward systemic pressure on trust, shared reality, and verification.
- GenAI makes realistic content and identity artifacts cheap to produce, expanding the pool of capable adversaries and enabling rapid experimentation.Prompts, templates, and turnkey services reduce specialized-skill requirements and shorten the path from intent to execution.
- Thousands of message, image, or persona variants can be generated and tested, turning defense from finding individual fakes into managing high-volume plausible streams.Coordinated networks can distribute these variants across platforms and accounts.
- Domain-specific generation improves deception by matching an institution, community, or individual’s language, norms, workflows, and expectations.The relevant advantage is fit as well as realism, because contextual matching removes cues that might otherwise trigger suspicion.
- Micro-segmentation lets adversaries tailor narratives to distinct fears, identities, and grievances, undermining collective rebuttal and amplifying incompatible interpretive frames.Different communities may receive different claims and corrections may not reach the audiences most affected.
- Automated conversational agents turn deception into an interactive process by building rapport, probing uncertainty, and sustaining manipulative relationships over time.This shifts the problem from static content authenticity toward belief formation and decision-making.
- Detection remains imperfect under transformations and adversarial adaptation, while watermarking is incomplete in open ecosystems; high-stakes decisions therefore require auditable provenance and process safeguards.The provenance gap persists for unauthenticated media and content whose watermarks are removed, weakened, or bypassed.
- Ubiquitous synthetic outputs can make authentic evidence dismissible as fake, increasing strategic denial and producing epistemic instability.Audiences may rationally default to suspicion when the evidentiary substrate itself is contested.
- Together, cost collapse, throughput, customization, micro-segmentation, interaction, and provenance gaps shift risk from isolated deception events to systemic pressure on trust and verification.The mechanisms reinforce one another by increasing volume and persuasive fit while making reliable verification difficult at scale.
4 How risks materialize: Representative risk realizations
The paper’s case bank shows that synthetic-reality risks have already appeared across fraud, elections, harassment, documentation, and model supply chains. Across cases, cheap high-conviction artifacts exploit workflow points where people and institutions rely on trust cues.
- The case bank uses mechanism-diverse, publicly documented incidents to illustrate operational harms rather than provide an exhaustive census.Its coverage spans fraud, elections, harassment, documentation, and supply-chain compromise.
- Public documentation is uneven across regions and sectors, so the reported cases should be interpreted as illustrative lower bounds rather than a complete census.
- Across cases, GenAI reduces the cost of high-conviction artifacts and exploits social and institutional workflows designed for expensive-to-fabricate evidence.
- Case A: Enterprise impersonation fraud: A fabricated group video meeting used familiar voices, faces, and operational routines to induce an employee to authorize large transfers.The reported Hong Kong case combined a phishing pretext with publicly sourced video and voice materials.
- Case B: Election-adjacent synthetic outreach: AI-generated robocalls mimicked a well-known political figure and delivered demobilizing messages timed to the New Hampshire primary.Subsequent enforcement targeted the communications infrastructure as well as the content.
- Case C: Synthetic sexual imagery: Non-consensual sexual deepfakes spread rapidly across major platforms, prompting temporary search restrictions to slow discovery and resharing.The case illustrates identity-targeted harassment in which a victim’s identity becomes a reusable asset for abuse.
- Case D: Fabricated documentation: AI-generated receipts and related documents increasingly defeat quick visual inspection through realistic formatting and metadata manipulation.Operational responses move toward automated detection, metadata forensics, and cross-validation.
- Case E: Generative-pipeline compromise: Malicious model uploads to a major model-sharing hub extended supply-chain compromise to the distribution of models as files.The reported pattern used model artifacts to execute malware when loaded.
5 Mitigation as a stack (not a silver bullet)
The paper treats mitigation as defense-in-depth because synthetic-reality risks can arise at multiple layers and no single tool or policy is sufficient. The proposed stack combines provenance, platform governance, institutional redesign, public resilience, and accountability.
- No single tool or policy can solve synthetic-reality risk because detection is brittle, watermarking is incomplete, and takedowns may arrive after harm.
- Provenance infrastructure: Provenance systems establish chain-of-custody and authenticity signals, especially when adopted end-to-end by high-stakes institutions and consumer interfaces.They remain limited for content generated outside authenticated pipelines or captured by compromised devices.
- Platform governance: Platforms can reduce harm by limiting amplification of unverified media, surfacing provenance and uncertainty, rate-limiting new accounts, and addressing coordinated inauthentic behavior.
- Institutional workflow redesign: Institutions should use out-of-band verification, multi-factor approvals independent of voice or face cues, authenticated channels, and provenance-aware evidentiary standards.The broader shift is from artifact-based trust toward workflow-based verification.
- Public resilience: Public resilience should emphasize calibrated skepticism, authenticated channels, awareness of manipulation tactics, and community-level correction mechanisms.The paper frames these practices as epistemic hygiene rather than universal individual authentication.
- Accountability: Policy can clarify liability and incentives through disclosure requirements, rapid responses to impersonation and intimate imagery, and authentication standards in sensitive domains.The stated objective is accountability while protecting legitimate creative and assistive uses.
- Layered synthesis: Mitigations should map to stack layers and failure modes, combining provenance, platform governance, institutional redesign, and public resilience because adversaries can route around isolated defenses.The realistic goal is risk reduction through defense-in-depth with measurement and monitoring.
6 Open problems and a research agenda
The paper calls for measuring epistemic security as systemic resilience rather than artifact authenticity, using operational metrics, interactive benchmarks, provenance evaluation, institutional redesign, and equity analysis.
- Research agenda: Epistemic-security research should measure whether socio-technical systems sustain shared reality and accountable decisions under adversarial pressure.Current information-integrity tooling is optimized for static artifacts rather than layered environments combining content, identity, interaction, and institutions.
- Operational metrics: The agenda proposes operational metrics covering provenance coverage, correction latency, and susceptibility to manipulation.These measures are intended to quantify the epistemic tax imposed by synthetic reality and identify where defensive investment has the greatest marginal benefit.
- Interactive benchmarks: Interactive benchmarks should test multi-turn persuasion, adaptive targeting, relationship persistence, and social-proof generation under realistic platform and adversarial constraints.
- Provenance and robustness: Provenance systems require end-to-end evaluation of cryptographic robustness, usability, adoption incentives, failure modes, attack surfaces, and detector uncertainty.Detection should function as probabilistic evidence suitable for institutional decision-making rather than as a binary oracle.
- Institutional redesign: Courts, elections, journalism, and finance need evidence regimes that remain robust when perceptual cues and documentation lose default credibility.Candidate responses include authenticated capture, verifiable logs, audit trails, uncertainty disclosure, and procedures for contested synthetic evidence.
- Equity and differential harm: Mitigation research must measure differential exposure, verification burden, and downstream harms because synthetic-reality risks disproportionately affect marginalized groups and people lacking authenticated channels or legal recourse.Evaluation should assess whether interventions shift burdens onto those least able to absorb them.
- Epistemic resilience: The overarching shift is from testing individual artifacts to assessing whether communities and institutions can converge on truth, assign responsibility, and act accountably.The paper frames this as an interdisciplinary research program requiring empirical measurement in real-world settings.
7 Conclusions
The paper concludes that GenAI can manufacture coherent synthetic realities that weaken shared epistemic ground, increasing verification burdens and potentially causing societies to discount digital evidence. It therefore advocates defense-in-depth and resilient verification regimes focused on the systems and processes sustaining trust.
- Conclusion: GenAI’s deeper risk is constructing synthetic realities in which manufactured identities, conversations, documentation, and social proof make false narratives difficult to falsify.
- Conclusion: The Generative AI Paradox is that ubiquitous, difficult-to-distinguish synthetic content may lead societies to discount digital evidence altogether.The paper links this possibility to higher verification costs, slower institutional processes, and weakened accountability.
- Observable pressures: The paradox yields measurable pressures including greater reliance on authenticated channels, longer correction latency, fragmented rebuttal reach, higher verification load, and more strategic denial claims.Together, these pressures constitute an epistemic tax on governance, commerce, and civic life.
- Mitigation: Addressing synthetic-reality risk requires defense-in-depth across provenance infrastructure, platform governance, institutional workflow redesign, and public resilience.The proposed goal is not perfect authenticity but resilient verification regimes that preserve trust where it matters most.
Funding
No external funding was reported.
- Funding: This research received no external funding.