Source-linked AI summary
GenAI Against Humanity: Nefarious Applications of Generative Artificial Intelligence and Large Language Models
Emilio Ferrara
TL;DR
GenAI and LLMs create risks including fraud, misinformation, harmful content, and synthetic identities, raising cybersecurity, ethical, and societal concerns. The paper synthesizes these abuses through a harm-and-intent taxonomy and reports that ChatGPT generated conspiracy-theorist-perspective content 80% of the time in one evaluation. It concludes that mitigation strategies, ethical guidelines, and continuous monitoring are urgently needed.
Problem
GenAI and LLMs can enable harms such as fraud, misinformation, harmful content, and blurred boundaries between reality and generated media, threatening cybersecurity, ethics, and societal structures.
Method
The paper synthesizes research on GenAI abuse and organizes misuse scenarios in a 3x4 matrix crossing types of harm with malicious intent.
Results
80% of the time, ChatGPT complied when asked to write from the perspective of conspiracy theorist Alex Jones.
Takeaways & Limitations
The findings underscore the need for robust mitigation strategies, ethical guidelines, and continuous monitoring of GenAI and LLM systems.
Takeaways & Limitations
The taxonomy’s intent dimension does not fully encompass all possible motivations behind GenAI misuse.
Abstract
from arXiv · showhide
Generative Artificial Intelligence (GenAI) and Large Language Models (LLMs) are marvels of technology; celebrated for their prowess in natural language processing and multimodal content generation, they promise a transformative future. But as with all powerful tools, they come with their shadows. Picture living in a world where deepfakes are indistinguishable from reality, where synthetic identities orchestrate malicious campaigns, and where targeted misinformation or scams are crafted with unparalleled precision. Welcome to the darker side of GenAI applications. This article is not just a journey through the meanders of potential misuse of GenAI and LLMs, but also a call to recognize the urgency of the challenges ahead. As we navigate the seas of misinformation campaigns, malicious content generation, and the eerie creation of sophisticated malware, we'll uncover the societal implications that ripple through the GenAI revolution we are witnessing. From AI-powered botnets on social media platforms to the unnerving potential of AI to generate fabricated identities, or alibis made of synthetic realities, the stakes have never been higher. The lines between the virtual and the real worlds are blurring, and the consequences of potential GenAI's nefarious applications impact us all. This article serves both as a synthesis of rigorous research presented on the risks of GenAI and misuse of LLMs and as a thought-provoking vision of the different types of harmful GenAI applications we might encounter in the near future, and some ways we can prepare for them.
INTRODUCTION
GenAI and LLMs offer powerful capabilities in language and multimodal content generation, but also enable diverse harms spanning cybersecurity, ethics, and society. The paper synthesizes research on these risks and calls for mitigation, ethical guidance, and monitoring.
- INTRODUCTION: A synthetic voice impersonating a CEO helped defraud a UK energy firm of $243,000 in 2019.The incident illustrates that AI-enabled fraud is already operational rather than merely speculative.
- INTRODUCTION: GenAI and LLMs can support botnets, harmful content, blurred distinctions between reality and generated media, fabricated evidence, and alibis.The paper presents these as applications that can harm cybersecurity, ethics, and societal structures.
- INTRODUCTION: The paper frames GenAI misuse as a dual-use challenge requiring robust mitigation strategies, ethical guidelines, and continuous monitoring.Its aim is both to synthesize research on abuse and to stimulate discussion of the technologies’ dual nature.
- INTRODUCTION: GenAI creates novel text, images, and audio from learned patterns, while LLMs specialize in generating coherent, contextually relevant text from extensive corpora.LLMs process input queries and training data to produce textual responses ranging from answers to other forms of generated content.
Technological Advancements and Democratization
GenAI became more capable and more accessible as technical advances, lower costs, open-source tools, user-friendly interfaces, and cloud services broadened participation. Governance efforts remain divergent across regions, complicating universal regulation.
- Technological Advancements and Democratization: The early 2020s democratized AI by enabling individuals and smaller organizations to use GenAI across sectors.This broader access differentiated the current landscape from previous AI generations in technological sophistication and societal impact.
- Technological Advancements and Democratization: Technical advances, especially in neural networks, produced more sophisticated models capable of understanding and generating complex data patterns.The passage attributes these advances to improvements in machine-learning algorithms.
- Technological Advancements and Democratization: Falling computing prices and open-source tools reduced the cost of developing and deploying GenAI systems.User-friendly interfaces and cloud-based services further increased accessibility for non-specialists.
- Technological Advancements and Democratization: The EU emphasizes privacy, transparency, accountability, and risk-based oversight, whereas China emphasizes national security, social stability, data control, and misuse prevention.These divergent priorities illustrate the difficulty of creating a universally accepted regulatory framework.
- Technological Advancements and Democratization: Regional GenAI policies are likely to influence global standards and practices.The paper connects differing governance approaches with broader regulatory challenges.
UNDERSTANDING GENAI ABUSE: A TAXONOMY
The paper organizes GenAI abuse in a 3x4 matrix crossing types of harm with malicious intent. This taxonomy links concrete misuse scenarios to categories that can guide threat anticipation and mitigation.
- Types of Harm: The taxonomy distinguishes harms to persons, financial and economic damage, information manipulation, and societal, sociotechnical, and infrastructural damage.Examples include identity theft, fraud, misinformation, threats to democratic processes, and attacks on technological systems.
- Malicious Intent: Its intent dimension covers deception, propaganda, and dishonesty.These involve misleading others, advancing political, ideological, or commercial agendas, and concealing or misrepresenting truth for ulterior motives.
- Scope: The intent dimension does not fully encompass all possible motivations for GenAI misuse.The authors present it as a guide for framing nefarious applications according to intent to harm.
- Matrix Structure: Each cell of the 3x4 matrix represents a combination of harm and malicious intent, such as identity theft through AI impersonation.Fake-news campaigns intended to influence public opinion represent another intersection: information manipulation and propaganda.
- Applications: Table 2 summarizes proof-of-concept scenarios involving dishonest, propagandist, or deceptive misuse of GenAI and LLMs.The framework is intended to help stakeholders anticipate threats and devise specific mitigation strategies.
A GLIMPSE INTO DAYS OF FUTURE PAST
GenAI and deepfakes extend familiar forms of media manipulation into realistic, scalable, and potentially inexpensive abuses. The paper highlights synthetic identities, impersonation, harassment, fabricated evidence, and alibis as concrete risks.
- A GLIMPSE INTO DAYS OF FUTURE PAST: GenAI and deepfakes create new challenges beyond traditional tampered footage and photoshopped multimedia.They can generate deepfakes, plausible deniability, subliminal messages, and deceptive content.
- Synthetic Identities: Synthetic identities can look real on social platforms despite representing nonexistent people, and they are already used in malicious activities.The paper notes that AI-generated identities are appearing online and being used by people with malicious intentions.
- Synthetic Identities: Fake personas can be purchased, generated freely, animated, and made to talk, creating additional opportunities for abuse.The paper describes services offering fake people and animated personas for stated benign purposes that can be easily misused.
- Malicious Personas: Generated identities can support espionage, propaganda, and harassment by disguising malicious actors behind attractive or friendly profiles.Examples include infiltration of intelligence communities, right-wing propaganda, and trolling targets.
- Plausible Deniability and Attribution Problems: Generative AI could construct scalable, inexpensive alibis or fabricate criminal evidence, undermining trust, credibility, and accountability.The paper links these capabilities to attribution problems and plausible deniability.
GenAI Against the People
GenAI expands cyber threats from vulnerable systems to the people behind them, enabling personalized scams, harassment, and more convincing social engineering. Data scraping supports targeted spear-phishing and other individualized attacks.
- GenAI-powered cyberattacks target human users as well as vulnerable systems, including automated ad hominem attacks, online harassment, and personalized scams.
- AI tools can scrape personal identifiable information and social-media data about potential victims to support more detailed social-engineering efforts.
- Spear phishing uses collected target data to craft personalized emails, and AI-enabled data scraping may make it more common and effective.
- GenAI enables automated harassment campaigns that can disrupt services, damage reputations, or target individuals through fake accounts and automated VoIP calls.
Automated Harassment.
GenAI can automate harassment, fabricate identities and transactions, and clone voices for scams or unauthorized access. These applications extend deception across social platforms, financial systems, and voice-authentication settings.
- Automated Harassment.: Automated harassment can produce relentless, potentially untraceable campaigns against businesses, private individuals, and public figures.Tactics include fake social-media accounts, lies, and automated VoIP calls.
- Automated Harassment.: LLMs combined with other GenAI tools can create synthetic personas, fake accounts, and fake transactions for scams and deception.
- Automated Harassment.: At least 3.5 million Wells Fargo accounts were opened using existing customer data without consent, including fake PINs and email addresses.Funds were transferred from legitimate to fraudulent accounts.
- Automated Harassment.: PayPal believed 4.5 million accounts were not legitimate and possibly fraudulent, while fake accounts also enabled spamming, fake reviews, and user-spoofing fraud.
- Automated Harassment.: Generative AI can clone voices and likenesses, enabling convincing fraudulent calls, ransom scams, and potential voice-authentication bypasses.Threat actors can use brief voice samples to generate scripts in a person’s voice.
Opening the Floodgates to Disinformation
LLMs can rapidly generate persuasive false narratives and conspiracy content at scale, raising concerns about their use as powerful misinformation tools. NewsGuard testing found ChatGPT complied with false-narrative requests about 80% of the time.
- LLMs can craft persuasive content that parrots false narratives and conspiracy theories at scale.
- Researchers described ChatGPT as a potentially powerful tool for spreading misinformation after it generated convincing conspiracy-based content.
- ChatGPT can produce convincing content rapidly, conceal its sources, and generate clean variations of disinformation-loaded material within seconds.
- 80% of the time, ChatGPT complied when NewsGuard researchers asked it to produce content based on false narratives.
- When prompted as Alex Jones about the Parkland shooting, ChatGPT falsely claimed that media and government used crisis actors to promote gun control.
All Systems Down
At planetary scale, GenAI could affect economies, democracy, infrastructure, privacy, and individual agency through surveillance, censorship, synthetic realities, and opinion manipulation. Its ability to create immersive environments also introduces risks of psychological and behavioral influence.
- All Systems Down: GenAI misuse may have especially catastrophic consequences in socio-technical systems and infrastructures deployed at planetary scale.The passage identifies potential effects on the economy, democracy, and infrastructure.
- All Systems Down: GenAI-enhanced surveillance could recognize and predict individual behavior with unprecedented accuracy, raising concerns about privacy and individual rights.
- All Systems Down: LLM-assisted content moderation can remove harmful content efficiently, but authoritarian misuse may suppress dissent and curate a single narrative.The passage connects these risks to democratic values.
- All Systems Down: GenAI can construct immersive, indistinguishable AR, VR, and metaverse environments that may manipulate perceptions of reality.
- All Systems Down: Synthetic environments could influence beliefs, emotions, and behaviors through personalized advertisements or virtual narratives promoting agendas and ideologies.
- All Systems Down: The paper emphasizes preserving a discernible boundary between virtual and physical worlds and maintaining agency within synthetic realities.
Systemic Aberrations.
GenAI’s broad capabilities create systemic risks across identity, historical knowledge, medicine, personalization, politics, and information quality. These risks include scalable fabrication, manipulation, fraud, and misleading content.
- Systemic Aberrations: Generative AI’s ability to recreate historical artifacts can also be misused to fabricate objects that mislead historians and collectors.The same restoration capability can support either cultural preservation or historical distortion.
- Systemic Aberrations: GenAI can scale the creation of fake identities, fabricated documents, and fraudulent evidence to quality levels comparable to costly custom-made fakes.These capabilities may enable synthetic personas and seemingly legitimate documentation.
- Systemic Aberrations: AI-generated medical images may support privacy-preserving training, but fabricated images pose risks of misdiagnosis, fraudulent research, and insurance scams.The passage presents medical image generation as both beneficial and hazardous.
- Systemic Aberrations: Personalized GenAI content can be weaponized to spread misinformation, propaganda, and messages that manipulate individual beliefs and behaviors.Personalized curation therefore has both user-experience benefits and malicious applications.
- Systemic Aberrations: Deepfakes and automated subliminal messages could support political misinformation, fabricated events, fake endorsements, and other influence strategies.The paper cites election campaigns, public diplomacy, and automated subliminal messaging as sensitive use cases.
- Systemic Aberrations: AI-generated books can lack fact-checking and quality assurance, sometimes presenting hallucinated information to inattentive readers.The paper identifies digital book marketplaces as an information-quality concern.
Recommendations
The paper recommends layered mitigation combining identity and content verification, transparency, monitoring, ethical guidance, education, and risk-benefit analysis. It cautions that many measures primarily address actors willing to comply.
- Recommendations: The recommendations provide a non-exhaustive set of plausible technical and socio-technical approaches for mitigating GenAI abuse.The proposed approaches span authentication, labeling, provenance, watermarking, monitoring, and governance.
- Recommendations: These approaches would work for complying actors rather than entities with mischievous or illicit intent.The paper therefore frames mitigation as requiring additional robust strategies beyond voluntary compliance.
- Recommendations: Responsible deployment should include continuous output monitoring, ethical guidelines, stakeholder education, transparency, and pre-deployment risk-benefit analysis.The analysis should weigh potential advantages against possible harms across short- and long-term implications.
- Recommendations: A balanced approach combines proactive mitigation and thorough risk-benefit analysis to preserve digital and physical-world integrity while pursuing innovation.The conclusion frames this balance as necessary for using generative AI constructively.
- Recommendations: Proof of identity can link AI-generated content or actions to legitimate sources through methods such as multifactor authentication, biometrics, or digital certificates.Authentication protocols can likewise verify whether AI-generated content, actions, or requests are legitimate.
- Recommendations: Audience disclaimers and content labels can disclose algorithmic production and distinguish AI-generated material from human-generated content.The paper presents transparency as a way for consumers to assess content more critically.
- Recommendations: Source verification, provenance tracking, and digital watermarking can help preserve integrity, establish traceability, and detect alterations to AI-generated content.Blockchain is offered as one way to trace provenance, while watermarking can distinguish generated content and reveal tampering.
CONCLUSIONS
The paper examines how GenAI and LLMs’ transformative capabilities can enable malicious applications affecting cybersecurity, ethics, and society. It synthesizes risks including misinformation, malicious content, malware, botnets, and radicalizing material, and calls for mitigation, ethical guidance, monitoring, and balanced deployment.
- CONCLUSIONS: GenAI and LLMs offer capabilities in natural language processing and multimodal content generation, but their proliferation has created malicious applications threatening cybersecurity, ethics, and society.The conclusion presents this dual nature as the paper’s central framing.
- CONCLUSIONS: The paper explores misuse in misinformation campaigns, malicious content that can bypass traditional security filters, and sophisticated malware attacks involving LLMs as intermediaries.Its scope centers on harmful applications of generative AI, with special emphasis on LLMs.
- CONCLUSIONS: It also examines societal implications including AI-powered social-media botnets and the generation of harmful or radicalizing content.These examples extend the analysis from technical misuse to broader social effects.
- CONCLUSIONS: The findings underscore the need for robust mitigation strategies, ethical guidelines, and continuous monitoring for responsible GenAI and LLM deployment.The paper aims to raise awareness of the technologies’ dual-edge nature while safeguarding against nefarious applications.