Source-linked AI summary
LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?
David Glukhov, Ilia Shumailov, Yarin Gal, Nicolas Papernot, Vardan Papyan
TL;DR
LLM defenses against misuse remain fallible, and the limits of external censorship mechanisms are unclear. The paper defines censorship and analyzes semantic censorship and Mosaic attacks, concluding that censorship challenges extend beyond an ML detection problem and should be approached as a security problem.
Problem
LLM defenses against misuse have been bypassed, while the limitations of external censorship mechanisms remain unclear.
Source: §1 · p. 1Method
The paper defines censorship, connects semantic censorship to undecidability and invertible transformations, and describes Mosaic Prompts that combine permissible outputs into impermissible ones.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors conclude that censorship should be treated as a security problem, adapting security methods to recognize, control, and mitigate potential risks.
Source: §6 · p. 9The analysis excludes censorship based on internal LLM representations and does not consider global methods; its strict output-censorship impossibility result does not apply to global censorship.
Source: §4.2 · p. 8
Abstract
from arXiv · showhide
Large language models (LLMs) have exhibited impressive capabilities in comprehending complex instructions. However, their blind adherence to provided instructions has led to concerns regarding risks of malicious use. Existing defence mechanisms, such as model fine-tuning or output censorship using LLMs, have proven to be fallible, as LLMs can still generate problematic responses. Commonly employed censorship approaches treat the issue as a machine learning problem and rely on another LM to detect undesirable content in LLM outputs. In this paper, we present the theoretical limitations of such semantic censorship approaches. Specifically, we demonstrate that semantic censorship can be perceived as an undecidable problem, highlighting the inherent challenges in censorship that arise due to LLMs' programmatic and instruction-following capabilities. Furthermore, we argue that the challenges extend beyond semantic censorship, as knowledgeable attackers can reconstruct impermissible outputs from a collection of permissible ones. As a result, we propose that the problem of censorship needs to be reevaluated; it should be treated as a security problem which warrants the adaptation of security-based approaches to mitigate potential risks.
1 Introduction
The introduction frames LLM censorship as a response to misuse risks and fallible defenses, then argues that semantic censorship has inherent limits and that Mosaic attacks can extend the challenge to other censorship methods. It proposes treating censorship as a security problem rather than an ML problem.
1 Introduction
- LLMs’ expanded capabilities and instruction following have raised misuse concerns, while fine-tuning and external censorship mechanisms have been empirically bypassed.The cited examples include social engineering and data exfiltration risks.Source: §1 · p. 1
- The paper defines censorship as regulating model inputs or outputs against provider-chosen semantic or syntactic constraints.It formalizes a mechanism as transforming a string into a permissible one when necessary.Source: §1 · p. 2
- Semantic output censorship has inherent limits: an impermissible string and its invertible transformation can be indistinguishable by permissibility.The paper connects the difficulty to undecidability and argues that a model output may be either permissible or an invertible transformation of an impermissible string.Source: §1 · p. 2
- Mosaic Prompts can assemble impermissible outputs from individually permissible components requested in separate contexts.The example is ransomware code built from benign functions; a user may also obtain the compositional structure from the model itself.Source: §1 · p. 2, Figure 1 · p. 2
- The authors argue that censorship should be treated as a security problem, with approaches such as access controls and user monitoring helping control vulnerabilities.They call for reevaluating how censorship and LLM safety and security are approached.Source: §1 · pp. 1, 3
2 Limitations of Semantic Censorship
The section shows that semantic censorship of program-like content faces theoretical limits: Rice’s theorem rules out general permissibility decisions for Turing-machine descriptions, while invertible string transformations create an output-censorship impossibility under stated assumptions.
2.1 Undecidability of Semantic Censorship
- For any nontrivial set of permissible languages, Rice’s theorem makes deciding whether a Turing machine’s language belongs to that set undecidable.The result means no general algorithm can determine permissibility for every program description.Source: §2.1 · p. 3
- The Rice’s theorem result applies to program descriptions; bounded inputs and outputs and limited memory constrain its direct fit to real computers.The paper notes that Turing machines do not technically exist in practice, while program-behavior verification remains difficult.Source: §2.1 · p. 4
2.2 Impossibility of Semantic Output Censorship
- The output-censorship result assumes a model can apply user-specified string transformations and that the recipient may know an arbitrary inverse transformation.Under these conditions, invariance requires transformed versions of impermissible outputs to remain impermissible.Source: §2.2 · p. 4
- Under those assumptions, the set of semantically permissible bounded-length strings must be empty or include every string.The proof derives this result by mapping an impermissible string to a permissible output with an invertible transformation.Source: §2.2 · p. 4
- When a user’s knowledge of the inverse transformation is unknown, verifying an output’s nontrivial semantic permissibility is impossible.The practical attack setting also includes transformations that do not require prior knowledge of permissible outputs.Source: §2.2 · pp. 4–5
- Restrictive syntactic censorship can limit outputs to a pre-approved set, but this restricts model usefulness and does not ensure semantic permissibility.Invertible transformations may make semantically impermissible strings syntactically permissible.Source: §2.2 · p. 5
3 Mosaic Prompts: A Strong Attack on Censorship
Mosaic Prompts combine individually permissible outputs to reconstruct impermissible strings, challenging censorship that evaluates each string in isolation. The paper outlines practical prerequisites and a worst-case attack, while noting that strong restrictions on inputs or outputs can constrain the attack or make verification costly.
3 Mosaic Prompts: A Strong Attack on Censorship
- Mosaic Prompts combine permissible outputs into impermissible strings, and defending against them can be futile when censorship sees only individual strings.Exceptions may require greatly restricting possible model inputs.Source: §3 · p. 5, §3.2 · p. 6
3.2 General Formulation of Mosaic Prompt
- Compositionality lets a user reconstruct an impermissible string from permissible expressions and a known structure.The general formulation represents the string as w = S({e1, e2, . . . , en}).Source: §3.1 · p. 6, §3.2 · p. 6
3.1 An Example of a Mosaic Prompt
- In the keylogger example, three separately requested components are composed into a malicious program, although each component is generally benign.The components are a phishing email, a keystroke-logging function, and a function that emails the logged data.Source: §3.1 · p. 6
3.2 General Formulation of Mosaic Prompt
- Separate contexts can hide the broader structure from censorship, which checks only whether each individual string is permissible.Mosaic attacks may evade input censorship when the inputs needed for permissible outputs are themselves permissible; stringent input rules are an exception.Source: §3.2 · p. 6
3.3 Extent of Limitations
- The bit-by-bit worst case assumes the model can encode strings and follow conditional instructions, capabilities the authors say current models may not perform perfectly.The construction considers a setting with only two permissible outputs.Source: §3.3 · pp. 6–7
- With more than one permissible output, repeated bit queries can recover a complete impermissible string, under assumptions about string transformations and input censorship.The extreme construction assumes the model can encode strings as bits and follow conditional instructions; the authors note that current models may not perform this perfectly.Source: §3.3 · pp. 6–7
3.4 Practical Limitations
- Mosaic attacks require users to know the target structure and obtain useful permissible subexpressions, and practical risks need case-by-case assessment.A tool such as an email API may itself compose permissible outputs into impermissible behavior.Source: §3.4 · p. 7
- Strong syntactic input and output censorship could reduce the model to a lookup table, but checking all bounded input-output combinations may be unreasonably expensive.The provider would need to construct the combinations and verify their permissibility against its standards.Source: §3.4 · p. 7
4 Discussion
The discussion argues that LLM censorship should be approached as a security problem, not solely as an ML detection task. It recommends context-dependent controls while identifying boundaries of the paper’s results.
4.1 Implications
- The authors argue that semantic censorship by LLM-based detectors is fundamentally misguided and call for a security-oriented approach to controlling risks.They note that censorship should account for the settings and risks of impermissible-content generation.Source: §4.1 · p. 7
- Syntactic censorship may help protect tools or data objects, but Mosaic Prompting can bypass even syntactic censorship.The authors therefore call for context-dependent definitions of impermissibility and suitable security measures.Source: §4.1 · p. 7
- An access-control framework could distinguish trusted from untrusted users or input sources, containing potentially malicious inputs and outputs without censorship.Censorship within such a framework could add functionality but also introduce risks.Source: §4.1 · pp. 7–8
4.2 Limitations of Results
- The paper studies censorship of model inputs and outputs, not methods based on internal LLM representations.The authors note that reliably detecting impermissible behavior with those methods may be hard to guarantee.Source: §4.2 · p. 8
- Global censorship using multiple information sources is not considered; the strict output-censorship impossibility result does not apply to it.The authors state that global methods would still face undecidability and Mosaic Prompting.Source: §4.2 · p. 8
5 Related work
Related work covers prior results on LLM alignment and safety, as well as attacks that exploit model behavior. The paper distinguishes its focus on censorship limitations and generalized attack strategies from those strands.
Limitations of LLM Safety
- Prior work shows that nonzero probability of misaligned outputs can leave prompts with arbitrarily misaligned expected outputs.Other cited results concern sacrifices needed for security against data poisoning and privacy loss from memorization.Source: §5 · p. 8
- Earlier impossibility results address whether a model is safe, robust, or aligned, while this paper studies inducing impermissible behavior in a given LLM under assumptions.Separate work addresses limits on attributing generated text to LLMs.Source: §5 · p. 8
Attacks on LLMs
6 Conclusion
The conclusion distinguishes semantic from syntactic censorship and argues that instruction-following capabilities make semantic output censorship impossible. It extends the concern to attacks that combine permissible outputs and calls for security-based approaches.
6 Conclusion
- The paper formally defines censorship and distinguishes semantic constraints from syntactic ones.Its conclusion argues that censorship should not be treated as an ML problem solvable by improving LLM detectors.Source: §6 · p. 8
- Semantic output censorship can be undecidable, and Mosaic Prompting can compose permissible outputs into impermissible ones.The authors connect these limitations to LLM instruction-following capabilities.Source: §6 · pp. 8–9
- The authors conclude that censorship should be treated as a security problem and call for adapting classical security methods to manage potential risks.They identify recognizing, controlling, and mitigating risks as aims for that adaptation.Source: §6 · p. 9
A Diffie-Helman and One-Time Pad Encryption Attack
The attack uses Diffie-Hellman-Merkle key exchange to establish a shared secret with an LLM, then uses that key for one-time-pad encryption of generated outputs. The encrypted outputs are intended to make determining permissibility very hard, if not impossible.
A Diffie-Helman and One-Time Pad Encryption Attack
- DHM key exchange lets the user and LLM establish a shared secret key without publicly exchanging secret information.The protocol is presented as operating over an insecure communication channel.Source: A Diffie-Helman and One-Time Pad… · p. 11
- DHM’s security depends on the computational hardness of recovering a secret exponent from the exchanged values when the prime is large.Both parties compute the same shared key from the public values and their respective secret exponents.Source: A Diffie-Helman and One-Time Pad… · p. 11
- A stream cipher combines message characters with a secret key through XOR; a uniformly random key at least as long as the message yields one-time-pad encryption.The passage describes one-time-pad encryption as impossible to crack.Source: A Diffie-Helman and One-Time Pad… · p. 12
- The attack presumes the LLM may memorize a secret key planted in training data, for example through poisoning or domain hijacking.The passage cites evidence that LLMs, particularly larger ones, can memorize training data.Source: A Diffie-Helman and One-Time Pad… · p. 12
- The attack produces encrypted outputs without requiring the user to know either the model’s impermissible strings or permissible target strings, making permissibility very hard, if not impossible, to determine.The passage attributes cryptographic guarantees to the generated outputs.Source: A Diffie-Helman and One-Time Pad… · p. 12
B Prompt Templates for Syntactic Censorship
Prompt Templates restrict permissible strings to a small predefined collection, turning censorship into classification over approved templates. The paper describes template vetting and downstream verification as a way to support security guarantees, while noting substantial restrictions on possible inputs.
B Prompt Templates for Syntactic Censorship
- Because misspellings can preserve harmful meaning while bypassing word filters, the section proposes restricting permissible outputs to a predefined set instead.Syntactic filters based on allowed word types are described as unreliable.Source: B Prompt Templates for Syntactic… · p. 12
- Prompt Templates restrict permissible strings to a small predefined collection, so censorship becomes classification over templates and can reject safe strings outside that collection.The approach restricts what a string can be, rather than only what it may contain.Source: B Prompt Templates for Syntactic… · p. 12
B.1 Definition of Prompt Templates
- A Prompt Template uses variable-name tokens as pointers to content inferred or generated from the original uncensored string, which is stored in external memory.The formal definition represents a template as string tokens and variable tokens.Source: §B.1 · p. 12
- The proposed template framework is intended to let other models or tools access uncensored data when needed, including within larger systems.The paper frames this as a setting where the output of those models may be censored.Source: §B.1 · p. 12
B.2 Prompt Template Examples
- Examples include email-task templates with variable slots for recipients, subjects, dates, senders, and requested information.The examples cover user requests and email-summary templates for tasks such as drafting, scheduling, searching, and summarizing.Source: §B.2 · pp. 12–13
- A large template collection can be classified by another LLM, while separating variables for classification or software-tool inference may reduce prompt-injection risk and improve utility.The examples concern user tasks and outputs involving potentially risky information.Source: §B.2 · p. 13
B.3 Practical Implications and Security Guarantees
- Preselected templates can be exhaustively vetted, and their effects on downstream models or tools can be verified to provide security guarantees for individual outputs.Prototype prompts may be found by clustering prior interactions for a use case.Source: §B.3 · pp. 13–14
- Under the stated assumption that users do not already know the impermissible output, the encryption-style attack is unlikely to recover it through a template because the approved set is small.The passage attributes the low probability to the relatively small number of preselected templates among all possible output strings.Source: §B.3 · p. 14
C A Framework for the design of Secure LLM-Based systems
The paper extends access-control models to LLM systems with users, models, tools, and data, using security levels and properties to constrain information flow. It presents censorship as a way to permit otherwise restricted flows while preserving specified security criteria, while noting that censorship itself has undecidability challenges.
C A Framework for the design of Secure LLM-Based systems
- The framework extends classic access-control models to LLM systems with tools, defining entities, actions, security levels, and properties that constrain information flow.It draws on the Bell-LaPadula and Biba models.Source: C A Framework for the design of Secure… · p. 14
- The access-control framework is restrictive on its own; verifiable censorship is presented as enabling otherwise disallowed information flows while preserving desired security criteria.The framework focuses on information integrity as a practical concern in settings involving prompt injection.Source: C A Framework for the design of Secure… · pp. 14–15
- The framework identifies users, tools, and files as system entities, then defines permissions and security levels to regulate their possible actions and access.Models receive a single level that determines their clearance as subjects and classification as tools.Source: C A Framework for the design of Secure… · p. 15
- LLMs can act as both subjects and tools, and may treat instructions embedded in object data as prompts from a subject, a challenge tied to prompt injection.The framework distinguishes prompting from access because LLMs may conflate data and instructions.Source: C A Framework for the design of Secure… · p. 15
- Its security properties restrict tool execution, prompting, data access, updates, and creation according to security levels, and the resulting model reduces to Biba properties that maintain integrity and prevent unauthorized tool use.The paper also states that these properties can be adapted to other criteria, including confidentiality.Source: C A Framework for the design of Secure… · pp. 15–16
- Censorship can create exceptions to some access-control properties, but determining whether tool instructions pose security risks remains difficult because semantic censorship faces undecidability challenges.The passage identifies input censorship for tool instructions and input or output censorship for other properties.Source: C A Framework for the design of Secure… · p. 16
- Prompt Template censorship maps inputs and outputs to an approved whitelist, preventing specified misuse and corruption across security levels.The paper describes censorship as facilitating information flow while maintaining security guarantees.Source: C A Framework for the design of Secure… · pp. 14, 16