Source-linked AI summary
Neural Linguistic Steganography
Zachary M. Ziegler, Yuntian Deng, Alexander M. Rush
TL;DR
The paper addresses linguistic steganography that must conceal messages in prevalent, innocuous language while resisting statistical detection. It combines arithmetic coding with modern language models and finds that the resulting system can fool both statistical and human adversaries.
Problem
Linguistic steganography needs a prevalent, innocuous, medium-independent cover signal while avoiding statistical patterns detectable by machine-based eavesdroppers.
Method
The paper combines arithmetic coding with large pretrained language models to map uniformly sampled messages into sequences of words.
Results
The system achieves near-optimal statistical security and generates cover texts that can fool human evaluators, including statistical indistinguishability from human-written sentences at 1 bit/word.
Takeaways & Limitations
Arithmetic coding with large pretrained language models can produce high-quality linguistic steganography that addresses both distributional and single-sample detection.
Takeaways & Limitations
Current language models produce the most convincing cover texts at low compression, where the KL is moderate, and machine-detectable statistical patterns remain possible.
Abstract
from arXiv · showhide
Whereas traditional cryptography encrypts a secret message into an unintelligible form, steganography conceals that communication is taking place by encoding a secret message into a cover signal. Language is a particularly pragmatic cover signal due to its benign occurrence and independence from any one medium. Traditionally, linguistic steganography systems encode secret messages in existing text via synonym substitution or word order rearrangements. Advances in neural language models enable previously impractical generation-based techniques. We propose a steganography technique based on arithmetic coding with large-scale neural language models. We find that our approach can generate realistic looking cover sentences as evaluated by humans, while at the same time preserving security by matching the cover message distribution with the language model distribution.
1 Introduction
Linguistic steganography hides messages in natural-language cover text, exploiting language's prevalence and medium independence. The paper combines arithmetic coding with modern language models to target both statistical security and human-perceived plausibility.
- Language is a useful cover signal because it is prevalent, innocuous, and independent of any single medium.
- Edit-based methods modify human-selected text, whereas generation-based methods generate entire text blocks while reversibly encoding messages in token choices.
- Earlier practical systems were mainly edit-based and encoded only small amounts of information.
- Prior neural generation-based methods improved fluency but used theoretically sub-optimal heuristics and weaker language models.
- The paper combines arithmetic coding with state-of-the-art language models to pursue near-optimal statistical security and human-indistinguishable cover text.
2 Background & Related work
The paper frames steganography through a cover distribution induced by invertible encoding and contrasts information-theoretic security with earlier quality-focused heuristics. Existing heuristic methods may produce fluent text while remaining vulnerable to machine-based detection.
- An invertible mapping transforms a uniformly distributed secret message into natural-language cover text and induces the cover distribution q.
- Security is measured by D_KL(q||P_true), while compression is characterized by the entropy H(q).
- Earlier generative methods use vocabulary bins or stepwise Huffman coding, with quality parameters that trade cover quality against encoded bits.
- These methods prioritize generated quality at maximal compression rather than minimizing KL divergence at maximal compression.
- Because their heuristics can create detectable statistical patterns, they may fool human eavesdroppers while remaining susceptible to machine-based eavesdroppers.
- Concurrent work analyzed KL values for a modified Huffman algorithm but did not perform human evaluation.
3 Arithmetic coding
The proposed steganographic encoder reverses arithmetic coding: it maps a uniformly sampled message into language-model tokens whose conditional probability bins contain the message's fractional representation. Temperature and top-k truncation provide a quality–compression trade-off.
- Arithmetic coding is asymptotically entropy-optimal and can be more efficient than Huffman coding because it avoids blocking.
- For steganography, a uniformly sampled message is selected first and then mapped into a sequence of words using the language-model distribution.
- The secret message is interpreted as a fraction in [0, 1), and encoding reads tokens from successive probability bins until the message is unambiguously defined.
- Decoding reverses encoding by progressively narrowing the possible-message range until the predetermined end token is included at both boundaries.
- For long sequences, arithmetic coding can match a target distribution exactly, yielding D_KL(q||p_s)=0 and H(q)=H(p_s).
- Temperature modulation and top-k truncation trade generation quality against compression when applying the method to pretrained language models.
4 Experiments and results
Experiments evaluate arithmetic coding with GPT-2 on information-theoretic security, human-perceived continuation quality, and text-to-text steganography. Arithmetic coding closely matches the language-model distribution, while human-plausible generations are strongest at lower compression.
- Experimental setup: Experiments use GPT-2 on CNN/DailyMail contexts, generating one sentence from a uniformly random message for each steganography algorithm.Arithmetic temperature, Huffman truncation length, and block size are varied for comparison.
- Metrics: Bits/word measures encoded message bits per sentence length, while KL divergence to the language-model distribution evaluates distributional similarity.Because the true language distribution is unavailable, KL is estimated against pLM for relative algorithm comparisons.
- Human evaluation: Human evaluation asks whether a generated sentence is a likely next sentence given a news-article context, using absolute judgments rather than pairwise comparisons.The study uses 80 generations from each source, randomly mixed into HITs containing five generations.
- Quantitative results: For 1–5 bits/word, arithmetic coding achieves the lowest KL, indicating the encoded-text distribution is closest to the language-model distribution.The unmodulated model reaches KL = 4e-8 nats, consistent with arithmetic coding matching the model distribution.
- Quantitative results: Within 3 bits/word, Huffman and arithmetic generations appear plausible continuations over 50% of the time; at 1 bit/word, both are statistically indistinguishable from human-written sentences.Huffman is slightly better in human judgments, but its transferred distribution differs statistically from natural text.
- Qualitative results: For natural-language messages, arithmetic coding first compresses the message into near-uniform bits and then encodes those bits into cover text using a predetermined context.The receiver reverses these steps; message and cover lengths remain comparable modulo small overhead, while their content can differ substantially.
5 Conclusion
The paper combines large pretrained language models with arithmetic coding to target both statistical and human steganalysis. It demonstrates convincing cover text at low compression, while improved language models could further raise performance.
- Conclusion: Large pretrained language models combined with arithmetic coding can fool statistical adversaries analyzing distributions and human adversaries inspecting individual samples.The conclusion describes this as high-quality, efficient steganography and realizable optimality of arithmetic coding.
- Conclusion: Current systems generate the most convincing cover texts at low compression, where KL divergence remains moderate.Future language-model improvements can be directly incorporated into the arithmetic approach.
A Human evaluation details
The human evaluation tests whether readers accept generated sentences as likely continuations of news-article contexts. It uses randomized sources, explicit contextual and factuality instructions, and quality-control exclusions.
- Sampling: The evaluation samples 80 generations from each of 11 sources, including arithmetic, Huffman, block encoding, and human-written continuations.Steganography messages are sampled uniformly at random, and generations are randomly mixed into HITs with five generations each.
- Judgment procedure: Workers judge whether each generation is a likely next sentence while considering contextual sense, factual correctness, and grammatical correctness.Human judgment is calculated from the workers’ mean response.
- Judgment procedure: The study uses absolute judgments because real viewers see one continuation rather than two alternatives for head-to-head comparison.This design is intended to align the evaluation with real-world use.
- Quality control: HITs failing attention checks, associated with workers who frequently fail them, or completed unusually quickly are excluded from analysis.These filters are applied before calculating results.
B Additional Examples
The examples use uniformly random messages and human-written CNN/DM context to generate one-sentence cover text with arithmetic, Huffman, and block steganography methods.
- The examples use the first three sentences of randomly selected CNN/DM articles as human-written context and uniformly sampled bit strings as messages.
- The generated cover text is truncated after one sentence for the example tables.
- The examples use parameter settings intended to give similar compression efficiencies across the arithmetic, Huffman, and block methods, and the first four generated examples for each model are uncurated.
- Table 2 presents additional examples produced with the arithmetic method.
- Table 3 presents additional examples produced with the Huffman method.
- Table 4 presents additional examples produced with the block method.