PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Large Language Models for Cryptanalysis and Side-Channel Vulnerabilities

Utsav Maskey, Chencheng Zhu, Usman Naseem

arXiv:2505.24621v3cs.CL

TL;DR

The paper addresses limited evaluation of LLM cryptanalysis, a security-relevant question given encoding-related generalization concerns. It introduces a benchmark and assesses LLM decryption and comprehension across ciphertexts. The findings indicate that models comprehend and decrypt only obfuscation methods seen in pre-training, while partial comprehension may leave them susceptible to generalization-based attacks.

  • Problem

    LLM cryptanalysis remains underexplored, despite concerns that encoding-related generalization failures can be exploited in jailbreak attacks.

    Source: §1 · p. 1

  • Method

    The paper introduces a diverse plaintext–ciphertext benchmark and evaluates LLM decryption, comprehension, and safety implications across zero-shot and few-shot settings.

    Source: §1 · pp. 1–2

  • Results

    LLMs comprehend and decrypt only obfuscation methods occurring in pre-training, and even failed full decryption can leave partial comprehension that may enable generalization-based attacks.

    Source: §6 · p. 5, §8 · p. 9

  • Takeaways & Limitations

    The benchmark and findings provide directions for analyzing LLM safeguards and considerations for aligning LLMs toward safety.

    Source: §8 · p. 9

    Evaluation covered a specific range of encryption schemes and general English texts, leaving other schemes and harmful adversarial texts for further exploration.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Recent advancements in large language models (LLMs) have transformed natural language understanding and generation, leading to extensive benchmarking across diverse tasks. However, cryptanalysis - a critical area for data security and its connection to LLMs' generalization abilities - remains underexplored in LLM evaluations. To address this gap, we evaluate the cryptanalytic potential of state-of-the-art LLMs on ciphertexts produced by a range of cryptographic algorithms. We introduce a benchmark dataset of diverse plaintexts, spanning multiple domains, lengths, writing styles, and topics, paired with their encrypted versions. Using zero-shot and few-shot settings along with chain-of-thought prompting, we assess LLMs' decryption success rate and discuss their comprehension abilities. Our findings reveal key insights into LLMs' strengths and limitations in side-channel scenarios and raise concerns about their susceptibility to under-generalization-related attacks. This research highlights the dual-use nature of LLMs in security contexts and contributes to the ongoing discussion on AI safety and security.

1 Introduction

LLM cryptanalysis is underexplored despite its relevance to security: encoding shifts can expose gaps in models’ generalization, while ciphertext decryption also tests mathematical reasoning and character-level processing. The paper introduces a diverse benchmark and evaluates decryption, comprehension, and safety implications.

1 Introduction

  • Cryptanalysis recovers plaintext from ciphertext without direct knowledge of the encryption process, but decryption challenges LLMs’ numerical reasoning and tokenization.Many encryption schemes operate at character or block level, unlike the word or sub-word tokens on which LLMs are primarily trained.Source: §1 · p. 1
  • LLM generalization failures across encodings can create opportunities for mismatched-generalization attacks, including jailbreak attempts using ciphers or other alternative formats.The paper also connects such failures to long-tailed knowledge and programmatic behaviors such as code injection and virtualization.Source: §1 · p. 1
  • The paper introduces a benchmark pairing diverse plaintexts with encrypted versions and evaluates state-of-the-art LLMs in zero-shot and few-shot settings.The plaintexts span domains, lengths, styles, and topics and include both human-generated and LLM-generated content.Source: §1 · pp. 1–2
  • The study examines how partial decryption and token inflation relate to potential jailbreak attacks, alongside models’ decryption accuracy and comprehension.Its stated contributions include discussion of safety implications arising from LLM generalization abilities.Source: §1 · p. 2

2 Related Work

Prior work applies machine learning to cipher analysis and evaluates LLMs on classical ciphers, while related benchmarks and studies address decryption and jailbreaks. This paper supplements those works with attention to partial comprehension, token inflation, and few-shot evaluation.

2.1 Existing Studies on ML Cryptanalysis

  • Neural networks have been applied to block-cipher distinguishing and key recovery, LWE key recovery, and translation-style decoding or classification of classical ciphers.The cited work includes ResNet-based analysis of Speck32/64, transformer-based work on low-dimensional LWE, and GAN approaches to generating synthetic ciphertexts.Source: §2.1 · p. 2

2.2 Existing LLM Evaluation

  • A prior LLM study reported 77% success unscrambling low-complexity ciphers, while CipherBench and CipherBank also explore LLM decryption, cipher probing, and jailbreaks.The reported success was attributed to subword tokenization, pattern recognition, and reasoning abilities.Source: §2.2 · p. 2
  • This benchmark supplements CipherBench and CipherBank with discussion of partial comprehension, token inflation, and few-shot evaluations on fast-thinking models.The supplied dataset-table captions describe samples distributed across text lengths, writing styles, and domains, with plaintexts paired with ciphertexts produced by nine algorithms.Source: §2.2 · p. 2, Table 1 · p. 3, Table 2 · p. 3

3 Dataset

The dataset pairs diverse human- and LLM-generated plaintexts with ciphertexts produced by nine encryption algorithms. Its design varies text length, domain, and writing style to examine model performance across different text properties.

3 Dataset

  • The dataset contains 4,509 entries: 501 plaintext samples encrypted with each of nine algorithms.The plaintext collection includes short and long texts, writing-style sets, domain sets, literature, and quotes.Source: §3 · p. 3

3.1 Text Length

  • Short texts contain up to 100 characters, while long texts contain approximately 300 characters.ChatGPT and Claude were used to generate texts of varying lengths, with balanced representation of short and long samples.Source: §3.1 · p. 3

3.2 Domains

  • The domain set covers scientific, medical, news-headline, technical, social-media, legal, business, literature, and common-English-quote texts.The stated aim is to assess adaptability across content types LLMs are capable of producing.Source: §3.2 · p. 3

3.3 Writing Style

  • Shakespearean and dialect texts provide less familiar writing styles for evaluating robustness to uncommon linguistic structures.The paper says these human-written datasets were used to avoid inherent bias during dataset generation.Source: §3.3 · p. 3

4 Methodology

The methodology encrypts plaintexts using a range of schemes and asks LLMs to decrypt them. Evaluation uses zero-shot and few-shot prompting with chain-of-thought, examining reconstruction, comprehension, and identification of the encryption method.

4.1 Encryption

  • The encryption workflow transforms plain text into encrypted ciphers for LLM decryption.The paper describes substitution as letter replacement, transposition as character rearrangement, and encoding as conversion to another format.Source: §4.1 · p. 3, Figure 1 · p. 4
  • The same 501 samples are encrypted with all nine schemes, maintaining equal representation across text domains, styles, and lengths.The methods include substitution, transposition, and encoding; the paper also includes medium- and hard-difficulty schemes alongside simpler obfuscations.Source: §4.1 · pp. 3–4

4.2 Decryption / LLM Cryptanalysis

  • LLMs receive ciphertexts in zero-shot and few-shot settings with chain-of-thought prompting, reflecting adversarial scenarios where fine-tuning is inconvenient.Models are expected to rely on prompts and their inherent knowledge without further guidance.Source: §4.2 · p. 4
  • The evaluation asks models to reconstruct plaintext, demonstrate partial comprehension, and identify the encryption method used.Method identification is assessed independently of whether full or partial decryption succeeds.Source: §4.2 · p. 4

5 Experimental Setup

The setup evaluates a mix of LLMs with zero-shot and few-shot prompts, using multiple text-generation and cryptography metrics to assess decryption.

5 Experimental Setup

  • The study evaluates one reasoning and six non-reasoning LLMs, including open-source and proprietary models, with temperature 0 and a 1536-token output limit.Table 4 lists the models, implementation details, and estimated sizes.Source: §5 · p. 4, Table 4 · p. 4
  • Few-shot prompts include nine examples, with one encryption-decryption pair for each encryption method.The study also uses a zero-shot prompt; the prompts are generic decryption prompts.Source: §5 · p. 4
  • The prompting approach is classified as Level 3 in TELeR, with detailed, multi-step instructions asking for explanations of the thought process.Source: §5 · p. 4
  • Decryption is assessed with BLEU, BERT Score, Exact Match, and Normalized Levenshtein Similarity.Table 5 reports overall zero-shot performance using Exact Match, BLEU, and Normalized Levenshtein Similarity across cipher types.Source: §5 · p. 5, Table 5 · p. 5

6 Experimental Results and Analysis

Across the tested ciphers, LLMs struggled with Medium and Hard methods, while performance on easier methods varied by model and cipher. The results also show that partial comprehension, tokenization, text length, writing style, and domain affect decryption or cipher identification.

Overall decryption performance

  • All models struggled with Medium and Hard ciphers; Claude Sonnet led on most easier schemes, while GPT-4o and GPT-4o-mini underperformed on Atbash and Bacon.Atbash uses alphabet reversal, while Bacon encodes each character as a five-character sequence.Source: §6 · p. 5

Partial comprehension and safety

  • GPT models could recognize Atbash patterns but produced imprecise decryptions, showing that comprehension did not ensure exact output and could still expose long-tail attack vulnerabilities.The authors attribute this likely to under-generalization and argue that safeguards should address partial comprehension.Source: §6 · pp. 5–6

Few-shot performance

  • Few-shot examples improved decryption most for simpler ciphers: GPT-4o’s Caesar EM rose from 0.66 to 0.90 and BLEU from 0.82 to 0.98.Gains were minimal to none for more complex ciphers, where most models’ EM remained near 0; o4-mini did better than GPT-4o on Atbash but remained below Claude-3.5 Sonnet.Source: §6 · p. 6

Tokenization and generalization

  • Token inflation and pretraining exposure help explain cipher-specific differences: Bacon’s encryption produced 7.93× more tokens and was difficult, while Morse remained learnable despite 6.9× inflation.The paper attributes Morse’s relative success to familiar dot-dash patterns in pretraining; it identifies token inflation as relevant to RSA as well.Source: §6 · p. 7

Length and writing style

  • Longer texts generally reduced exact-match accuracy, while BLEU and NL stayed relatively consistent; Shakespearean style lowered EM for Sonnet and GPT models.The EM declines for Shakespearean texts were 0.09 for Sonnet, 0.16 for GPT-4o, and 0.13 for GPT-4o Mini; Claude’s EM changed by -0.01 with length, versus -0.10 for GPT-4o and -0.19 for GPT-4o-mini.Source: §6 · pp. 7–8, Figure 2 · p. 7

Text domains

  • Across nine domains, Sonnet’s EM exceeded 0.35, GPT-4 variants ranged from 0.13–0.22 on medical and social-media content, and other models rarely exceeded 0.05.Gemini and Mistral models maintained BERT scores of 0.80–0.83 despite low EM.Source: §6 · p. 8, Table 8 · p. 8

Cipher identification

  • GPT-4o and Claude Sonnet had the strongest cipher-identification results, improving from zero-shot F1 scores of 0.43 and 0.37 to few-shot scores of 0.69 and 0.66.The authors frame cipher identification as interpretative rather than traditional classification and note its potential security risks in malicious prompts and jailbreaks.Source: §6 · p. 9, Table 10 · p. 9

7 Discussion

The discussion interprets benchmark scores as reflecting both comprehension and vulnerability to exploitation. Qualitative examples show that models can reconstruct sentence structure yet make consequential decryption errors.

7 Discussion

  • Higher benchmark scores indicate greater ability to comprehend and decrypt transformed text, while lower scores generally indicate greater resistance to exploitation.The analysis considers EM, NL, and BLEU scores.Source: §7 · p. 9
  • For safety-aligned models, Maskey et al. (2025) suggest high decryption scores for benign texts and low scores for prompts intended to elicit harmful responses.The evaluation is intended to guide safeguards that address partial comprehension concerns.Source: §7 · p. 9
  • Qualitative examples show accurate sentence reconstruction can coexist with a missed critical word, while other errors alter a name or substitute “patience” for “patients.”These examples indicate fragile lexical replication despite strong overall comprehension.Source: §7 · p. 9

8 Conclusion

The conclusion presents a benchmark and evaluation framework for LLM cryptanalysis. It reports partial comprehension even when models fail to fully decrypt complex ciphers and connects this behavior to potential jailbreaks.

8 Conclusion

  • The paper introduces a benchmark dataset and evaluation framework for assessing LLM cryptanalysis capabilities on encrypted texts.The conclusion describes these as tools for evaluating model capabilities.Source: §8 · p. 9
  • Even when LLMs cannot fully decrypt complex ciphers, they exhibit some partial comprehension and may be susceptible to generalization-based attacks.The authors discuss token inflation and partial decryption in relation to potential jailbreaks.Source: §8 · p. 9
  • The authors present their findings and evaluation methods as directions for analyzing safeguards and considerations for aligning LLMs toward safety.Source: §8 · p. 9

Limitations

The authors identify limits in cipher coverage, prompting, and text scope, and call for further work on long-tail ciphers, harmful adversarial texts, and model behavior.

Limitations

  • The study evaluates comprehension on general English text; the authors say comprehension on harmful adversarial texts should also be explored.Source: Limitations · p. 10
  • The authors call for further exploration of identifying ciphers or long-tail texts and of the mechanisms behind performance differences across LLMs.They also report improved comprehension of long-tail texts when the tokenizer processes them effectively.Source: Limitations · p. 10
  • The evaluation covers only a specific range of encryption schemes, so other schemes that may pose different challenges could be overlooked.Source: Limitations · p. 10
  • The generic decryption prompt may not be optimal for models with different prompting guidelines, such as Instruct models.Source: Limitations · p. 10

Ethical Considerations

The paper describes steps to limit offensive content and balance data and code release, reviews possible mitigations, and notes a potential safety–performance trade-off.

Ethical Considerations

  • The authors report redacting offensive materials and balancing data and code release to follow responsible-research and ethical standards.Source: Ethical Considerations · p. 10
  • Perplexity filters may flag and drop cipher inputs, while Morse code can avoid these filters.Source: Ethical Considerations · p. 10
  • The discussed mitigation approaches may be effective but can trade safety for performance, potentially affecting overall model utility.Source: Ethical Considerations · p. 10

A.1 Data Generation Prompts

This appendix section provides supplemental documentation and comparisons for the benchmark, covering text-generation and decryption prompts, dataset construction, cipher difficulty and evaluation measures, and analyses of random-baseline bias, cipher classification, and sequence-length effects. It also reports comparisons across writing styles, text lengths, and zero-shot and few-shot performance across text types, along with the metrics used.

A.1 Data Generation Prompts

  • Text-generation prompts requested 50 English samples under 100 characters, 50 around 300 characters, and 500 English samples spanning domains such as medical and scientific.Source: §A.1 · p. 13

A.2 Decryption Prompt

  • The decryption prompt requests method identification, stepwise pattern and frequency analysis, multiple or hybrid-cipher attempts, assumptions, partial answers, and a fixed report; its few-shot version supplies nine examples.Source: §A.2 · p. 13

A.3 Dataset and Statistics

  • The dataset comprises 501 unique plaintexts encrypted with each of nine ciphers, totaling 4,509 pairs, across length, writing-style, domain, literature, and quote subsets.The subsets include 76 short, 68 long, 68 style, 231 domain, 30 literature, and 28 quote samples.Source: §A.3 · p. 14, Table 11 · p. 15

A.4 Decryption Difficulty Analysis

  • The appendix classifies Caesar, Atbash, Morse, and Bacon as trivial; Rail Fence, Vigenere, and Playfair as medium; and RSA and AES as hard, citing key-search difficulty, factoring, and large AES key spaces.Source: §A.4 · p. 14

A.5 Evaluating Metrics

  • Evaluation uses exact match for binary correctness, BLEU for n-gram overlap, BERTScore for semantic similarity, and normalized Levenshtein for character-order similarity and partial-match assessment.Source: §A.5 · p. 15

A.6 Random-guessing baseline (“Lorem Ipsum”)

  • Random text produced NL similarity of approximately 0.18–0.19 across ciphers, while BLEU and EM remained near zero, so NL values near 0.18 mark the incorrect-output floor.The authors rely primarily on EM and BLEU for strict correctness and interpret NL above the random baseline as graded similarity.Source: §A.6 · p. 15, Table 13 · p. 16

A.7 Cipher Classification

  • Few-shot F1 reached 0.69 for GPT-4o and 0.66 for Claude Sonnet, versus zero-shot 0.43 and 0.37; GPT-4o-mini and Gemini reached 0.44 and 0.46, while Mistral gains were comparatively low.Cipher identification was assessed from ciphertext alone within the decryption prompt, and the authors note that few-shot improvements could support obfuscation-based jailbreaks.Source: §A.7 · p. 16

A.8 Text Length Additional Analysis

  • For Claude-3.5 Sonnet, EM fell from 0.42 to 0.37 and NL from 0.55 to 0.36 as length increased; GPT-4o fell from 0.29 to 0.08 EM and 0.46 to 0.16 NL.The comparison is between 30 characters and 500 tokens; the passage reports decreases of 12% and 35% for Claude and 72% and 65% for GPT-4o, respectively.Source: §A.8 · p. 16, Table 14 · p. 16

A.9 Other Tables and Figures

  • Table 15 compares writing styles using Exact Match and Normalised Levenshtein Similarity, with Normal Text averaging other text types.The caption specifies the metrics and the meaning of the Normal Text category but gives no scores.Source: Table 15 · p. 16
  • Table 16 compares LLM performance on short and long texts using Exact Match and Normalized Levenshtein.The caption specifies the text-length categories and metrics but gives no scores.Source: Table 16 · p. 16
  • Table 17 compares zero-shot performance across text types using EM, BLEU, NL, BERT, and LD.The caption does not provide the table’s values or identify a best-performing model.Source: Table 17 · p. 17
  • Table 18 compares few-shot performance across text types using EM, BLEU, NL, BERT, and LD.The caption does not provide the table’s values or identify a best-performing model.Source: Table 18 · p. 17

Related papers

Loading 2505.24621v3…