Beyond Semantic Manipulation: Token-Space Attacks on Reward Models

1 Introduction

• Because reward models are imperfect surrogates trained on finite, noisy preference data, optimizing them can reward hacking rather than improved task performance.The introduction frames reward hacking as exploiting reward-model vulnerabilities instead of improving the underlying objective.Source: §1 · p. 1
• Prior attacks exploit readable-text cues such as verbosity or sycophancy, often through manually constructed inputs, leaving automatic token-space discovery open.Examples include overly long responses and sycophantic or deceptively persuasive tones.Source: §1 · p. 1
• TOMPA bypasses decode–re-tokenize, mapping policy tokens directly to the reward model and using black-box scalar feedback to discover adversarial sequences.The introduction also reports that such patterns become highly rewarded only at sufficient sequence lengths.Source: §1 · p. 2
• On Skywork-Reward-V2-Llama-3.1-8B, TOMPA earns +33.64 mean reward versus GPT-5's +17.48 and exceeds it on 98.0% of prompts.The paper reports a consistent failure mode across multiple state-of-the-art reward models.Source: §1 · p. 2
• TOMPA's high-reward outputs are gibberish, while a random out-of-distribution baseline receives strongly negative rewards and near-zero beat rates.The random baseline comparison indicates the result is not explained by arbitrary out-of-distribution inputs.Source: §1 · p. 2

2.1 Reward Hacking and Overoptimization in RLHF

• Reward models can diverge under distribution shift despite shared training data, and optimization can amplify their errors into high-reward behavior without better task performance.The section describes this vulnerability as especially relevant when a policy departs from its initialization.Source: §2.1 · p. 3
• Prior RLHF reward hacking exploits semantic cues such as verbosity, sycophancy, and deceptive persuasiveness; LLM judges also respond to simple textual manipulations [Raina et al., 2024; Zhao et al., 2025].Other examples include appended phrases, response-order swaps, and text mimicking social consensus or reasoning structure.Source: §2.1 · p. 3
• Some non-linguistic attacks use structured gibberish or malformed tokens, but these patterns are typically manually designed for specific failure modes.The section positions TOMPA's automated discovery as a departure from such hand-crafted attacks.Source: §2.1 · p. 3

2.2 RL-based Adversarial Attacks on Reward Models

• RL-based red-teaming has searched for dialogue-model failures [Perez et al., 2022], and ARLAS automates adversarial prompt generation [Wang et al., 2025b], but these attacks remain semantic.Adv-RM also uses RL to maximize a target reward model while encouraging disagreement with auxiliary reward models, producing interpretable natural-language inputs [Bukharin et al., 2025].Source: §2.2 · pp. 3–4
• TOMPA differs across optimization, content, and access: it uses RL to find non-linguistic token patterns with only black-box access to the target reward model.Table 1 characterizes it as the only approach combining RL discovery, non-linguistic adversarial content, and black-box access.Source: §2.2 · p. 4, Table 1 · p. 4

3.1 Preliminaries: Group Relative Policy Optimization (GRPO)

• GRPO removes the value network and estimates response advantages by standardizing rewards within each generated group.A reward function assigns each response a scalar score.Source: §3.1 · p. 4
• The GRPO objective uses probability-ratio clipping for stable updates and a per-token KL penalty to limit deviations from the reference policy.The ratio is defined using the current and previous policies’ token probabilities.Source: §3.1 · p. 5
• The method targets unverifiable settings such as creative writing, where reward models replace rule-based verification because no objective ground truth is available.GRPO was originally designed for tasks with verifiable rewards.Source: §3.1 · p. 5

3.2 Token Mapping Perturbation Attack

• TOMPA maps policy-generated tokens directly into the reward model’s vocabulary, bypassing the standard text decoding and re-tokenization interface.The mapped tokens are represented using the reward model’s embedding matrix.Source: §3.2 · p. 5
• TOMPA iteratively generates responses, applies the mapping, queries the reward model through forward passes, and updates the policy with RL.The policy searches for high-reward token patterns without requiring human-readable adversarial inputs.Source: §3.2 · pp. 5–6

4.1 Experimental Setup

• Evaluation used 100 NoveltyBench prompts, while training sampled 10,000 WildChat prompts across two policy–reward-model tokenizer pairings.Training used 10 epochs, a batch size of 64, 8 rollouts per prompt, a 2048-token response limit, and a 10^-6 learning rate.Source: §4.1 · p. 6

4.2 Quantitative Results

• The attack achieved a 98.0% beat rate against both state-of-the-art reward models; against Llama-3.1-8B, its mean reward was +33.64, nearly doubling the gold-answer score of +17.48.The Random OOD baseline yielded strongly negative mean rewards and near-zero beat rates across both settings.Source: §4.2 · p. 7, Table 2 · p. 7

4.4 The Role of Response Length in Reward Hacking

• Rewards were low or negative below 1024 tokens, then Llama rose from near +7 at 1792 to over +30 at 2048, indicating length-dependent scoring failures.The responses were progressively truncated for this inference-time ablation.Source: Figure 3 · p. 8, §4.4 · pp. 8–9

4.3 Qualitative Results

• Decoded attacks collapse into repetitive non-linguistic patterns: cross-lingual and code fragments for Qwen3, and reserved tokens for Llama.Unlike semantic manipulation that remains human-readable, these token-space patterns need not preserve syntax or meaning.Source: Figure 2 · p. 7, §4.3 · p. 8

4.5 Optimization Dynamics of Reward Hacking

• Across 1,500 steps, training moves from strongly negative rewards toward high-reward patterns and surpasses the respective GPT-5 reference scores (+8.12 and +17.48).The Llama-targeted run grows slowly during the first 500 steps, then surges once high-reward token combinations are identified.Source: Figure 4 · p. 9, §4.5 · p. 9

• TOMPA applies token mapping perturbation at the policy–reward-model interface and optimizes adversarial sequences directly in token space.Source: §5 · p. 9
• The attack consistently outperforms GPT-5 reference answers on state-of-the-art reward models while generating entirely nonsensical text.Source: §5 · p. 9
• The authors plan to study improving reward-model robustness through methods such as adversarial training.Source: §5 · p. 9