Source-linked AI summary
Nonsense Helps: Prompt Space Perturbation Broadens Reasoning Exploration
Langlin Huang, Chengsong Huang, Jinyuan Li, Donghong Cai, Yuyi Yang, Jiaxin Huang
TL;DR
GRPO can lose its training signal when all sampled responses to a hard question fail, while ordinary resampling may remain trapped by the current policy’s reasoning paths. The paper proposes LoPE, which prepends task-irrelevant Lorem Ipsum sequences before resampling; it reports consistent gains across model sizes and finds that effective perturbations are Latin-based and low-perplexity.
Problem
GRPO’s relative advantages collapse when every rollout fails, and increasing the sampling budget may still have a low success rate on difficult questions.
Method
LoPE prepends randomly assembled Lorem Ipsum sequences to prompts for failed-question resampling within GRPO training.
Results
LOPE consistently improves mathematical reasoning performance, with average gains of +2.79 points on Qwen3-1.7B-Base, +4.62 points on Qwen3-4B-Base, and +6.20 points on Qwen2.5-Math-7B.
Takeaways & Limitations
Effective prompt perturbations are Latin-based and relatively low-perplexity, supporting LoPE as a baseline for broadening exploration in LLM reinforcement learning.
Takeaways & Limitations
Standard resampling is unlikely to significantly improve success when the current generation policy cannot solve a difficult question within its initial rollouts.
Abstract
from arXiv · showhide
Reinforcement learning with verifiable rewards, particularly Group Relative Policy Optimization (GRPO), has significantly advanced the reasoning capabilities of Large Language Models (LLMs). However, in complex tasks, GRPO frequently suffers from the ``zero-advantage problem'': when all sampled rollouts for a query fail, the relative advantage collapses to zero. Consequently, the model loses effective training signals for these questions, wasting the training data and computational budget. While simply increasing the sampling budget for these questions is a common remedy, the static sampling policy inherently constrains reasoning exploration, limiting the success rate. In this paper, we propose Lorem Perturbation for Exploration (LoPE), a simple yet effective training framework to break this exploration bottleneck. We posit that task-irrelevant prompt-space perturbations can shift the model's output distribution enough to unlock orthogonal reasoning pathways for hard questions. Specifically, LoPE prepends sequences stochastically assembled from Lorem Ipsum vocabulary (a pseudo-Latin placeholder text) to the prompts before resampling. Experiments across 1.7B, 4B, and 7B models demonstrate that LoPE significantly outperforms resampling with the original prompts. Further analysis reveals that other Latin-based random sequences with low perplexity are also effective perturbations. Our results establish LoPE as a strong baseline for broadening exploration in LLM reinforcement learning.
1 Introduction
GRPO loses training signal when every rollout for a hard question fails, and simply resampling the original prompt may remain ineffective. LoPE addresses this bottleneck by prepending task-irrelevant Lorem Ipsum perturbations before resampling, improving exploration and training outcomes.
- When all sampled responses fail, GRPO’s relative advantages collapse to zero, wasting training data and rollout computation.
- Increasing the sampling budget gives hard questions more attempts but can still produce a low resample success rate under the current policy.
- LoPE prepends randomly sampled Lorem Ipsum vocabulary to failed-question prompts, creating a task-irrelevant prompt-space perturbation before resampling.
- +2.79 points on Qwen3-1.7B-Base, +4.62 points on Qwen3-4B-Base, and +6.20 points on Qwen2.5-Math-7B were the reported average gains across mathematical reasoning benchmarks.
- Other effective perturbations use pseudo-Latin vocabularies and maintain low perplexity, supporting LoPE as a baseline for broadening exploration.
2 Background: Group Relative Policy Optimization (GRPO)
GRPO improves reasoning by comparing multiple responses sampled for each question and normalizing their rewards into relative advantages. When every response fails, this normalization produces no useful gradient for the question.
- GRPO samples a group of G responses for each query from the old policy and uses their relative correctness for optimization.
- Unlike PPO-based approaches, GRPO does not require an explicit reward model and instead leverages relative rewards within each sampled group.
- The importance sampling ratio compares the current policy probability with the old policy probability for each response token.
- GRPO normalizes each response’s scalar reward within its group to compute the rollout-level advantage.
- If every sampled response fails, the reward vector is zero and every advantage becomes zero, yielding a zero-gradient training batch.
3 The Limitation of Logit-Space Exploration
Standard resampling and higher-temperature sampling may not escape the model’s local reasoning basin on difficult questions. LoPE instead perturbs the prompt with meaningless pseudo-Latin text, producing distinct exploration patterns and recovering additional solutions.
- After all initial rollouts fail, standard resampling is unlikely to substantially improve success on questions difficult under the current generation policy.
- The comparison tests naive prompting at temperature 0.6, naive prompting at temperature 1.2, and Lorem-perturbed prompting at temperature 0.6.
- Lorem Ipsum supplies meaningless pseudo-Latin text that mimics natural-language structure without conveying semantic content; words are sampled from a pool of 63 Latin words.
- 50 of 352 hard questions were uniquely resolved by Lorem-perturbed responses, while neither the base nor high-temperature methods resolved them.
- Base prompting concentrates responses near zero entropy and perplexity one, whereas Lorem perturbation removes the near-zero entropy spike and slightly increases uncertainty.
- High-temperature sampling produces much higher entropy and perplexity, which can hurt reasoning quality and accuracy.
4 Lorem Perturbation for Exploration (LoPE)
LoPE resamples questions whose original rollouts all fail using randomly prepended Lorem Ipsum perturbations, then integrates successful responses into training while preserving context alignment.
- Rollout with Perturbation: LoPE triggers perturbed resampling only when all G responses generated under the naive prompt fail.The perturbation is a random Lorem Ipsum sequence prepended to the original prompt.
- Regroup LLM Responses: Resampled responses are combined with failed original responses, replacing up to G −1 failures while retaining at least one incorrect response.This preserves a group size of G and keeps relative advantages non-zero.
- Pseudo Rollouts: LOPE pairs resampled outputs with the naive prompt during training to align response contexts and reduce biased advantage estimation.Because resampled responses are generated off-policy, LOPE applies an importance-sampling correction.
- Training Procedure: LOPE removes KL regularization because the constraint could counteract the distributional shifts intended to broaden exploration.The method otherwise follows a GRPO-like training procedure with perturbed resampling for zero-advantage cases.
5 Training Signal Shaping
Training signal shaping addresses two problems introduced by LOPE’s off-policy resampling: suppressed gradients for unlikely tokens and biased advantages from selecting only some responses.
- Overview: Training signal shaping combines policy shaping and advantage shaping to mitigate importance-sampling and advantage-estimation problems.The two components target different distortions created by pseudo-rollout training.
- Policy Shaping: Policy shaping amplifies gradients for low-probability tokens while constraining gradient magnitude for high-probability tokens.This prioritizes rare reasoning steps that the naive policy would otherwise underweight; γ is set to 0.1.
- Advantage Shaping: Advantage shaping computes advantages over all G + G′ responses rather than only the G selected responses.The discarded responses are mostly failures, so excluding them understates question difficulty and suppresses positive advantages.
- Advantage Shaping: Advantage shaping amplifies positive advantages by a factor of 2.1 to 5.0.The analysis quantifies the effect of shaping on rare successful responses.
- Full Training Objective: The complete objective combines standard GRPO updates on original rollouts with policy-shaped updates for resampled responses.The formulation incorporates the shaped training signals into a single LOPE objective.
6 Experiment
LOPE is evaluated across three base models and multiple mathematical reasoning benchmarks against GRPO and naive-prompt resampling. It achieves the highest average performance overall and maintains stronger training-time question-level exploration.
- Experimental Setup: The evaluation uses MATH-500, GSM8K, AMC, AIME 2024, and AIME 2025 with model-specific accuracy metrics.Training uses OpenR1-Math-46k-8192, with G = 8 and G′ = 24 for the resampling comparison.
- Main Results: LOPE with training signal shaping consistently outperforms GRPO and naive-prompt resampling across model scales.The comparison covers Qwen3-1.7B-Base, Qwen3-4B-Base, and Qwen2.5-MATH-7B.
- Successful Training-Time Exploration: LOPE achieves a significantly higher question-level success rate than naive resampling while achieving comparable response-level accuracy.This pattern is observed during Qwen3-1.7B-Base training, with related observations for the larger models in Appendix B.
- Successful Training-Time Exploration: LOPE maintains a higher resample success rate throughout GRPO training and provides effective signals on more questions.Figure 4 tracks question-level pass@G′ and response-level Mean@G′ for Qwen3-1.7B-Base.
7 Analysis: What Makes a Good Prompt Space Perturbation?
The analysis identifies effective prompt-space perturbations as language-like, low-perplexity noise that shifts generation while preserving question comprehension. Excessively noisy or English-based perturbations can distort understanding or interfere with the reasoning context, reducing performance.
- 7.1 Exploring Alternative Prompt Space Perturbation: The study compares randomly generated perturbations including fake English, ASCII, and token sequences, using lengths uniformly sampled between 100 and 300 tokens.The analysis also evaluates Lorem-based and other language-model-generated perturbations.
- 7.2 How LLMs Perceive Prompt-Space Noise: Perplexity analysis evaluates 500 sequences of 200 tokens per method, using mean perplexity as distributional distance and standard deviation as perturbation-strength consistency.Question text from the 500-question evaluation subset provides the natural-language reference.
- 7.2 How LLMs Perceive Prompt-Space Noise: Near-natural perturbations have mean perplexity below 100, including Lorem Ipsum at 25.12 versus 4.82 for the Question Text reference.Filtered Latin Natural Language, Latin Unigram Model, English Unigram Model, and Latin 3-Gram Model also fall below 100.
- 7.2 How LLMs Perceive Prompt-Space Noise: Lorem Ipsum combines the lowest synthetic mean perplexity with a tightly concentrated distribution, reporting standard deviation 2.84.Other low-mean perturbations can have substantially wider dispersion, including Filtered Latin Natural Language with standard deviation 42.63.
- 7.2 How LLMs Perceive Prompt-Space Noise: Entropy distributions for most perturbations overlap with Question Text, while Random Tokens show a significant right-shifted deviation.This token-level analysis measures average entropy across constituent question tokens.
- 7.2 How LLMs Perceive Prompt-Space Noise: Near-natural or moderately out-of-distribution perturbations preserve semantic representations near the Question Text, whereas Random Token representations drift far away.The representation analysis uses 10 questions solved by all methods within 8 rollouts, with t-SNE applied to mean-pooled final-layer question representations.
- 7.2 How LLMs Perceive Prompt-Space Noise: Excessively high-perplexity perturbations can corrupt question understanding and hinder discovery of correct solutions during reinforcement-learning training.The analysis concludes that low-perplexity, near-natural perturbations shift generation without corrupting the core question representation.
- 7.3 The Recipe for an Effective Perturbation: The strongest perturbations are Latin-based and low-perplexity: Filtered Latin Natural Language and Latin Unigram Model exceed average score 39.6, while English-based noise can interfere with English reasoning context.The results support pseudo-Latin vocabularies and low perplexity as the defining characteristics of effective perturbations.
8 Related Work
Prior work addresses zero-reward signals in GRPO-style reinforcement learning through rollout and guidance strategies, while related studies show that prompt context and meaningless tokens can alter model behavior.
- Zero-Advantage Recovery in RLVR: Related reinforcement-learning work targets zero-reward signals on hard prompts through adaptive budgets, targeted exploration, scaffolded hints, and off-policy guidance.These approaches focus on improving rollout efficiency and recovering useful training signals.
- Context-level Perturbation: Context-level studies show that prompt context, formatting, demonstrations, and instruction structure can substantially influence model generation and reasoning performance.The cited work includes both implicit context effects and prompting sensitivity.
- Context-level Perturbation: More recent studies report that synthetic or meaningless tokens can shift model activations and alter reasoning behavior.This connects prompt-space perturbation with broader evidence that non-semantic context changes affect generation.
9 Conclusion
LOPE prepends randomly generated Lorem Ipsum to prompts during GRPO resampling to broaden exploration. The paper reports consistent gains and finds that effective perturbations are Latin-based and low-perplexity.
- 9 Conclusion: LOPE prepends a randomly generated Lorem Ipsum sequence to the naive prompt, enabling exploration of alternative reasoning trajectories for previously failed questions.The method is used during GRPO training as a prompt-space perturbation strategy.
- 9 Conclusion: Systematic comparisons indicate that effective perturbations are Latin-based and have relatively low perplexity, highlighting controlled perturbation as important for improving exploration.The conclusion states that LOPE's success is not isolated to a single perturbation format.
Ethics Statement
The ethics statement identifies safety and controllability risks from automatically generated or excessively strong perturbations. It emphasizes controlled, language-like noise while leaving safety and robustness as open challenges.
- Ethics Statement: Automatically generated perturbation sequences are not guaranteed to be free of toxic, biased, or inappropriate words or expressions.The risk follows from generating the sequences automatically.
- Ethics Statement: Excessively strong perturbations may produce incoherent outputs and, in some cases, undesirable or harmful content.The paper identifies reduced controllability over model outputs as a corresponding limitation.
- Ethics Statement: Moderate, language-like perturbations are reported as less likely to negatively affect behavior, but ensuring their safety and robustness remains an open challenge.The authors emphasize controlled perturbation as a mitigation approach.
- Ethics Statement: Future work will study how perturbation types and strengths affect behavior while seeking to reduce toxic or harmful outputs without losing exploration benefits.The stated goal is to preserve improved exploration while minimizing safety risks.
B Training-Time Resample Accuracy for Qwen3-4B-Base and Qwen2.5-Math-7B
During training, LoPE achieves a significantly higher question-level resample success rate than the compared original-prompt resampling approach on both evaluated models.
- LoPE consistently achieves a significantly higher question-level resample success rate during training.The comparison covers Qwen3-4B-Base and Qwen2.5-Math-7B.
C.1 Policy Shaping
Policy shaping modifies resampling gradients to emphasize low-probability tokens while bounding gradient peaks, and advantage shaping further increases the training weight of rare correct trajectories on hard questions.
- Policy shaping: Policy shaping applies f(x) = x/(x + γ) to reshape gradients in the resampling objective.The shaping function is introduced to address limitations of vanilla and GRPO-clipped gradient bounds.
- Policy shaping: The shaping gradient peak shifts from πθ = 1/2 toward the low-probability regime, emphasizing unfamiliar rewarded tokens.The peak location is approximately π⋆θ ≈ γπθold.
- Policy shaping: The shaped peak remains bounded by 1/4, avoiding the vanilla bound’s unbounded growth as πθold →0.GRPO clipping instead truncates gradients to zero in the relevant region, sacrificing those learning signals.
- Policy shaping: The shaped weighting assigns greater importance to low-probability effective actions and attenuates updates for tokens the model has mastered.This reweights parameter updates across the probability spectrum.
- Advantage shaping: With G = 8 and G′ = 24, advantage amplification ranges from 2.10× at c = 1 to 5.00× at c = 7.The analysis applies to hard questions where initial samples fail and c < G is typical.
- Advantage shaping: Advantage amplification directly increases gradient weight for rare correct trajectories that drive learning on hard questions.The connection follows from the advantage term multiplying the training objective and gradient.