PaperlayerView exact revision on arXiv

Source-linked AI summary

SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment

Yixuan Tang, Yi Yang

arXiv:2512.02807v1cs.CL

TL;DR

Human-preference alignment depends on external supervision, while existing approaches also face limitations such as sparse, subjective judgments and prompt-sensitive self-evaluation. This paper uses stable rank from model representations as a quality signal for response selection and reinforcement learning, reporting gains across its evaluations. Its results suggest internal representation geometry can provide an alignment signal without external supervision.

  • Problem

    LLM alignment relies on external supervision, while human judgments are subjective and sparse and self-evaluation methods can be prompt-sensitive and biased.

    Source: §1 · p. 1

  • Method

    The paper uses stable rank, a hidden-state geometric quality signal, for Best-of-N selection and as the reward in SR-GRPO.

    Source: §1 · p. 2, §8 · p. 8

  • Results

    84.04% accuracy on RewardBench and average Best-of-N gains of 11.3 percentage points over greedy decoding accompany SR-GRPO improvements of 10% on STEM and 19% on mathematical reasoning.

    Source: §8 · pp. 8–9

  • Takeaways & Limitations

    The findings suggest internal representation geometry can provide a quality signal for LLM alignment without external supervision.

    Source: §8 · pp. 8–9

    The quality interpretation rests on the hypothesis that high-quality generations have higher effective dimensionality, while low-quality outputs exhibit rank collapse.

    Source: §1 · p. 2

Abstract

from arXiv · show

Aligning Large Language Models (LLMs) with human preferences typically relies on external supervision, which faces critical limitations: human annotations are scarce and subjective, reward models are vulnerable to reward hacking, and self-evaluation methods suffer from prompt sensitivity and biases. In this work, we propose stable rank, an intrinsic, annotation-free quality signal derived from model representations. Stable rank measures the effective dimensionality of hidden states by computing the ratio of total variance to dominant-direction variance, capturing quality through how information distributes across representation dimensions. Empirically, stable rank achieves 84.04% accuracy on RewardBench and improves task accuracy by an average of 11.3 percentage points over greedy decoding via Best-of-N sampling. Leveraging this insight, we introduce Stable Rank Group Relative Policy Optimization (SR-GRPO), which uses stable rank as a reward signal for reinforcement learning. Without external supervision, SR-GRPO improves Qwen2.5-1.5B-Instruct by 10% on STEM and 19% on mathematical reasoning, outperforming both learned reward models and self-evaluation baselines. Our findings demonstrate that quality signals can be extracted from internal model geometry, offering a path toward scalable alignment without external supervision.

1. Introduction

The introduction argues that external supervision and self-evaluation have important limitations, then proposes stable rank as a geometric quality signal and SR-GRPO as an annotation-free alignment method. It reports gains from using stable rank for both response selection and reinforcement learning.

1. Introduction

  • Human judgments are subjective and context-dependent, annotations are sparse, reward models can be vulnerable to reward hacking, and self-evaluation has prompt sensitivity and biases.Existing alternatives still rely on preference data, feasible automatic verification, or external signals rather than information in model representations.Source: §1 · p. 1
  • Stable rank measures hidden-state effective dimensionality as total variance relative to dominant-direction variance, treating broader information spread as a quality signal.The paper hypothesizes that high-quality generations maintain higher effective dimensionality, while low-quality outputs exhibit rank collapse.Source: §1 · pp. 1–2
  • 84.04% accuracy on RewardBench and average Best-of-N gains of 11.3% over greedy decoding show stable rank's value as a zero-shot reward proxy and selection signal.The Best-of-N gains are reported across four model families on STEM and mathematics benchmarks.Source: §1 · p. 2
  • SR-GRPO improves reasoning-task performance by 10 to 19% over GRPO with learned reward models at zero annotation cost, and by 8 to 9% over self-reward baselines.Consistent gains on DeepSeek-R1-Distill-Qwen-1.5B are reported as evidence of robustness across model families.Source: §1 · p. 2
  • Stable rank's associated quality dimensions include semantic coherence, information density over verbosity, and sensitivity to reasoning words at key turns.The paper links these dimensions to an aggregate quality signal rather than a single superficial feature.Source: §1 · p. 2
  • The contributions extend stable rank beyond reinforcement learning to zero-shot response evaluation and Best-of-N decoding.SR-GRPO uses stable rank as a dense reward without preference datasets or external verifiers.Source: §1 · p. 2

2. Stable Rank as an Intrinsic Quality Metric

This section defines stable rank as a measure of hidden-state effective dimensionality and evaluates it as both a zero-shot preference proxy and a Best-of-N selection signal.

2.1. Definition

  • Stable rank measures effective dimensionality: a dominant singular value yields SR(H) near 1, while balanced singular values make it approach rank(H).The paper extracts final-layer hidden states and computes stable rank from the activation matrix; final-layer embeddings provide the strongest quality signal in its cross-layer analysis.Source: §2.1 · pp. 2–3

2.2. Empirical Validation: Stable Rank as Zero-Shot Reward Proxy

  • 84.04% accuracy on RewardBench for Qwen3-8B lets stable rank exceed Pointwise (83.70%) and IPO (78.02%); on Qwen2.5-1.5B-Instruct, its 75.95% beats the best baseline, IPO (65.85%), by 10.1 percentage points.The benchmark compares stable rank with prompt-based pointwise and pairwise scoring and IPO across five models.Source: §2.2 · p. 3, Table 1 · p. 3

2.3. Empirical Validation: Best-of-N Decoding with Stable Rank

  • At N=16, stable-rank selection improves over greedy decoding by 20.5% on Llama-3.2-1B, 17.0% on Qwen2.5-1.5B, 10.2% on DeepSeek-R1, and 8.5% on Phi-3.5-mini.Stable rank also beats random selection on Llama-3.2-1B at N=16: 26.5% average accuracy versus 19.8%.Source: §2.3 · p. 4

3. Stable Rank Group Relative Policy Optimization (SR-GRPO)

SR-GRPO uses stable rank as a proxy reward to train a policy initialized from a reference model. Group-relative updates use rewards computed on that frozen reference model.

3. Stable Rank Group Relative Policy Optimization (SR-GRPO)

  • Stable rank supplies a dense reinforcement-learning reward that requires no labeled data and can be computed for generated responses.The method aims to optimize a trainable policy initialized from a reference model to generate higher-quality responses.Source: §3 · p. 4, §3.1 · p. 4

3.2. Training Procedure

  • GRPO compares responses within a sampled group to obtain relative quality signals without training a separate value-function critic.The approach uses stable rank as its geometric reward signal.Source: §3.2 · p. 4
  • Rewards are computed on the frozen reference model and standardized within each group; freezing it provides a stationary signal the policy cannot manipulate by changing its own geometry.The training procedure samples responses, computes stable-rank rewards and group-standardized advantages, then updates the policy; stable-rank computation is described as negligible relative to the transformer forward pass.Source: §3.2 · p. 4

4. Alignment Experiments

SR-GRPO uses stable rank rewards without preference labels, and Table 2 reports that it outperforms reward-model and self-evaluation baselines across benchmark categories. The experiments evaluate two models on STEM, mathematical reasoning, and general chat.

4.1. Experiment Setup

  • Training uses SmolTalk2 prompts without preference labels; stable rank is computed on the frozen base model while LoRA trains the policy.Evaluation covers GPQA and MMLU-redux for STEM, four math benchmarks, and WildBench for general chat.Source: §4.1 · pp. 4–5

4.2. Results Analysis

  • SR-GRPO outperforms reward-model and self-evaluation baselines across all benchmark categories without external labels.WildBench scores are reported as WB-Elo.Source: Table 2 · p. 5, §4.2 · p. 6
  • On Qwen2.5-1.5B-Instruct, mathematical reasoning accuracy rises 4.4 percentage points (28.0% to 32.4%); DeepSeek-R1-Distill-Qwen-1.5B improves from 58.5% to 64.7%.On DeepSeek, OlympiadBench improves from 43.6% to 58.5%.Source: §4.2 · p. 6
  • Baseline results are mixed: the reward model drops Qwen2.5-1.5B GPQA from 19.0% to 15.7%, while Self-Reward matches SR-GRPO on AIME at 13.3%.Perplexity performs well on DeepSeek but is less consistent across both models; IPO degrades several benchmarks.Source: §4.2 · p. 6
  • WildBench WB-Elo gains are 26.2 points for Qwen2.5-1.5B and 19.0 points for DeepSeek-R1.The authors report that gains extend beyond task-specific correctness to open-ended conversation quality.Source: §4.2 · p. 6

5. What Stable Rank Captures

The paper analyzes stable rank’s correlations with 37 text-quality metrics across 5,970 responses and 2,985 chosen/rejected pairs. The findings associate stable rank with coherence, information density, and selected discourse-marker patterns.

5.1. Stable Rank Captures Semantic Coherence

  • Progression score (ρ = 0.313) and QA alignment consistency (ρ = 0.316) correlate positively with stable rank, while coherence standard deviation correlates negatively (ρ = −0.356).The authors interpret this pattern as favoring topical focus and smooth development over erratic transitions.Source: §5.1 · p. 6

5.2. Stable Rank Distinguishes Density from Verbosity

  • Stable rank correlates negatively with token count (ρ = −0.294) and sentence count (ρ = −0.368), but positively with lexical diversity (ρ = 0.238) and compression ratio (ρ = 0.233).The authors interpret these correlations as favoring information density over verbose, repetitive text.Source: §5.2 · p. 7

5.3. Stable Rank Highlights Key Logical Markers

  • Common additive, conditional, and enumeration markers correlate negatively with stable rank (ρ = −0.156, −0.163, and −0.148), as does total marker count (ρ = −0.204).The paired-difference analysis controls for prompt context across 2,985 chosen/rejected pairs.Source: §5.3 · p. 7
  • Contrastive and causal marker frequencies have positive Pearson correlations with stable rank (r = 0.187 and r = 0.139), but small Spearman correlations.The authors suggest that a few well-placed markers matter more than frequent use.Source: §5.3 · p. 7

5.4. Summary

  • The reported correlations are moderate in magnitude (|ρ| ≈ 0.2–0.4), consistent in direction, and statistically significant (p < 0.001).The authors conclude that stable rank is associated with multiple quality dimensions rather than a single superficial feature.Source: §5.4 · p. 7

6. Ablation Studies

Three ablations examine alternative intrinsic-dimension metrics, context length, and prompt format.

Alternative intrinsic dimension metrics

  • Stable rank achieves 84.04% RewardBench accuracy, above PCA 95% variance (61.91%), effective rank (54.50%), and condition number (36.04%).The authors attribute its advantage to aggregating information across the singular-value spectrum through the Frobenius norm.Source: §6 · p. 7

Context length

  • Accuracy falls from 83.85% at 512 tokens to 62.59% at 128 tokens; Code falls from 87.91% to 24.80%.Extending beyond 512 tokens yields less than 0.2 percentage points of gain.Source: §6 · pp. 7–8

Prompt format

  • Across six input templates and three models, overall accuracy varies by at most 3 percentage points, with no format consistently best.The authors say this robustness allows simple formats without extensive tuning.Source: §6 · p. 8

7. Related Work

Related work covers external-feedback alignment, automatic or model-derived signals, and links between representation geometry and generation quality. This paper’s stated distinction is to use stable rank during optimization rather than only for post-hoc diagnosis.

7.1. Alignment with External Feedback Signals

  • RLHF and preference-based methods rely on external preference data; DPO, KTO, Bayesian RLHF, and MaxMin-RLHF optimize directly on curated comparisons.The passage also notes noisy, costly preference data and reward-model vulnerability to reward hacking and distribution shift.Source: §7.1 · p. 8

7.2. Alignment with Automatic Signals

  • Verifiable rewards use programmatic checkers but do not generalize to open-ended tasks, while self-evaluation derives signals from model outputs or token probabilities.Factoscope and INSIDE show that hidden states encode factuality and hallucination risk; this method uses activation geometry as a policy-optimization reward.Source: §7.2 · p. 8

7.3. Representation Geometry and Generation Quality

  • Prior work links representation rank and geometry to generation quality, while this paper uses stable rank as a live reward during policy optimization.The paper characterizes this as a shift from post-hoc diagnosis to optimization.Source: §7.3 · p. 8

8. Conclusion

Stable rank provides an intrinsic signal for assessing response quality and aligning models, and SR-GRPO uses it as a reinforcement-learning reward without external supervision.

8. Conclusion

  • Stable rank achieves 84.04% accuracy on RewardBench and improves Best-of-N sampling by 11.3 percentage points on average across reasoning benchmarks.The conclusion presents it as a geometric signal from LLM hidden states for text-quality assessment and alignment.Source: §8 · p. 8
  • 10% on STEM and 19% on mathematical reasoning: SR-GRPO improves Qwen2.5-1.5B-Instruct, outperforming learned reward models and self-evaluation baselines.The approach uses stable rank as its reinforcement-learning reward without external supervision.Source: §8 · pp. 8–9
  • Stable rank captures semantic coherence, information density, and sensitivity to key reasoning structures.The authors conclude that internal representation geometry provides sufficient signal for effective LLM alignment.Source: §8 · p. 9

A. Cross-Layer Stable Rank Analysis

Stable rank’s quality signal varies by transformer layer: final layers perform best across model families, with task-specific patterns. The supplied appendix passages also describe baseline procedures and the scope of the RewardBench comparison.

A. Cross-Layer Stable Rank Analysis

  • Final-layer stable rank reaches 70–85% accuracy on most categories, while layers 5–25 achieve near-random ∼50% accuracy across all categories and three model families.The analysis evaluates Qwen3-8B, Qwen2.5-1.5B-Instruct, and Phi-3.5-mini-instruct on Chat, Chat Hard, Safety, Code, and Math.Source: A. Cross-Layer Stable Rank Analysis · p. 13
  • Math and Code improve most in final layers; Qwen2.5-1.5B reaches 95–98% accuracy on Math at layer 27, while Safety shows a U-shape on two models.Chat and Chat Hard rise sharply at the end, whereas Safety also performs reasonably at layers 0–2 on Qwen3-8B and Phi-3.5-mini.Source: A. Cross-Layer Stable Rank Analysis · p. 13
  • The layer analysis supports extracting stable rank from the final transformer layer for SR-GRPO, where the quality signal is most pronounced.The passage contrasts 70–85% final-layer accuracy with 45–50% in middle layers.Source: A. Cross-Layer Stable Rank Analysis · p. 13

B.1.1. POINTWISE SCORING PROMPT

  • Pointwise Scoring asks an LLM to assess each response against a rubric, provide detailed feedback, and return an integer score from 1 to 5.The score is extracted from the [RESULT] token; its reference answer and rubrics are adapted from RewardBench materials.Source: §B.1.1 · p. 14

B.1.2. PAIRWISE COMPARISON PROMPT

  • Pairwise Comparison asks an LLM to judge two responses against a rubric and choose the better one; each pair is tested in both orders, counting only agreements.The preference is parsed from the [RESULT] token.Source: §B.1.2 · pp. 14–15

B.2. IPO Implementation

  • IPO prompts task-specific Yes/No verification, scores responses from final-position Yes/No logits, and predicts the response with the higher IPO score as preferred.The supplied prompts cover Code, Math, Safety, and default Chat tasks.Source: §B.2.2 · p. 15, §B.2.1 · p. 15

C. RewardBench Results

  • Table 5 presents a complete RewardBench performance comparison across reward methods, including pointwise scoring and pairwise comparison.The supplied caption describes these two baselines as independent 1–5 grading and direct evaluation of two responses, respectively.Source: C. RewardBench Results · p. 15, Table 5 · p. 16

D. Best-of-N Complete Results

Stable-rank Best-of-N selection beats random selection across models and sampling budgets, with especially strong reported gains on math-heavy benchmarks. The supplied passages also detail training, metric-analysis methods, and correlations between stable rank and text features.

D. Best-of-N Complete Results

  • At N=16, Llama-3.2-1B gains +33.8% relative improvement over random selection, while random selection falls 9.9% below greedy decoding.The passage attributes the random-selection drop to the possibility of choosing low-quality samples.Source: D. Best-of-N Complete Results · p. 16, Table 6 · p. 17
  • At N=16, stable rank selection yields particularly strong improvements on math-heavy benchmarks, raising DeepSeek-R1 from 82.8% to 86.8% on MATH500 and from 43.6% to 59.7% on OlympiadBench.Its gains increase with N for most models.Source: D. Best-of-N Complete Results · p. 16

E. SR-GRPO Training Details

  • SR-GRPO training uses LoRA and a frozen reference model for stable-rank rewards, with adapters disabled during reward calculation to prevent reward hacking.The two models use distinct step counts and prompt-length settings; both use the common GRPO group size K=8 and KL penalty β=0.04.Source: E. SR-GRPO Training Details · pp. 16–17

F.1. Statistical Methodology

  • Metric analysis uses sample-level correlations across 5,970 responses and paired differences across 2,985 chosen–rejected pairs to control for prompt variation.Paired-difference analysis examines whether within a prompt, higher stable rank aligns with more or fewer instances of a linguistic feature.Source: §F.1 · pp. 17–18

F.2. Metric Definitions

  • The analysis groups response quality metrics into semantic coherence, information density, and linguistic structure.Its implementation uses sentence embeddings and sentence tokenization, with syllable counting for readability and marker counts normalized per sentence or 100 tokens.Source: §F.2 · p. 18, §F.3 · p. 18

F.5. Key Findings

  • Stable rank correlates positively with progression score (ρ=0.313) and QA alignment consistency (ρ=0.316), and negatively with coherence standard deviation (ρ=−0.356).The analysis interprets these associations as relating to topic flow, prompt alignment, and erratic transitions between adjacent sentences.Source: Table 8 · p. 19, §F.5 · p. 20
  • Token count (ρ=−0.294) and sentence count (ρ=−0.368) correlate negatively with stable rank, while lexical diversity (ρ=0.238) and compression ratio (ρ=0.233) correlate positively.The authors interpret this pattern as distinguishing information density from verbosity.Source: §F.5 · p. 20, Table 9 · p. 20
  • Additive, conditional, and enumeration markers correlate negatively with stable rank, while contrastive and causal markers show positive Pearson but weak Spearman correlations.The reported paired-difference correlations are ρ=−0.156, −0.163, and −0.148 for the first three categories; contrastive r=0.187 and causal r=0.139, with weak Spearman correlations.Source: §F.5 · p. 20, Table 10 · p. 21

G. Ablation: Comparison with Other Intrinsic Dimension Metrics

On RewardBench, stable rank outperforms three alternative intrinsic-dimension metrics. The analysis attributes this advantage to how stable rank combines information across the singular-value spectrum.

G. Ablation: Comparison with Other Intrinsic Dimension Metrics

  • The comparison evaluates final-layer Qwen3-8B representations on RewardBench using stable rank, condition number, PCA 95% variance, and entropy-based effective rank.Condition number uses the inverse ratio as its score; PCA counts components needed to capture 95% of variance.Source: G. Ablation: Comparison with Other… · p. 21
  • 84.04% overall RewardBench accuracy for stable rank exceeds PCA 95% variance (61.91%), effective rank (54.50%), and condition number (36.04%).Scores use Qwen3-8B final-layer representations.Source: G. Ablation: Comparison with Other… · p. 21
  • On Math, stable rank reaches 84.34% while effective rank and condition number fall below 13%; on Safety, it reaches 76.20% while alternatives remain below 46%.PCA 95% variance also struggles on Math (27.96%) and Safety (45.45%).Source: G. Ablation: Comparison with Other… · p. 21
  • Stable rank aggregates the singular-value spectrum with the Frobenius norm and balances representational richness against coherence, unlike discrete PCA counting or entropy weighting.The authors suggest this makes it robust to outliers that affect condition number.Source: G. Ablation: Comparison with Other… · p. 21

H. Ablation: Context Window Size

Stable rank accuracy degrades when context is truncated at short lengths but quickly saturates, and remains robust across tested prompt formats. Examples connect low stable rank with repetition loops and redundant verbosity.

H. Ablation: Context Window Size

I. Ablation: Input Prompt Format

  • Across prompt formats, overall accuracy varies by at most 3 percentage points per model with no consistent winner; the authors recommend a native chat template or minimal format.Category-level changes can be larger, but show no systematic pattern across models.Source: §I · p. 24, Table 15 · p. 24

J. Stable Rank Example Prompts

  • Repetition loops collapse variance into one direction, matching stable rank’s strongest negative correlation with coherence standard deviation (ρ = −0.356).The example comparison describes this degeneration in an Image Processing response.Source: J. Stable Rank Example Prompts · p. 24, Table 16 · p. 25
  • Redundant verbosity can dilute a correct solution; stable rank correlates negatively with token count (ρ = −0.294) and total discourse markers (ρ = −0.204).The example contrasts an over-elaborated solution with one that stops at the answer.Source: J. Stable Rank Example Prompts · p. 24, Table 16 · p. 25
Loading 2512.02807v1…