Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectors

1 INTRODUCTION

• Frozen-model distillation with shared, input-invariant vectors at selected layers tests whether RL-induced gains can be recovered through low-dimensional activation interventions.The approach identifies an effective manifold in activation space and characterizes its capacity and separation from the variance-dominant representation subspace.Source: §1 · pp. 1–2
• Over 85% of the full fine-tuning gain is recovered on most evaluated tasks with one input-invariant steering vector per controlled layer.Recovery depends on distribution gap and injection depth: wider gaps call for more flexible parameterizations, and fixed-vector interventions degrade near the output because of an expressiveness bottleneck.Source: §1 · p. 2, Figure 1 · p. 2
• RL-induced control directions are reproducible across training configurations and concentrate in the low-variance complement of the activation principal subspace.Across tasks, direction alignment strongly reflects transfer gains; later extracted directions can independently support target behavior despite declining standalone performance.Source: §1 · p. 2, Figure 1 · p. 2
• Alpha-Stabler achieves more stable reward growth over thousands of steps than gradient clipping and other gradient projection methods, with no significant additional computation or trainable parameters.Its Predictor monitors principal-subspace intrusion against the frozen base model's principal subspace; its Controller removes the principal-subspace component of activation gradients while preserving the orthogonal complement.Source: §1 · p. 2, Figure 1 · p. 2

2.2 COMPRESSIBILITY OF RL LEARNING SIGNALS

• A single steering vector per controlled layer recovers over 85% of the RL gain on most evaluated tasks, under both on-policy and off-policy distillation.The evaluation uses teachers trained with GRPO or DAPO across mathematical, scientific, code-generation, and instruction-following tasks.Source: §2.2 · p. 3, §2.1 · p. 3
• Full-parameter training, LoRA, and vector steering perform comparably with small teacher–student gaps, but larger gaps yield that same capacity ordering, with steering worst.Non-RL teachers also substantially degrade steering performance, with some settings collapsing as the gap widens; fixed offsets express less input-dependent correction than LoRA or full-parameter updates.Source: §2.2 · pp. 3–4

2.3 LAYER-WISE COMPRESSIBILITY OF RL UPDATES

• Fixed-vector steering performs well at lower and intermediate layers but deteriorates near the output, while selected-layer full-parameter updating and LoRA remain effective across depths.Shallow steering gradually approaches intermediate-layer performance during training, whereas high-layer steering plateaus early despite comparable gradient magnitudes.Source: §2.3 · p. 4, Figure 3 · p. 4
• Rank-1 gating reduces teacher KL divergence and improves converged performance at all four tested high-layer sites; a parameter-matched static control does not reproduce the improvement.Increasing gating rank yields further gains toward the full-parameter reference at r = 16.Source: §2.3 · p. 4, Figure 3 · p. 4
• Mean gradient similarity rises from about 0.019 at lower and intermediate layers to about 0.14 near the output, yet alignment alone does not explain steering’s depth-dependent performance.At higher layers, gating coefficients concentrate near 0 and 1; the results associate token-dependent modulation with improved high-layer interventions.Source: §2.3 · pp. 4–5, Figure 4 · p. 5
• Effective manifold capacity can be very low, but depends on whether the intervention form matches the required correction.Fixed-vector recovery is not unlimited: the required capacity varies with the balance between shared teacher-induced shifts and context-dependent fluctuations.Source: §2.3 · p. 5

3.1 PROBING RESIDUAL EFFECTIVE DIRECTIONS WITH MULTI-VECTOR STEERING

• 67.5% versus 62.0% standalone accuracy on SciKnowEval separates the first and eighth orthogonal directions by only 5.5 percentage points.The eighth direction remains independently effective despite orthogonality to the preceding seven.Source: §3.1 · p. 5, Figure 5 · p. 6
• Standalone accuracy generally declines with extraction order, but larger models retain higher-order directions closer to the leading direction than smaller models.The decline across directions is sharper in smaller models.Source: §3.1 · p. 6

3.2 TASK-DEPENDENT GEOMETRY AND PRINCIPAL-SUBSPACE SEPARATION

• Steering directions show stronger same-index than cross-index alignment across teacher parameterizations, with broadly consistent geometry across evaluated configurations and distillation regimes within a task.The comparisons cover full-parameter and LoRA teachers and on- versus off-policy distillation.Source: §3.2 · p. 6, Figure 6 · p. 7
• Directional alignment across six ordered topic pairs correlates with normalized transfer gains, with a linear fit yielding R2 = 0.91.Stronger alignment corresponds to better transferability across the evaluated scientific topics.Source: §3.2 · p. 6
• Across four scientific topics, extracted directions place only about 1%–2% of their energy in the Top-10% principal subspace, versus a 10% isotropic reference; restricting vectors to the Top-30% subspace raises energy to about 15% without meaningfully improving accuracy.The results locate effective steering primarily in the low-variance complement rather than the principal subspace.Source: §3.2 · p. 7
• During stable training, PSI stays below the 10% isotropic reference, then becomes volatile before visible score degradation and rises as collapse progresses.The section summarizes effective steering geometry as lying mainly in the low-variance complement, with cross-topic alignment associated with transfer.Source: §3.2 · p. 7

4.1 ONLINE MONITORING AND CONTROL

• The Predictor activates control after sustained PSI elevation, while the Controller removes principal-subspace components from activation gradients and preserves the orthogonal complement.The Predictor uses warning and lower release thresholds; the Controller leaves forward activations unchanged.Source: §4 · p. 7, §4.1 · p. 8

4.2 EXPERIMENTAL EVALUATION

• Under Alpha-Stabler, PSI remains near its early-training range while task scores continue to improve; uncontrolled runs show rising PSI alongside sharp score deterioration.Figure 8 compares training dynamics with and without the framework.Source: §4.2 · pp. 8–9, Figure 8 · p. 9
• Triggered control matches always-on control, delayed intervention worsens score and peak PSI, and Top-PC projection outperforms alternative subspaces and global gradient clipping.Stabilization holds across tested learning rates, with low single-digit peak PSI under control and comparable final scores across settings.Source: §4.2 · p. 9, Figure 8 · p. 9

• The theoretical framework rests on empirically motivated assumptions that have not yet been derived from first principles.Source: §5 · p. 9
• The analysis and experiments are restricted to RLVR tasks such as mathematical reasoning and code generation.Source: §5 · p. 9
• Whether the proposed geometry holds for RLHF, multi-turn agentic decision-making, or open-ended generation remains unvalidated.Future work could relax the assumptions via neuron attribution and causal tracing, extend validation to those settings, and explore off-principal localization for regularization or adaptive training signals.Source: §5 · p. 9

• Effective Manifold Capacity describes how low-dimensional interventions reproduce RLVR gains, with required capacity depending on how well the intervention matches the needed correction.Source: §6 · p. 9
• Control Manifold Separation places reproducible task-related control directions in the low-variance complement of the activation principal subspace.Source: §6 · p. 9
• Alpha-Stabler monitors principal-subspace intrusion as an early warning and projects its component out of activation gradients during backpropagation.The conclusion reports stabilization with almost no additional computational overhead.Source: §6 · p. 10
• The work recasts RLVR from black-box parameter optimization as an interpretable geometric process for post-training.Source: §6 · p. 10

• Generative AI tools assisted with language editing, manuscript organization, and formatting.Source: AI USE STATEMENT · p. 10
• The authors critically reviewed and revised AI-assisted changes for technical accuracy, consistency, and fidelity.Source: AI USE STATEMENT · p. 10
• The authors take full responsibility for the final manuscript, including its methods, arguments, results, and conclusions.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Appendix B documents procedures, configurations, datasets, verifiers, evaluation protocols, prompts, hyperparameters, hardware, and example commands; Appendices C–D add vector analyses, assumptions, proofs, and Alpha-Stabler details.Algorithm 1 gives warm-up calibration, monitoring, and control procedures; source code is in the supplementary materials.Source: REPRODUCIBILITY STATEMENT · p. 10

A RELATED WORK

• Prior work covers RL for reasoning and agentic tasks, on-policy training dynamics, and activation steering or representation engineering.The cited work includes studies of sampling efficiency, entropy, scaling, parameter dynamics, and low-dimensional behavioral directions.Source: A RELATED WORK · p. 13

B.1 PRELIMINARIES

• The preliminaries distinguish the distillation teacher, which supplies targets, from the reference model, which anchors RL regularization.Source: §B.1 · p. 14
• RLVR assigns rewards from verifiable outcomes, while practical GRPO and DAPO updates use algorithm-specific advantage estimates and clipped surrogate objectives.The underlying regularized RL objective separates task reward from reference-policy regularization; the reward is typically observed after the full response.Source: §B.1 · p. 14
• The OPD optimization approximation retains only the immediate token-level contribution and treats its log-probability difference as a stop-gradient weight.This approximation omits future-token contributions.Source: §B.1 · p. 14
• On-policy distillation uses student-generated trajectories and reverse KL, while off-policy distillation uses teacher-generated trajectories and forward KL, implemented as sequence negative log-likelihood.The off-policy objective is equivalent to minimizing the likelihood loss on teacher-generated trajectories.Source: §B.1 · pp. 14–15

B.2 EXPERIMENTAL SETUP

• Comparisons hold the base model, prompt set, sampling recipe, and evaluation protocol fixed where applicable, while making parameterization-specific optimization differences explicit.Matched on-policy sampling recipes do not guarantee identical trajectories because samples depend on the current policy.Source: §B.2 · pp. 15–16
• The study evaluates on-policy RL, on-policy distillation, and off-policy distillation; distillation teachers are the RL-tuned versions of their students’ base models by default.For off-policy distillation, prompt construction is matched to on-policy runs, while trajectory distributions and objectives remain distinct.Source: §B.2 · pp. 15, 17
• Experiments span five models from 1.5B to 14B parameters, two model families, GRPO and DAPO, and four task families with task-specific verifiers and held-out sets.The tasks are mathematical and scientific reasoning, code generation, and instruction following.Source: §B.2 · p. 15
• Each of the four domains uses its own training and held-out evaluation data, answer-format instructions, and verifier-compatible scoring.The off-policy comparison also includes distinct math supervision corpora for standard SFT and distillation.Source: §B.2 · pp. 15–16
• Mean@4 is the arithmetic mean of four independently sampled responses per held-out prompt, not best-of-four selection or pass@4.Each response is scored with its task-specific evaluation criterion.Source: §B.2 · p. 16
• Full fine-tuning updates all parameters, LoRA uses low-rank adapters while retaining context-dependent corrections, and vector steering trains context-invariant residual-stream biases with the backbone frozen.The default steering configuration uses one vector at each controlled layer in S = {5, . . . , 20}.Source: §B.2 · p. 16

C.1 VOCABULARY-SPACE READOUT

• The readout applies RMS normalization and the unembedding matrix to each steering vector, then uses normalized entropy to quantify distribution concentration.The random baseline passes normalized isotropic Gaussian vectors through the same readout pipeline.Source: §C.1 · p. 21
• A clear vocabulary-level association does not ensure effective task adaptation, because a fixed vector applies the same correction across positions without adjusting to context.The gains from high-layer gating are consistent with input-dependent modulation helping explain depth sensitivity.Source: §C.1 · p. 22
• Effective lower- and intermediate-layer steering has a diffuse vocabulary readout, while near-output steering becomes more concentrated and performs worse.Beginning around L26, normalized entropy falls below the random baseline, and top-logit items increasingly include discourse-related terms such as “Wait” and “But.”Source: §C.1 · pp. 21–22

C.2 LOCAL REPRESENTATION GEOMETRY UNDER SINGLE-VECTOR STEERING

• Lower- and intermediate-layer injections retain high clean–steered principal-subspace overlap, while overlap falls most at the latest injection sites.The overlap compares the orientations of the top-k principal subspaces measured at layer L + 1.Source: §C.2 · p. 23
• The top-k explained-variance ratio stays near one at lower and intermediate depths and departs further from one near the output, with modest variance changes.Reduced alignment at the latest injection sites accompanies comparatively small changes in variance captured by leading components.Source: §C.2 · p. 23
• Relative activation-centroid displacement is small for lower and intermediate injections but rises near the output, where steering performs less well.Earlier-layer steering’s strong performance coincides with only a small relative mean shift at the next layer.Source: §C.2 · p. 24
• High-layer steering’s weaker task performance coexists with changed representations, and gating results suggest input-dependent modulation helps address this expressiveness bottleneck.These observations are consistent with the layer-wise analysis rather than establishing that geometric changes cause performance degradation.Source: §C.2 · pp. 22, 24
• Controlled and uncontrolled runs show essentially the same score growth before collapse, indicating almost no additional computational overhead from stabilization.This comparison covers learning rates 10^-4, 5×10^-5, and 10^-5.Source: Figure 12 · p. 25
• Alpha-Stabler calibrates a principal-subspace intrusion diagnostic during warm-up, then monitors shifts and projects gradients when intrusion thresholds persist.The algorithm uses a 50-update warm-up and monitors selected layers at quarter, half, and three-quarter depth.Source: §C.2 · p. 26

D.3 THE RLVR TEACHER SHIFT: EXPONENTIAL TILTING AND SHARED STRUCTURE

• Shared teacher structure requires shared success profiles, a small relative logit residual, and a consistent representation pullback; these are additional conditions, not consequences of binary rewards alone.The analysis also distinguishes activation covariance principal structure from sensitivity-weighted conditions required for its recovery bound.Source: INFORMAL PREVIEW · p. 27, §D.1 · p. 29, §D.3 · pp. 31–32

D THEORY OF LOW-DIMENSIONAL RLVR STEERING

• Under localization and shared-component conditions, the optimal constant intervention recovers at least ρ ≥ κ/((1+η)(1+ε)) of locally representable target energy.Here ρ is not a task-score gain recovery rate, and the conditions’ validity for a particular teacher requires separate justification.Source: INFORMAL PREVIEW · p. 27, D THEORY OF LOW-DIMENSIONAL RLVR… · p. 27

D.2 DISTILLATION REDUCES TO WEIGHTED LEAST SQUARES

• Local distillation reduces to Fisher-weighted least squares, fitting context-dependent teacher targets with a constant vector in an output-sensitivity metric.This is a fixed-distribution local surrogate, not an identity for the full on-policy training trajectory; the fitting target is not an observed hidden-state difference.Source: §D.2 · pp. 30–31

D.3 THE RLVR TEACHER SHIFT: EXPONENTIAL TILTING AND SHARED STRUCTURE

• The idealized KL-regularized RL optimum exponentially tilts complete-response probabilities by reward, while next-token changes depend on candidate-token continuation-success probabilities.Binary rewards do not by themselves impose a common direction across contexts.Source: §D.3 · pp. 31–32
• A small absolute residual does not ensure high relative alignment when the shared amplitude is small; the Fisher decomposition requires a suitably small residual relative to that amplitude.Under the stated relative condition, the orthogonal residual is bounded by θ/(1−θ) times a(x)∥s∥Fx.Source: §D.3 · p. 32
• A small-positive-β uniform residual bound requires success probabilities bounded away from zero, a condition not automatic for binary rewards.The regime concerns an idealized optimum, not small realized KL divergence or a zero-penalty checkpoint; Corollary D.1 still requires its relative Fisher-residual condition.Source: §D.3 · p. 33
• A bound on representation pullback residual alone does not establish a shared correction direction or the moment conditions needed for Assumption D.5.Even equal metrics need not yield a common direction: reversing Jx preserves Gx but reverses the pullback.Source: §D.3 · p. 33
• An exact regularized optimum, shared success profile with small relative residual, and consistent pullback successively yield shared next-token, logit, and representation components.The final lower bound additionally requires off-principal localization and the moment conditions of Assumptions D.4–D.5; these conditions concern different objects and must be checked independently.Source: §D.3 · p. 33
• The exponential-tilt identities apply to an exact optimum with β > 0, not automatically to trained checkpoints, including DAPO checkpoints with zero explicit KL coefficient.Fitting and recovery can instead take t(x) as given under their stated local and structural conditions; clipping and small updates do not establish an exact tilt identity, metric homogeneity, or a shared direction.Source: §D.3 · p. 33

D.4 SINGLE-VECTOR RECOVERY: AN IDENTITY AND A LOWER BOUND

• The optimal constant vector minimizes the quadratic surrogate, but recovery of target energy does not by itself guarantee exact teacher reproduction or empirical task-score gain recovery.Exact reproduction additionally requires local reachability and negligible approximation error; relating surrogate recovery to nonlinear KL or task scores needs further assumptions.Source: §D.4 · pp. 33–34

D.4.1 SIGNAL-TO-FLUCTUATION FORM

• In the constant-metric model, single-vector recovery is a signal-to-fluctuation ratio, with recovery at least u exactly when SNR is at least u/(1−u).Theoretical recovery identities do not imply a universal capacity trend with teacher–student gap or injection depth; the reported trends are empirical.Source: §D.4.1 · pp. 34–35

D.4.2 EXPLICIT THREE-CONSTANT BOUND

• Low principal energy is a sufficient condition for high constant-vector recovery under the stated decomposition, but it is not necessary and is not established as prevalent by the theorem.A constant nonzero target entirely in the principal subspace can still have full recovery.Source: §D.4.2 · p. 35

D.5 SEQUENTIAL-ORTHOGONAL STEERING AND CONSTRAINED FITTING

• Sequential fitting constrains each new direction to be orthogonal to earlier directions, but later useful interventions do not establish distinct target spectral components or intrinsic manifold dimension.The spectral expression characterizes a more flexible oracle class; a trained gating architecture may fit less well.Source: §D.5 · pp. 35–36

D.6 EFFECTIVE DIRECTIONS AND THE PRINCIPAL SUBSPACE

• 1%–2% of learned-vector Euclidean energy lies in the top-10% activation principal subspace, below the approximately 10% isotropic reference.This measurement concerns learned vectors, not the sensitivity-weighted energy of the context-dependent target family; Euclidean localization alone need not imply low sensitivity-weighted intrusion.Source: §D.6 · pp. 36–37

D.6.1 A WEIGHT-SPACE ACCOUNT OF ACTIVATION LEAKAGE

• The weight-to-activation leakage bound is conditional and first-order; small weight updates or zero principal weight mass alone do not ensure activation localization.The leakage factor exposes basis mismatch and Jacobian mixing, while extending the result beyond first order requires a nonlinear remainder bound.Source: §D.6.1 · p. 37, PART III. GEOMETRIC ANALYSIS OF… · p. 37

Reference geometry

• The reference basis uses the leading 10% of hidden dimensions, estimated after a 50-update unprojected warm-up and then held fixed.The analysis conditions on estimated projectors and does not provide a finite-sample PCA guarantee.Source: §D.7 · p. 38

Intrusion measurement

• PSI compares principal-shift energy with total shift energy, summing token energies before normalization and aggregating layer energies before taking the ratio.When monitoring a token subset, these identities apply to that subset, not automatically to all tokens or the population.Source: §D.7 · p. 38

Predictor and threshold calibration

• The Predictor checks every three updates and flags intrusion after three consecutive valid checks above threshold, following stable warm-up calibration.The MAD multiplier gives no false-alarm probability for dependent observations, and warning lead times require empirical evaluation.Source: §D.7 · p. 38

Controller

• At an active hook, the Controller removes the incoming gradient’s principal component while retaining its complement; this is the unique minimal-change removal and cannot increase local norm.At an inactive site, the incoming gradient is unchanged.Source: §D.7 · p. 39

From gradients to activation changes

• For one fixed-batch SGD step, projected gradients do not alone eliminate first-order principal displacement; that depends on the coupling ΠKQ.Zero first-order principal displacement and preserved complement displacement require the additional condition ΠKQ = 0.Source: §D.7 · p. 39

Optimizer and multi-site scope

• The single-site SGD analysis does not guarantee behavior for adaptive optimizers or multiple control sites; those require the actual update decomposition and full modified gradient flow.The estimate is conditional on a specified step and its local regularity conditions.Source: §D.7 · pp. 39–40

Cumulative PSI and stability

• A local leakage bound does not imply monotone cumulative PSI or a no-collapse guarantee; long-run bounds require control of couplings, updates, remainders, optimizer state, and observation distributions.Warning utility and stabilization are treated as empirical outcomes.Source: §D.7 · p. 40

Implementation and overhead

• Parameter-free projection does not establish low end-to-end overhead; measurements must include reference evaluation, actor passes, covariance estimation, calibration storage, and communication.The required implementation costs vary with monitoring and forward-pass choices.Source: §D.7 · p. 40

Recovery

• Shared teacher targets can support strong constant-vector recovery under local approximation, moment, and metric conditions.Constrained and adaptive fitting represent different intervention capacities; neither orthogonality nor low learned-vector PC energy alone determines target rank or localization.Source: §D.8 · p. 40

Gradient control

• Alpha-Stabler uses a fixed activation reference for monitoring and local gradient control, with exact projection properties but optimizer- and coupling-dependent activation displacement.These results motivate the controller but do not establish its training-wide effects.Source: §D.8 · p. 40

Empirical scope

• The analysis does not replace empirical evidence for task-score recovery, early warning, stability, or runtime overhead.Table 5 summarizes recovery, steering geometry, and gradient-control results alongside their assumptions and empirical scope.Source: Table 5 · p. 41