PaperlayerView exact revision on arXiv

Source-linked AI summary

RAPTOR: Ridge-Adaptive Logistic Probes

Ziqi Gao, Yaotian Zhu, Qingcheng Zeng, Xu Zhao, Ziqing Wang, Feng Ruan, Kaize Ding

arXiv:2602.00158v2cs.LGcs.AI

TL;DR

Probe-then-steer pipelines need concept vectors that are accurate, directionally stable, and inexpensive. RAPTOR uses validation-tuned ridge logistic probes and normalized weights, finding competitive accuracy and stability with lower cost while theoretically characterizing regularization in an idealized high-dimensional model.

  • Problem

    Probe-derived concept vectors must be accurate, directionally stable under perturbations, and inexpensive to obtain for probe–then–steer workflows.

  • Method

    RAPTOR fits validation-tuned ℓ2-regularized logistic probes on frozen layer representations and normalizes their weights into concept vectors, with theory based on a Gaussian teacher–student model.

  • Results

    Across the benchmark, RAPTOR matches strong alternatives in accuracy while offering superior directional stability and negligible computational cost.

  • Takeaways & Limitations

    A tuned ridge-logistic probe is a strong reference point that can often make substantially more complex estimators unnecessary for standard activation-steering tasks.

  • Takeaways & Limitations

    The theoretical analysis uses a stylized Gaussian teacher–student model that simplifies the distribution of real LLM representations.

Abstract

from arXiv · show

Probing studies what information is encoded in a frozen LLM's layer representations by training a lightweight predictor on top of them. Beyond analysis, probes are often used operationally in probe-then-steer pipelines: a learned concept vector is extracted from a probe and injected via additive activation steering by adding it to a layer representation during the forward pass. The effectiveness of this pipeline hinges on estimating concept vectors that are accurate, directionally stable under ablation, and inexpensive to obtain. Motivated by these desiderata, we propose RAPTOR (Ridge-Adaptive Logistic Probe), a simple L2-regularized logistic probe whose validation-tuned ridge strength yields concept vectors from normalized weights. Across extensive experiments on instruction-tuned LLMs and human-written concept datasets, RAPTOR matches or exceeds strong baselines in accuracy while achieving competitive directional stability and substantially lower training cost; these quantitative results are supported by qualitative downstream steering demonstrations. Finally, using the Convex Gaussian Min-max Theorem (CGMT), we provide a mechanistic characterization of ridge logistic regression in an idealized Gaussian teacher-student model in the high-dimensional few-shot regime, explaining how penalty strength mediates probe accuracy and concept-vector stability and yielding structural predictions that qualitatively align with trends observed on real LLM embeddings.

1 Introduction

RAPTOR addresses probe-then-steer requirements by combining accuracy, directional stability, and computational efficiency in a validation-tuned ridge-logistic probe. Across broad empirical and theoretical analyses, it matches strong baselines while reducing training overhead and clarifying regularization’s role.

  • Motivation: Probe-then-steer pipelines use learned directions to modify frozen model representations without updating model weights.Additive steering injects a concept direction into a layer representation at inference time.
  • Motivation: Effective probes must provide accurate predictions, directionally stable concept vectors, and inexpensive training across layers, concepts, and models.The paper treats these three criteria as a joint lens for comparing probe choices.
  • RAPTOR: RAPTOR fits one ℓ2-regularized logistic probe per model, layer, and concept, selecting ridge strength λ by validation and normalizing its weights into a concept vector.The design uses logistic regression for a linear accuracy baseline and ridge regularization for limited-data robustness.
  • Empirical evaluation: Across instruction-tuned models, concept datasets, and network layers, RAPTOR matches or exceeds strong baselines in accuracy while improving directional stability and reducing training overhead.Qualitative steering examples support the connection between robust concept vectors and more reliable downstream control.
  • Theory: A high-dimensional Gaussian teacher–student analysis characterizes how λ shapes probe accuracy and concept-vector stability in the proportional regime.The resulting predictions qualitatively align with dominant trends observed on real datasets.

2 Related Work

The paper situates RAPTOR at the intersection of diagnostic probing, concept-direction estimation, and additive activation steering. It builds on linear probes and learned intervention directions while targeting probe–then–steer use cases.

  • Probing: Probing trains auxiliary predictors on frozen layer representations to measure what information models encode.Linear probes are commonly used to map linguistic and semantic properties across layers and architectures.
  • RAPTOR pipeline: Figure 1 summarizes RAPTOR’s operational path from layerwise embeddings and standardized features to a normalized concept vector used for additive steering.The pipeline selects λ on validation data before applying hℓ,T ← hℓ,T + αvℓ at inference.
  • Concept directions: Probe weights can be interpreted as concept directions for representation analysis, editing, or removal.Related approaches include TCAV-style directions, nullspace procedures, amnesic removal, subspaces, and distributions over directions.
  • Probe–then–steer pipelines: Probe–then–steer pipelines reuse a probe’s learned direction as the intervention direction added to internal activations.This operational pattern underlies several recent activation-steering methods.

3 Methodology

RAPTOR uses validation-selected ridge logistic regression to extract concept directions from standardized layer representations, then applies those directions through additive steering. The method is designed for stable, efficient probing in high-dimensional small-sample settings.

  • 3.1 Ridge-Adaptive Logistic Probe: High-dimensional, small-sample probing often produces nearly linearly separable hidden states, making unregularized logistic regression numerically unstable.The unregularized objective may lack a finite maximizer, while an ℓ2 penalty restores existence and uniqueness.
  • 3.1 Ridge-Adaptive Logistic Probe: RAPTOR selects a single ridge strength λ by validation, using it as an interpretable control for statistical regularization and training stability.This keeps the estimator minimal while supporting a high-dimensional characterization of accuracy–stability behavior.
  • 3.1 Ridge-Adaptive Logistic Probe: The implementation uses warm starts and early stopping to improve practical efficiency across many probe configurations.These are standard training configurations rather than additional estimator components.
  • 3.2 Method setup: For each layer, the method forms labeled hidden-state data and uses stratified training, validation, and test splits with labels mapped to {±1}.The split structure supports validation-based selection of λ.
  • 3.2 Method setup: Embeddings are standardized using training-split statistics and the same transformation is applied to validation and test data.Train-only standardization prevents leakage and controls feature scales during ridge tuning.
  • 3.2 Steering setup: After fitting ridge-regularized logistic regression with an intercept, parameters are converted back to the original embedding space so the concept vector can be injected into native representations.The learned direction is then used for additive steering, with injection strength set by a calibration rule without retraining.

4 Experiments

RAPTOR is evaluated against xRFM and GCS across model, dataset, and layer grids for accuracy, directional robustness, training cost, and steering control. It improves accuracy broadly, is more robust than xRFM while near GCS, trains faster than both baselines, and achieves near-perfect probe-coordinate control with task-dependent intervention costs.

  • Probe accuracy: Across 42 model–dataset settings, RAPTOR improves average accuracy over GCS in all settings and best-layer accuracy in 41/42, with one tie.Compared with xRFM, it matches or exceeds best-layer accuracy in 26/42 settings and outperforms average accuracy in 27/42.
  • Probe accuracy: 0.874 best-layer accuracy is RAPTOR’s grid average, versus 0.854 for GCS (+1.96 points) and 0.871 for xRFM (+0.29 points).GCS gains are largest on HateXplain (+3.51 points) and Sarcasm (+2.12 points) in mean best-layer accuracy.
  • Directional robustness: RAPTOR consistently improves directional robustness over xRFM under 20% training-data ablations, while GCS remains most stable overall and RAPTOR is typically close.Robustness is summarized by mean and best-layer absolute-cosine similarity across layers.
  • Training cost: RAPTOR is consistently faster than xRFM and GCS across the full 7 × 6 model–dataset grid in median per-layer training time.Training cost is measured as wall-clock time under the same hardware and data pipeline, with probing performed independently per layer.
  • Steering results: Adaptive steering achieves near-perfect probe-coordinate control across datasets and directions, while intervention rates range roughly from 0.54 to 0.83 of evaluated pairs.Median steering strengths range about 3.6 to 12.4, but maximum |α| reaches 249, with a heavier tail for away-direction control.

5 Mechanistic Analysis

The analysis studies ridge logistic regression in high-dimensional few-shot settings, where regularization shapes both test accuracy and concept-vector stability. A Gaussian teacher–student model and CGMT yield deterministic characterizations linking the ridge strength to signal alignment and orthogonal energy.

  • High-dimensional probing can make unregularized logistic regression ill-posed, motivating ridge regularization as a primary determinant of concept-vector quality.The representation dimension p may be comparable to or exceed the sample count n, and maximum-likelihood estimates may fail to exist or diverge.
  • Ridge strength λ controls the learned direction’s decomposition into signal-aligned and orthogonal components, thereby linking regularization to directional stability.Suppressing orthogonal energy or increasing the signal component improves cosine similarity between directions learned from perturbed data.
  • CGMT characterizes ridge logistic regression in the proportional regime through deterministic order parameters and an auxiliary scalar.The analysis takes n, p →∞ with n/p →δ and uses parameters (¯α, ¯σ, ¯γ) to describe the estimator.
  • The asymptotic test accuracy is expressed through the joint distribution of teacher and learned scores and the fixed-point solution.The resulting accuracy characterization depends on (¯α, ¯σ), with the Gaussian score representation providing the bridge to out-of-sample classification.
  • Empirical subsampling experiments test whether performance trends are primarily governed by the aspect ratio δ = n/p.The study sweeps six sampling fractions with five seeds and compares RAPTOR’s held-out accuracy with a theory-inspired structure predictor.

6 Discussion and Conclusion

The discussion presents RAPTOR as a minimal, validation-tuned ridge-logistic standard for concept extraction in activation steering. Benchmark results support strong accuracy, directional stability, and low computational cost, while the theoretical model remains stylized relative to real LLM representations.

  • RAPTOR reduces probe design to a single validation-selected ridge parameter λ for inference-time intervention.The paper frames this as a minimal approach to extracting concept vectors from frozen representations.
  • Across the benchmark, RAPTOR matches strong alternatives in accuracy while offering superior directional stability and negligible computational cost.The reported practical implication is that more complex estimators are often unnecessary for standard activation-steering tasks.
  • The Gaussian teacher–student theory provides insight into regularization and stable directions but simplifies the distribution of real LLM representations.The authors identify more realistic feature dependencies as a direction for future work.

A CGMT derivation of Theorem 5.1

The appendix derives Theorem 5.1 through a self-contained CGMT route from the ridge-logistic empirical-risk problem to a deterministic characterization. The resulting fixed-point equations are equivalent to those of Salehi et al. after matching notation.

  • The appendix provides a self-contained CGMT derivation connecting the empirical-risk formulation to deterministic fixed-point equations.
  • After matching notation, the derived equations coincide with Salehi et al.’s characterization.
  • The main text states the fixed-point system and limiting test accuracy, while the appendix supplies the derivational route.

A.1.1 Data model and proportional regime

The derivation specifies a Gaussian teacher–student data model, rescales ridge logistic regression, separates teacher-aligned and orthogonal components, and applies CGMT to obtain scalar fixed-point and accuracy characterizations.

  • Data model and proportional regime: The model uses Gaussian features in the proportional regime, with n and p growing while n/p →δ.Labels are generated by a logistic teacher whose dependence on the feature matrix is mediated by the teacher-direction scores.
  • CGMT reduction: Fenchel duality converts the logistic-loss objective into a convex saddle formulation with dual variables w ∈[0, 1]^n.The dual variable arises from the coordinate-wise convex conjugate representation of the log-partition function.
  • Data model and proportional regime: The analysis rescales β through z := β/√p, preserving the optimizer while making the Gaussian bilinear form standard.The high-dimensional limit is derived for z and then translated back to β.
  • CGMT reduction: Gaussian orthogonal decomposition isolates label dependence in the teacher score and leaves an independent Gaussian matrix in the orthogonal bilinear term.This independence reduction is the condition that enables application of CGMT.
  • CGMT reduction: CGMT replaces the primary optimization with an auxiliary optimization whose optimizer converges to the primary optimizer under strict-separation conditions.The subsequent scalar reduction uses norm linearization, proximal operators, and concentration of empirical averages.
  • Fixed-point and accuracy characterization: The resulting fixed-point system determines the estimator’s aligned and orthogonal energies and the asymptotic test accuracy.The final performance characterization evaluates the sign classifier under the Gaussian teacher–student model.

A.8.3 What are the independent variables, and how does Acc vary?

The theory expresses accuracy through δ, λ, and κ via fixed-point parameters, with an effective margin m summarizing the learned classifier’s role. For fixed κ, accuracy increases with m, reaching chance when m=0.

  • Accuracy is a deterministic function of δ, λ, and κ through the fixed-point solution (ᾱ, σ̄).The dependence is mediated by the theorem’s fixed-point characterization.
  • The learned classifier enters the accuracy expression only through the effective margin m, while the teacher enters through κ.
  • For fixed κ, accuracy increases with effective margin m because test scores become more aligned with teacher scores and contain less orthogonal noise.
  • When m=0, the test score is independent of y and accuracy equals 1/2.This is the chance-accuracy limit described in the analysis.

B Experimental details

RAPTOR’s experimental procedure trains per-layer probes using train, validation, and test splits, tunes λ when validation data are available, then refits on the combined training data. The resulting logistic weights are converted from standardized coordinates for evaluation and concept-vector use.

  • Algorithm 2 takes per-layer representations, binary labels, and train/validation/test index splits as inputs.It produces per-layer concept vectors together with learned probe parameters.
  • When validation data are available, RAPTOR tunes λ, combines training and validation examples, and refits the logistic probe on the full development set.
  • After fitting in standardized coordinates, RAPTOR folds the learned weights back through the feature scales to obtain concept vectors in the original representation coordinates.
  • The implementation searches C over logspace(−4, 2, 100) with warm starts and selects the value giving the highest validation accuracy.Because C=1/λ, the selected inverse-regularization parameter determines the ridge strength.

B.2 Linear separable

The experiments motivate ridge logistic probing because layer representations are frequently linearly separable in the few-shot setting. Steering then uses learned directions with adaptive, minimal-strength interventions, optional reliability filtering, and cost measurements across layers and prompts.

  • Linear separability: 90.5% of layers are separable overall, with dataset-level counts ranging from 22/28 to 28/28.Separability is diagnosed when standardized LinearSVC reaches zero training error for some tested penalty.
  • Steering protocol: Steering applies additive interventions along learned concept directions and chooses per-layer strengths to reach extreme probe probabilities.Towards targets are approximately 0.9999 and away targets approximately 0.0001.
  • Steering protocol: The intervention objective minimizes |α| subject to probe-target success, with a no-op when the baseline already meets the target.
  • Steering protocol: Reliability filtering skips layers whose probe accuracy falls below τ, while reported metrics include success rate, intervention rate, and the distribution of |αℓ|.
  • Steering outcomes: Succ.=1.000 across all runs after filtering, while intervention rates vary and steering-strength costs have heavy-tailed p90 and maximum values.Filtering can reduce unnecessary interventions while preserving overall success.
  • Steering outcomes: Late layers tend to have heavier steering-strength tails, and away-direction control exhibits more extreme maxima.

B.6 Qualitative steering examples

The paper illustrates steering with qualitative examples across factuality, hate, sarcasm, and sentiment settings, while also validating theory-inspired accuracy trends on real embeddings. The examples demonstrate behavioral changes but include artifacts under large interventions, and the structural predictor shows positive accuracy-ranking correlations.

  • Qualitative steering: The sanity checks illustrate typical behavior as well as artifacts such as language mixing under large interventions.
  • Qualitative steering: Qualitative steering examples cover Counterfact, HateXplain, Sarcasm, and STSA settings in both towards and away directions.
  • Qualitative steering: The examples show changes in factual claims, toxicity-related language, sarcasm, and sentiment between original and steered completions.
  • Theory validation: The real-embedding experiment varies δ=n/p through stratified subsampling and compares empirical held-out accuracy with a theory-inspired scalar predictor.
  • Theory validation: Spearman ρs measures ranking consistency across δ, while Pearson r measures agreement in accuracy magnitude; correlations are consistently positive and often high.
  • Layerwise evaluation: Figures 4 and 5 plot mean layerwise probing accuracy by dataset and model, with each model stack ordered consistently across columns.
Loading 2602.00158v2…