PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning a Generative Meta-Model of LLM Activations

Grace Luo, Jiahai Feng, Trevor Darrell, Alec Radford, Jacob Steinhardt

arXiv:2602.06964v1cs.LGcs.AIcs.CL

TL;DR

Existing activation-analysis methods rely on restrictive structural assumptions, motivating models that learn activation structure directly. The paper trains diffusion-based Generative Latent Priors on one billion residual-stream activations and finds that their learned priors improve steering fluency while their meta-neurons isolate interpretable concepts, with both uses scaling with diffusion loss.

  • Problem

    Existing approaches for analyzing and manipulating activations often rely on linearity or other structural assumptions and can produce corrupted activations that degrade fluency.

  • Method

    The paper trains a diffusion model called GLP on one billion residual-stream activations to learn their distribution and uses it as a prior and feature extractor.

  • Results

    GLP improves steering fluency at the same steering effect, while its meta-neurons outperform SAE features and raw LLM neurons on 1-D probing; both applications improve with scale and track diffusion loss.

  • Takeaways & Limitations

    Generative meta-models offer a promising interpretability primitive that sidesteps restrictive structural assumptions and whose downstream utility improves predictably with compute.

  • Takeaways & Limitations

    The study models single-token residual-stream activations independently at a single layer and uses an unconditional GLP.

Abstract

from arXiv · show

Existing approaches for analyzing neural network activations, such as PCA and sparse autoencoders, rely on strong structural assumptions. Generative models offer an alternative: they can uncover structure without such assumptions and act as priors that improve intervention fidelity. We explore this direction by training diffusion models on one billion residual stream activations, creating "meta-models" that learn the distribution of a network's internal states. We find that diffusion loss decreases smoothly with compute and reliably predicts downstream utility. In particular, applying the meta-model's learned prior to steering interventions improves fluency, with larger gains as loss decreases. Moreover, the meta-model's neurons increasingly isolate concepts into individual units, with sparse probing scores that scale as loss decreases. These results suggest generative meta-models offer a scalable path toward interpretability without restrictive structural assumptions. Project page: https://generative-latent-prior.github.io.

1. Introduction

GLP uses a diffusion model to learn activation structure without restrictive assumptions, addressing the risk that existing methods produce off-manifold activations that degrade fluency. Its learned prior improves steering and probing, with downstream utility scaling predictably with compute.

  • Existing activation-analysis and intervention methods often assume linearity or other structures, which can produce corrupted activations and degraded LLM fluency.
  • GLP is a diffusion model trained on one billion residual stream activations to learn the activation distribution directly.The model uses Frechet Distance and PCA to check whether generated activations resemble real ones.
  • Across sentiment control, SAE feature steering, and persona elicitation, GLP post-processing improves fluency at the same steering effect.Diffusion sampling projects off-manifold interventions back toward the natural activation manifold while preserving semantic content.
  • GLP intermediate representations produce meta-neurons that outperform SAE features and raw LLM neurons on 1-D probing tasks.These results suggest that GLP isolates interpretable concepts into individual units.
  • Across 0.5B–3.3B-parameter models, diffusion loss follows a smooth scaling law, while steering and probing improve with compute and track the loss.The diffusion loss therefore serves as both a training objective and a reliable predictor of downstream utility.
  • GLP treats the trained meta-model as a prior or feature extractor, offering an interpretability path that avoids hand-crafted structural assumptions.

2. Generative Latent Prior

GLP is an activation diffusion model built with flow matching and a feedforward MLP denoiser. It is trained on single-token residual-stream activations using a large-scale producer-consumer data pipeline.

  • 2.1. Diffusion Objective: Flow matching linearly interpolates between real activation z_0 and Gaussian noise ϵ, then reverses this process to generate activations from noise.The reverse process starts from z_1 ∼ N(0, I) and iteratively samples toward clean data.
  • 2.1. Diffusion Objective: GLP trains a neural-network denoiser to approximate the target velocity u = ϵ − z_0.
  • 2.2. Architecture: The denoiser is a stack of Llama3-style feedforward MLP blocks with SwiGLU layers, residual connections, and timestep conditioning.The models are unconditional and omit attention because they model single-token rather than multi-token activations.
  • 2.3. Data Pipeline: GLP is trained on residual-stream activations extracted from documents using a producer-consumer buffer that manages runtime and memory tradeoffs.
  • 2.3. Data Pipeline: The large-scale corpus uses 1 billion FineWeb tokens, with activations collected from token positions up to 2048 tokens and usually from the middlemost layer.
  • 2.4. Scaling GLP: GLP achieves lower Frechet Distance than SAE reconstructions and improves with parameter scale, despite generating from pure noise.

3. Scaling GLP

GLP models activation distributions directly without structural assumptions, with quality assessed by distributional metrics and visualizations. Its diffusion loss scales predictably with compute and tracks downstream steering and probing utility.

  • Generation quality: GLP learns activation distributions directly, using unconditional diffusion models trained on Llama activations rather than imposing structural assumptions.The evaluation pipeline compares generated samples with real activations using Frechet Distance and PCA.
  • Generation quality: GLP samples become relatively indistinguishable from real activations at about 20 diffusion steps in top-2 PCA components.Frechet Distance provides a quantitative comparison alongside the PCA visualization.
  • Generation quality: GLP achieves lower Delta LM Loss than comparable SAE reconstructions despite not being trained for reconstruction.The evaluation also reports transfer from Llama8B-Base to Llama8B-Instruct with minor degradation.
  • Scaling laws: The diffusion loss follows a power law with compute, with an estimated irreducible error E = 0.52 and improvement rate α = 0.169.The fit uses C = 6ND and L(C) = E + A · C^-α.
  • Scaling laws: Steering performance and probing accuracy improve with compute and closely track diffusion loss, making that loss a reliable proxy for downstream utility.The reported scaling relationship supports using diffusion loss as a metric to optimize.

4. On-Manifold Steering with GLP

GLP post-processes steered activations by denoising them back toward the learned activation manifold. Across SAE, persona, and sentiment steering, this improves the tradeoff between concept strength and fluency, with stronger results from better-trained GLPs.

  • On-manifold steering: GLP post-processing addresses steering’s concept–fluency tradeoff by correcting off-manifold activations through diffusion sampling.The method initializes sampling from an off-manifold activation at an intermediate timestep rather than from pure noise.
  • Improving SAEs: GLP post-processing improves the concept–fluency Pareto frontier over SAE steering alone in Llama8B-Base.Figure 5 varies the steering coefficient; scores use an LLM judge on a 0-2 scale with 95% bootstrap confidence intervals.
  • Improving SAEs: GLP improves SAE steering alignment with feature descriptions when decoder directions are off-manifold, indicating that off-manifold artifacts contribute to steering failures.The study evaluates 500 random directions and compares outputs against Neuronpedia feature descriptions.
  • Persona elicitation: GLP expands the persona-steering Pareto frontier over Persona Vectors alone for three behavioral traits in Llama8B-Instruct.The figure compares concept and fluency scores on a 0-100 scale.
  • Scaling behavior: GLPs trained with more compute achieve better sentiment-steering performance, especially when the steering vector norm exceeds the average activation norm.The highlighted regime aggregates results for relative coefficient r ≥ 1.

5. Interpreting with GLP

GLP is evaluated as a nonlinear feature encoder for interpreting LLM activations through single-unit concept probes. Its meta-neurons outperform baselines and improve with scaling, while probing identifies interpretable activation patterns.

  • Baseline comparison: GLP meta-neurons substantially outperform all baselines on aggregated 1-D probing across Llama1B and Llama8B.Table 4 aggregates results over 113 tasks with 95% bootstrap confidence intervals.
  • Method: 1-D probing tests whether a single GLP meta-neuron can predict a binary concept, assessing concept isolation and interpretability.The evaluation covers 113 binary classification tasks spanning language, knowledge, biology, and mathematics.
  • Method: Meta-neurons are extracted from GLP’s internal SwiGLU-gate representations after noise is added to inputs at a selected timestep.The procedure uses a single forward pass through the diffusion model.
  • Baseline comparison: Raw MLP neurons are the strongest Llama8B baseline, while SAEs are slightly worse than raw layer outputs.This comparison indicates that native MLP representations already provide some disentanglement.
  • Baseline comparison: Llama1B GLP outperforms all Llama8B raw activations, suggesting GLP can improve parsimonious representations without relying solely on larger language models.The result is reported for 1-D probing under the stated candidate-filtering and validation-selection procedure.
  • Scaling: 1-D probing performance continues improving across GLP model and data scales, with no curve showing a plateau.The cleanest scaling trend is reported for inputs at timestep t = 0.5.

6. Related Work

GLP models activation distributions rather than network weights or merely verbalizing activations. This positions the method as a nonlinear meta-model used directly for steering and probing.

  • Meta-models: Meta-model research treats neural networks as a data modality, but modeling weights requires expensive data generation and techniques for permutation symmetry.GLP avoids these challenges by modeling activations instead of weights.
  • Meta-models: Unlike prior diffusion work on DINO activations for generation or conditioning, GLP uses the meta-model itself as a steering prior and probing encoder.The distinction concerns how the learned diffusion model is used downstream.
  • Activation modeling: Many activation-interpretability methods impose linear assumptions, including sparse autoencoders and vector-arithmetic approaches.GLP instead models the activation distribution with a nonlinear diffusion model.
  • Activation modeling: Natural-language activation-description methods verbalize activations rather than modeling their distribution, making them complementary to GLP.The cited methods include SelfIE and LatentQA.
  • Diffusion language models: Diffusion language models generate language from token or embedding processes, whereas GLP is designed to manipulate activations from existing language models.The related approaches are trained to compete with autoregressive language models rather than understand them.

7. Discussion

The discussion presents GLP as a scalable interpretability primitive with two downstream roles: preserving on-manifold steering and extracting interpretable concepts. The authors also identify important scope limitations and future applications.

  • Main findings: GLP serves as both a prior that keeps steering interventions on-manifold and a feature extractor whose meta-neurons isolate interpretable concepts.Both applications improve with scale and track diffusion loss.
  • Main findings: The authors suggest generative meta-models can provide interpretability without restrictive structural assumptions.This conclusion is tied to the observed use cases and their scaling behavior.
  • Limitations: The current approach models single-token residual-stream activations independently at one layer, limiting cross-position and activation-type coverage.The authors propose multi-token modeling and extensions to other activation types or layers.
  • Limitations: GLP is unconditional, and conditioning on clean rather than noised activations may reduce information loss for steering.This is presented as a direction for future work, not as an established improvement.
  • Future directions: High GLP diffusion loss might flag unusual or out-of-distribution activations, extending diffusion-based typicality ideas to interpretability.The paper presents this as a possible future application.

B. Scaling: Extended Results

The paper also compares single-layer and multi-layer GLPs by training a model across all 16 Llama1B layers and conditioning it on layer position.

  • Multi-layer modeling: A multi-layer GLP is trained on activations from all 16 layers of Llama1B.The model is adapted to represent layer position explicitly.
  • Multi-layer modeling: Layer position is encoded with a sinusoidal embedding and added to the timestep embedding.This conditions the shared multi-layer architecture on which layer produced the activation.
  • Scaling comparison: The single-layer and multi-layer models are compared on middle-layer activations, where the single-layer model is specialized.The comparison is presented as an analysis of their scaling behavior.

B.2. Additional PCA Visualizations

Additional visualizations compare GLP and SAE reconstructions in PCA space and examine how diffusion-model scale relates to steering performance. They also report multi-layer representation distances and steering evaluation details.

  • PCA visualizations: GLP and SAE reconstructions are relatively indistinguishable from real activations in the top-2 PCA components.The comparison is more favorable to SAE reconstructions because GLP generates unconditionally from pure noise.
  • Multi-layer scaling: Multi-layer GLPs have final representation Frechet Distance 0.66 versus 0.55 for specialized single-layer GLPs.The comparison uses activations from the middlemost layer, where the single-layer model is specialized.
  • Multi-layer scaling: Figure 8 loss curves are used to estimate the FLOPsmulti-layer/FLOPssingle-layer exchange rate at matched diffusion loss.The single-layer compute is obtained by piecewise linear interpolation.
  • Steering evaluation: Steering visualizations plot on-manifold performance against diffusion loss and separately report concept and fluency scores.The steering evaluation uses a five-point sentiment classifier for concept and negative log-likelihood for fluency.
  • Steering evaluation: The steering results average relative steering coefficients ≥1, where GLP is most helpful and checkpoint performance is more dispersed.The larger spread in this regime is shown in the steering-coefficient analysis.

C.5. Experimental Configurations

The experimental configurations cover steering benchmarks, sentiment evaluation, qualitative comparisons, and probing analyses. Additional evaluations vary steering scale, scoring procedures, and diffusion-input noisiness.

  • Experimental setup: The on-manifold steering experiments use configurations detailed in Table 9, with qualitative examples reported separately for each steering setting.The qualitative tables cover checkpoint evaluation, SAE steering refinement, and persona elicitation.
  • Sentiment control: The sentiment figure reports concept with a five-point classifier and fluency with negative log-likelihood, including 95% bootstrap confidence intervals from 10,000 resamples.Higher concept scores are better, while lower fluency negative log-likelihood is better.
  • Probing configurations: Figure 15 compares probing performance across compute and diffusion loss while varying noisy versus relatively clean diffusion inputs.Scaling trends are cleaner at timestep t = 0.5 than at t = 0.1.
  • Probing configurations: Dense probing uses all available features and shows cleaner scaling trends for noisier inputs, with better performance as GLP compute increases.Dense probing selects the best-performing layer using validation AUC.

D.3. Additional 1-D Probing Results

Additional 1-D probing results validate the feature-selection procedure, examine feature availability and meta-neuron locations, and compare probing across compute, loss, and diffusion-input noise.

  • Feature selection: Pre-filtering produces no observable difference in 1-D probe results compared with exhaustively probing all available features.The comparison reports average AUC with 95% confidence intervals.
  • Feature availability: Llama1B GLP has more available features than the SAE, while Llama8B GLP has fewer; GLP nevertheless significantly outperforms SAE in probe AUC in both cases.The analysis reports no noticeable relationship between feature count and 1-D probe performance.
  • Meta-neuron locations: The middlemost diffusion layer contains the most semantically rich meta-neurons among the best-performing Llama8B GLP features.Figure 16 locates the best-performing meta-neuron for each 1-D probing task and colors layers by task-specific frequency.
  • Dense probing: Dense probing uses all available features and compares scaling across FLOPs versus diffusion loss and noisy versus clean inputs.Unlike 1-D probing, dense probing does not pre-filter layer features.
Loading 2602.06964v1…