Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders

Yi Jing; Zao Dai; Jinwu Hu; Zijun Yao; Lei Hou; Juanzi Li; Xiaozhi Wang

Source-linked AI summary

Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders

Yi Jing, Zao Dai, Jinwu Hu, Zijun Yao, Lei Hou, Juanzi Li, Xiaozhi Wang

arXiv:2605.27354v1May 26, 2026cs.LGcs.AIcs.CL

TL;DR

Post-training RL data engineering largely relies on external signals, leaving model-internal information underexplored. SAERL uses SAE activations to model data diversity, difficulty, and quality and maps them to concrete engineering operations, yielding consistent accuracy and convergence-efficiency gains across model scales and RL algorithms. The paper also identifies mathematical-reasoning scope and limited supervision as boundaries of the evidence.

Problem
External post-training data-engineering signals can be costly, while model-internal feedback remains underexplored for reinforcement-learning post-training.
Method
SAERL uses SAE activations to model diversity, difficulty, and quality for batching strategy, curriculum ordering, and data filtering.
Results
SAERL demonstrates consistent accuracy and convergence-efficiency gains across model scales and RL algorithms, with a single SAE transferring across model families.
Takeaways & Limitations
Model internals are a powerful and practical source of signals for post-training data engineering, complementary to external feedback-based approaches.
Takeaways & Limitations
Empirical validation focuses on mathematical reasoning with verifiable rewards, and transfer to other post-training settings remains to be established.

Abstract

from arXiv · show

Model internals encode rich information about how a large language model (LLM) processes its training data; however, post-training data engineering largely relies on external signals and ignores rich intrinsic signals lying in model internals. We propose SAERL, a data engineering framework for LLM reinforcement learning (RL). It models three intrinsic data properties: diversity, difficulty, and quality, using model internals extracted with Sparse Autoencoder (SAE), an advanced mechanistic interpretability tool. Each property grounds a concrete data engineering operation: SAE-space clustering with moderate batch mixing for batch diversity control, a difficulty proxy for easy-to-hard curriculum ordering, and a quality probe for data filtering. SAERL improves average accuracy by 3.00% over vanilla GRPO and reaches target accuracy with 20% fewer training steps on Qwen2.5-Math-1.5B, with consistent gains across model scales and RL algorithms. Experiments show that SAE transfers effectively across model families and scales, serving as a lightweight and reusable data engineering tool. These results demonstrate that model internals are a powerful and practical source of signals for post-training data engineering.

1 Introduction

Post-training data engineering strongly shapes training efficiency and final performance, yet RL pipelines mainly use costly external signals while leaving model-internal feedback underexplored. SAERL addresses this gap by using SAE-derived diversity, difficulty, and quality signals in concrete data operations.

Post-training data engineering affects training efficiency and final performance through sample selection, ordering, and batching strategies.
Existing pipelines typically rely on external signals such as human preferences, verifier outcomes, rollout pass rates, and difficulty signals for selection and curriculum learning.
External signals can be costly to obtain and apply throughout training, while model-internal data-feedback signals remain largely underexplored for RL post-training.
SAE activations provide sparse, fine-grained, and disentangled perspectives on LLM internals.
SAERL models diversity, difficulty, and quality with SAE activations and maps them to batching strategy, curriculum ordering, and data filtering.
SAERL improves post-training data engineering by jointly modeling data diversity, sample difficulty, and data quality with SAEs.

2 Motivating Finding

Motivating studies test whether SAE representations encode diversity-, difficulty-, and quality-relevant information. Across these properties, SAE features support semantic coverage measurement, difficulty-aware curricula, and quality-aware filtering.

Together, the preliminary findings identify diversity, difficulty, and quality as actionable intrinsic data properties for SAERL.
Data diversity: SAE activations encode topic-level semantic structure, supporting their use for measuring data coverage and diversity.Linear probes substantially outperform the majority-class baseline across topic granularities, including 82 leaf topics.
Data difficulty: SAE features strongly predict in-distribution difficulty and retain a positive signal under distribution shift.The difficulty signal extends beyond shallow cues such as length or topic.
Data quality: SAE features outperform mean and metadata-only baselines for quality prediction on PRM800K.Test Pearson correlation improves from 0.2100 with metadata features to 0.3715 with SAE features.

3 Methodology

SAERL is an offline RL post-training framework that uses pooled SAE activations as a shared representation for data quality, difficulty, and diversity. It maps these signals to filtering, cluster-first curriculum construction, and moderate batch mixing.

SAERL maps diversity, difficulty, and quality to batching strategy, curriculum ordering, and data filtering, respectively.
SAE representation: SAERL uses pooled token-level SAE activations, separated across prompt and solution spans, as a shared representation for data engineering.Mean and max pooling capture sustained and localized activation patterns.
Batch diversity: SAERL clusters samples in SAE space and applies moderate cross-cluster batch mixing to control batch diversity.Moderate mixing improves over pure-cluster batches, whereas excessive mixing hurts optimization.
Curriculum ordering: SAERL estimates difficulty from SAE representations and uses calibrated scores to construct cluster-first easy-to-hard curricula.Samples are sorted within clusters into batches, then batches are interleaved across clusters by stage.
Data filtering: An SAE-based quality probe filters noisy data toward target-distribution samples before curriculum ordering.The probe is trained as a supervised classifier and selects samples using a threshold or top-k ranking.

4 Main Experiment

SAERL is evaluated on mathematical reasoning across model scales, benchmarks, and RL algorithms, with comparisons spanning accuracy, convergence efficiency, and noisy-data selection. The experiments show stronger performance, faster convergence, and effective SAE-based quality discrimination.

Experiment Setup: SAERL is evaluated on Qwen2.5-Math-1.5B and Qwen2.5-Math-7B using DeepMath-103K, with GRPO and DAPO implementations.A shared SAE trained on Qwen3-1.7B layer-27 activations guides data engineering across model families and larger scales.
Evaluation: The evaluation covers six mathematical-reasoning benchmarks spanning lower, middle, and competition-level difficulty.GSM8K and AMC23 are lower difficulty; MATH500 and MinervaMath are mid-level; OlympiadBench and AIME24 are competition-level.
Training Performance: SAERL improves average accuracy across RL algorithms, baselines, and model scales.At 1.5B, it improves both GRPO and DAPO; at 7B, SAERLG achieves the best average result among compared methods.
Training Efficiency: SAERL improves convergence across model scales and RL algorithms, with SAERLD fastest on average at 1.5B and SAERLG fastest at 7B.Training steps measure how quickly methods reach a shared target accuracy, with lower values indicating greater efficiency.
Training Efficiency: SAERL obtains convergence gains with lower preprocessing overhead than methods requiring generated labels or multiple rollouts.ADARFT takes approximately 17.33 H100 GPU hours, whereas SAE encoding 103,022 samples takes about 0.5 H100 GPU hours.
Noisy-Data Selection: The SAE-only source/style probe achieves 0.9911 ROC-AUC and 0.9910 AP, while top-50k selection reaches 99.92% DEEPMATH purity.On the mixed raw pool, the p95 threshold retains 103,121 samples with 95.37% DEEPMATH purity and 95.46% recall.

5 Analysis

SAERL’s gains arise from the joint use of difficulty ordering, SAE-space grouping, and moderate batch mixing. The analysis shows that moderate diversity improves performance and convergence, while SAE clusters expose curriculum-relevant structure beyond topic labels.

Component contribution: Removing difficulty sorting causes the largest degradation, establishing the easy-to-hard trajectory as SAERL’s central component.
Component contribution: Difficulty sorting alone is insufficient because removing clustering and mixing also reduces performance, indicating useful local coherence from SAE-space grouping.
Batch diversity control: mix8 achieves the best step-800 performance and reaches the fixed mean@8 threshold τ = 43.0% in the fewest training steps.Performance improves from mix0 to mix8, then declines at stronger mixing levels.
Component contribution: Moderate batch mixing is most effective when layered onto a cluster-first, easy-to-hard curriculum; mixing without difficulty sorting can weaken the curriculum.
Batch diversity control: mix16 and mix32 reduce final accuracy and convergence speed despite higher measured diversity, indicating a non-monotonic diversity–performance relationship.Excessive cross-cluster mixing disrupts within-batch gradient coherence more than it reduces cluster-local bias.
Robustness: SAERL outperforms GRPO under Avg@8 at both B = 128 and B = 512, while larger batches narrow the Pass@8 gap.The results suggest that larger batches may dilute the benefit of an ordered learning trajectory.
Interpretability: SAE clusters poorly align with human topic labels but capture problem format, reasoning structure, solution profile, and difficulty relevant to curriculum construction.The reported alignment with topic metadata is purity = 0.1095 and NMI = 0.0881.
Interpretability: Among LightGBM’s top-20 difficulty features, 19 are SAE-derived, and each sample can be audited through activation groups, feature signals, and curriculum position.

6 Related Work

Related work increasingly uses adaptive post-training data engineering and model internals as training signals. SAERL differs by using sparse, fine-grained SAE features rather than relying mainly on external or coarse internal signals.

Post-training data engineering: Post-training data engineering has progressed from task- or replay-based curricula to quality-aware selection and optimization- or resource-aware data selection.
Post-training data engineering: Existing approaches still rely mainly on external or scalar signals, whereas SAERL grounds data engineering in model-internal structure.
Model internals as training signals: Model-internal training signals include logits or losses, gradients or influence estimates, and hidden states or activations for selection and intervention.
Model internals as training signals: SAERL uses sparse autoencoder features to provide sparse, fine-grained activation signals for RLVR post-training, extending prior SAE applications in diversity and preference modeling.

7 Conclusion

SAERL frames model-internal sparse representations as actionable signals for post-training data engineering. Its experiments show consistent accuracy and convergence-efficiency gains across model scales and RL algorithms, with one SAE transferring across model families.

SAERL uses SAE activations as a shared representation space for diversity, difficulty, and quality, mapping them to batching, curriculum ordering, and data filtering.
Experiments on mathematical reasoning demonstrate consistent gains in accuracy and convergence efficiency across model scales and RL algorithms.
A single SAE transfers effectively across model families, supporting model internals as a practical source of post-training data-engineering signals.

Limitations

The empirical validation is limited to mathematical reasoning with verifiable rewards, and SAERL still depends on modest supervision and unproven assumptions about SAE-space proximity.

Domain scope: Validation focuses on mathematical reasoning with verifiable rewards, leaving transfer to code, agents, tools, multi-step decisions, and general instruction following unestablished.
Limited supervision: SAERL is not fully unsupervised because its difficulty proxy uses labeled difficulty data and its quality probe relies on source or distribution labels.
Theoretical scope: SAE-space proximity is treated as a proxy for semantic similarity and gradient coherence, but the analysis does not prove a causal relationship with training dynamics.

Ethical Considerations

The work addresses ethical risks of using model-internal SAE representations in post-training data engineering. The authors limit experiments to mathematical reasoning and recommend safeguards for broader use.

Potential Risks: SAERL’s internal signals could be misused to optimize data for unsafe behaviors.The stated risk concerns applications beyond the paper’s mathematical-reasoning setting.
Potential Risks: The experiments are restricted to mathematical reasoning with verifiable rewards.
Potential Risks: The authors recommend safety filtering and human oversight for broader applications.
Artifacts: Released artifacts are described as publicly available research materials used under their applicable licenses and terms.The authors also state that released code and processed artifacts will follow corresponding license requirements.
Intended Use: SAERL is intended for research uses including curriculum construction, batch organization, data filtering, and interpretability-oriented analysis.

A A Bias–Variance View of Moderate Batch Mixing

The bias–variance analysis explains why moderate cross-cluster mixing can improve optimization: pure clusters reduce variance but may introduce bias, whereas excessive mixing weakens gradient coherence. The section also describes the SAE-based pipeline used to construct and filter curricula.

Bias–Variance Analysis: Pure-cluster batches have low variance but may be biased when their cluster mean differs from the target gradient.The MSE decomposes into cluster-local bias and estimation variance.
Bias–Variance Analysis: Moderate mixing can reduce cluster-local bias, while excessive mixing weakens within-batch gradient coherence.When the mixing optimum is interior, mixing utility has an interior maximum.
Curriculum Construction: The curriculum clusters samples in SAE-augmented feature space, estimates difficulty from a small labeled subset, and interleaves local easy-to-hard batches across clusters.The described implementation uses MiniBatchKMeans with K = 10 and an ElasticNet difficulty proxy calibrated with cluster residual shrinkage.
Moderate Batch Mixing: Moderate batch mixing exchanges only a tail block with a nearby batch having similar difficulty and sequence length but a different dominant cluster.This preserves most local curriculum structure while introducing limited cross-cluster mixing.
Quality Filtering: The quality probe treats raw data selection as binary classification over SAE representations and supports threshold-based or fixed-size top-k selection.
Cluster Structure: SAE clusters do not simply reproduce human-defined topic labels, instead reflecting activation-space structure such as reasoning format, solution profile, and difficulty.Reported cluster-topic alignment is low, with purity 0.1095, NMI 0.0881, and ARI 0.0394.

D.2 Feature-Level Signals

Difficulty prediction relies primarily on SAE-derived features rather than metadata. Solution-side mean activations provide the strongest correlational signal, while prompt-side maxima add localized problem-statement cues.

Feature-Level Signals: 19 of the top-20 and 97 of the top-100 gain-ranked difficulty features are SAE-derived.Only 1 top-20 feature and 3 top-100 features are metadata features.
Feature-Level Signals: Solution-side mean features dominate the ranked signals, consistent with sustained solution-side activations correlating with difficulty.
Feature-Level Signals: Prompt-side max features contribute localized predictive cues associated with symbolic structure or diagrammatic form.
Semantic Inspection: High-activation SAE features show recurring semantic tendencies across mathematical areas and formats, including algebra, integration, combinatorics, geometry, and number theory.These observations are semantic tendencies rather than strict one-feature-one-meaning mappings.
Pipeline Inspection: The curriculum pipeline records representations, metadata, cluster assignments, difficulty predictions, calibrated scores, batch assignments, and tail-swap partners for inspection.