PaperlayerView exact revision on arXiv

Source-linked AI summary

Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders

Zhongzhi Li, Xuansheng Wu, Yijiang Li, Lijie Hu, Ninghao Liu

arXiv:2602.10388v4cs.CLcs.AI

TL;DR

Post-training data must be diverse, but conventional text- and embedding-based metrics provide limited evidence about task-relevant features. The paper introduces FAC and FAC Synthesis, using sparse autoencoders to identify missing features and guide targeted generation; experiments report stronger feature coverage and downstream performance across four tasks, with 2,000 synthetic samples matching MAGPIE while using 150× less data. A stated limitation is that single-layer SAE features may not capture sophisticated reasoning based on distributed multi-layer circuits.

  • Problem

    Collecting comprehensive post-training data is difficult, and existing diversity metrics weakly capture the task-relevant features associated with downstream performance.

  • Method

    FAC measures coverage of task-relevant internal features, while FAC Synthesis uses sparse autoencoders to identify missing features and generate targeted synthetic samples.

  • Results

    Across four tasks and three model families, FAC Synthesis improves FAC and consistently outperforms baseline synthesis methods; it matches MAGPIE with 2,000 synthetic samples versus 150× more data.

  • Takeaways & Limitations

    Task-relevant feature coverage provides a practical basis for measuring diversity and guiding data-centric optimization of LLM post-training.

  • Takeaways & Limitations

    Single-layer SAE features may be insufficient for sophisticated reasoning behaviors that depend on distributed multi-layer circuits.

Abstract

from arXiv · show

The diversity of post-training data is critical for effective downstream performance in large language models (LLMs). Many existing approaches to constructing post-training data quantify diversity using text-based metrics that capture linguistic variation, but such metrics provide only weak signals for the task-relevant features that determine downstream performance. In this work, we introduce Feature Activation Coverage (FAC) which measures data diversity in an interpretable feature space. Building upon this metric, we further propose a diversity-driven data synthesis framework, named FAC Synthesis, that first uses a sparse autoencoder to identify missing features from a seed dataset, and then generates synthetic samples that explicitly reflect these features. Experiments show that our approach consistently improves both data diversity and downstream performance on various tasks, including instruction following, toxicity detection, reward modeling, and behavior steering. Interestingly, we identify a shared, interpretable feature space across model families (i.e., LLaMA, Mistral, and Qwen), enabling cross-model knowledge transfer. Our work provides a solid and practical methodology for exploring data-centric optimization of LLMs.

1. Introduction

Post-training dataset diversity is important but difficult to collect systematically, while common text- and embedding-based metrics weakly represent task-relevant features. The paper introduces FAC and FAC Synthesis to measure and improve feature coverage for downstream performance.

  • Dataset diversity supports effective post-training, but collecting comprehensive long-tailed samples is difficult.
  • Existing diversity metrics mainly measure lexical, syntactic, or generic semantic variation rather than task-relevant model features.
  • FAC quantifies data diversity through coverage of task-relevant features in a model’s internal feature space.
  • FAC Synthesis uses missing task-relevant features to guide generation of synthetic samples.
  • Pearson r = 0.95 and Spearman ρ = 0.90 quantify FAC’s reported correlation with downstream task performance.
  • FAC Synthesis matches prior SOTA MAGPIE using 2,000 synthetic samples, whereas MAGPIE requires 150× more data.

2. Related Work

Existing diversity metrics and synthesis methods often operate in text or generic embedding spaces, emphasizing linguistic variation or simple prompting rather than task-relevant feature coverage.

  • Text- and embedding-space metrics often quantify lexical, syntactic, or semantic variation without capturing task-relevant latent features.
  • Current LLM data synthesis methods rarely guide generation toward diversity and mainly rely on simple prompting.

3. Preliminaries

The paper constructs its feature space with sparse autoencoders that represent LLM activations as sparse, interpretable latent features. The encoder-decoder is trained to reconstruct inputs while controlling activation sparsity.

  • Sparse autoencoders extract interpretable features from LLM internal activations.
  • An encoder maps an input embedding x ∈ R^d to sparse activations z = σ(xW), and a decoder reconstructs x̂ = zW^⊤.
  • The SAE uses k ≫ d features, with each feature intended to capture a distinct latent pattern potentially relevant to the task.
  • Training minimizes reconstruction error plus λ∥z∥1, where λ controls sparsity.

4. Quantify Generalization of Synthetic Data

The paper characterizes synthetic-data generalization error through a distribution gap and a sampling error. This framework motivates reducing the gap between the target task distribution and the synthetic-data distribution.

  • Effective synthetic data requires both a distribution close to the target domain and representative finite samples.
  • The generalization-error bound separates a distribution gap between D and Dgen from sampling error on Sgen.
  • The two-term bound identifies distribution matching and finite-sample representativeness as separate requirements for synthetic-data effectiveness.
  • Total variation distance ΔTV(D, Dgen) measures the gap between the task-domain and synthetic-data distributions.
  • Sampling error measures the difference between expected risk under Dgen and empirical risk on synthetic data.

5. Reduce Distribution Gap in Feature Space

The paper reduces the target-to-generated distribution gap in SAE feature space rather than text space, focusing synthesis on task-relevant features absent from generated data. FAC measures coverage of these features, while missing-feature synthesis guides data augmentation.

  • Text-space selection is sensitive to linguistic variation, whereas SAE features encode task-aligned semantics and functional properties.
  • The distribution gap is bounded by divergence in SAE feature space plus an additional non-optimizable conditional term.
  • The framework estimates the target feature distribution with a relevant anchor corpus and synthesizes samples that activate missing features.
  • FAC measures the fraction of task-relevant features covered by generated data relative to those present in the target-domain anchor distribution.
  • Missing features are those present under the anchor distribution but absent under the generated distribution, forming the synthesis targets.

6. Reduce the Sampling Error under Dgen

The paper treats sampling error as a consequence of finite synthetic datasets and connects it to uncertainty in the generated data. FAC Synthesis addresses this uncertainty through contrastive, SAE-filtered generation targeting missing features.

  • Finite synthetic datasets can imperfectly estimate the training objective even when their distribution matches the target domain.
  • The PAC-Bayesian analysis bounds sampling error using mutual information between the synthetic dataset and post-trained model parameters.
  • Reducing uncertainty in the synthesized dataset, measured by H(Sgen), is crucial for reducing sampling error.
  • Naive prompting provides limited control over target-feature expression, causing variable synthetic samples and high dataset uncertainty.
  • FAC Synthesis constructs contrastive pairs, generates candidates conditioned on them, and retains samples that strongly activate each missing feature.
  • The two-step procedure reduces conditional entropy and estimation error by constraining generation toward target features.

7. Experiments

The experiments evaluate FAC Synthesis across four downstream tasks and investigate performance, feature relevance, cross-model transfer, human reasonableness, and hyperparameter sensitivity. The framework uses SAE-based feature extraction, missing-feature identification, and filtered synthesis alongside established baselines.

  • Research questions: The evaluation asks whether coverage-guided data improves fine-tuned performance, whether missing features relate to performance, and whether they transfer across models.
  • Experimental setup: Experiments cover Toxicity Detection, Reward Modeling, Behavior Steering, and Instruction Following using corresponding public benchmarks.
  • Evaluation: Toxicity Detection uses AUPRC, Reward Modeling uses Accuracy, and Instruction Following reports Win Rate and Length-Controlled Win Rate.
  • FAC Synthesis: FAC Synthesis extracts SAE-activated features from anchor and seed data, identifies their task-relevant difference, and generates samples for missing features.
  • Baselines: The method is compared with instruction-expansion, self-evolution, alignment-constrained, and other LLM-based synthesis baselines.

1 Our proposed method outperforms baselines across

Across four tasks, the experiments report that explicitly goal-directed synthesis is generally more reliable than unguided instruction expansion or self-evolution. FAC Synthesis targets missing task-relevant SAE features and is reported as the best-performing method across all tasks.

  • Across all four tasks, the results indicate that explicitly goal-directed data synthesis is generally more reliable.
  • Table 1 compares performance on Toxicity Detection, Reward Modeling, Behavior Steering, and Instruction Following, with the best result in each column bolded.For Behavior Steering, SCR is the accuracy difference between activation multipliers of 1 and -1.
  • Instruction expansion and self-evolution methods can be competitive but show unstable performance across tasks without efficient task-specific generation guidance.
  • Objective-driven methods enforcing alignment constraints tend to yield more consistent gains across tasks.
  • FAC Synthesis targets missing task-relevant SAE features and consistently yields the best performance across all tasks.

2 FAC serves as a strong predictor of downstream per-

FAC is strongly associated with downstream AUPRC, and broader coverage of task-relevant missing features improves performance more than simply increasing sample count. Two-step synthesis also produces higher FAC than one-step synthesis at matched activation thresholds.

  • r = 0.95 between FAC and AUPRC indicates a strong linear relationship across observed results.The paper contrasts this with weak correlations for word-level, syntax-level, and embedding-level diversity metrics.
  • Increasing the proportion of covered missing features produces monotonic performance improvement in both synthesis variants.
  • Two-step synthesis consistently achieves higher FAC than one-step synthesis under the same SAE activation threshold.The two-step strategy incorporates contrastive guidance to activate target features more reliably.

1 FAC is the primary driver of performance gains.

The paper evaluates FAC-guided synthesis across model families, qualitative feature analyses, and hyperparameter settings. Results support transfer across LLaMA, Mistral, and Qwen, while showing that generator choice, activation thresholds, and synthesis budget affect outcomes.

  • Cross-model generalization: Shared synthetic data consistently improves performance across LLaMA, Mistral, and Qwen backbones despite differing initial baselines.The result implies a shared SAE feature space across these model architectures.
  • Cross-model generalization: Using LLaMA-3.1-8B-Instruct features for Qwen2-7B-Instruct yields AUPRC gains ranging from 1.60% to 5.13% across three generators.
  • Qualitative analysis: SAE feature analyses find coherent semantic patterns in activated spans, and targeted synthesis can instantiate the corresponding behaviors.One example associates activated spans with rob and steal, representing criminal intent.
  • Hyperparameter sensitivity: Performance peaks at an intermediate decoding temperature, while LLaMA-3.1-8B-Instruct outperforms GPT-4o mini across all tested temperatures.The authors associate conservative decoding with insufficient exploration and overly random decoding with off-target content.
  • Hyperparameter sensitivity: When δ becomes overly large, the missing-feature target set becomes too sparse, constraining coverage and degrading performance.Larger δ otherwise identifies fewer missing task-relevant features and reduces target synthesis samples.
  • Hyperparameter sensitivity: AUPRC increases with more samples per missing feature, but decreasing DES indicates diminishing marginal gains as synthesis size grows.The authors report that most gains require only a small number of samples per feature.

8. Conclusion

FAC Synthesis targets missing task-relevant SAE features and improves FAC across four tasks, while the paper identifies boundaries involving feature representation, transferability, and safety-sensitive use.

  • Conclusion: FAC Synthesis identifies missing task-relevant SAE features and generates targeted synthetic samples, improving FAC and outperforming baselines across four tasks.The framework’s broader scope includes robustness, generality, and limitation analyses in supplementary experiments.
  • Conclusion: Single-layer SAE features may be insufficient for sophisticated reasoning behaviors that depend on distributed multi-layer circuits.The paper cites preliminary GSM8K and LiveCodeBench results as evidence for this scope boundary.
  • Impact and safety: Feature-targeted synthesis can amplify harmful content, so the paper recommends safety-improving objectives, filtering, dataset review, and human oversight.The release plan limits potentially harmful synthetic examples and provides safe-use guidance.
  • Theoretical analysis: The theoretical analysis decomposes the generalization gap into distribution-gap and sampling-error terms.The distribution gap is further decomposed into a feature-marginal term and residual conditional mismatch, while sampling error is bounded using mutual information.
  • Theoretical analysis: Covering missing features reduces a feature-distribution surrogate, although smoothing is required to make the surrogate finite.The surrogate penalizes missing features and overly broad supports, while remaining missing features decrease the corresponding term.
  • Theoretical analysis: Activating task-relevant features decreases conditional entropy H(X | Z), narrowing uncertainty in generated samples.The analysis interprets feature alignment as making outputs more determined by the activated features.
  • Representation: The paper frames SAE feature coverage as a more suitable representation for coverage-guided synthesis than dense embeddings.The appendix discusses SAE features as providing an exact characterization of feature coverage.

D. Proof of Minimizing the Distribution Gap between PZ and QZ

The proof replaces an intractable continuous feature-distribution KL objective with a binary activation-support surrogate. This surrogate penalizes missing or irrelevant feature coverage, and adding samples for missing features reduces it, with smoothing handling partial coverage.

  • Objective construction: The method thresholds continuous SAE activations to obtain binary feature-expression indicators and defines task-relevant coverage through their supports.The extractor maps inputs to SAE activations, while the thresholded indicators make the objective tractable.
  • Objective construction: The surrogate KL objective compares uniform distributions over task-feature supports induced by task and synthetic data.It is designed to avoid directly optimizing the continuous feature-distribution divergence.
  • Objective behavior: Missing features cause infinite surrogate divergence before smoothing because task support assigns them positive mass while synthetic support assigns zero.When all task features are covered, the support inclusion condition removes this missing-support divergence.
  • Objective behavior: The surrogate also penalizes overly broad synthetic supports, encouraging coverage without activating irrelevant features.Thus, the objective is sensitive to both absent task-relevant features and unnecessary expressed features.
  • Feature augmentation: Adding synthetic samples that activate missing features makes the coverage surrogate smaller, and smoothing yields a finite divergence for partial progress.The remaining-missing-feature term decreases as additional missing features are activated, while smoothing prevents undefined divergence.
  • Feature augmentation: The first surrogate term is not necessarily monotone as the expressed feature set expands because probability mass spreads across covered features.Smoothing still provides a uniform lower bound for covered-feature probabilities.

E. Proof of Lemma 6.1 (Upper Bound of Sampling Error)

The paper bounds sampling error under a sub-Gamma loss assumption and connects the bound to mutual information. It also analyzes uncertainty reduction and contrasts exact SAE coverage with an approximate dense-cluster proxy.

  • Sampling-error bound: Under the sub-Gamma loss assumption, the sampling error is bounded through an annealed-risk deviation function and its convex conjugate.The bound concerns deviation between expected synthetic-distribution risk and empirical risk on the finite synthetic sample.
  • Sampling-error bound: Choosing an oracle prior reduces the expected conditional KL divergence to mutual information, yielding an information-theoretic sampling-error bound.The resulting complexity term involves I(Sgen; W), linking the synthetic dataset to post-training parameters.
  • Feature alignment: Increasing activation probabilities for missing features brings the synthetic activation distribution QA closer to the task distribution PA under the Bernoulli surrogate.The mean-field minimizer satisfies qi = pi for every feature.
  • Uncertainty reduction: Enforcing more missing features monotonically decreases conditional entropy, concentrating synthetic samples within the target feature region.The result follows from the shrinking feasible region under additional activation constraints.
  • Uncertainty reduction: Once the joint missing-feature event has nontrivial probability, reducing conditional entropy on that event quantitatively reduces uncertainty in synthetic samples there.The conditional distribution is well-defined when the event has positive probability.
  • Coverage comparison: SAE coverage is exact by construction, whereas dense-cluster coverage is only a proxy and introduces a feature-level disagreement gap.The dense baseline declares a feature covered when its associated cluster appears, which need not mean the feature itself is covered.

I.1. Introduction to the tasks in the experiments

The experiments evaluate FAC Synthesis across multiple model families, tasks, feature-selection settings, synthesis strategies, and training regimes. Results generally support task-relevant SAE feature coverage as a useful guide for improving diversity and downstream performance.

  • Experimental setup: Experiments use LLaMA-3.1-8B-Instruct as the default backbone and additionally evaluate Mistral-7B-Instruct and Qwen2-7B-Instruct for cross-model generalization.Activations are extracted from layer 16 for LLaMA and Mistral and layer 14 for Qwen.
  • Feature identification: 84%–86% of selected features are human-validated as task-relevant, while the unclear rate remains about 5%.A feature is confirmed relevant or irrelevant only when both annotators agree.
  • Diversity metric: FAC correlates strongly with downstream performance, reaching Pearson r = 0.95 and Spearman ρ = 0.90 for Toxicity Detection.Reward Modeling also shows strong correlations of r = 0.85 and ρ = 0.84.
  • Feature coverage: +34.67% Sycophancy and +18.67% Survival improvements result when selecting 100% rather than 30% of features.The same comparison reports +3.52% Toxicity Detection AUPRC, +6.12% Reward Modeling Average Accuracy, and +10.88% Instruction Following WR.
  • Synthesis strategy: Two-step synthesis increases FAC over one-step synthesis across tasks, including +28.57% on Survival at threshold 1.0.At the same threshold, FAC also rises by +5.0% for Toxicity Detection and +3.84% for Reward Modeling.
  • Representation comparison: FAC-guided SAE feature selection reaches 62.60 AUPRC, exceeding dense cluster coverage at 45.57 and random SAE selection at 50.61.The comparison indicates that SAE features are more effective than dense clusters for guiding synthesis.
  • Training settings: LoRA fine-tuning produces the best overall Reward Modeling average accuracy across diverse RewardBench subsets.Compared with head-only training, LoRA enables backbone adaptation and yields stronger improvements across subsets.

L.16. Preliminary Results on Reasoning-Heavy Benchmarks

Preliminary experiments test FAC Synthesis on GSM8K mathematical reasoning and LiveCodeBench code generation, where relevant behaviors may be distributed across multiple layers. The method improves both benchmarks, but gains are smaller than on the paper’s main tasks.

  • Benchmark scope: FAC Synthesis is evaluated on GSM8K and LiveCodeBench as preliminary reasoning-heavy benchmarks.These tasks involve multi-step reasoning and program synthesis, unlike the main evaluations.
  • Results: +0.31 to +1.51 GSM8K accuracy points are obtained across three backbone models.The reported range summarizes before-and-after fine-tuning with FAC-guided synthetic data.
  • Results: 0.00 to +1.53 LiveCodeBench pass@1 points are obtained across three backbone models.The gains are positive overall but vary by backbone.
  • Scope boundary: The gains are smaller than those on the main tasks, suggesting that reasoning-heavy capabilities may require richer multi-layer feature representations.The paper identifies single-layer SAE features as only partially sufficient for these benchmarks.
Loading 2602.10388v4…