PaperlayerView exact revision on arXiv

Source-linked AI summary

FineScope : SAE-guided Data Selection Enables Domain Specific LLM Pruning and Finetuning

Chaitali Bhattacharyya, Hyunsei Lee, Junyoung Lee, Shinhyoung Jang, Il hong Suh, Yeseong Kim

arXiv:2505.00624v3cs.CLcs.AI

TL;DR

High-quality, domain-aligned fine-tuning data is scarce, limiting domain adaptation in settings where such datasets are rarely available. FineScope uses SAE-selected examples to guide structured pruning and fine-tuning, and the authors report improved performance across diverse domains while reducing model size.

  • Problem

    Domain adaptation often lacks high-quality, domain-aligned fine-tuning data, despite compressed models depending heavily on dataset quality and relevance.

    Source: §1 · p. 1

  • Method

    FineScope uses SAE representations to select relevant samples from an unlabeled corpus, then uses the curated data to guide structured pruning and modified self-distillation fine-tuning.

    Source: §1 · p. 2

  • Results

    Across diverse domains, FineScope consistently improves performance while significantly reducing model size.

    Source: §5 · p. 11

  • Takeaways & Limitations

    The authors conclude targeted data selection is crucial for effective LLM deployment in resource-constrained settings.

    Source: §5 · p. 11

    The domain-critical component set is not unique: different pruning procedures or tie-breaking can retain different sets with similar loss, and the unknown domain distribution is approximated using the selected dataset.

    Source: §7.3 · p. 15

Abstract

from arXiv · show

Training large language models (LLMs) from scratch requires significant computational resources, driving interest in developing smaller, domain-specific LLMs that maintain both efficiency and strong task performance. Medium-sized models such as LLaMA, llama} have served as starting points for domain-specific adaptation, but they often suffer from accuracy degradation when tested on specialized datasets. We introduce FineScope, a framework for deriving compact, domain-optimized LLMs from larger pretrained models. FineScope leverages the Sparse Autoencoder (SAE) framework, inspired by its ability to produce interpretable feature representations, to extract domain-specific subsets from large datasets. We apply structured pruning with domain-specific constraints, ensuring that the resulting pruned models retain essential knowledge for the target domain. To further enhance performance, these pruned models undergo self-data distillation, leveraging SAE-curated datasets to restore key domain-specific information lost during pruning. Extensive experiments and ablation studies demonstrate that FineScope achieves highly competitive performance, outperforming several large-scale state-of-the-art LLMs in domain-specific tasks. Additionally, our results show that FineScope enables pruned models to regain a substantial portion of their original performance when fine-tuned with SAE-curated datasets. Furthermore, applying these datasets to fine-tune pretrained LLMs without pruning also improves their domain-specific accuracy, highlighting the robustness of our approach.

1 Introduction

FineScope addresses the shortage of high-quality, domain-aligned fine-tuning data by joining data selection with pruning and fine-tuning. It uses SAE-selected samples to guide adaptation, and the paper reports improved performance alongside reduced model size.

1 Introduction

  • High-quality, domain-aligned fine-tuning data is scarce, although compressed-model adaptation depends heavily on dataset quality and relevance.Existing options include general-purpose instruction data and manually curated corpora, which may be noisy, expensive, or misaligned.Source: §1 · p. 1
  • Across domains, FineScope outperforms standard fine-tuning and improves pruned-model performance while reducing model size; Figure 1 reports higher accuracy under aggressive pruning.Figure 1 compares fine-tuning with Self-Instruct and FineScope across STEM, Social Sciences, and Humanities at different pruning ratios.Source: Figure 1 · p. 2, §1 · p. 2
  • FineScope unifies domain-specific data selection with pruning and fine-tuning, so selected data guides both compression and adaptation.The framework uses a modified self-distillation step to help recover task-relevant knowledge lost during compression.Source: §1 · pp. 1–2
  • SAEs trained on intermediate activations identify semantically relevant samples from large unlabeled corpora, starting from a small seed set.The curated samples form a compact dataset reflecting the target domain.Source: §1 · p. 2

2 Related Work

  • Domain-specialized models often depend on large, high-quality datasets and full-scale retraining, while efficient adaptation with limited compute and annotation remains under-addressed.The related-work discussion positions automatic selection from unlabeled corpora and model compression as a response to those constraints.Source: §2 · p. 2
  • Existing pruning is often independent of downstream fine-tuning data, whereas FineScope informs pruning with samples selected for target-domain relevance.The paper also contrasts representation-guided selection with approaches that keep the model fixed or rely on surface-level similarity or metadata.Source: §2 · p. 3

3 FineScope

FineScope selects domain-relevant examples with an SAE, then uses them to guide structured pruning and teacher-guided fine-tuning of a compact model.

3.1.1 Training SAE

  • The SAE learns structured, low-dimensional representations from intermediate LLM activations, using representative activations to highlight significant neurons while discarding less relevant signals.Its training uses reconstruction loss plus an ℓ1 sparsity penalty on encoder activations.Source: §3.1.1 · p. 4

3.1.2 Top-K Activation Selection for SAE Training

  • To reduce SAE training cost, FineScope preselects a fixed subset of activation coordinates using input sensitivity, estimated with a Hutchinson-style estimator requiring a small number of Jacobian vector products.The sensitivity scores are aggregated over a reference corpus to rank coordinates for selection.Source: §3.1.2 · pp. 4–5

3.1.3 Dataset Curation

  • Starting from a small set of target-domain seed examples, FineScope selects the top-M corpus samples by similarity in SAE embedding space to form a curated dataset.The selection uses cosine similarity, with M set to 100 across SAEs in the evaluation.Source: §3.1.3 · p. 5

3.2.1 Pruning

  • Domain-specific examples guide structured pruning, with gradient-based block attribution preserving components estimated to matter for target-domain performance.Block-wise pruning removes redundant components; the pruning ratio controls the trade-off between compactness and domain-specific performance.Source: §3.2.1 · pp. 5–6

3.2.2 Fine-Tuning via Teacher-Guided Distillation (TGD)

  • Teacher-guided distillation transfers knowledge from the unpruned original model to the pruned student, supporting knowledge recovery and regularization against the small curated dataset.The teacher is used to construct the distilled dataset; the student is trained independently without direct access to teacher logits or intermediate representations.Source: §3.2.2 · p. 6

4 Evaluation

FineScope is evaluated on broad domain benchmarks, mathematical subdomains, and coding tasks. The results cover domain-specific tuning, pruning and distillation choices, TopK trade-offs, synthetic-data comparisons, and SAE embedding selection.

4.1 Experimental Results

  • FineScope gains 3.8% over Alpaca tuning and 4.45% over OpenInstruct on average across MMLU domains and model types.MathCoder-CL gains 8.28% in STEM, 7.8% in Social Sciences, and 7.9% in Humanities.Source: §4.1 · p. 7, Table 1 · p. 7
  • Pruned FineScope models, with approximately 30% fewer parameters, outperform GPT-3 and OLMO-7B in most settings, though GPT-3 leads in Social Sciences and GPT-3 175B in Humanities.Unguided pruning degrades Vicuna by up to 50.17% on average across domains.Source: §4.1 · p. 7
  • Math-domain tuning improves over Alpaca-tuned baselines by 7.01 for Vicuna, 7.71 for MathCoder-CL, and 18.45 for LLaMa 3.1.FineScope restores performance after pruning while reducing model size to approximately 71% of the original.Source: §4.1 · pp. 7–8
  • On HumanEval and MBPP, FineScope improves coding performance, including pruned LLaMa 3.1 scores of 0.49 and 0.43 versus Full OpenInstruct’s 0.30 and 0.29.Across the three models, pruning alone reduces code-generation performance, and Alpaca tuning does not recover the loss.Source: §4.1 · p. 8, Table 3 · p. 8

4.2 Effect of Teacher-Guided Distillation (TGD) and Pruning Dataset

  • Teacher-guided distillation improves performance over standard fine-tuning by 1.9% in STEM, 5.8% in Social Sciences, and 10.26% in Humanities.Domain-aligned pruning data followed by same-domain fine-tuning yields an average accuracy gain of 11.8% over pruning with a general-purpose corpus.Source: §4.2 · p. 9, Table 4 · p. 9

4.3 Effect of TopK on Computation

  • Increasing SAE TopK improves average accuracy across STEM, Social Sciences, and Humanities but lengthens training; K=128 balances accuracy and tractable training time.K=256 brings only marginal accuracy gains at significantly higher computational overhead.Source: §4.3 · pp. 9–10, Figure 3 · p. 9

4.4 Comparison with Synthetic Dataset

  • FineScope’s 2.1K-sample dataset achieves accuracy comparable to synthetic STEM-Saraswati and both outperform general-purpose Alpaca and OpenInstruct datasets on STEM tasks.STEM-Saraswati is generated using GPT-4.Source: §4.4 · p. 10, Figure 4 · p. 10

4.5 Effect of SAE-Embedding

  • SAE-Embedding curation outperforms BERT, all-mpnet-base-v2, and activation-magnitude selection across models and domains.For Vicuna, SAE embeddings yield average fine-tuning gains of +5.08 for the original model and +2.06 for the pruned model.Source: §4.5 · p. 10, Table 5 · p. 11

5 Conclusion

FineScope combines SAE-guided data selection with structured pruning and fine-tuning to build compact, domain-specialized LLMs. Across diverse domains, it improves performance while reducing model size.

5 Conclusion

  • Across diverse domains, FineScope improves performance while significantly reducing model size through SAE-guided data selection, structured pruning, and fine-tuning.The supplement adds implementation details and experiments on robustness, generality, and hyperparameter sensitivity.Source: §5 · pp. 11–12
  • FineScope selects semantically relevant samples from large unlabeled corpora to build compact, high-quality datasets that guide both compression and adaptation.Source: §5 · p. 11
  • The supplement examines whether FineScope generalizes across Qwen variants, pruning methods, pruning ratios, and seed settings.It also reports computation-cost breakdowns and qualitative analyses of representations and seed examples.Source: §5 · p. 12

6 Sparse Autoencoder Objective Details

The supplement details the SAE training setup and objective, combining scale-normalized reconstruction, a dead-latent auxiliary loss, and sparsity regularization.

6 Sparse Autoencoder Objective Details

  • The expanded SAE objective details define activations, encoder and decoder parameters, and residuals for reproducibility; loss implementations are from the repository.Source: §6 · p. 12, §6.2 · p. 12

6.1 Setup

  • A mini-batch of activations from a frozen pretrained transformer hookpoint is processed by the SAE into reconstructions, residuals, and latent activations.Source: §6.1 · p. 12

6.2 Scale-Normalized Reconstruction Loss (FVU)

  • The scale-normalized FVU reconstruction objective makes errors comparable across layers and activation scales, stabilizing training and making loss weights more transferable.Source: §6.2 · p. 12

6.3 Auxiliary Dead-Latent Residual Loss (AuxK)

  • The AuxK loss uses selected dead latents to predict the residual, improving feature utilization; it is zero when no dead latents are present.When dead latents exist, the method decodes up to k_aux of them for the auxiliary residual prediction.Source: §6.3 · p. 13

6.4 Sparsity Regularization

  • An ℓ1 penalty on post-ReLU encoder activations encourages sparse, interpretable codes whose cosine similarity reflects overlap in salient features.Source: §6.4 · p. 13

6.5 Final SAE Objective

  • The final SAE objective combines reconstruction, dead-latent auxiliary loss, and explicit sparsity regularization, with α and λ as nonnegative hyperparameters.Source: §6.5 · p. 13

7 Design Rationale: Embeddings, Top-K Selection, and Pruning

FineScope uses SAE codes for retrieval, Jacobian sensitivity to select activation coordinates, and domain-conditioned importance to guide structured pruning. The design choices aim to preserve representations and components relevant to the target domain.

7.1 SAE Encoder Codes as Domain-Aligned Embeddings

  • SAE encoder codes serve as retrieval embeddings: cosine similarity favors shared internal features and coefficient patterns rather than surface-form overlap.Under the sparse-coding view, nearest neighbors share latent features activated internally by the model, concentrating retrieval on domain-relevant behavior.Source: §7.1 · pp. 13–14

7.2 Jacobian Sensitivity as a Coordinate-Selection Criterion

  • Jacobian row-norm selection retains coordinates responsive to input variation, preserving more input-induced activation variation for a compact retrieval representation.Source: §7.2 · pp. 14–15
  • Unlike magnitude or variance heuristics, Jacobian sensitivity targets input-conditioned variation rather than changes potentially driven by noise, position, or distributional artifacts.The supplement motivates sensitivity as closer to semantic dependence needed for retrieval.Source: §7.2 · p. 15

7.3 Domain-Conditioned Structured Pruning and Domain-Critical Components

  • Because the target distribution is unknown, domain-conditioned pruning uses the curated dataset as its proxy and scores components by their estimated effect on domain loss.The first-order approximation motivates retaining high-importance blocks and removing those with small predicted domain-loss effects.Source: §7.3 · p. 15
  • The retained domain-critical components are not unique: different pruning procedures or tie-breaking can yield different sets with similar loss.Source: §7.3 · p. 15

8 Effect of Different Top-K selection method for SAE training

At fixed coordinate count, Jacobian-sensitivity selection outperforms random, activation-statistic, and PCA alternatives across three subdomains. The comparison holds the SAE architecture and training budget constant.

8 Effect of Different Top-K selection method for SAE training

  • The selector comparison isolates coordinate choice by fixing Kact, SAE architecture, and training budget.Source: §8 · p. 16, Table 6 · p. 16
  • Performance improved from random selection through activation statistics and PCA to input-sensitivity selection; the largest gaps were in Algebra and Counting and Probability.The results indicate that first-order sensitivity information yields stronger SAE representations than activation-driven or unsupervised-subspace alternatives.Source: §8 · p. 16
  • 12.91/10.12/7.01 on Pre-algebra/Algebra/C&P were FineScope’s best results across all three subdomains.Table 6 compares coordinate selectors at fixed Kact.Source: Table 6 · p. 16, §8 · p. 16

9 Different Pruning Methods

FineScope’s domain-conditioned pruning data shifts importance estimates toward domain-critical components, with sharp gains over Alpaca in Algebra and Counting. Table 7 compares multiple pruning strategies under matched pruning ratios while varying the data.

9 Different Pruning Methods

  • FineScope data yields sharp Alpaca-to-FineScope improvements in Algebra and Counting, suggesting generic pruning data can undervalue specialized reasoning computations.The passage interprets domain-aligned data as better preserving components important to the target distribution.Source: §9 · p. 16
  • Domain-conditioned pruning uses target-domain data as a proxy for expected loss, shifting importance estimates toward domain-critical components.Source: §9 · p. 16
  • Table 7 compares Magnitude Pruning, FLAP, and LLMPruner at matched pruning ratios while varying pruning or tuning data between Alpaca and FineScope.The baselines span magnitude-based and LLM-oriented structured pruning approaches.Source: §9 · p. 16, Table 7 · p. 17

10 Experiments on a Different Model Family: Qwen

FineScope consistently improves over the Alpaca baseline across three Qwen generations and the evaluated mathematical domains. The gains are relatively larger for older, weaker models than for Qwen 3-8B.

10 Experiments on a Different Model Family: Qwen

  • Relative gains are larger for the older and weaker Qwen models than for Qwen 3-8B.Source: §10 · p. 17
  • The reported benefits persist when overall accuracy is relatively low, rather than being limited to high-capacity or high-performing settings.Source: §10 · p. 17
  • FineScope consistently improves over Alpaca across Qwen 3-8B, Qwen 2.5-7B, and Qwen 2-7B on evaluated mathematical domains.Table 8 reports the performance analysis across these Qwen models.Source: Table 8 · p. 17, §10 · p. 17

11 Computation cost

For Qwen 8B, saliency computation through the JVP pipeline dominates runtime, while SAE training contributes a smaller, more uniform share. Runtime generally increases with layer depth, with a transient peak attributed to execution-path effects rather than sustained arithmetic cost.

11 Computation cost

  • Saliency computation consistently exceeds SAE optimization cost and generally increases with layer depth.Source: §11 · p. 17
  • The transient saliency-time spike is most consistent with one-off initialization or execution-path selection costs, not sustained per-layer arithmetic growth.The passage identifies possible kernel-selection, fallback, or allocator-related effects.Source: §11 · p. 17
  • Saliency/JVP computation primarily determines Qwen 8B’s end-to-end cost, while SAE training contributes a smaller, more uniform runtime share.Figure 5 breaks down per-layer runtime on four GPUs.Source: §11 · p. 17, Figure 5 · p. 18

12 Extended Pruning Ratio

On LLaMA 3.1 8B, FineScope is more resilient than SelfInstruct as pruning increases. SelfInstruct begins degrading at 25% pruning, while FineScope maintains stable accuracy up to 35%.

12 Extended Pruning Ratio

  • The pruning-ratio comparison uses the SelfInstruct dataset with the LLaMA 3.1 8B model.Source: §12 · p. 18
  • Both methods lose accuracy as pruning increases, but FineScope tolerates more aggressive pruning before accuracy collapses.Source: §12 · p. 18
  • As pruning increases, both methods' accuracy declines, but FineScope consistently outperforms SelfInstruct across all settings; SelfInstruct begins degrading at 25% pruning, while FineScope maintains stable accuracy up to 35%.Source: §12 · p. 18

13 Discussion for Seed Choice Sensitivity

Accuracy changes little as the number of initial seeds varies from 5 to 25, while expanding the selected seed set improves accuracy substantially until gains begin to saturate. FineScope’s seed-selection analysis also describes representations becoming more domain-specific in deeper layers and provides representative human-defined prompts for three domains.

13.1 Effect of Number of Initial Seeds

  • Changing initial seed count K from 5 to 25 has only a marginal accuracy effect: STEM remains near 31%, while Social Sciences and Humanities reach around 36–37%.Social Sciences and Humanities improve slightly up to approximately K = 15–20, then largely plateau; the authors conclude that a modest number of initial seeds is sufficient.Source: §13.1 · p. 18, Figure 7 · p. 19

13.2 Effect of Number of Selected Seeds

  • Accuracy rises substantially from 25 to about 80 selected seeds, then gains saturate; Social Sciences peaks around 100–125, while Humanities improves to about 140 before a marginal drop.Selected seeds are the expanded set chosen by retrieval and selection to guide subsequent data collection; the study uses a moderate selection size given its fixed computation budget.Source: Figure 8 · p. 19, §13.2 · p. 19

14.2 Example Seeds

  • Representative user-defined seed prompts are provided for STEM, Social Sciences, and Humanities as the pipeline’s primary human-specified inputs for extracting domain-specific data.The figures show domain seed samples, including questions about short-run inflation and the decline of the Roman Republic.Source: §14.2 · pp. 20, 23, Figure 10 · p. 22, Figure 11 · p. 22

14.1 Seed Selection

  • Cluster analysis suggests lower layers capture broad linguistic features, while deeper layers form more compact, semantically meaningful domain-specific clusters.FineScope selects seeds from layers balancing generalization and domain specificity, using SAE-based representations for more interpretable, high-level domain alignment than raw embedding similarity.Source: §14.1 · pp. 19–20, Figure 9 · p. 21
Loading 2505.00624v3…