PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Data Selection via Manifold Coverage and Sparse Feature Coverage in LLM Post-training

Peng Sun, Yi Yang, Antong Zhang, Chunxiao Li, Yanbo Wang, Dianbo Liu, xin chen, Kai Yu, Lu Chen, Tianfan Fu

arXiv:2608.16927v1cs.LGcs.CV

TL;DR

Data selection in original embedding spaces can entangle global semantics, fine-grained supervision differences, and local noise. MASS addresses this with coarse-to-fine manifold and sparse-feature coverage, consistently outperforming baselines across datasets and budgets.

  • Problem

    Existing selection methods measure diversity in original embedding spaces, where global semantics, fine-grained supervision differences, and local noise are entangled.

  • Method

    MASS uses dense-autoencoder manifold coordinates for coarse grouping, then TopK sparse-autoencoder features and quality signals for within-group selection.

  • Results

    MASS consistently outperforms existing baselines across datasets and budgets, reaching ARP 102.12 on Vision-Flan at 10% and exceeding Full Data by 2.12 points.

  • Takeaways & Limitations

    The results support hierarchical coverage as an effective framing for selecting compact supervised fine-tuning subsets across varied data distributions and task complexities.

  • Takeaways & Limitations

    MASS depends on external embeddings and quality scores, cannot compensate for gaps in the candidate pool, and does not study task-specific data selection.

Abstract

from arXiv · show

As supervised fine-tuning data continues to scale, selecting high-value subsets from large candidate pools is crucial for reducing training cost and improving model performance. Existing methods often measure diversity directly in the original embedding space, where geometric metrics entangle dominant semantic directions, fine-grained supervision differences, and local noise. We address this limitation by formulating data selection as a coarse-to-fine hierarchical coverage problem and propose MASS. MASS learns low-dimensional principal manifold coordinates with a dense autoencoder for coarse semantic grouping, and then performs quality-aware sparse feature coverage within each group using a TopK sparse autoencoder. Experiments on Vision Flan and LLaVA-CoT show that MASS consistently outperforms strong data selection baselines across multiple budgets, and in several settings matches or surpasses full data training with only a small subset of data.

1 Introduction

The paper frames supervised fine-tuning data selection as hierarchical coverage because original embedding geometry conflates semantic structure, fine-grained differences, and noise. MASS addresses this with manifold-based grouping and sparse feature coverage, outperforming baselines across settings and sometimes surpassing full-data training.

  • Motivation: Growing supervised fine-tuning datasets make selecting high-value subsets important for reducing training cost and improving model performance.Existing methods characterize distributions with embeddings and preserve diversity through similarity measurement and clustering.
  • Problem: Original embedding geometry mixes dominant semantic directions, fine-grained attributes, response-format variation, and local noise.The data distribution has high global variation, while local neighborhoods have lower effective dimensionality and more consistent nonlinear-manifold structure.
  • Contribution: The paper reformulates data selection as hierarchical coverage combining coarse-grained principal-manifold coverage with fine-grained supervision-feature coverage.This coarse-to-fine formulation separates global grouping from local feature coverage rather than applying flat diversity measurement in the original embedding space.
  • Results: MASS consistently outperforms existing data-selection baselines across multiple budgets and sometimes surpasses full-data training using only a small subset.It is evaluated on Vision-Flan and LLaVA-CoT across general instruction and reasoning tasks; contributions also report validation across datasets, budgets, embedding sources, and target models.
  • Method: MASS uses a DAE to construct principal-manifold groups, a TopK SAE for within-group sparse-feature coverage, and quality signals for sample selection.The method explicitly combines coarse semantic grouping with fine-grained supervision-feature coverage.

2 Related Work

Prior data-selection methods primarily target sample importance or coverage diversity, while autoencoders and sparse autoencoders provide complementary tools for low-dimensional representation and selective feature discovery. MASS builds on these foundations by using TopK sparse feature coverage on joint input-response supervision representations to diversify fine-grained signals within manifold regions.

  • Data Selection: Data selection seeks high-value subsets that reduce training cost while preserving or improving model performance, commonly through sample importance or coverage diversity.The passage identifies training-dynamics methods including EL2N, LESS, and OPUS as examples of importance-based selection.
  • Autoencoders and Manifold Representation: Autoencoders learn compact low-dimensional representations by encoding and decoding inputs under a reconstruction objective.Prior studies use such latent representations for nonlinear dimensionality reduction because they can preserve principal directions of variation.
  • Sparse Autoencoders: Sparse autoencoders reconstruct representations using few activated features, producing more selective and interpretable latent features.They originate from sparse coding and dictionary learning and are widely used for mechanistic interpretability of language-model activations.
  • Sparse Autoencoders: Unlike prior interpretability-focused work, MASS applies TopK SAE to joint input-response supervision representations for sparse feature coverage-based data selection.This improves the diversity of fine-grained supervision signals within each principal manifold region while addressing sparsity, dead features, and reconstruction-quality tradeoffs.

3 Method

MASS formulates data selection as budgeted subset selection and uses a coarse-to-fine pipeline that combines manifold coverage with sparse supervision-feature coverage and quality scores. It separates input semantics from complete input-response supervision to guide grouping and within-group selection.

  • Selection Objective: MASS seeks a budgeted subset D′ ⊆ D with |D′| = P whose training performance approaches or surpasses training on D.
  • Dual View Encoding: MASS encodes each sample with condition and supervision embeddings, using input semantics for manifold grouping and complete input-response supervision for sparse feature coverage.Both embeddings come from a frozen embedding model and are independently centered and L2-normalized.
  • DAE for Principal Manifold Encoding: A dense autoencoder compresses condition embeddings into stable 32-dimensional principal manifold coordinates for coarse semantic grouping.The autoencoder is trained to reconstruct the original condition embedding from the low-dimensional coordinate.
  • SAE for Sparse Feature Encoding: A TopK sparse autoencoder represents supervision embeddings with sparse features, using 131072 nonnegative activations and K = 64 in the main experiments.The sparse vector has at most K nonzero activations and is decoded to reconstruct the supervision embedding.
  • Coarse to Fine Selection: Within KMeans-derived groups, MASS greedily selects candidates by combining normalized SAE coverage gain with external quality scores ηi ∈ [0, 1] until each group reaches its assigned budget.Feature coverage gain accounts for maximum observed activations and current feature coverage counts.

4 Experiments

Experiments evaluate MASS on Vision-Flan and LLaVA-CoT across three subset budgets against nine data-selection baselines. MASS achieves the best reported performance across both datasets and all budgets, exceeding Full Data at two Vision-Flan budgets.

  • Experimental Setup: MASS is evaluated on Vision-Flan for general instruction tasks and LLaVA-CoT for reasoning tasks, covering different task complexities and data distributions.The target models are LLaVA-V1.5-7B for Vision-Flan and Llama-3.2-11B-Vision-Instruct for LLaVA-CoT.
  • Experimental Setup: The study tests 5%, 10%, and 15% data budgets on both datasets and compares MASS with 9 data selection baselines.Baselines include Random Selection, XMAS, COINCIDE, SemDeDup, D2 Pruning, PRISM, ScalSelect, CLIP Score, and EL2N.
  • Experimental Setup: Evaluation uses 12 benchmarks per dataset, with Vision-Flan targeting general capabilities and LLaVA-CoT covering reasoning and general capabilities.All experiments use three random seeds: 0, 42, and 99, with average performance reported.
  • Results: 99.50, 102.12, and 103.23 are MASS’s ARP values on Vision-Flan at the 5%, 10%, and 15% budgets, respectively, outperforming all baselines.At 10% and 15%, MASS exceeds Full Data by 2.12 and 3.23 points, respectively.
  • Results: 99.65, 99.83, and 102.82 are MASS’s ARP values on LLaVA-CoT across the 5%, 10%, and 15% budgets, achieving the best performance at every setting.Across both datasets and all three budgets, MASS consistently achieves the best results, supporting hierarchical coverage rather than global sampling in the original embedding space.

5 Analysis and Ablation Studies

MASS benefits from hierarchical coarse-to-fine coverage: DAE manifold coordinates improve coarse grouping, while SAE fine-feature coverage and quality scores provide complementary gains. Ablations show sensitivity to representation capacity and sparsity, but effectiveness remains robust across embedding sources and target models.

  • DAE Manifold Coordinates: 102.12 ARP with DAE coordinates exceeds 101.59 with raw embeddings, supporting more reliable coarse-grained grouping.DAE coordinates reduce interference from redundant dimensions, local perturbations, and non-dominant variations.
  • DAE–SAE Complementarity: 102.12 ARP for full MASS exceeds 97.71 with DAE alone, 101.44 with SAE alone, and 95.30 with random selection.DAE preserves coarse manifold coverage, while SAE provides stronger fine-grained feature coverage; combining them performs best.
  • DAE Configuration: 102.71 ARP at latent dimension 64 outperforms 100.66, 98.86, 102.12, and 101.77 at dimensions 8, 16, 32, and 128.Both overly small and overly large DAE latent dimensions hurt final performance.
  • SAE Configuration: 102.12 ARP at TopK 64 exceeds 100.81, 100.11, and 101.21 at TopK values 16, 32, and 128.Smaller TopK limits activated features, whereas larger TopK weakens sparsity and may add non-essential features.
  • SAE Configuration: 103.23 ARP with 131072 SAE features exceeds 102.81, 100.98, and 99.19 with 65536 features at the 15%, 10%, and 5% budgets.A larger SAE feature space strengthens fine-grained feature representation and within-cluster sparse feature coverage.
  • Quality Signals and Embedding Sources: 102.44 ARP with LLaVA-OneVision scoring and 101.68 ARP with GME embeddings show that quality signals and representation changes preserve strong selection performance.MASS without quality scores reaches 99.96 versus 95.30 for random selection, while the original scorer reaches 102.12 ARP; GME is slightly lower than the original embedding result.
  • Robustness Across Target Models: MASS consistently outperforms random selection across target models: 89.49, 93.09, and 94.15 versus 85.12, 92.26, and 91.14 on Vision-Flan, and 96.52, 96.02, and 95.36 versus 94.30, 95.25, and 94.77 on LLaVA-CoT.These results cover different parameter scales within a model family and different model families.

6 Conclusion and Limitations

MASS formulates supervised fine-tuning data selection as coarse-to-fine hierarchical coverage, using learned manifold coordinates for grouping and sparse autoencoder features for within-group selection. Its effectiveness depends on external signals and cannot resolve distributional gaps already present in the candidate pool.

  • Conclusion: MASS treats supervised fine-tuning data selection as a coarse-to-fine hierarchical coverage problem.
  • Conclusion: MASS uses a dense autoencoder for low-dimensional principal-manifold grouping, then applies TopK sparse-autoencoder selection within each group.
  • Limitations: MASS depends on external embedding and quality-scoring models, making selected subsets vulnerable to severely biased external signals.
  • Limitations: MASS cannot compensate for missing task types or supervision patterns when the candidate pool contains fundamental distributional gaps.

A Embedding Geometry Analysis

Vision-Flan and LLaVA-CoT embeddings have high global linear complexity but much lower, nonlinear local dimensionality, motivating MASS’s coarse-to-fine hierarchical coverage strategy. MASS uses a dense autoencoder for principal manifold coordinates and a sparse autoencoder for within-region supervision-feature coverage.

  • Global embedding complexity: 379 PCA components explain 90% of Vision-Flan variance, versus 459 for LLaVA-CoT, despite both datasets having embedding dimension 2048.Their PCA effective ranks are 255.06 and 275.17, respectively.
  • Local dimensionality: Vision-Flan and LLaVA-CoT have much lower local dimensionality, with TwoNN intrinsic dimensions of 8.25 and 11.88, respectively.Mean local PCA dimensions are 17.38 and 20.88, while median dimensions are 19 and 21.
  • Nonlinear local structure: Isomap preserves neighborhoods better than PCA on both datasets: 0.499 versus 0.383 for Vision-Flan and 0.380 versus 0.281 for LLaVA-CoT.This comparison indicates that local neighborhood structure is nonlinear rather than adequately represented by a global linear projection.
  • Motivation: Direct selection in the original embedding space can conflate global semantic variation, local fine-grained differences, and task-irrelevant perturbations.A single distance or diversity criterion may therefore fail to distinguish these levels of variation.
  • MASS formulation: MASS performs coarse-to-fine hierarchical coverage by learning principal manifold coordinates with a DAE, then selecting sparse supervision features with an SAE within each semantic region.The DAE constructs coarse-grained semantic regions, while the SAE enables fine-grained coverage selection within each region.

B Implementation Details of MASS

MASS uses dual-view embeddings to separate input conditions from joint input-response supervision, then combines dense principal-manifold encoding, sparse feature encoding, and normalized multimodal quality scores for selection.

  • Dual View Encoding: For each sample, MASS embeds the condition using only the input and embeds supervision using the concatenated input-response pair.This dual-view encoding distinguishes condition information from supervision information.
  • Dense Autoencoder for Principal Manifold Encoding: The dense autoencoder L2-normalizes inputs, adds spherical noise of magnitude 0.03, renormalizes them, and uses a GELU MLP encoder with dimensions 1024, 512, and 128.It learns low-dimensional principal manifold coordinates from input-side condition embeddings.
  • Sparse Autoencoder for Sparse Feature Encoding: The sparse autoencoder centers and L2-normalizes supervision embeddings, then uses ReLU TopK activation selection in an overcomplete feature space before reconstruction.It models fine-grained sparse features formed jointly from inputs and responses.
  • Coarse to Fine Selection: MASS obtains external quality scores from a multimodal model using ratings from 1 to 5, normalized to [0, 1] as 0, 0.25, 0.5, 0.75, and 1.The discrete rating-to-score mapping is explicitly defined for coarse-to-fine selection.

C Datasets and Preprocessing Details … E Evaluation

The paper evaluates MASS on Vision-Flan and LLaVA-CoT using minimally processed datasets and specified training settings. Vision-Flan provides broad fine-grained visual-task coverage, while LLaVA-CoT supplies structured image-text reasoning data.

  • C.1 Datasets: Vision-Flan contains 187 fine-grained visual tasks with expert-written and validated instructions for diverse visual-task coverage.It is a high-quality human-annotated visual instruction tuning dataset constructed from publicly available academic vision datasets.
  • C.1 Datasets: LLaVA-CoT-100k contains approximately 99K image-question-answer pairs annotated with summary, caption, reasoning, and conclusion stages.The data combines general-purpose and science-targeted VQA datasets to support structured reasoning.
  • C.2 Preprocessing Procedures: Vision-Flan is used without additional filtering or format conversion.
  • C.2 Preprocessing Procedures: Removing LLaVA-CoT samples without associated images leaves 98572 image-text reasoning samples.The reasoning process and final answer are wrapped with <think></think> and <answer></answer>, respectively.
  • D Training Hyperparameters: LLaVA-V1.5-7B and LLaVA-V1.5-13B follow hyperparameter settings from the official LLaVA training code.The passage identifies these settings as the basis for training those two target models.
  • D Training Hyperparameters: Qwen2-VL-2B-Instruct and Llama-3.2-11B-Vision-Instruct use a conservative learning rate of 1 × 10−5 for full-parameter fine-tuning.Both models are already instruction-tuned checkpoints.
  • D Training Hyperparameters: Qwen2-VL-2B-Instruct and Llama-3.2-11B-Vision-Instruct use batch size 64 because batch size 128 leads to too few optimiza.

E.1 Evaluation Benchmarks · E.2 Detailed Evaluation Results

The evaluation uses capability-specific benchmark suites for Vision-Flan and LLaVA-CoT, while excluding four uninformative benchmarks for the latter model. Detailed results compare MASS with baselines and report manifold, complementarity, and sensitivity analyses.

  • E.1 Evaluation Benchmarks: Vision-Flan evaluation covers multimodal understanding, visual question answering, OCR, document understanding, chart understanding, and science-diagram reasoning.
  • E.1 Evaluation Benchmarks: LLaVA-CoT evaluation combines multimodal mathematical and logical reasoning benchmarks with general vision-language benchmarks.
  • E.1 Evaluation Benchmarks: Four LLaVA-CoT benchmarks—We-Math, LogicVista, DynaMath, and MMStar—are excluded because full-data training performs extremely poorly on them.
  • E.1 Evaluation Benchmarks: The LLaVA-CoT model is evaluated on the remaining eight benchmarks, with the full benchmark list provided in Table 6.
  • E.2 Detailed Evaluation Results: Tables 7 and 8 provide benchmark-level comparisons of MASS and all data selection baselines on Vision-Flan and LLaVA-CoT.
  • E.2 Detailed Evaluation Results: Additional benchmark-level results cover the paper’s analysis and ablations, including DAE manifold-coordinate ablations in Table 9 and DAE–SAE complementarity in Table 10.
  • E.2 Detailed Evaluation Results: Table 11 reports a sensitivity analysis of the DAE.

F Time Cost Analysis

MASS requires approximately 15.39 GPU hours on a single NVIDIA A6000 for LLaVA-CoT, with most computation spent on quality scoring and embedding extraction. These extracted results can be reused in other data-usage scenarios, including subsequent curriculum learning.

  • F Time Cost Analysis: 15.39 GPU hours: MASS’s total time cost on LLaVA-CoT using a single NVIDIA A6000.Its computational cost is slightly higher than that of some baselines.
  • F Time Cost Analysis: Most MASS computation is spent on quality scoring and embedding extraction, whose results can be reused for subsequent curriculum learning.The passage also notes reuse in other data-usage scenarios.
Loading 2608.16927v1…