PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind the Tail: Data Selection Through Sparse Concepts

Ziheng Zhang, Neha Hulkund, Colin Lee, Zheda Mai, Hui Shen, Ying Ding, Tanya Berger-Wolf, Sara Beery, Wei-Lun Chao, Jianyang Gu

arXiv:2610.21007v1cs.CV

TL;DR

Data selection can shape model behavior by reallocating visual evidence, yet principled guidance on concept allocation across learning settings is limited. This paper selects samples using sparse-autoencoder concept support, diminishing returns, and neighborhood support. Its subsets outperform strong baselines across classification and multimodal pretraining, and the analyses connect concept coverage with test accuracy and robustness.

  • Problem

    Data selection reallocates unevenly supported visual concepts, but a principled way to determine useful concept allocation across learning scenarios is lacking.

    Source: §1 · p. 1

  • Method

    The method extracts sparse visual concepts with autoencoders, tracks selected-set support, and combines diminishing returns with neighborhood support to favor under-supported evidence.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4

  • Results

    Selected subsets consistently outperform strong data-selection baselines across classification and multimodal pretraining; zero-shot accuracy improves by 2.4% at the 10% budget.

    Source: §1 · p. 2, §4.1 · p. 6

  • Takeaways & Limitations

    Enhanced concepts align with larger test-accuracy improvements, and broader concept coverage can improve robustness to out-of-distribution and spurious contexts.

    Source: §1 · p. 2, §5 · p. 8, §6 · p. 10

    Selected subsets reflect their source datasets’ characteristics, so the authors recommend validating them before use beyond research settings.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Data selection reallocates training support across visual concepts, including objects, attributes, color, texture, and backgrounds. Different allocation strategies may produce varied downstream model behavior, especially for tail concepts supported by only a few samples. Discarding them can hurt test-time robustness, while overemphasizing samples lying on category boundaries may introduce unstable training signals. This work explicitly investigates this relationship across fine-grained, spurious-correlation, and general image classification, as well as multimodal pretraining. Specifically, we use sparse autoencoders (SAEs) to extract concept activations from individual images. During selection, we accumulate the dataset-level concept support for each added sample. Applying diminishing returns to this distribution down-weights well-represented concepts and shifts the focus to the distribution tail. We then incorporate local support to find samples that reliably strengthen under-supported concepts. Empirical results show consistent improvements over strong data selection baselines across all benchmarks. We also reveal the connection between concepts our method enhances and the induced behavioral changes, including improved robustness against out-of-distribution and spurious contexts. This work establishes concept support, beyond individual sample importance, as a useful principle for data-efficient and robust learning.

1 INTRODUCTION

Data selection reallocates visual evidence that can shape model behavior, but the concept allocation useful across learning scenarios is not yet principled. The paper uses sparse concepts to study this relationship and reports gains across classification and multimodal pretraining.

1 INTRODUCTION

  • Data selection shapes model behavior as well as training cost, because training data can encode useful recognition traits and spurious backgrounds such as bird feeders.The paper frames concept allocation as one way to study how that evidence is represented in a selected subset.Source: §1 · p. 1
  • The paper uses sparse autoencoders to represent recurring visual patterns and examine how concept-support allocations relate to downstream behavior across fine-grained, spurious-correlation, and general settings.Different settings may demand repeated support for discriminative evidence or more support for less frequent concepts.Source: §1 · p. 2
  • The selection objective accumulates concept support, applies diminishing returns to saturated concepts, and uses local support to favor recurring underrepresented concepts over isolated occurrences.This makes it possible to vary the balance between local support and tail concepts.Source: §1 · p. 2
  • Selected subsets consistently outperform strong data-selection baselines across classification and multimodal pretraining, while allocation changes correspond to downstream behavior.Broader concept coverage can improve robustness to spurious context, while aggressively prioritizing tail concepts can hurt when recurring discriminative evidence matters more.Source: §1 · p. 2

2 RELATED WORK

Prior data-selection research has emphasized sample utility, representativeness, coverage, and redundancy, while sparse concepts have separately supported interpretability and emerging selection approaches. This work directly optimizes concept support together with local support across several settings.

2 RELATED WORK

  • Existing data-selection approaches use geometry, training dynamics, optimization, self-supervised scores, utility-and-coverage combinations, data attribution, and pretrained representations.Examples include K-Center, forgetting-based selection, Moderate Coreset, CCS, D2-Pruning, InfoMax, RAM-APL, and image–text filtering.Source: §2 · p. 3
  • Sparse autoencoders decompose dense activations into sparse, often interpretable features, with vision and vision–language extensions capturing localized and hierarchical visual patterns.Related concept-based selection includes target-distribution matching, difficulty estimation, diversity-driven selection, and dynamic concept exposure.Source: §2 · p. 3
  • This work optimizes concept support together with local support as an interpretable selection criterion for classification and multimodal pretraining.It examines how distributions produced by selection strategies relate to downstream behavior across dataset types.Source: §2 · p. 3

3 METHODOLOGY

The method treats subset selection as reallocating support across visual concepts under a limited budget. It combines diminishing returns on concept coverage with neighborhood-based local support, aiming to favor under-supported evidence without elevating isolated samples.

3 METHODOLOGY

  • SAE activations from DINOv3 patch features are max-pooled into sparse image vectors that represent each image’s concept support.The method considers objects, attributes, colors, and textures as possible concepts.Source: §3 · p. 3
  • Because concept support is long-tailed, removing rare fine-grained or context evidence can narrow training support and leave similar test cases out of distribution.Tail concepts are vulnerable to data reduction, motivating explicit tracking of accumulated support.Source: §3 · p. 3
  • Concept coverage identifies underrepresented evidence, while local support helps distinguish meaningful source structure from isolated samples; aggressive tail expansion can introduce chaotic training signals.The pipeline therefore progresses from identifying valuable concepts to selecting samples that reliably support them.Source: §3 · p. 3, Figure 2 · p. 4

3.1 CONCEPT COVERAGE AS A PROBE FOR THE DISTRIBUTION TAIL

  • A logarithmic diminishing-return utility lowers the marginal gain of repeatedly supporting prevalent concepts, making under-supported concepts more competitive without specifying a tail threshold or target frequency.The paper compares alternative concave utilities in §D.Source: §3.1 · p. 4

3.2 LOCAL SUPPORT TO EXPOSE VALUABLE SAMPLES

  • Neighborhood support augments a candidate’s concept contribution with credit recurring among nearby samples, while isolated rare concepts receive limited local support.β controls local-support strength, and a_ti is cosine similarity normalized across the neighboring set; ablations examine β and normalization.Source: §3.2 · p. 4
  • Lazy greedy selection exploits the monotone submodular objective’s (1 − 1/e) approximation guarantee and sparse activations to reduce marginal-gain computation.The procedure avoids recomputing every candidate’s marginal gain at each iteration.Source: §3.2 · p. 4

4 EXPERIMENTS

The experiments test concept-aware subset selection across image-classification and multimodal settings under matched selection budgets. Results favor the method over strong baselines, while ablations examine representation choices, similarity normalization, and task-dependent local support.

4 EXPERIMENTS

  • Classification uses NABirds, Stanford Cars, Waterbirds, and Spawrious with ResNet-50; multimodal experiments select CC12M subsets for CLIP ViT-B/16 evaluation.Methods share selection budgets and downstream training configurations; the baselines include Random, K-Center, and task-specific selection methods.Source: §4 · p. 5

4.1 MAIN RESULTS

  • At 10%, NABirds improves over the strongest baseline by 3.2%, while Waterbirds gains are 7.6% at 10% and 7.5% at 30%; Stanford Cars leads at multiple budgets.The method remains competitive on Spawrious, where EL2N and CCS perform better but are generally weaker on other benchmarks.Source: §4.1 · pp. 5–6
  • At 10% CC12M, zero-shot accuracy improves by 2.4%; I2T and T2I Recall@1 improve by 2.4% and 1.1% at 30%, and by 2.3% and 2.0% at 70%.The method uses image-level concept support without information from the text distribution.Source: §4.1 · p. 6

4.2 ABLATION STUDY

  • SAE concept coverage is strongest on NABirds, while SAE local support improves Waterbirds over dense DINO; SAE dimensions show multiple activation peaks unlike DINO's generally normal profiles.The authors argue that these distributions provide more indicative signals for accumulated and rare concepts.Source: §4.2 · p. 6, Figure 4 · p. 6
  • The selection method remains effective across DINOv2, DINOv3, CLIP, and SigLIP feature spaces, rather than depending on one pretrained backbone or objective.The proposed objective locates tail concepts from the accumulated distribution when an encoder extracts diverse concepts.Source: §4.2 · pp. 6–7
  • Row-normalizing neighbor similarity improves performance in five of six settings and bounds each target sample's contribution across its neighbors.The normalization prevents samples in dense neighborhoods from contributing disproportionately large local support.Source: Table 3 · p. 6, §4.2 · p. 7
  • Across fine-grained, spurious, and general cases, β = 0.05 improves over concept coverage alone; fine-grained and spurious datasets favor β = 0.05, while general datasets favor β = 2.The experiments use the corresponding preferred value for each dataset group.Source: §4.2 · p. 7, Figure 5 · p. 7

5 ANALYSIS AND DISCUSSION

The analysis links concept allocation in selected subsets to downstream recognition and robustness. Across the NABirds and Waterbirds case studies, local support and broader visual evidence matter alongside tail enrichment and group composition.

5 ANALYSIS AND DISCUSSION

  • Compared with random selection’s 7,973, our subset increases effective number from 865 to 1,124 and lowers prevalence ratio from 1.00 to 0.74; the three compared selectors remain at 0.97.Our subset allocates a greater share of its concept support to tail concepts.Source: §5 · p. 8
  • Test samples carrying enhanced concepts show substantially larger accuracy improvements than samples carrying unchanged or reduced concepts, although all three categories improve.The analysis uses NABirds and ImageNet-1K linear probing from multimodal pretraining.Source: §5 · p. 8

5.1 CASE STUDY: NABIRDS

  • In NABirds, accuracy improves on samples carrying enhanced concepts such as beaks, wings, hands, signs, and feeders; hand-activated test accuracy rises from 28.6% to 51.4%.Only 160 of 23,929 training images activate the hand concept; our subset retains 60 across 50 categories, compared with random selection’s 20 across 20 categories.Source: §5.1 · p. 8
  • Removing local support reduces accuracy from 42.9% to 37.9%, while pure-rarity selection enriches tail concepts 8.2× but yields 33.1% accuracy.Pure rarity can favor atypical examples with little local support; further tail enrichment beyond the full objective does not improve NABirds accuracy.Source: §5.1 · pp. 8–9, Figure 7 · p. 9

5.2 CASE STUDY: WATERBIRDS

  • Our Waterbirds subset contains 18.8% bias-conflicting examples, while other compared approaches select less than 10%; broader concept coverage diversifies object–background combinations and weakens the background shortcut.Table 5 reports subset composition and robustness to background changes; its caption defines counter-correlation as the fraction of bias-conflicting examples.Source: §5.2 · p. 9, Table 5 · p. 9
  • Matching Waterbirds group composition raises WGA from 52.3% to 63.7%, still below our method; easy-example selection reaches 64.0% WGA versus 52.2% for difficult examples.The results suggest that robustness reflects both more favorable group composition and richer visual evidence within groups.Source: §5.2 · p. 9

6 CONCLUSION

The conclusion frames data selection as reallocating concept support under a limited data budget. It reports effectiveness across classification and multimodal pretraining and connects enhanced context concepts with robustness to out-of-distribution and spurious connections.

6 CONCLUSION

  • The paper formulates data selection as reallocating visual-concept support, using sparse autoencoders and an objective balancing concept coverage with local support.The objective seeks to address tail-concept gaps while avoiding isolated samples.Source: §6 · p. 9
  • Experiments suggest the proposed concept-based selection is effective across classification and multimodal pretraining tasks.The paper probes model properties using concepts enhanced by the selected dataset.Source: §6 · p. 9
  • Context concepts that do not directly provide category-related information improve robustness to out-of-distribution and spurious connections.The conclusion presents interpretable, fine-grained concept measurement as a principle for data selection.Source: §6 · p. 10

ETHICS STATEMENT

The ethics statement reports that the study used no human subjects or personal or identifiable information. It describes the data as publicly available and recommends validating selected subsets before use beyond research settings.

ETHICS STATEMENT

  • The study involved no human subjects and collected no personal or identifiable information.Source: ETHICS STATEMENT · p. 10
  • All experiments use publicly available datasets in accordance with their respective licenses.Source: ETHICS STATEMENT · p. 10
  • Selected subsets naturally reflect their source datasets’ characteristics, so the authors recommend validation before use beyond research settings.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement identifies released resources and points to appendices for experimental setup, SAE details, and implementation at scale.

REPRODUCIBILITY STATEMENT

A IMPLEMENTATION DETAILS

The implementation uses SAE-derived image concepts and a shared selection and training setup across tasks. For CC12M, an approximate neighbor graph and lazy greedy selection make it feasible to rank the full pool.

A.1 DEFAULT CONFIGURATION

  • The default setup uses a TopK SAE on DINOv3-B patch activations, max-pools activations into image-level concept vectors, and applies task-specific β values with k = 3 neighbors.Classification uses ResNet-50, while CC12M multimodal pretraining uses ViT-B/16.Source: §A.1 · p. 16

A.2 TRAINING HYPERPARAMETERS

  • ImageNet-1K uses a larger learning rate, cosine schedule, and longer training budget; training steps are held constant across selection rates.Source: §A.2 · p. 16

A.3 SCALABLE SELECTION: NEIGHBOR GRAPH AND LAZY GREEDY

  • For CC12M, approximate neighbor retrieval and exact re-ranking build the graph in about 34 minutes, and lazy greedy completes the full ranking in about 1.3 hours.The ranking is reused across selection ratios.Source: §A.3 · p. 16

B.1 TRAINING SETUP

This appendix describes SAE training and evaluation, defines concept-support selection variants, and specifies visualizations used to inspect concepts and model attributions.

B.1 TRAINING SETUP

  • ReLU, TopK, and BatchTopK SAEs use Matryoshka reconstruction; TopK variants enforce sparsity through activation and add AuxK loss to mitigate dead latents.The nested objective encourages early latents to capture broad structure and later latents to refine reconstruction.Source: §B.1 · p. 17

B.2 EVALUATION METRICS AND RESULTS

  • L0 counts active latents per token, while dead latents remain inactive or rarely active; both characterize SAE dictionary use alongside reconstruction fidelity.Source: §B.2 · p. 18
  • Normalized MSE and cosine similarity assess reconstruction fidelity, while CKNNA measures local alignment between backbone activations and SAE representations.Source: §B.2 · p. 18
  • Downstream fidelity compares predictions from original and SAE-reconstructed activations using KL divergence and Top-1 accuracy decrease, with lower values indicating better preservation.Source: §B.2 · p. 18
  • The evaluation compares sparsity, reconstruction, representation preservation, and downstream fidelity; TopK-32 provides a favorable balance and is used in the main experiments.Source: §B.2 · p. 18, Table 10 · p. 18

C.1 CONCEPT SUPPORT AND CONCEPT GROUPS

  • Concept support is measured for subsets against random subsets at the same budgets; concepts are enhanced above the 97.5th percentile, reduced below the 2.5th, and otherwise classified by the stated middle-percentile band.Source: §C.1 · p. 19

C.2 PURE-RARITY SELECTOR

  • The pure-rarity selector scores each image by the summed inverse prevalence of its active concepts, favoring images carrying many rare concepts without coverage or local-support terms.Source: §C.2 · p. 19

C.3 VISUALIZATION METHODS

  • SAE concept visualizations retrieve images with strong concept activation and overlay patch activations; Finer-CAM maps regions driving predictions from the ground-truth category against the three highest-logit categories.The representation ablation table compares SAE and dense DINO features for concept coverage and local support, reporting NABirds accuracy and Waterbirds WGA.Source: §C.3 · p. 19, Table 11 · p. 20

D.1 CONCEPT AND LOCAL REPRESENTATIONS

These ablations examine representation choices, local-support settings, utility functions, SAE pretraining data, and downstream architecture. Results support SAE-based selection choices and show that the method’s advantage persists across architectures.

D.1 CONCEPT AND LOCAL REPRESENTATIONS

  • Using SAE features for both concept coverage and local support achieves the best result; removing the concept component causes the largest drop, including Waterbirds 10% WGA falling from 68.6% to 38.0%.At NABirds 10%, SAE-based coverage exceeds all DINO-based concept variants by at least 6.2%.Source: §D.1 · p. 20
  • Row normalization improves performance in five of six settings; β = 0.05 performs best overall for fine-grained and spurious-correlation tasks, while general classification performs best at β = 2.Source: §D.1 · p. 20, Table 13 · p. 20, Table 14 · p. 20
  • Across NABirds and Waterbirds, k = 3 gives the most consistent performance, while larger neighborhoods show no systematic improvement.Source: §D.1 · pp. 20–21, Table 12 · p. 20

D.3 CHOICE OF UTILITY FUNCTION

  • Nonlinear utilities generally outperform the linear alternative, supporting diminishing returns; the logarithmic utility is chosen for strong overall performance across datasets and selection rates.All compared utilities share the same initial marginal value.Source: §D.3 · p. 21, Table 15 · p. 21

D.4 SAE FEATURES VS. SELECTION OBJECTIVE

  • K-Center on SAE features improves over DINO-feature K-Center in five of six settings but remains below the proposed method in all six, indicating that most of the gain comes from the objective.On NABirds at 10%, SAE features add 1.1 points to K-Center and the objective adds another 10.2.Source: §D.4 · p. 21, Table 16 · p. 21

D.5 SAE PRETRAINING DATA

  • An ImageNet-1K-pretrained SAE performs comparably to target-trained SAEs on NABirds and Waterbirds, supporting use of a single pretrained concept space across these tasks.Source: §D.5 · p. 22, Table 17 · p. 22

D.6 DOWNSTREAM ARCHITECTURE

  • The method’s advantage persists when the same subsets are evaluated with ViT-B/16; it achieves the best NABirds accuracy, Waterbirds WGA, and overall accuracy across all three rates.At 10%, Waterbirds WGA changes from 68.6% to 63.7% for the method, versus 57.5% to 21.7% for K-Center and 31.6% to 3.9% for Forgetting.Source: §D.6 · p. 22, Table 18 · p. 22

E.1 IMAGENET-1K AND DISTRIBUTION SHIFT

On ImageNet-1K, selection gains are modest on standard validation but more pronounced on several robustness benchmarks. Additional analyses show how selection redistributes concept support and how models trained on selected subsets attend to concepts and image regions.

E.1 IMAGENET-1K AND DISTRIBUTION SHIFT

  • On ImageNet-1K, smaller gains may reflect that random subsets already preserve much evidence needed for standard classification; characterizing support allocation in broad datasets remains future work.Source: §E.1 · p. 23
  • The method improves ImageNet-1K validation accuracy over random selection at all three rates, with modest gains; on robustness tests it leads on ImageNet-A, ImageNet-R, and ImageNet-D at each rate, while CCS does better on ImageNet-V2 at 10% and 50%.Source: §E.1 · p. 23, Table 19 · p. 23

E.2 CONCEPT-SUPPORT PROFILES OF ALL SELECTORS

  • Across 10% NABirds and Stanford Cars subsets, other selectors stay near the random concept-support profile, while this method spreads support more evenly and shifts it toward less prevalent concepts.Source: §E.2 · p. 23, Table 20 · p. 24

E.3 ADDITIONAL SAE CONCEPT VISUALIZATIONS

  • Enhanced concepts include uncommon context such as hands, sign edges, human figures, and suspension bridges; reduced concepts include recurring patterns such as waves, sky, trees, and tires.These examples illustrate redistribution of the fixed training budget away from repeatedly represented patterns.Source: §E.3 · p. 23, Figure 8 · p. 24

E.4 ADDITIONAL FINER-CAM VISUALIZATIONS

  • Across additional Finer-CAM examples, models trained on the method’s subsets more often attribute to target objects, while random and prior-work subsets more often emphasize surrounding context.Source: §E.4 · p. 24, Figure 9 · p. 25

F RELATION TO CONCEPT-BASED DATA SELECTION

  • This work studies support allocation across concepts in a fixed selected subset, complementing prior approaches that select examples using concept diversity, difficulty, or target distributions.Source: F RELATION TO CONCEPT-BASED DATA… · p. 25

Related papers

Loading 2610.21007v1…