Source-linked AI summary
Diversity Without Labels: Task-Agnostic Coreset Selection Across Perception and Action
Julia Hindel, Benjamin Ramtoula, Kyuhwan Yeon, Nick Heppert, Paul Newman, Daniele De Martini, Abhinav Valada
TL;DR
Coreset selection seeks useful training subsets, but existing approaches can require task-specific designs, extensive labels, or costly comparisons. CoreDNA pools vision-model neuron activations into per-image and dataset-level distributions and uses them for task-agnostic selection. Across perception and action tasks, it matches or outperforms most baselines.
Problem
Coreset methods must identify useful subsets, while existing approaches can require proxy training, extensive annotations, task-specific tailoring, or image-to-image comparisons.
Source: §1 · p. 1Method
CoreDNA pools vision-foundation-model neuron activations into per-image and dataset-level distributions and selects coresets using activation statistics.
Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5Results
Across image classification, semantic segmentation, depth estimation, and imitation learning, CoreDNA matches or outperforms most baselines while remaining computationally efficient.
Source: §1 · p. 2, §5 · p. 10Takeaways & Limitations
Last-layer mean, standard deviation, and cross-layer rank agreement correlate best with downstream performance, supporting activation-based selection across task and data types.
Source: §5 · pp. 9–10Determining the optimal retention ratio is an active research direction in dataset pruning and is not addressed here.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
While data collection for model training is relatively easy, curating high-quality datasets from large collections is inherently challenging. Coreset estimation methods address this problem by identifying the best data subsets to train models on. However, existing approaches often require proxy model training, extensive annotations, or task-specific tailoring. Unsupervised alternatives typically exploit geometric properties in foundation model feature spaces, but are limited to pairwise image-to-image comparisons or comparisons against a fixed subset. We introduce CoreDNA, an unsupervised, task-agnostic coreset estimation approach that pools neuron activations from vision foundation models into per-image and per-dataset distributions. By comparing each image's distribution to the dataset-level distribution, CoreDNA can capture both low-level visual structure and high-level semantic information across layers. We find that high-quality samples concentrate in the tails of the last layer's activation statistics, which we validate by relating them to training-based coreset methods. Building on these findings, we develop a computationally efficient selection mechanism to construct task-agnostic, high-quality coresets for selective annotation and model training. Across image classification, semantic segmentation, depth estimation, and imitation learning for end-to-end driving and mobile manipulation, CoreDNA matches or outperforms the state of the art, demonstrating unprecedented generalization across perception and action tasks. These results highlight the effectiveness of compact activation-based representations for data selection.
1 INTRODUCTION
Coreset selection can make training more efficient, but identifying useful subsets is difficult when datasets are repetitive and informative edge cases are underrepresented. Existing methods may depend on task-specific designs, full labels, or costly comparisons, motivating CoreDNA’s dataset-relative activation approach.
1 INTRODUCTION
- Training on informative subsets can improve efficiency while preserving performance, but repetitive scenarios can leave rare, safety-critical or informative edge cases underrepresented.In robotics, fixed time- or location-based selection may oversample similar scenarios.Source: §1 · p. 1
- Existing one-shot methods may be task-specific or rely on sample-hardness signals from training on fully labeled data.Task-specific examples include classification and imitation learning; hardness may be measured by loss or misclassification.Source: §1 · p. 1
- Unsupervised coreset methods use foundation-model feature diversity, but commonly compare images pairwise or against a sampled reference distribution rather than the dataset’s own distribution.Pairwise comparisons are computationally expensive and do not scale to larger coresets; activation distributions can represent both low-level and semantic attributes.Source: §1 · p. 2
- CoreDNA pools foundation-model neuron activations into per-image and dataset-level distributions, then uses efficient density- and coverage-based selection across perception and action tasks.High-utility samples tend to lie in the tails of final-layer activation statistics; CoreDNA matches or outperforms state-of-the-art training-based and training-free methods.Source: §1 · p. 2
2 RELATED WORK AND BACKGROUND
Coreset methods select compact training subsets using either training-derived signals or training-free geometric criteria. CoreDNA instead represents activation distributions relative to the source dataset.
2 RELATED WORK AND BACKGROUND
- Training-based methods use losses, gradients, or bilevel optimization, often requiring proxy or target-model training and sometimes labeled data.Other approaches use pseudo-labeling or object-level representations to relax full-dataset labeling requirements.Source: §2 · p. 2
- CoreDNA’s Figure 1 summarizes activation-histogram pooling into per-layer statistics and selection using sliced-EMD or farthest-point sampling.The selected axes are last-layer mean and standard deviation plus cross-layer dispersion.Source: Figure 1 · p. 3
- Training-free methods commonly select diverse samples using geometric criteria in foundation-model feature spaces, including clustering, representative-point selection, density, or coverage.Related methods also use attention patterns, segmentation complexity, feature matching, or foundation-model agreement.Source: §2 · p. 3
3 TECHNICAL APPROACH
The technical approach formalizes one-shot coreset selection, derives compact image descriptors from pooled foundation-model activations, analyzes their relation to coreset quality, and uses selected axes for scalable sampling.
3.1 PROBLEM DEFINITION
- Coreset estimation selects a predefined α fraction of a dataset in one pass to maximize downstream performance, with training-free, task-agnostic, scalable selection as desiderata.Choosing the optimal retention ratio is outside this work’s scope.Source: §3.1 · p. 3
3.2 PER-IMAGE PATTERN EXTRACTION
- For each image, neuron-wise EMD compares activation histograms with dataset-level histograms, and per-layer mean, standard deviation, and maximum summarize the distances.The representation also considers rank-disagreement statistics, δmax and δavg, to capture when feature axes rank an image differently.Source: §3.2 · p. 4
- Same-layer µℓ and σℓ correlate at r = 0.93–0.95, while µ1 and µ12 correlate at r = 0.43; their extreme 10% tails overlap at Jaccard 0.12–0.27.Rank-disagreement statistics are mostly orthogonal to µ12 and σ12, with mean absolute correlations of 0.04–0.08 across five datasets; SemanticKITTI-DVPS and LIBERO-10 are exceptions.Source: §3.2 · pp. 4–5
3.3 ANALYSIS ON CORESET METHODS
- For video, a frame index encourages temporal diversity; for autonomous agents, pose, velocity, and acceleration can be compressed with PCA and added as a selection axis.Source: §3.3 · p. 6
- Last-layer µ12, σ12, and δmax correlate best with fidelity, while the three-axis space yields correlations of 0.308 (coverage) and 0.357 (fidelity).The three-axis space improves over either two-axis space on most datasets; Cityscapes shows no correlation with fidelity metrics.Source: §3.3 · pp. 5–6, Table 1 · p. 6
- Hardness patterns vary by dataset: Cityscapes’ high-µ12 tail is +0.63 SD, while PascalVOC’s low-µ12 tail reaches +0.80 SD.The relation is weak or absent on CIFAR10 and SemanticKITTI-DVPS, motivating both coverage- and fidelity-based selection.Source: §3.3 · p. 6
3.4 COREDNA CORESET SELECTION PROCEDURE
- Fidelity selection minimizes sliced Wasserstein distance, while coverage selection uses farthest-point sampling (FPS), which favors sparse and extreme regions.The respective costs are O(EP log P) and O(PC); axes are standardized and pairs correlated above 0.95 are removed.Source: §3.4 · pp. 6–7
4 EXPERIMENTAL EVALUATION
The evaluation spans perception, end-to-end driving, and visuo-motor policy learning, with CoreDNA performing consistently well across many settings. Results vary by dataset and retention regime, while both frame selection and auxiliary demonstration selection show benefits in imitation learning.
4.1 PERFORMANCE ON PERCEPTION DATASETS
- At α=0.7, CoreDNA beats unsupervised and training-based methods by at least 0.64 pp on Cityscapes; at α=0.1, it beats both by 2 pp on PascalVOC.Neither farthest-point nor sliced-Wasserstein sampling dominates universally; the better choice depends on dataset and task.Source: §4.1 · p. 7, Table 2 · p. 8
- At α=0.3 for segmentation and α=0.1 for depth, SemanticKITTI-DVPS is within ±1 pp of full-data performance; CoreDNA beats baselines by at least 0.82 pp on high-ratio Cityscapes-DVPS segmentation.Temporal stride is difficult to beat at α=0.1–0.3, while unsupervised methods do better at α=0.01 and α≥0.5.Source: Table 3 · p. 8, §4.1 · p. 8
4 EXPERIMENTAL EVALUATION
- CoreDNA performs consistently well across perception settings, often outperforming stronger, more cumbersome baselines and remaining competitive when it does not lead.The evaluation covers 41 dataset-ratio settings across seven benchmarks.Source: §4 · p. 7, §4.1 · p. 8
4.2 PERFORMANCE ON IMITATION LEARNING
- On LIBERO-10, CoreDNA matches or improves on full-frame sampling in both 5- and 50-demonstration variants, though α=0.9 may be suboptimal.The benchmark comprises ten long-horizon manipulation tasks.Source: §4.2 · p. 9
- At 50% retention, farthest-point sampling nearly matches full-data NAVSIM performance and beats WS and random at both tested ratios.The preferred selection axes vary: state and position perform best at α=0.1, while excluding position is preferable at α=0.5.Source: Table 4 · p. 9, §4.2 · p. 9
- Selecting 10% of LIBERO-90 as auxiliary demonstrations raises success rate and outperforms DataMIL.The selected demonstrations rank closest to the five fixed LIBERO-10 demonstrations, rather than matching LIBERO-90’s own distribution.Source: §4.2 · p. 9
5 CONCLUSION
CoreDNA uses pooled activation distributions and recording priors for one-shot, task-agnostic coreset estimation. It performs well across perception and action tasks while remaining computationally efficient.
5 CONCLUSION
- Last-layer mean, standard deviation, and cross-layer rank agreement correlate best with downstream performance, which the authors attribute to hard samples concentrating in distribution tails.Source: §5 · p. 9
- Adding recording priors such as frame index or position as an axis addresses redundancy in non-curated datasets.Source: §5 · p. 9
- CoreDNA selection is task- and data-type-agnostic across classification, segmentation, depth estimation, and imitation learning, outperforming most baselines while remaining computationally efficient.The authors suggest these findings may help extend coreset estimation beyond curated datasets toward real-world action domains.Source: §5 · p. 10
A.1 OVERVIEW
The appendix outlines additional selection settings, experimental details, and extended results, followed by analyses and ablations.
A.1 OVERVIEW
- Sections A.2 and A.3 cover additional CoreDNA selection settings and experimental setups for the evaluated tasks.Source: §A.1 · p. 14
- Sections A.4 and A.5 present extended results for end-to-end driving and visuo-motor policy learning, while A.6 expands performance-correlation experiments with criteria and per-dataset results.Source: §A.1 · p. 14
- Section A.7 compares image-pattern rankings across backbones and justifies histogram bin size; A.8 ablates axis choices and selection, and A.9 tests redundancy with duplicated CIFAR10.Source: §A.1 · p. 14
A.2 ADDITIONAL HYPERPARAMETERS AND PSEUDO-CODE
The appendix specifies sliced Wasserstein coreset selection, including its inputs, projection-based scoring, and spacing constraint.
A.2 ADDITIONAL HYPERPARAMETERS AND PSEUDO-CODE
- The algorithm takes feature points F, budget C, projections E, and minimum gap g, and returns C selected indices.Source: §A.2 · p. 14
- It samples E−1 random unit directions alongside e1, ranks points along each direction, and accumulates scores using a rank-based spacing weight.Source: §A.2 · p. 14
- Points are considered in ascending score order, subject to gap g from already selected points; remaining slots are filled by score if the gap prevents reaching C.Source: §A.2 · p. 14
- The method uses 16-bin per-image and per-dataset histograms and 10 sliced-Wasserstein projections, with further settings ablated in the appendix.Source: §A.2 · p. 14
A.3 DETAILED EXPERIMENTAL SETTING
The appendix details training and evaluation setups across five task families, using published baseline code and three-seed experiments.
A.3 DETAILED EXPERIMENTAL SETTING
- Experiments cover classification, segmentation, metric depth, end-to-end driving, and imitation learning, using published baseline code, three seeds, and consistent training iterations per dataset.Source: §A.3 · p. 14
- CIFAR10 classification uses ResNet18, 40,000 training iterations, random crops and flips, and accuracy as its reported metric.Source: §A.3 · p. 14
- Segmentation uses DeepLabv3 with a ResNet-101 backbone on four datasets, reports mIoU, and extends Forgetting, EL2N, and AUM using an 80%-correct-pixels criterion.Source: §A.3 · p. 14
- Metric depth uses ZoeDepth-Small with a Depth Anything V2 backbone on two datasets, reports RMSE, and defines correctness by relative error below 0.1 on at least 10% of the image.Source: §A.3 · pp. 14–15
- NAVSIM driving trains Latent TransFuser on navtrain and evaluates on held-out navtest, reporting Extended Predictive Driver Model Score.Source: §A.3 · p. 15
- LIBERO-10 manipulation fine-tunes OCTO and reports success over 50 roll-outs; the five demonstrations per task were selected manually after filtering failed demonstrations.Source: §A.3 · p. 15
A.4 DETAILED PERFORMANCE ON END-TO-END DRIVING
The driving evaluation summarizes performance with EPDMS, combining safety, compliance, progress, and comfort metrics. With reduced training data, CoreDNA’s strongest gains over random selection are in safety-related metrics and extended comfort, alongside a small reduction in ego progress.
A.4 DETAILED PERFORMANCE ON END-TO-END DRIVING
- EPDMS includes collision, drivable-area, direction, traffic-light, progress, time-to-collision, lane-keeping, history-comfort, and extended-comfort metrics.The score combines multiplier metrics NC, DAC, DDC, and TLC with weighted metrics EP, TTC, LK, HC, and EC; weights are {5, 5, 2, 2, 2}.Source: §A.4 · p. 15
- At α = 0.1, CoreDNA (FPS) improves on NC (+1.1), DAC (+1.5), TTC (+1.4), and EC (+3.6) over random selection, while EP is lower (−0.4).At α = 0.5, CoreDNA (FPS) also gains mainly on DAC (+1.1), with smaller gains on NC, DDC, and TTC and EP lower (−0.7).Source: §A.4 · p. 15
- With less training data, DAC and EC degrade markedly, while TLC and HC remain nearly saturated even at 10% of the data.The passage associates DAC with precise road-boundary perception and EC with noisy trajectory predictions.Source: §A.4 · p. 15
A.5 DETAILED PERFORMANCE FOR END-TO-END MANIPULATION IMITATION LEARNING
In LIBERO-10 manipulation, the full method does not win most per-task comparisons, although it follows the mean success-rate trend across demonstrations. The authors attribute better or comparable performance to pruning irrelevant frames and identify additional modalities as a future direction.
A.5 DETAILED PERFORMANCE FOR END-TO-END MANIPULATION IMITATION LEARNING
- The full method wins four of ten tasks with 5 demonstrations and three of ten with 50 demonstrations; in the LIBERO-90 mix-in, it wins one task.Data-MIL has the best performance on three of ten tasks.Source: §A.5 · pp. 15–16, Table 8 · p. 16
- The authors attribute better or comparable performance to pruning irrelevant frames from continuously recorded demonstrations.They expect force input could add information when images are mostly uninformative during dexterous interaction.Source: §A.5 · p. 16
- Unlike autonomous driving, adding low-dimensional joint states does not produce a similar performance increase in manipulation.Source: §A.5 · p. 16
A.6 DETAILED PERFORMANCE CORRELATION EXPERIMENTS
The appendix evaluates activation-derived selection spaces, score correlations, representation choices, and robustness to redundant data. Results favor a three-axis correlation space and show that unsupervised and diversity-augmented selectors remain stable under duplication, while some choices trade performance across datasets or retention settings.
A.6 DETAILED PERFORMANCE CORRELATION EXPERIMENTS
- With δmax, layer ℓ12 has the highest fidelity correlation (0.340), while ℓ6 and the all-layer mean lead coverage at 0.297 and 0.293.Cityscapes shows no correlation for any fidelity metric; the analysis continues with ℓ12 because its fidelity is strongest and the difficulty signal grows with depth.Source: §A.6 · p. 17, Table 9 · p. 18
- The three-axis space correlates best with performance on average: coverage 0.308 and fidelity 0.357, versus 0.276–0.280 and 0.340–0.341 for two-axis spaces.Adding either δmax or σ12 to the other axis improves correlations on six of seven datasets; the authors call the axes a sensible default, not an optimum.Source: §A.6 · p. 17, Table 11 · p. 19
A.7 VDNA BINS AND BACKBONE
- On CIFAR10’s 50,000 training images, the estimated 1000-bin representation occupies 28GB versus 3.1GB at 16 bins, with no additional gain in ordering agreement over 16 bins.The 128-bin representation occupies 11GB; at 2 bins, ordering fidelity is ρ = 0.976 to 0.990.Source: §A.7 · p. 18
- Across datasets, backbone orderings agree moderately (Spearman ρ = 0.814 to 0.877), but tail overlap is 0.410 to 0.511 versus 0.087 to 0.163 around the median.The chance overlap for two independent random subsets is 0.053.Source: §A.7 · p. 18, Figure 5 · p. 19
A.8 ABLATION OVER AXIS CHOICES AND SELECTION MECHANISM
- On CIFAR10, δavg performs best at α = 0.1, but δmax is retained as the universal choice; using only µ12 is comparable, while σ12 or δmax alone loses at least 1.23 pp.Using all layer means and standard deviations is not best at α = 0.1 and is only on par with the chosen space at α = 0.5.Source: §A.8 · p. 18, Table 12 · p. 20
- For sliced Wasserstein selection, reducing g costs 0.34 pp at α = 0.1, larger g hurts more, and E = 10 is optimal in both settings with other E values losing at least 0.43 pp.Source: §A.8 · p. 18, Table 13 · p. 20
A.9 CIFAR10 DUPLICATION EXPERIMENT
- On CIFAR10-Dupl at C = 25000, AUM, Forgetting, and EL2N drop by 63.36, 32.50, and 63.65pp, while CoreDNA, ZCore, FreeSel, ELFS, CCS, and BOSS remain within a few points of original-dataset performance.The authors link the training-dynamics methods’ collapse to duplicated samples distorting difficulty estimates and concentrating selection on redundant hard examples.Source: §A.9 · p. 19, Table 14 · p. 20