PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Dataset Selection for High-Quality Data Sharing

Xiaona Zhou, Yingyan Zeng, Ran Jin, Ismini Lourentzou

arXiv:2512.10952v2cs.LGcs.AI

TL;DR

Existing data selection methods often operate on individual samples, while real-world data is organized into datasets that vary in relevance. DaSH models utility at both group and dataset levels; across benchmarks, it outperforms existing selection baselines, with accuracy gains up to 26.2%.

  • Problem

    Existing selection methods often select individual samples and assume data sources are uniformly relevant, despite datasets varying in relevance, redundancy, and quality.

    Source: §1 · p. 1

  • Method

    DaSH uses hierarchical Bayesian inference to model dataset utility at both group and dataset levels and explore candidates using observed model performance.

    Source: §1 · p. 2

  • Results

    Across two public datasets, DaSH improves accuracy by up to 26.2% on DIGIT-FIVE and 10.8% on DOMAINNET.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support automating practical data curation across heterogeneous sources, with reported robustness to imperfect grouping and limited exploration budgets.

    Source: §7 · p. 7

    Evaluation focuses on publicly available image datasets, which may not capture the complexity of other modalities or selection environments.

    Source: Limitations · p. 14

Abstract

from arXiv · show

The success of modern machine learning hinges on access to high-quality training data. In many real-world scenarios, such as acquiring data from public repositories or sharing across institutions, data is naturally organized into discrete datasets that vary in relevance, quality, and utility. Selecting which repositories or institutions to search for useful datasets, and which datasets to incorporate into model training are therefore critical decisions, yet most existing methods select individual samples and treat all data as equally relevant, ignoring differences between datasets and their sources. In this work, we formalize the task of dataset selection: selecting entire datasets from a large, heterogeneous pool to improve downstream performance under resource constraints. We propose Dataset Selection via Hierarchies (DaSH), a dataset selection method that models utility at both dataset and group (e.g., collections, institutions) levels, enabling efficient generalization from limited observations. Across two public benchmarks (Digit-Five and DomainNet), DaSH outperforms state-of-the-art data selection baselines by up to 26.2% in accuracy, while requiring significantly fewer exploration steps. Ablations show DaSH is robust to low-resource settings and lack of relevant datasets, making it suitable for scalable and adaptive dataset selection in practical multi-source learning workflows.

1 Introduction

The introduction frames dataset selection as finding useful whole datasets in heterogeneous source pools, a setting that instance-level methods do not model. It presents DaSH’s hierarchical approach as a way to estimate utility across groups and datasets and select efficiently.

1 Introduction

  • Existing selection methods often choose individual samples and assume uniform relevance, although datasets differ in relevance, redundancy, and quality.Datasets are commonly acquired or shared in discrete units and organized by source, institution, or repository.Source: §1 · p. 1
  • The paper formalizes selecting a subset of unknown-relevance datasets that improves model performance without exhaustively evaluating candidates.The task reflects resource, bandwidth, or labeling constraints in multi-source data acquisition.Source: §1 · pp. 1–2
  • DaSH uses hierarchical Bayesian modeling to estimate utility at both group and dataset levels, supporting structured exploration from observed model performance.Groups are based on dataset origin, such as an institution or collection.Source: §1 · p. 2
  • Across two public datasets, DaSH improves accuracy by up to 26.2% on DIGIT-FIVE and 10.8% on DOMAINNET.The introduction reports consistent performance gains over data selection methods.Source: §1 · p. 2

2 Related Work

  • Prior selection work largely overlooks dataset hierarchies; DaSH applies hierarchical bandit ideas to identify groups that jointly maximize downstream performance.The paper describes this as the first use of hierarchical bandits for dataset selection and reports gains in accuracy and efficiency over non-hierarchical alternatives.Source: §2 · p. 2

3 Method

DaSH formalizes dataset selection as choosing external datasets to improve performance over local-only training, then uses hierarchical Bayesian inference to estimate utility at group and dataset levels. It selects first across groups and then within a chosen group, updating both levels from reward observations.

3 Method

  • Dataset selection seeks an external subset that maximizes model performance gain over training on local data alone.The pool consists of groups containing one or more datasets.Source: §3 · p. 2

DaSH Initialization

  • DaSH models group and dataset utility with hierarchical Bayesian distributions, selecting a group and then a dataset within it based on sampled posterior values.Reward observations update both the selected group and dataset distributions.Source: DaSH Initialization · pp. 2–3, Figure 2 · p. 3

DaSH Posterior Computation

  • The group posterior mean combines the prior mean with aggregated mean rewards, while each dataset posterior depends only on its own rewards and approaches its empirical mean as observations accumulate.The group-level prior's influence decays as observations increase.Source: DaSH Posterior Computation · p. 3

Dataset Selection Based on Posterior Distributions

  • DaSH selects groups and datasets whose posterior means exceed a percentile-based threshold, with the percentile set to match training costs or exploratory needs.Selecting the top-x groups or datasets is an alternative.Source: Dataset Selection Based on Posterior… · pp. 3–4

Algorithmic Complexity

  • Hierarchical sampling costs O(n + m_i) per step, compared with O(|D|) for flat selection, and each reward updates both dataset- and group-level posteriors.The paper reports that sharing feedback across datasets in a group reduces evaluations needed to reach a fixed target accuracy.Source: Algorithmic Complexity · p. 4

4 Experiments

Experiments on DIGIT-FIVE and DOMAINNET assess dataset selection across domain-specific subsets and grouping strategies. DaSH approaches the global upper bound on DIGIT-FIVE and outperforms baselines on both benchmarks.

4 Experiments

  • DIGIT-FIVE contains five digit domains and DOMAINNET four object-style domains, each divided into three disjoint subsets for the experiments.The study compares perfect same-domain groups with mixed groups combining subsets from different domains.Source: §4 · p. 4

Experimental Results

  • External datasets consistently underperform local baselines on DIGIT-FIVE, whereas DOMAINNET shows more favorable cross-domain transfer, including gains for REAL trained on CLIPART subsets.The heatmaps compare local accuracy, augmented training with same- and cross-domain subsets, and the same-domain global reference.Source: §4 · p. 4, Figure 3 · p. 4
  • 78.3% average accuracy on DIGIT-FIVE leaves DaSH 0.5% below the 78.8% global upper bound and well above the 51.2% local lower bound.The comparison averages five independent runs across five domains.Source: Experimental Results · p. 5, Table 1 · p. 5
  • DaSH falls behind ActiveFT and BiLAF by 26.2% and 20.4%, respectively, on DIGIT-FIVE, while maintaining top performance with low variance across target domains.The authors note particularly low baseline accuracy on MNIST-M and SYN, which have substantial distributional divergence from other datasets.Source: Experimental Results · p. 5, Table 1 · p. 5
  • 3.3–10.8%: DaSH outperforms all baselines on DOMAINNET, although its margins are narrower than on DIGIT-FIVE.The authors attribute the narrower margins in part to shared ResNet-18 features reducing domain differences.Source: Experimental Results · p. 5, Table 2 · p. 5

5 Ablation Studies

The ablations examine hierarchical grouping, exploration budgets, initialization strength, and cross-domain grouping. Results show that hierarchical variants retain advantages under constrained exploration and imperfect or atypical group structures.

5 Ablation Studies

  • The ablations isolate hierarchical modeling, imperfect group definitions, Bayesian posterior updates, exploration–exploitation, selection granularity, and efficiency.They assess robustness across grouping noise and varying selection settings.Source: §5 · p. 5

Impact of Hierarchical Grouping

  • 20–60 steps: DaSH saves this many exploration steps per DIGIT-FIVE domain versus DaS (flat) without sacrificing accuracy, while imperfect grouping often lies near the perfect-grouping frontier.DaSH (mixed) has only modest drops in more challenging domains such as SYN, QUICKDRAW, and REAL.Source: Impact of Hierarchical Grouping · pp. 5–6, Figure 4 · p. 6

Comparison Under Limited Exploration

  • +8.8% on MNIST, +1.8% on USPS, +9.8% on MNIST-M, and +4.5% on SYN: DaSH and DaSH (mixed) gains over DaS (flat) under a 15-step DIGIT-FIVE budget.Both hierarchical variants outperform the flat baseline in 4 of 5 domains; mixed-grouping accuracy differs from perfect grouping by 1–2% in most domains.Source: Comparison Under Limited Exploration · p. 6, Figure 5 · p. 6

Effectiveness Under Weak Initialization

Robustness under Cross-Domain Grouping

6 Qualitative Analysis

Qualitative examples show DaSH selecting data more consistent with the target domain than several baseline methods across DIGIT-FIVE and DOMAINNET.

6 Qualitative Analysis

  • Baselines often select visually similar but incorrect domains, while DaSH selects relevant data across both benchmarks and diverse categories.For MNIST, only FreeSel selects a sample from MNIST; the remaining baselines fail to retrieve meaningful samples in the described comparison.Source: §6 · p. 7

7 Conclusion

The conclusion presents DaSH as a framework for selecting training datasets from diverse sources, reporting stronger performance and efficiency under realistic constraints while identifying future directions for broader selection criteria and settings.

7 Conclusion

  • DaSH consistently outperforms non-hierarchical and instance-level selection baselines and remains robust to imperfect grouping and limited exploration budgets.The conclusion frames these results as evidence for automating practical data curation across heterogeneous sources.Source: §7 · p. 7
  • DaSH models relationships between datasets and their sources to improve selection efficiency and downstream performance.Source: §7 · p. 7
  • Future work includes multi-objective criteria such as utility, fairness, and domain coverage, and application to evolving multi-institutional sharing platforms.Source: §7 · p. 7

Reproducibility Checklist

The reproducibility checklist marks the paper as including a conceptual outline or pseudocode for its AI methods and pedagogical references for less-familiar readers.

Reproducibility Checklist

  • The checklist marks a conceptual outline and/or pseudocode description of introduced AI methods as present.Source: §1 · p. 10
  • The checklist marks statements distinguishing opinions, hypotheses, and speculation from objective facts and results as present.Source: §1 · p. 10
  • The checklist marks pedagogical references for less-familiar readers seeking background for replication as present.Source: §1 · p. 10

2. Theoretical Contributions

The checklist indicates that the paper does not make theoretical contributions; its follow-up fields on assumptions, claims, proofs, and related matters are unanswered.

2. Theoretical Contributions

  • The checklist states that the paper makes no theoretical contributions.Source: §2 · p. 10
  • The checklist's follow-up fields on assumptions and restrictions, novel claims, proofs, and proof intuition contain response prompts rather than answers.Source: §2 · p. 10
  • The checklist's fields on citations to theoretical tools, empirical demonstration of theoretical claims, and experimental code for eliminating or disproving claims also remain unanswered.Source: §2 · p. 10

3. Dataset Usage

The dataset checklist reports use of datasets, motivation for the selected datasets, and appropriate citations and public availability for datasets drawn from existing literature.

3. Dataset Usage

  • The checklist states that the paper relies on one or more datasets and gives a motivation for using the selected datasets.Source: §3 · p. 10
  • The checklist marks data-appendix and publication-availability questions for newly introduced datasets as not applicable.Source: §3 · p. 10
  • For datasets drawn from existing literature, the checklist marks appropriate citations and public availability as present.Source: §3 · p. 10
  • The checklist marks the question about describing nonpublic datasets and explaining why public alternatives are inadequate as not applicable.Source: §3 · p. 10

4. Computational Experiments

The paper reports computational experiments and provides documentation on evaluation, reproducibility, and implementation practices.

4. Computational Experiments

  • The checklist reports experiments, metric motivation, run counts, distributional analysis, and statistical testing as included.It marks metric explanations, algorithm-run counts, variation or confidence measures, and significance tests as present.Source: §4 · pp. 10–11
  • The checklist marks preprocessing and experiment-analysis code, public code release, implementation comments, random-seed procedures, and computing infrastructure as documented.These items are marked yes in the paper’s checklist.Source: §4 · p. 10
  • Final hyperparameters and the parameter-development search details are marked not applicable.The checklist marks final-parameter reporting and parameter search details as NA.Source: §4 · pp. 10–11

Datasets

The experiments use DIGIT-FIVE and DOMAINNET, organizing dataset subsets into groups to examine selection under different source structures.

Datasets

  • DIGIT-FIVE comprises five handwritten-digit subsets, with preprocessed CNN features and each subset randomly divided into three mutually exclusive groups.The subsets are MNIST, MNIST-M, USPS, SVHN, and SYN.Source: Datasets · p. 12
  • DOMAINNET experiments use 15 classes from four of its six domains, with each selected domain randomly divided into three mutually exclusive groups.The selected domains are CLIPART, QUICKDRAW, REAL, and SKETCH.Source: Datasets · p. 12
  • The experiments test perfect, mixed, and cross-domain groupings, including groups that combine subsets from different domains.Mixed grouping reflects organizations contributing data spanning multiple domains; the cross-domain DIGIT-FIVE groups contain no same-domain datasets.Source: Datasets · p. 12

Implementation details

This section describes DaSH’s implementation and analysis, including scalability tests, behavior when relevant sources are absent, and the scope of the evaluation. It reports consistent accuracy gains with larger dataset pools and asymptotically optimal regret, while noting that evaluation is limited to public image datasets.

Implementation details

  • DIGIT-FIVE uses a one-layer CNN and DOMAINNET a three-layer classifier; K-means supplies five near-centroid points per class, and stopping uses fewer steps than exhaustive exploration.The exhaustive representative-point totals are 750 steps for DIGIT-FIVE and 1125 for DOMAINNET.Source: Implementation details · pp. 12–13

DaSH Dataset Selection Algorithm

Robustness to Absence of Relevant Sources

Scalability to Larger Dataset Pools

  • 15-versus-51-dataset configurations show consistent accuracy gains across five domains; a 4× larger SVHN pool needs only 2.6× more selection steps.Per-step computational cost remains constant, and total selection steps increase sublinearly with pool size.Source: Table 5 · p. 13, Scalability to Larger Dataset Pools · p. 13

Optimality Analysis

  • DaSH achieves asymptotically optimal O(log T) regret as posterior distributions concentrate and the probability of selecting a suboptimal dataset vanishes.The analysis considers both group-level and dataset-level posterior distributions.Source: Optimality Analysis · pp. 13–14

Limitations

  • The evaluation covers publicly available image datasets, which may not capture the complexity of other data modalities or selection environments.The authors identify time-series and graph data as future domains for extending the evaluation.Source: Limitations · p. 14

Broader Impacts

  • The authors frame hierarchical selection as supporting robust, efficient, and transparent data integration under real-world data ecosystem constraints.They describe this as supporting ML systems that reflect the constraints and diversity of those ecosystems.Source: Broader Impacts · p. 14
Loading 2512.10952v2…