PaperlayerView exact revision on arXiv

Source-linked AI summary

CEDD-optimizer: Enabling Cost-Efficient Dataset Distillation on Geographically Distributed Edge Systems

Dai Liu, Eishi Arima, Martin Schulz

arXiv:2609.10151v1cs.DC

TL;DR

Distributed DD faces a tuning gap because edge locations have different energy and transfer prices, and DD settings affect cost and test accuracy. CEDD-optimizer models these factors and optimizes settings per edge device; its three-step tuning scheme uses distilled datasets for downstream runs, limiting their overhead.

  • Problem

    Cost-efficient DD tuning in geographically distributed, non-uniform edge environments remains largely unexplored, although DD hyperparameters affect transfer, energy use, and test accuracy.

    Source: §I · p. 1

  • Method

    CEDD-optimizer selects DD hyperparameters for each edge device using predictive models and a solver that minimizes total system cost for a target test accuracy.

    Source: §I · p. 2

  • Results

    The three-step tuning scheme's multiple downstream training runs add limited overhead because training uses distilled datasets.

    Source: 7) Cost Reduction by DD for Downstream… · p. 10

  • Takeaways & Limitations

    The framework targets deep-learning workflows across edge devices or the edge-to-cloud continuum while accounting for geographically dependent network and electricity costs.

    Source: §VIII · p. 13

Abstract

from arXiv · show

Centralized learning is a fundamental paradigm in modern AI, in which data are collected from distributed edge devices and aggregated at a central host for model training. However, this pipeline is often bottlenecked by the substantial communication overhead of data collection. Dataset Distillation (DD), with its high compression ratio, is therefore attractive for centralized learning on distributed data. Yet, the cost efficiency of DD in non-uniform edge environments remains largely unexplored. Geographically distributed edge devices face different energy and data-transfer prices, while key DD hyperparameters, such as the target compression ratio and number of distillation steps, substantially affect energy use, transfer overhead, and downstream test accuracy. These hyperparameters must therefore be tuned both locally and globally. We propose the Cost-Efficient Dataset Distillation optimizer (CEDD-optimizer), a hyperparameter-tuning framework for cost-efficient distributed DD. It minimizes total cost under a training-quality constraint by optimizing hyperparameter settings across edge devices while accounting for environmental non-uniformity. The framework comprises two modules: CEDD-calibrator and CEDD-solver. CEDD-calibrator identifies parameters for our energy and training-quality models: the former is determined through offline calibration, whereas the latter is estimated online through a three-step tuning scheme. Based on these models, CEDD-solver solves the cost-minimization problem to steer and improve the distributed DD workflow. Experiments across various image datasets show that our approach achieves up to a 20.8 improvement over the baseline DD method under the same quality constraint.

I. INTRODUCTION

The introduction frames distributed dataset distillation as a cost-aware tuning problem: edge locations differ in transfer and energy prices, while DD settings affect both operational cost and model quality. CEDD-optimizer selects settings per edge device using predictive models and an optimization solver.

I. INTRODUCTION

  • Traditional compression either offers limited compression ratios or sacrifices data details, while DD's costly hyperparameter tuning for geographically variable prices remains underexplored.The optimization target is minimizing total operational cost while satisfying a model-quality requirement.Source: §I · pp. 1–2
  • CEDD-optimizer selects distilled data per class (DPC, α) and iteration count (IC, β) separately for each edge device in a non-uniform deployment.DPC determines transferred-data volume, while IC governs local computation and energy consumption.Source: §I · p. 2
  • CEDD-calibrator estimates energy and test-accuracy models, and CEDD-solver uses them to minimize system cost for a target downstream test accuracy.The framework models energy use, test accuracy, and distilled-data volume as functions of DD hyperparameters.Source: §I · p. 2
  • Experiments across four edge devices and five benchmark datasets report significant cost reductions while maintaining competitive accuracy.The evaluation also analyzes scalability and trade-offs under varying system conditions.Source: §I · p. 2

A. Dataset Distillation

  • The work focuses on iterative DD, using original Dataset Condensation as its baseline while targeting parameters shared by other iterative methods.It presents geographical location-based tuning to minimize total cost with marginal accuracy loss in edge deployments.Source: A. Dataset Distillation · p. 2

B. Edge Computing

Dataset distillation replaces original data with a compact synthetic dataset, and its compression and energy trade-offs motivate tuning across a distributed centralized-learning workflow. The section describes that workflow and the distinct effects of the key DD hyperparameters.

B. Edge Computing

  • Prior edge-computing work addresses cost management, compression, and resource constraints; this work treats DD as orthogonal to federated learning and focuses on centralized pipelines.Source: B. Edge Computing · pp. 2–3

A. Motivation: High Compression Ratio of DD

  • DD achieves compression ratios orders of magnitude better than the compared methods, and its total energy cost is smallest in this experiment.The evaluation used MNIST and CIFAR10 accuracy thresholds of 97% and 60%, respectively, and the same NVIDIA Jetson Xavier NX platform for energy measurements.Source: Figure 3 · p. 3, A. Motivation: High Compression Ratio… · p. 3

B. Application Scenario

  • The assumed centralized workflow moves data from geographically distributed end devices to local edge DD, cloud aggregation, host training, and cloud model storage.Source: B. Application Scenario · p. 3

C. Leveraging Hyperparameters in DD

  • DPC α controls distilled-dataset size, while α and iteration count β both affect downstream test accuracy and edge DD energy overhead.The DD computation per loop is modeled as quadratic in α, while execution time and energy are linear in β.Source: C. Leveraging Hyperparameters in DD · p. 4

D. Impact of Geographical Locations on Pricing

Geographic variation in energy and transfer prices motivates cost-aware DD tuning across edge devices. The formulation selects per-edge hyperparameters to minimize total cost while meeting a downstream accuracy target.

D. Impact of Geographical Locations on Pricing

  • Energy prices vary significantly by country, while communication costs vary by an order of magnitude across edge-device locations.The paper identifies tuning DD under these geographic price variations as a previously unaddressed challenge.Source: D. Impact of Geographical Locations on… · p. 4

A. Problem Formulation

  • For each edge, the formulation selects compression and iteration settings based on its data and energy and transfer prices, alongside cloud storage cost.The resulting distilled datasets are aggregated for downstream training, whose accuracy must meet the target.Source: A. Problem Formulation · p. 5
  • The objective minimizes energy, data-transfer, and storage costs subject to a target test-accuracy constraint.The compression-setting upper bound is tied to a per-node target compression ratio to control data-transfer traffic.Source: A. Problem Formulation · p. 5

B. Cost and Accuracy Modeling

The cost model combines energy, transfer, and storage costs, with DD energy and distilled-data size modeled as functions of hyperparameters.

1) Cost Modeling:

  • Total cost comprises DD energy, distilled-data transfer, and storage costs, represented by three dot products of prices and resource amounts.The energy and data-volume vectors contain per-edge values that depend on each edge’s data and hyperparameters.Source: 1) Cost Modeling · p. 5
  • The empirical DD energy model treats iteration count β as scaling the loop cost and approximates compression-setting α's effect on per-loop energy quadratically.The initialization overhead is represented separately by an intercept.Source: 1) Cost Modeling · p. 6
  • Distilled-dataset size is estimated using a simple analytical model.The model is used for the size function in the cost calculation.Source: 1) Cost Modeling · pp. 5–6

2) Statistical Accuracy Modeling:

The accuracy model summarizes hyperparameter settings across edges with their means and standard deviations, then uses a second-order polynomial to estimate downstream test accuracy.

2) Statistical Accuracy Modeling:

  • The model treats accuracy as permutation-invariant to hyperparameter-vector elements, particularly for IID data.The paper notes non-IID cases separately.Source: 2) Statistical Accuracy Modeling · p. 6
  • The proposed test-accuracy model uses a second-order polynomial over the means and standard deviations of α and β.The authors stop at second-order moments after motivating a polynomial representation of descriptive statistics.Source: 2) Statistical Accuracy Modeling · p. 6
  • Keeping the model simple holds the number of unknowns constant as edge count grows and limits calibration overhead and overfitting.The authors report that the model is lightweight but accurate enough across various edge counts in their later evaluation.Source: 2) Statistical Accuracy Modeling · p. 6

A. Solution Overview

CEDD-optimizer pairs an offline energy-model calibrator with an online solver that tunes distributed DD. Its online scheme gathers compact training examples, fits an accuracy curve, and then optimizes edge settings, stopping early if the target is met.

A. Solution Overview

  • CEDD-calibrator identifies energy-model parameters offline, while CEDD-solver uses the models and prices to optimize hyperparameters for minimum cost at a target accuracy.Energy calibration is performed once per edge device; accuracy-model coefficients can be learned online or supplied from prior knowledge.Source: A. Solution Overview · p. 6

B. Three-Step Hyperparameter Tuning Scheme

  • The first tuning step runs DD on every edge with α and β at their minimum settings, producing an initial set of distilled datasets.The paper describes this step’s energy overhead as almost negligible under its O(α^2β) energy model.Source: B. Three-Step Hyperparameter Tuning… · pp. 6–7
  • The second step samples settings near their minima and assigns them to edges through minimum-weight matching to reduce DD energy and transfer cost.The sampling ranges ∆α and ∆β control total cost, including DD energy and subsequent transfer.Source: B. Three-Step Hyperparameter Tuning… · p. 7
  • The scheme trains downstream models on combinations of the collected distilled datasets, fits an accuracy curve, and then optimizes settings using that curve, the energy model, and prices.It can stop after the first or second step if the target accuracy is met; the procedure takes three or fewer steps.Source: B. Three-Step Hyperparameter Tuning… · pp. 6–7

VI. EVALUATION

The evaluation covers heterogeneous edge hardware, standard image datasets and data distributions, and vanilla Dataset Condensation as the DD baseline. Energy-model calibration measures runtime and power across settings of α on each platform.

A. Evaluation Setup

  • The default evaluation parameters are listed in Table II, with adjustments documented for individual experiments.The listed setup provides the reference configuration for the evaluation.Source: A. Evaluation Setup · p. 7, Table II · p. 7

1) Hardware Platforms:

  • CEDD-optimizer is evaluated on four edge and mobile platforms: Jetson Xavier NX, Jetson Orin Nano, MacBook Pro, and TQMx80UC.The platforms span embedded AI systems, a mobile platform, and an x86 edge module.Source: 1) Hardware Platforms · p. 7

2) Baseline Datasets and Networks:

  • Experiments use MNIST, FashionMNIST, CIFAR-10, SVHN, and ImageNette, with strong and weak scaling and IID, near-IID, and quantity-based non-IID settings.Near-IID and non-IID data use Dirichlet concentration parameters 1.0 and 0.5, respectively.Source: 2) Baseline Datasets and Networks · pp. 7–8

3) DD Implementation:

  • Vanilla Dataset Condensation is the DD baseline because it captures structures common to mainstream implementations, and the parametrization is presented as applicable to other DD frameworks.The evaluated downstream network follows the architecture of Gidaris and Komodakis.Source: 3) DD Implementation · p. 8, 2) Baseline Datasets and Networks · p. 8

4) Energy Model Calibration:

  • The energy model is calibrated by measuring runtime and power across eight α settings on four platforms, separating initialization from the DD main loop.Measurements use platform-specific power tools, with clock frequency at its default setting.Source: 4) Energy Model Calibration · p. 8

5) Price Parameter Setup:

The evaluation sets geographically variable energy and network prices, then validates the energy and test-accuracy models and their fitting-sample requirements. It compares CEDD-optimizer against DD and conventional sampling baselines.

5) Price Parameter Setup:

  • The cost model uses regional electricity and SiteLink network prices, with storage priced at $0.023 per GB regardless of edge location.Locations are randomly assigned from 74 selected countries.Source: 5) Price Parameter Setup · p. 8

6) Baseline Methods:

  • Comparisons include a maximum-parameter DD baseline, conventional random sampling, and random, heuristic, and exhaustive search; compression methods are excluded for missing the target ratio or accuracy.The baseline DD setting uses (α, β) = (αmax, βmax).Source: 6) Baseline Methods · p. 8

1) Validation of the Energy Model:

2) Validation of Accuracy Model:

  • For MNIST, FashionMNIST, SVHN, and CIFAR-10, test-accuracy absolute error and MAPE are below 0.5 and 1%, respectively, when N > 10.The reported error generally decreases as the number of edge nodes increases, then saturates.Source: 2) Validation of Accuracy Model · p. 8, Figure 12 · p. 9

3) Sample Count Selection in CEDD-calibrator:

  • Eight accuracy-model fitting points and three energy-model points suffice for the reported fit, while the calibrator samples 18 and five points, respectively.Across datasets, model MAPE is below 6% for accuracy and 3% for energy.Source: Figure 13 · p. 9, 3) Sample Count Selection in… · p. 9

4) Cost and Test Accuracy Benefits by CEDD-optimizer:

CEDD-optimizer reduces cost while meeting accuracy targets, and its trade-offs vary with calibration ranges, system heterogeneity, downstream training, and scaling policy. The reported scaling analysis identifies solver overhead as a potential large-scale bottleneck.

4) Cost and Test Accuracy Benefits by CEDD-optimizer:

5) Impact of ∆α and ∆β on Accuracy Model Quality in Our Three Step Tuning:

  • Setting (∆α, ∆β) = (25, 15) balances the trade-off across experiments; smaller ranges cover fewer basic-statistic regions and degrade test-accuracy-model accuracy, while (5, 5) yields no solver solution.Source: 5) Impact of ∆α and ∆β on Accuracy… · p. 9

6) Sensitivity to Device/Location Heterogeneity:

7) Cost Reduction by DD for Downstream Learning:

8) Scalability Assessment:

  • Solver overhead scales as O(N^3) and takes 3.4 seconds at N = 100, potentially becoming a bottleneck at larger scale.The paper identifies replacing SLSQP with a more efficient heuristic as one possible response.Source: 8) Scalability Assessment · p. 10
  • Under weak scaling, total cost excluding solver overhead is O(N) with or without DD, so DD’s cost-reduction ratio remains O(1); under strong scaling, it falls by O(1/N).The authors expect weak scaling to be more common because additional edges typically collect more data.Source: 8) Scalability Assessment · pp. 10–11, Table III · p. 10

9) Comparison to Iterative Search Methods:

Non-uniform hyperparameter tuning outperforms uniform setups across the evaluated dataset and distribution scenarios, while solver and search comparisons expose overhead trade-offs. The reported benefit also increases with network depth under the stated assumption.

9) Comparison to Iterative Search Methods:

  • CEDD-optimizer limits search overhead by restricting the search space, limiting iterations while fitting a curve, and using a solver to find an optimum.Exhaustive and heuristic search generally satisfy the constraint under IID and near-IID data, while random search can fail within 20 attempts.Source: 9) Comparison to Iterative Search… · p. 11

10) Benefit of Non-Uniform Hyperparameter Setup:

11) Impact of Network Depth on Cost Saving:

12) Impact of Solver Selection in CEDD-optimizer:

  • At N = 10, SLSQP takes 0.006s versus 2.431s for TRC, 110.686s for IDE, and 2.623s for MINLP, with relative costs of 1.33, 0.94, and 0.91 for those alternatives.SLSQP’s solution is reported as comparable to the discrete solvers, which become impractical as N scales.Source: 12) Impact of Solver Selection in… · pp. 11–12, Figure 18 · p. 12

13) Order Selection in Test Accuracy Model:

The section compares eight candidate accuracy models and selects model D by balancing prediction error against the calibration sample cost of adding model complexity.

13) Order Selection in Test Accuracy Model:

B. First-order means and variances (5 unknowns): a0 +

  • 0.73 absolute error and 1.31% MAPE for model D closely match models E and F, while increasing polynomial degree does not significantly reduce error.The model uses second-order moments, taking only means and variances, as a cost-accuracy trade-off.Source: B. First-order means and variances (5… · p. 12
  • Adding descriptive statistics gradually reduces error, but models with more unknowns need more online calibration samples; model H requires at least 18.The authors stop at second-order moments to balance model error against calibration cost.Source: B. First-order means and variances (5… · p. 12

VII. DISCUSSION

The discussion describes additional benefits and possible extensions of CEDD-optimizer while identifying boundaries around federated learning, non-IID data, and time-varying electricity prices. The conclusion frames the work as cost-aware DD hyperparameter tuning for geographically distributed edge settings.

A. Privacy Protection

  • DD's lossy synthesis makes original sensitive information extremely hard to restore, a privacy benefit the paper notes is outside its scope.The discussion cites theoretical and empirical studies reporting this benefit –, [102], [113].Source: A. Privacy Protection · p. 13

B. Cross-Architecture Performance

  • Cross-architecture performance reported in prior studies supports using one simple architecture for DD at the edges.The downstream network architecture is not tied to the architecture used for DD [12],, [15],,,.Source: B. Cross-Architecture Performance · p. 13

C. Coordination with Clock Frequency Scaling

  • Clock-frequency scaling can be incorporated by finding the best clock setup during offline energy calibration and applying it during online DD.This extension changes energy-model coefficients as functions of clock frequency.Source: C. Coordination with Clock Frequency… · p. 13

D. Application to Continual Learning

  • Continual learning is a potential use case, where its continuous data stream could help optimize CEDD-optimizer's accuracy-model calibration.The discussion notes prior studies of DD in continual learning –[26].Source: D. Application to Continual Learning · p. 13

E. Application to Federated Learning

  • Price-variation-aware optimization could extend to federated learning, but its differing edge computation and communication patterns require a modified optimization strategy.The paper focuses on centralized learning; integration with federated learning is beyond its scope.Source: E. Application to Federated Learning · p. 13

F. Handling Non-IID Data

  • CEDD-optimizer mainly targets IID and near-IID data; non-IID accuracy modeling may need weighted statistics, whose information weights remain difficult to quantify.Sampling data rather than using all local data is suggested to limit overhead when estimating those weights.Source: F. Handling Non-IID Data · p. 13

G. Temporal Variability of Electricity Prices

  • Time-varying electricity prices are excluded; forecast-based DD trigger timing is proposed as an additional optimization knob.The discussion links price variability to renewable-energy availability and power demand.Source: G. Temporal Variability of Electricity… · p. 13

VIII. CONCLUSIONS

  • CEDD-optimizer tunes DD hyperparameters across geographically distributed edge devices to maximize downstream accuracy while minimizing economic costs such as network traffic and electricity.The paper evaluates the framework across scenarios and parameter configurations for edge and edge-to-cloud deep-learning workflows.Source: §VIII · p. 13
Loading 2609.10151v1…