PaperlayerView exact revision on arXiv

Source-linked AI summary

FedPS: Federated data Preprocessing via aggregated Statistics

Xuefeng Xu, Graham Cormode

arXiv:2602.10870v1cs.LGcs.AI

TL;DR

Federated learning research often assumes data is already prepared, although privacy, communication, and heterogeneity make centralized or uncoordinated preprocessing impractical. FedPS uses aggregated statistics and data sketches to provide federated preprocessing methods and models, and experiments indicate improved accuracy over local preprocessing and raw-data baselines. Privacy-preserving preprocessing for complex statistics remains future work.

  • Problem

    Federated learning often assumes preprocessed data, despite privacy, communication, and heterogeneity challenges that complicate consistent preparation.

  • Method

    FedPS combines aggregated-statistics workflows, data sketches, preprocessing methods, and federated models for horizontal and vertical settings.

  • Results

    Federated preprocessing significantly surpasses local preprocessing and raw-data baselines, particularly in heterogeneous data contexts.

  • Takeaways & Limitations

    FedPS offers a systematic and flexible framework connecting data preparation with federated model training.

  • Takeaways & Limitations

    Privacy-preserving preprocessing for complex statistics such as quantiles is left for future work because of latency, privacy, accuracy, and communication trade-offs.

Abstract

from arXiv · show

Federated Learning (FL) enables multiple parties to collaboratively train machine learning models without sharing raw data. However, before training, data must be preprocessed to address missing values, inconsistent formats, and heterogeneous feature scales. This preprocessing stage is critical for model performance but is largely overlooked in FL research. In practical FL systems, privacy constraints prohibit centralizing raw data, while communication efficiency introduces further challenges for distributed preprocessing. We introduce FedPS, a unified framework for federated data preprocessing based on aggregated statistics. FedPS leverages data-sketching techniques to efficiently summarize local datasets while preserving essential statistical information. Building on these summaries, we design federated algorithms for feature scaling, encoding, discretization, and missing-value imputation, and extend preprocessing-related models such as k-Means, k-Nearest Neighbors, and Bayesian Linear Regression to both horizontal and vertical FL settings. FedPS provides flexible, communication-efficient, and consistent preprocessing pipelines for practical FL deployments.

1 Introduction

Federated learning depends on preprocessing, yet decentralized privacy, communication, and heterogeneous data make consistent preparation difficult. FedPS addresses this gap with aggregated-statistics workflows, data sketches, and federated preprocessing methods and models.

  • Motivation: Preprocessing is vital for cleaning, structuring, and analyzing data, but remains largely neglected in federated learning despite its importance to model performance.Common tasks include missing-value handling, feature scaling, and categorical encoding.
  • Challenges: Privacy constraints, limited communication, and client heterogeneity complicate the design of consistent federated preprocessing pipelines.Centralizing raw data violates federated learning’s privacy constraints, while heterogeneous distributions make coordination difficult.
  • Challenges: Local preprocessing preserves privacy but can distort global structure under non-IID data, including making previously separable classes non-separable after independent standardization.Experiments reportedly show that inconsistent local preprocessing may perform worse than using raw data.
  • FedPS Framework: FedPS provides a unified framework that maintains cross-client consistency through summarization, aggregation, and parameter distribution.Its workflow is paired with preprocessing methods spanning scaling, encoding, transformation, discretization, and imputation.
  • FedPS Framework: Data-sketching techniques enable communication-efficient computation of complex global statistics, while FedPS analyzes sufficient statistics and communication costs for scalable deployment.The framework also develops federated Bayesian linear regression for horizontal and vertical settings.
  • Results: Empirical results across various datasets indicate that federated preprocessing surpasses local preprocessing and raw-data baselines, particularly with heterogeneous data.The paper presents this pattern as evidence of federated preprocessing’s efficacy.

2 Preliminaries

The preliminaries define core preprocessing and federated learning concepts, emphasizing distributed statistics, data heterogeneity, sketches, and Bayesian linear regression.

  • 2.1 Data Preprocessing: Data preprocessing prepares raw data for modeling through scaling, encoding, discretization, imputation, and other transformations.These operations may target individual tabular features or multiple columns.
  • 2.2 Federated Learning: Federated learning lets clients collaboratively train models without sharing raw data, but communication cost and heterogeneous data distributions complicate coordination.FedAvg reduces communication rounds through multiple local updates before aggregation.
  • 2.2 Federated Learning: Horizontal FL partitions examples across clients with a shared feature space, whereas vertical FL partitions feature spaces across clients sharing identifiers.The paper mainly focuses on horizontal FL and adds vertical extensions when needed.
  • 2.3 Aggregated Statistics: Clients can aggregate basic statistics exactly, while quantiles and frequent items require compact sketches for practical communication costs.KLL provides additive error guarantees, REQ provides multiplicative error guarantees, and frequent-item sketches approximate common-item counts.
  • 2.4 Bayesian Linear Regression: Bayesian linear regression places a prior over parameters and produces a Gaussian posterior for missing-value imputation.Its posterior is characterized by mean ˆω and covariance Σ under Gaussian noise.
  • 2.4 Bayesian Linear Regression: The inverse covariance matrix can be computed through the singular value decomposition of X, avoiding explicit inversion of a large dense matrix.Using Λ = S2 yields Σ−1 = V(αI + βΛ)−1V⊤.

3 Federated Data Preprocessing

FedPS preprocesses federated data by aggregating client statistics, deriving global parameters, and broadcasting them for local application across diverse preprocessing methods.

  • 3.1 The Framework: FedPS organizes federated preprocessing into local statistic computation, server aggregation, parameter derivation, broadcasting, and local transformation.This workflow is paired with a suite of preprocessing methods.
  • 3.1 The Framework: The framework covers scaling, encoding, transformation, discretization, and imputation through methods compatible with widely used preprocessing libraries.Its examples span simple aggregations, sketches, and iterative model-based procedures.
  • 3.1 The Framework: StandardScaler aggregates sums, squared sums, and sample counts to compute global mean and variance, then applies the resulting transformation locally.The procedure requires one data pass and one communication round.
  • 3.1 The Framework: FedPS supports uniform, quantile-based, and clustering-based discretization, plus federated KNNImputer and IterativeImputer implementations.KNNImputer aggregates candidate neighbor distances, while IterativeImputer uses federated Bayesian linear regression across iterative feature updates.
  • 3.1 The Framework: Aggregated basic statistics remain exact despite client distribution shifts, while quantile and frequent-item sketches retain theoretical guarantees under heterogeneity.This makes the statistics suitable for heterogeneous federated environments.
  • 3.2 Communication Overhead Analysis: Communication overhead depends on the sufficient statistics, aggregation frequency, and whether preprocessing uses sketches, iterations, or pairwise sample interactions.The analysis compares communication rounds and per-client costs for different preprocessors.
  • 3.2 Communication Overhead Analysis: Simple statistics require O(m) communication per client, whereas normalizer, sketch-based, iterative, clustering, and neighbor methods incur costs tied to their operations.Normalizer requires O(n) total cost; more complex procedures have higher overhead from summaries or sample interactions.
  • 3.2 Communication Overhead Analysis: Encoding and quantile methods use category-dependent or sketch-dependent communication, with TargetEncoder costing O(d) and frequent-item sketches costing O(k log d) per feature.KLL and REQ sketches provide alternative quantile communication guarantees.

4 Federated Bayesian Linear Regression

FedPS extends Bayesian Linear Regression preprocessing to horizontal and vertical federated settings by aggregating privacy-preserving sufficient statistics. The formulations preserve centralized posterior means while controlling communication through additive statistics and iterative scalar or vector exchanges.

  • Horizontal Federated BLR: Horizontal federated BLR aggregates clients’ local X(c)⊤Y(c) and X(c)⊤X(c) contributions to obtain exact sufficient statistics.These statistics decompose additively across clients holding complete feature vectors for partitioned samples.
  • Horizontal Federated BLR: The horizontal procedure is equivalent to centralized BLR and avoids iterative FedAvg-style averaging because aggregation produces exact sufficient statistics.Only scalar errors are communicated during iterative refinement.
  • Horizontal Federated BLR: Theorem 4.1 gives horizontal federated BLR a per-client communication cost of O(m min(n, m)).Local eigenvalue decomposition reduces the initial X(c)⊤X(c) communication cost from O(m^2).
  • Vertical Federated BLR: Vertical federated BLR replaces X⊤X with the additively decomposable XX⊤ statistic, eliminating cross-client feature products.This reformulation addresses the off-diagonal interactions required by standard BLR when features are partitioned across clients.
  • Vertical Federated BLR: Theorem 4.2 shows that the reformulated vertical expression computes the same posterior mean as standard BLR.The equivalence is established using the Woodbury matrix identity.
  • Vertical Federated BLR: Theorem 4.3 gives vertical federated BLR a per-client communication cost of O(n min(n, m) + nt).The initial sufficient-statistic aggregation dominates, while iterative refinement contributes O(nt) communication across t iterations.

5 Empirical Evaluation

The evaluation compares no, local, and federated preprocessing under IID and non-IID partitions using tabular datasets and two representative preprocessing methods. Preprocessing improves performance in both settings, while federated preprocessing is especially beneficial when client distributions differ.

  • Experiment Setup: The experiments evaluate no preprocessing, local preprocessing, and federated preprocessing under IID and non-IID data partitions.Centralized preprocessing is excluded as infeasible for realistic federated learning, while transfer preprocessing is unsuitable when client distributions differ.
  • Experiment Setup: The study focuses on OrdinalEncoder and StandardScaler with FedAvg-trained Logistic Regression and a two-hidden-layer MLP.Each experiment runs for 100 communication rounds.
  • IID Data Partitioning: 17% accuracy improvement is observed on the Cover dataset with the MLP model under IID partitioning.Adult improves by 5% for Logistic Regression and 12% for the MLP, while Bank improves by around 1 to 2%.
  • IID Data Partitioning: Under IID partitioning, local and federated preprocessing behave similarly because local statistics align closely with global statistics.Preprocessing consistently improves model performance across all evaluated datasets and models.
  • Non-IID Data Partitioning: Under non-IID partitioning, preprocessing substantially outperforms no preprocessing, especially on the Adult and Cover datasets.Non-IID clients receive heterogeneous label distributions sampled using a Dirichlet distribution with α = 0.5.
  • Non-IID Data Partitioning: 11% lower accuracy is observed for local preprocessing than federated preprocessing on Cover with the MLP model under non-IID partitioning.For Logistic Regression on Cover, the gap is 8%, and local preprocessing performs worse than no preprocessing.
  • Communication Cost: Approximately 0.6 KB per client is incurred by StandardScaler on Adult, confirming low communication overhead for this method.Measured communication costs align with the theoretical analysis and expose trade-offs between preprocessing complexity and efficiency.

6 Discussion

FedPS broadens federated preprocessing beyond existing frameworks’ limited sets of simple aggregation methods and explicitly addresses communication flexibility. Privacy-preserving preprocessing remains an open issue because complex and iterative statistics create latency, privacy, accuracy, and communication trade-offs.

  • Existing Frameworks: Existing federated preprocessing frameworks support only a small set of techniques, mainly based on simple aggregation.Examples include MinMaxScaler using global minima and maxima and StandardScaler using global means and variances.
  • FedPS Compared with Prior Work: FedPS supports a broader range of preprocessing techniques with configurable parameters and explicit communication-overhead analysis.Its OneHotEncoder handling restricts category counts and filters low-cardinality items using frequent-item sketches.
  • Privacy: Privacy-preserving preprocessing is left for future work because complex statistics and iterative procedures require careful trade-offs among latency, privacy, accuracy, and communication cost.The discussion specifically identifies quantiles and intermediate iterative results as privacy-design challenges.

7 Conclusion

FedPS combines aggregated statistics, data sketches, and federated models into a unified preprocessing suite. The paper concludes that proper preprocessing improves accuracy, whereas inconsistent local preprocessing can reduce performance under non-IID data.

  • Conclusion: FedPS combines aggregated statistics, data sketches, and federated models in a unified suite for federated data preprocessing.The framework is presented as a bridge between data preparation and model training.
  • Conclusion: Experiments show that proper preprocessing substantially improves model accuracy, while inconsistent local preprocessing can reduce performance under non-IID data.The conclusion positions FedPS as a systematic and flexible framework for federated preprocessing.

A.1 Federated k-Means

Federated k-Means lets clients cluster locally while sharing only cluster-level sums and counts, which the server aggregates to update global centroids. The process repeats until convergence or a maximum iteration limit.

  • A.1 Federated k-Means: Federated k-Means assigns each data point to its closest centroid and updates each centroid using the mean of its assigned points.The centroid update is µj = sum of points in cluster Sj divided by nj, the cluster size.
  • A.1 Federated k-Means: In horizontal federated k-Means, the server broadcasts centroids while clients compute local cluster assignments, sums, and counts.Clients keep their data local and return aggregate statistics for each cluster.
  • A.1 Federated k-Means: The server aggregates client-level cluster sums and counts to update the global centroids.This aggregation replaces centralized access to all individual data points.
  • A.1 Federated k-Means: The client-server assignment and aggregation cycle continues until convergence or the maximum number of iterations is reached.The algorithm outputs the resulting centroids after the stopping condition is met.

A.2 Federated Nearest Neighbors Regression

Federated k-NN regression finds global nearest neighbors without centralizing all client data. Horizontal federation exchanges local nearest-distance candidates, while vertical federation aggregates partial distances across feature-holding clients.

  • A.2 Federated Nearest Neighbors Regression: k-NN regression predicts a target by averaging the target values of the k closest samples, optionally using distance-based weights.The method typically measures closeness with Euclidean distance.
  • A.2 Federated Nearest Neighbors Regression: In horizontal federation, each client sends its local k smallest distances, and the server merges them to identify the global top k neighbors.The server then requests the corresponding target values from the relevant clients.
  • A.2 Federated Nearest Neighbors Regression: In vertical federation, clients compute partial distance contributions for shared samples, which the server aggregates into full distances.The server selects the global nearest neighbors and sends their indices to the client responsible for prediction.
  • A.2 Federated Nearest Neighbors Regression: The client holding the selected targets computes their arithmetic or weighted mean to produce the prediction.The resulting mean is returned as the predicted value µ.

B Federated Data Preprocessors

FedPS supports federated preprocessing through global statistics and sketches across scaling, encoding, transformation, discretization, and imputation. It also covers federated implementations of preprocessing-related operations and compares support, accuracy, and communication costs.

  • Scaling: Scaling preprocessors derive transformations from global extrema, quantiles, norms, or other aggregated statistics.Examples include MinMaxScaler, RobustScaler, MaxAbsScaler, and Normalizer in horizontal or vertical federation.
  • Encoding: Categorical encoders require the global union of categories, while frequent-item sketches can control infrequent or output categories.Label encoders typically process one column, whereas OneHotEncoder and OrdinalEncoder often process multiple columns.
  • Transformations: PowerTransformer estimates its parameter from global sums and variances, then applies StandardScaler to obtain zero mean and unit variance.QuantileTransformer instead uses global quantiles from a quantile sketch to map data toward a uniform or Gaussian distribution.
  • Discretization: Discretization converts continuous variables into categories, while Binarizer applies a fixed threshold without federated computation.SplineTransformer chooses knot positions using global minima and maxima or global quantiles.
  • Imputation: SimpleImputer replaces missing values with means, medians, or most-frequent values using global sums and counts, quantile sketches, or frequent-item sketches.The required federated statistic depends on the selected imputation strategy.
  • Evaluation and coverage: Table 4 compares FedAvg test accuracy for Logistic Regression and MLP across IID and non-IID settings with different preprocessing options.Tables 3, 5, and 6 respectively summarize dataset statistics, communication cost per client, and preprocessor support across frameworks and FedPS.
Loading 2602.10870v1…