Source-linked AI summary
Federated Unsupervised Representation Learning
Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yueting Zhuang, Xiaolin Li
TL;DR
Federated Unsupervised Representation Learning seeks a common unsupervised representation model across private, heterogeneous client data, despite inconsistent and misaligned local representation spaces. FedCA combines contrastive local training with a shared representation dictionary and public-data alignment, and experiments show it outperforms baseline approaches in IID and Non-IID settings. The paper concludes that these modules provide a stronger baseline for FURL, while future work extends the method to cross-modal settings.
Problem
FURL must learn common representations from private unlabeled data despite client distribution shifts and cross-client representation misalignment.
Method
FedCA combines contrastive local training with a server-aggregated representation dictionary and an alignment model trained on a small public dataset.
Results
FedCA outperforms approaches that solely combine federated learning with unsupervised methods in IID and Non-IID evaluations.
Takeaways & Limitations
FedCA provides a stronger baseline for federated unsupervised representation learning while preserving private data on client devices.
Takeaways & Limitations
The paper leaves extension of FedCA to cross-modal clients with image, video, text, and audio data as future work.
Abstract
from arXiv · showhide
To leverage enormous unlabeled data on distributed edge devices, we formulate a new problem in federated learning called Federated Unsupervised Representation Learning (FURL) to learn a common representation model without supervision while preserving data privacy. FURL poses two new challenges: (1) data distribution shift (Non-IID distribution) among clients would make local models focus on different categories, leading to the inconsistency of representation spaces. (2) without the unified information among clients in FURL, the representations across clients would be misaligned. To address these challenges, we propose Federated Constrastive Averaging with dictionary and alignment (FedCA) algorithm. FedCA is composed of two key modules: (1) dictionary module to aggregate the representations of samples from each client and share with all clients for consistency of representation space and (2) alignment module to align the representation of each client on a base model trained on a public data. We adopt the contrastive loss for local model training. Through extensive experiments with three evaluation protocols in IID and Non-IID settings, we demonstrate that FedCA outperforms all baselines with significant margins.
Introduction
Federated unsupervised representation learning must handle limited, heterogeneous client data that can produce inconsistent and misaligned representation spaces. FedCA addresses these challenges with a shared representation dictionary, public-data alignment, and contrastive local training.
- FURL combines federated learning with unsupervised representation learning to use unlabeled client data while preserving privacy.
- Limited and distribution-shifted client data can make local models encode inconsistent representation spaces focused on different categories.For example, clients may separately observe cats and dogs versus cars and planes.
- Even with IID data, training randomness can misalign otherwise consistent representations across clients, harming the aggregated model.One model’s representations may correspond to another’s after a rotation.
- FedCA uses a server-maintained dictionary of client representations to address representation-space inconsistency and a public-data alignment model to align clients.Local models are trained with contrastive learning using both modules.
- FedCA outperforms naive methods that solely combine federated learning with unsupervised approaches.The authors present it as the first algorithm designed for FURL.
Related Work
Prior work provides strong unsupervised representation-learning methods and federated-learning foundations, but FedCA focuses specifically on representation consistency and alignment in federated unsupervised learning.
- Federated learning methods commonly prioritize shared parameter consistency, whereas this paper emphasizes consistency of learned representations.
- Unsupervised learning includes generative and discriminative approaches, with contrastive learning belonging to the discriminative family.
- Concurrent federated unsupervised-learning works primarily combine federated learning with unsupervised approaches without designing for the identified FURL challenges.
- FedCA introduces an algorithm that targets inconsistency and misalignment of representations in federated unsupervised representation learning.
Preliminary
The preliminary framework defines federated aggregation and unsupervised contrastive representation learning. FedAvg averages locally updated models, while contrastive learning separates positive and negative representations and uses the encoder outputs for downstream tasks.
- Federated Learning: Federated learning trains a shared model across private client datasets while keeping the data on devices.
- Federated Learning: FedAvg repeatedly selects clients, updates their model parameters locally with stochastic gradient descent, and aggregates them by weighted averaging.The process continues until the global model converges.
- Unsupervised Contrastive Learning: Unsupervised contrastive learning reduces distances between positive representations and increases distances between negative representations.
- Unsupervised Contrastive Learning: SimCLR creates two augmented views for each of N minibatch samples, treating same-image views as positive and different-image views as negative.
- Unsupervised Contrastive Learning: The encoder f extracts representation vectors h, the projection head g maps h to z, and downstream tasks use h.
Method
FedCA combines a dictionary module and an alignment module with contrastive learning to address inconsistent and misaligned representations in FURL. The dictionary aggregates client representations, while the alignment module regularizes local models toward a public-data base model.
- Challenges: FURL faces inconsistent representation spaces because clients may hold data from only a few classes, causing aggregated models to perform poorly.
- Dictionary Module: The dictionary module aggregates normalized client projections into a shared dictionary used as additional negative samples during local contrastive training.Clients send projections to the server, which gathers them into a shared dictionary; the resulting logits have size N × (N + K).
- Dictionary Module: Temporal ensembling maintains local projection histories to reduce inconsistencies between representations generated by successive global and local models.
- Alignment Module: Randomness can misalign representations even when clients encode consistent spaces, with vanilla local models showing angles beyond 20° on CIFAR-10.
- Alignment Module: The alignment module trains a model on a small public dataset and adds a regularization term that encourages local models to replicate its outputs.
- Alignment Module: Angles between aligned local representations can be controlled within 10° after 10 training rounds, supporting the module’s intended alignment effect.
- FedCA Algorithm: FedCA defines total local loss as contrastive loss plus β-scaled alignment loss and combines the dictionary and alignment modules in its federated algorithm.
Experiments
The paper presents an empirical study of FedCA for learning representations from decentralized and unlabeled data.
- The experiments empirically study FedCA for federated unsupervised representation learning from decentralized, unlabeled data.
Experimental Setup
The experiments use several unsupervised baselines and evaluate federated learning across IID and Non-IID settings with multiple datasets and encoders.
- Baselines: FedAE, FedPR, and FedSimCLR serve as baselines by combining FedAvg with AutoEncoder, Predicting Rotation, and SimCLR.
- Datasets: The datasets include CIFAR-10, CIFAR-100, and MiniImageNet, with 50,000 training and 10,000 test images for CIFAR datasets.
- Federated Setting: Experiments use one server and five clients, with five local epochs and all clients selected in each round.
- Federated Setting: Both IID and Non-IID federated settings are considered, with clients sampling data differently across the two conditions.
- Model Configuration: Baselines are evaluated with 5-layer CNN and ResNet-50 encoders, producing 2048-dimensional representations and 128-dimensional contrastive-learning vectors.
Evaluation Protocols and Results
The paper evaluates learned representations through linear, semi-supervised, and transfer-learning protocols across several datasets. FedCA outperforms the baseline methods, with particularly strong performance reported in Non-IID settings.
- Linear Evaluation: Linear evaluation fixes the encoder and trains a supervised linear classifier, reporting top-1 test accuracy on CIFAR-10, CIFAR-100, and MiniImageNet.
- Linear Evaluation: Federated averaging with contrastive learning performs better than other unsupervised approaches, while FedCA outperforms all evaluated baselines.
- Semi-Supervised Learning: Semi-supervised evaluation pretrains representations in FURL and fine-tunes them using either 1% or 10% labeled client data.
- Semi-Supervised Learning: FedCA outperforms other approaches in semi-supervised learning, especially in the Non-IID setting.
- Transfer Learning: Transfer learning initializes models with FURL representations and trains them with an MLP and encoder on other datasets.
- Transfer Learning: FedCA achieves excellent transfer-learning performance and outperforms all baseline methods in the Non-IID setting.
Ablation Study
The ablation study isolates alignment, dictionary, and temporal-ensembling components on CIFAR-10 under IID and Non-IID conditions. Each module improves performance, and their combination works collaboratively.
- Ablation Design: The study compares FedSimCLR with variants using alignment, dictionary, temporal-ensembled dictionary, and both FedCA modules.
- Module Effects: The alignment module improves accuracy by 1.4% in both IID and Non-IID settings.
- Module Effects: The dictionary module without temporal ensembling increases accuracy by 2.5% under IID and 2.7% under Non-IID conditions.
- Temporal Ensembling: Temporal ensembling contributes more in Non-IID than IID settings, where learned features are described as stable enough to reduce its importance.
- Combined Model: Combining alignment with the temporally ensembled dictionary produces excellent performance, indicating that the modules work collaboratively.
Conclusions
FedCA addresses FURL’s representation-space inconsistency and misalignment through dictionary and alignment modules, enabling consistent and aligned local models while protecting data privacy. Experiments show FedCA outperforms methods that solely combine federated learning with unsupervised approaches, while cross-modal extension remains future work.
- FedCA’s dictionary and alignment modules enable distributed local models to learn consistent and aligned representations while protecting data privacy.The dictionary module addresses representation-space inconsistency, while the alignment module addresses cross-client representation alignment.
- FedCA outperforms algorithms that solely combine federated learning with unsupervised approaches and provides a stronger baseline for FURL.
- Extending FedCA to cross-modal scenarios involving images, videos, texts, and audios is identified as future work.