Collaborative Unsupervised Visual Representation Learning from Decentralized Data

Weiming Zhuang; Xin Gan; Yonggang Wen; Shuai Zhang; Shuai Yi

Source-linked AI summary

Collaborative Unsupervised Visual Representation Learning from Decentralized Data

Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, Shuai Yi

arXiv:2108.06492v1Aug 14, 2021cs.DCcs.AIcs.CVcs.LG

TL;DR

The paper asks how decentralized unlabeled images can support visual representation learning without sharing private raw data. FedU uses federated contrastive learning with online-encoder aggregation and divergence-aware predictor updates for non-IID clients. It outperforms one-party training and other methods in reported non-IID evaluations.

Problem
Decentralized unlabeled image data cannot be centralized because of privacy constraints, while existing methods do not provide generic representation learning from such data without privacy risks.
Method
FedU trains contrastive Siamese networks at clients, aggregates online encoders, and uses DAPU to dynamically update predictors under non-IID data.
Results
FedU outperforms one-party training by over 5% and other methods by more than 14% in linear and semi-supervised evaluation on non-IID data.
Takeaways & Limitations
Federated unsupervised representation learning with encoder-only communication and adaptive predictor updates achieves superior performance across the reported evaluation protocols.
Takeaways & Limitations
The paper assumes parties learn a shared generic representation from unlabeled client datasets without sharing data among parties.

Abstract

from arXiv · show

Unsupervised representation learning has achieved outstanding performances using centralized data available on the Internet. However, the increasing awareness of privacy protection limits sharing of decentralized unlabeled image data that grows explosively in multiple parties (e.g., mobile phones and cameras). As such, a natural problem is how to leverage these data to learn visual representations for downstream tasks while preserving data privacy. To address this problem, we propose a novel federated unsupervised learning framework, FedU. In this framework, each party trains models from unlabeled data independently using contrastive learning with an online network and a target network. Then, a central server aggregates trained models and updates clients' models with the aggregated model. It preserves data privacy as each party only has access to its raw data. Decentralized data among multiple parties are normally non-independent and identically distributed (non-IID), leading to performance degradation. To tackle this challenge, we propose two simple but effective methods: 1) We design the communication protocol to upload only the encoders of online networks for server aggregation and update them with the aggregated encoder; 2) We introduce a new module to dynamically decide how to update predictors based on the divergence caused by non-IID. The predictor is the other component of the online network. Extensive experiments and ablations demonstrate the effectiveness and significance of FedU. It outperforms training with only one party by over 5% and other methods by over 14% in linear and semi-supervised evaluation on non-IID data.

1. Introduction

FedU addresses the overlooked problem of learning visual representations from decentralized unlabeled data while preserving privacy. It combines federated contrastive learning with encoder-only aggregation and divergence-aware predictor updates to handle non-IID data, outperforming one-party and competing methods in reported evaluations.

Motivation: Decentralized data are often non-IID, and existing unsupervised methods may learn poor representations when trained on only one party.Privacy constraints also prevent centralizing client data, while some prior federated approaches risk privacy leakage by sharing client features.
Contribution: FedU learns generic visual representations collaboratively from decentralized unlabeled data while preserving data privacy.The framework centralizes learned representations rather than raw data.
Results: FedU outperforms one-party training by over 5% and other methods by more than 14% under linear and semi-supervised evaluation on non-IID data.Experiments on CIFAR datasets also report superior performance across linear, semi-supervised, and transfer-learning protocols.
Method: FedU aggregates and updates only online encoders, while retaining target encoders to preserve locally representative regression targets.This communication design follows analysis of Siamese networks under non-IID data.
Method: DAPU dynamically determines predictor updates according to the divergence caused by non-IID data.It is introduced as the second method for addressing decentralized-data heterogeneity.

2. Related Work

Unsupervised representation learning includes generative, discriminative, and contrastive approaches, while federated learning coordinates decentralized clients but struggles with non-IID data. FedU addresses unlabeled, non-IID decentralized learning using encoder aggregation and divergence-aware predictor updates.

2.1. Unsupervised Representation Learning: Contrastive learning is presented as the state-of-the-art discriminative approach, using positive and negative pairs or, in BYOL and SimSiam, only positive pairs.Negative pairs may come from memory banks such as MoCo or large batches such as SimCLR.
2.1. Unsupervised Representation Learning: Generative methods reconstruct or generate pixels, whereas discriminative methods learn representations through proxy tasks such as in-painting and jigsaw puzzles.
2.2. Federated Learning: Non-IID federated data causes weight divergence and performance degradation, motivating methods such as public-data sharing, distillation, and client regularization.These approaches were investigated under supervised learning and are not directly applicable when data is unlabeled.
2.2. Federated Learning: Existing unlabeled federated-learning studies focus mainly on specific applications, bypass non-IID issues, or introduce potential privacy risks.FedCA addresses non-IID data by collecting client features and distributions, which can compromise privacy.
2.2. Federated Learning: FedU aggregates clients’ online encoders and updates predictors with divergence-aware rules to address non-IID decentralized unlabeled learning.The framework coordinates clients through a central server while retaining the proposed encoder-aggregation and predictor-update mechanisms.

3. Methodology

FedU learns visual representations from decentralized unlabeled data while preserving privacy by aggregating online encoders rather than raw data. It addresses non-IID data through online-encoder communication and divergence-aware predictor updates within a contrastive Siamese framework.

Problem and framework: FedU coordinates clients with unlabeled data to learn a generic representation without centralizing raw data, addressing privacy constraints and non-IID data.Each client retains its raw data while the server coordinates model exchange.
FedU overview: The server aggregates client online encoders using data-volume weighting, then distributes the global encoder and predictor for client-side model updates.Clients upload online encoders after local training, and the server coordinates aggregation and redistribution each round.
Local training: Each client performs contrastive learning with an online encoder-predictor network and a target encoder that supplies regression targets through exponential moving average updates.The target encoder is updated with the online encoder’s parameters using EMA rather than gradient descent.
Communication Protocol: FedU uploads clients’ online encoders for aggregation and updates only those online encoders with the global encoder, while retaining target encoders for stable local regression targets.The online encoder reflects the latest local representation, whereas the target encoder captures historical representations.
Divergence-aware Predictor Update: DAPU updates predictors locally when encoder divergence is large and globally when divergence is small, balancing local specificity against generalizability under non-IID data.The divergence measure compares local online-encoder parameters with the previous global encoder, using a controllable threshold.

4. Experimental Evaluation

FedU is evaluated on CIFAR datasets through linear, semi-supervised, and transfer-learning protocols under federated settings. It outperforms other methods across these evaluations, with particularly large gains over prior federated approaches in non-IID settings.

Experimental Setup: The evaluation uses CIFAR-10 and CIFAR-100 for linear and semi-supervised learning, and Mini-ImageNet for transfer learning.The study uses multiple backbones and federated settings, with ResNet-18 and ResNet-50 encoders.
Linear Evaluation: FedU outperforms other methods in linear evaluation, including supervised FedAvg on non-IID CIFAR-10.Table 1 compares top-1 accuracy across IID and non-IID CIFAR settings.
Semi-Supervised Learning: FedU exceeds FedCA by more than 22% on IID CIFAR-10 and more than 40% on non-IID CIFAR-10 in semi-supervised evaluation.The method also consistently outperforms single-client training by around 3% across settings.
Transfer Learning: FedU outperforms other baseline methods when representations trained on Mini-ImageNet are transferred to CIFAR datasets.The advantage is much stronger on CIFAR-100, while the gain on CIFAR-10 is slight because random initialization already performs relatively well.

5. Ablation Study

Ablations support aggregating and updating online encoders and dynamically updating predictors with DAPU. Performance also depends on the predictor threshold, local-epoch trade-off, and total training rounds.

Communication Protocol: Aggregating and updating the online encoder achieves the best performance among encoder-update choices, while DAPU further improves representation clustering.Updating only the target encoder performs poorly because it supplies regression targets for the online encoder; updating both encoders also limits local adaptation.
Divergence-aware Predictor Update: DAPU outperforms always updating with either the local or global predictor by around 5% under non-IID CIFAR settings.Its representations also cluster better than those produced by always using the local predictor.
Divergence-aware Predictor Update: FedU reaches its best CIFAR-10 performance at threshold µ = 0.2.The encoder divergence decreases as training proceeds, motivating an optimal threshold.
Hyperparameter Analysis: Increasing local epochs decreases accuracy while reducing communication costs, so E = 1 is suggested when bandwidth is not bounded.The total client computation is fixed while the number of local epochs varies.
Hyperparameter Analysis: Increasing total training rounds improves performance, especially when the number of rounds is small.FedU already outperforms other methods with R = 100, although additional rounds can improve accuracy further.

6. Conclusion

FedU learns generic visual representations from decentralized unlabeled data while preserving privacy through online-encoder aggregation and DAPU. The framework outperforms existing methods across the evaluated protocols, with larger-scale datasets and application scenarios left for future work.

Conclusion: FedU learns generic representations from multiple parties using decentralized unlabeled data while preserving data privacy.Its two main methods aggregate and update online encoders and dynamically update predictors with DAPU.
Conclusion: FedU outperforms existing methods on all evaluation protocols examined in the paper.The authors identify larger-scale datasets and specific application scenarios as future directions.