Source-linked AI summary
A generic framework for privacy preserving deep learning
Theo Ryffel, Andrew Trask, Morten Dahl, Bobby Wagner, Jason Mancuso, Daniel Rueckert, Jonathan Passerat-Palmbach
TL;DR
Privacy-preserving deep learning lacks a general framework that combines secure processing with a familiar deep learning interface. The paper introduces a PyTorch-based framework using tensor chains to support federated learning, MPC, and DP, and reports early results showing privacy-related accuracy trade-offs alongside substantial runtime overhead.
Problem
Securely trained models can remain vulnerable to reverse-engineering attacks, while privacy-preserving methods need a usable framework for deep learning.
Method
The framework uses chains of specialized tensors to expose federated learning, MPC, and DP through a transparent PyTorch interface.
Results
The framework supports SPDZ and moment-accountant methods in federated learning, while DP achieves 25-30 MSE versus 20-24 for the baseline with (0.5, 10^-5)-differential privacy.
Takeaways & Limitations
The framework combines MPC and DP functionality with PyTorch-compatible tensor commands for privacy-preserving federated learning.
Takeaways & Limitations
The current implementation has substantial training-time overhead and does not yet detect or defeat malicious attempts to corrupt data or models.
Abstract
from arXiv · showhide
We detail a new framework for privacy preserving deep learning and discuss its assets. The framework puts a premium on ownership and secure processing of data and introduces a valuable representation based on chains of commands and tensors. This abstraction allows one to implement complex privacy preserving constructs such as Federated Learning, Secure Multiparty Computation, and Differential Privacy while still exposing a familiar deep learning API to the end-user. We report early results on the Boston Housing and Pima Indian Diabetes datasets. While the privacy features apart from Differential Privacy do not impact the prediction accuracy, the current implementation of the framework introduces a significant overhead in performance, which will be addressed at a later stage of the development. We believe this work is an important milestone introducing the first reliable, general framework for privacy preserving deep learning.
1 Introduction
The paper introduces a transparent PyTorch framework for privacy-preserving deep learning, combining federated learning, multiparty computation, and differential privacy through standardized worker communication and tensor chains.
- The framework exposes federated learning, MPC, and DP through an intuitive interface for PyTorch users.
- A standardized worker-communication protocol makes federated learning possible within the framework.
- The chain abstraction on tensors overrides or encodes operations such as sending and sharing tensors between workers.
- The framework provides elements for implementing differential-privacy and multiparty-computation protocols.
- Its extensible design allows external contributors to add new federated-learning, MPC, or DP methods.
2 A standardized framework to abstract operations on Tensors
The framework represents tensor states and transformations as chains of specialized SyftTensors while preserving the native PyTorch interface and supporting local, remote, and debugging workers.
- Tensor-chain abstraction: SyftTensors represent data states or transformations and can be chained through child and parent attributes.
- Tensor-chain abstraction: Operations are first applied to the Torch tensor, then forwarded through the chain, preserving the native Torch interface.
- LocalTensor: The LocalTensor performs the native operation on the head tensor, while its looped two-node chain avoids recreating a child tensor object.
- PointerTensor: Sending a tensor creates a two-node chain containing an empty tensor and a PointerTensor identifying data ownership and remote storage.
- Workers: Virtual Workers replicate command chains locally for debugging without network communication, while network workers use sockets or WebSockets.
3 Towards a Secure MPC framework
The framework builds MPC and DP capabilities around chainable tensor abstractions, with federated workers, SPDZ-based MPC, and differentially private training components.
- MPC implementation: SPDZ-based MPC uses PointerTensors to distribute secret shares and MPCTensors to support operations including multiplication and neural-network matrix multiplication.FixedPrecisionTensor converts floating-point values into fixed-precision integers because SPDZ assumes integer data.
- MPC implementation: The framework assigns model ownership to a local worker that controls remote workers, while allowing remote shared tensors for data the owner cannot see.This unequal-player design differs from the cited MPC protocol, whose players are equal.
- MPC limitations: The current implementation does not ensure that every player behaves honestly, with MAC authentication proposed as a future improvement.The limitation concerns malicious behavior during secure multiparty computation.
- Differential privacy: Differential privacy is implemented using a privacy accountant and a sanitizer that clips gradients and adds Gaussian noise during stochastic training.In the federated setting, gradients are sanitized on remote workers before secured gradients reach the local worker.
- Differential privacy: The framework also describes worker-local sampling and an unpublished teacher-student differential privacy approach planned for integration.The teacher-student method trains a student model from noisy, aggregated votes of pre-trained teacher models.
4 Results and discussion
Experiments on Boston Housing and Pima Diabetes show substantial framework overhead, while differential privacy trades accuracy for stronger privacy and slows convergence.
- Execution performance: 46 times slower than regular PyTorch, the framework shows the same performance overhead on Boston Housing and Pima Indian Diabetes experiments.The comparison uses training-time measurements for neural-network and classifier experiments.
- Differential privacy results: 25-30 MSE versus 20-24 in the baseline model, the differentially private model retains strong (0.5, 10−5)-differential privacy.Increasing ϵ improves the model at the expense of data privacy.
- Differential privacy results: +50% overhead, Boston Housing training takes approximately 30.0ms per batch with differential privacy versus 19.8ms for the baseline.The paper describes this overhead as reasonable for adding privacy, while noting that convergence is much slower with DP enabled.
5 Conclusions
The paper presents a PyTorch-based privacy-preserving federated learning framework that combines MPC and DP through exchanged tensor chains, while identifying training efficiency and malicious MPC behavior as unresolved issues.
- Contributions: The framework combines PyTorch API commands with MPC and DP functionalities in tensor implementations exchanged between local and remote workers.Its design relies on chains of tensors for privacy-preserving federated learning.
- Limitations: Decreasing training time remains the foremost issue because the current pure Python implementation introduces substantial overhead.The paper attributes room for improvement to reliance on pure Python rather than optimized low-level libraries.
- Limitations: The paper also identifies securing MPC against malicious attempts to corrupt data or the model as a remaining concern.This concern complements the implementation’s lack of a mechanism ensuring honest player behavior.