Reconstructing Training Data from Trained Neural Networks

Niv Haim; Gal Vardi; Gilad Yehudai; Ohad Shamir; Michal Irani

Source-linked AI summary

Reconstructing Training Data from Trained Neural Networks

Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, Michal Irani

arXiv:2206.07758v3Jun 15, 2022cs.LGcs.CRcs.CVcs.NEstat.ML

TL;DR

The paper asks whether trained neural-network classifiers encode specific training samples and develops a reconstruction scheme based on implicit-bias theory. It shows that significant portions of training data can sometimes be reconstructed from classifier parameters alone, raising privacy concerns, while experiments focus on binary MLPs and small-scale settings.

Problem
The paper asks whether specific training samples are encoded in trained classifier parameters and can be recovered from network parameters, a question with practical and theoretical importance.
Method
The paper uses a reconstruction scheme derived from theoretical results on implicit bias in gradient-based neural-network training, finding a dataset that satisfies equations determined by trained parameters.
Results
The paper shows that significant portions of actual training data can sometimes be reconstructed from trained classifier parameters, including visually similar reconstructions on MNIST and CIFAR10.
Takeaways & Limitations
The results provide a proof of concept that training-data reconstruction can act as a privacy attack capable of revealing sensitive data.
Takeaways & Limitations
Experiments use relatively small-scale models, while convolutional networks, larger models and datasets, and other data types remain future directions; convergence and solution uniqueness are also unresolved.

Abstract

from arXiv · show

Understanding to what extent neural networks memorize training data is an intriguing question with practical and theoretical implications. In this paper we show that in some cases a significant fraction of the training data can in fact be reconstructed from the parameters of a trained neural network classifier. We propose a novel reconstruction scheme that stems from recent theoretical results about the implicit bias in training neural networks with gradient-based methods. To the best of our knowledge, our results are the first to show that reconstructing a large portion of the actual training samples from a trained neural network classifier is generally possible. This has negative implications on privacy, as it can be used as an attack for revealing sensitive training data. We demonstrate our method for binary MLP classifiers on a few standard computer vision datasets.

1 Introduction

The paper asks whether specific training samples are encoded in trained classifier parameters and presents a scheme that reconstructs significant portions of training data, with privacy implications.

The paper investigates whether specific training samples are encoded in trained classifier parameters and can be recovered from them.
The proposed scheme reconstructs significant portions of training data from trained network parameters alone, without additional information about the data.
The approach reverses the usual learning direction by finding training data given a trained model, demonstrated on CIFAR10 images from a binary classifier.
The reconstruction scheme has potential privacy consequences because an adversary might recover sensitive training data from a trained model.
The method is presented as the first work to show that actual training samples can be reconstructed from a trained neural network classifier.

2 Related Work

Related work analyzes learned representations, reconstructs class representatives, and studies implicit bias, but does not demonstrate exact training-sample reconstruction from classifiers.

Model visualization commonly optimizes inputs to maximize class outputs or intermediate activations, often using priors because unconstrained optimization produces noise.
Model-inversion privacy attacks reconstruct class representatives rather than actual training-set samples, while membership inference requires guessing a specific input.
The paper’s approach is based on theoretical characterizations of implicit bias in gradient-based training of homogeneous neural networks.

3 Background and Reconstruction Scheme

The reconstruction scheme uses implicit-bias results for homogeneous neural networks to recover training samples by optimizing conditions relating trained parameters to data-point gradients. It exploits overparameterization and can recover margin samples, while accounting for unknown margins and labels.

Implicit Bias: Gradient flow with logistic loss converges directionally to a KKT point for homogeneous ReLU networks under stated assumptions.The assumptions include reaching a time at which the empirical loss is below 1.
Implicit Bias: At a KKT point, the trained parameters are a linear combination of network derivatives evaluated at training samples.Only samples on the margin can contribute because non-margin samples have zero coefficients.
Dataset Reconstruction: The scheme reconstructs unknown inputs and auxiliary coefficients by minimizing losses encoding stationarity, dual feasibility, and optional data priors.The trained parameters and assigned labels are given, while the inputs and coefficients are optimized with a nonconvex method such as SGD.
Dataset Reconstruction: Using m ≥ 2n candidate samples with half positive and half negative labels permits a solution without knowing the training set’s label distribution.Extra points with zero coefficients or duplicated points can complete a zero-loss solution.
Dataset Reconstruction: The practical formulation avoids fixing the margin to 1 because finite training yields an unknown margin, while the directional stationarity conditions remain scale-invariant.This allows reconstruction to use conditions that hold for parameters pointing in the limiting direction.
Dataset Reconstruction: Overparameterization provides a heuristic for recoverability: Eq. (2) supplies p equations for O(nd) unknowns when p > nd.The method should primarily recover training samples lying on the margin, because those samples determine the relevant parameter relation.

4 A Simple Experiment in Two Dimensions

A toy two-dimensional experiment trains a wide classifier on labeled points arranged on a unit circle, then optimizes the reconstruction loss from random candidate points. The procedure recovers the input samples up to noise and improves after filtering weakly weighted and duplicate candidates.

Setup: The toy dataset contains 20 alternating-label training samples arranged on the unit circle in R2.A three-layer model with 1000 neurons per layer correctly classifies the training set.
Setup: The trained model’s output over the input plane is visualized with blue and red regions representing smaller and larger outputs.This visualization provides the classifier landscape used alongside the dataset and candidate reconstructions.
Reconstruction: Starting from 100 random candidate points split evenly between classes, optimizing Eq. (8) reconstructs all input samples up to noise.The reconstruction uses no prior knowledge of the data, with L_prior ≡ 0.
Post-processing: Filtering candidates with λ_i < 5 and greedily removing near-duplicates improves the reconstruction by discarding weakly weighted and redundant points.Points with λ_i = 0 do not affect the network parameters, so their inputs can take arbitrary values.

5 Results

Experiments on binary MLP classifiers show that reconstruction quality depends on training setup and sample position, while recovering actual training images from model parameters is feasible. The method produces similar images on MNIST and CIFAR10, though some reconstructions remain noisy or fail.

The top 45 CIFAR10 reconstructions are paired with their corresponding nearest neighbors from the model’s training set.
Reconstruction experiments use balanced binary MNIST odd/even and CIFAR10 vehicle/animal classification datasets.
Small initializations and longer training improve reconstruction quality, and the approach can also reconstruct data from models trained with mini-batch SGD.
Good reconstructions, such as SSIM > 0.4, occur for samples near the model’s margin, whereas increasing training size makes reconstruction more difficult.
High-quality reconstructions remain possible for a 3-layer non-homogeneous MLP with biases in all hidden layers.
Model inversion reconstructed 7 of 20 samples in the 2D experiment, while the proposed scheme reconstructed all 20.
For MNIST and CIFAR10, only 3 or 4 first-layer weights resembled training samples, compared with the proposed scheme’s reconstruction of dozens.

6 Discussion and Conclusion

The paper argues that trained-network parameters can contain enough information to reconstruct substantial amounts of training data, with implicit-bias theory providing an analytic explanation. Its scope remains limited by small-scale experiments, challenging convolutional optimization, and unresolved theoretical questions.

The parameters of trained networks may contain enough information to fully reconstruct training samples and a substantial amount of the training set.
The experiments use relatively small-scale models, while convolutional-network optimization remains challenging and is left for future research.
Implicit-bias theory provides an analytic explanation for why training samples can be reconstructed from trained-network parameters.
Larger models and datasets, tabular or textual data, and extensions to multi-class classifiers are identified as future directions.
It remains unclear why the reconstruction optimization converges to actual training samples when solutions may not be unique.

Checklist

The checklist records affirmative responses on contributions, limitations, societal impacts, theoretical assumptions, reproducibility, training details, and asset attribution, while describing established privacy attacks and reconstruction settings.

The paper states that its main claims accurately reflect its contributions and scope, and that it discusses limitations and potential negative societal impacts.
The theoretical results state their full assumptions, and complete proofs are not required because the work relies on known theoretical results.
The experiments provide code, data, instructions, training details, and compute information needed to reproduce the reported results.
The checklist notes that error bars were not applicable and that privacy attacks, including membership inference and model extraction, are reviewed.
The work uses publicly available assets and reports creator citations and asset licenses; human-subject and participant procedures are marked not applicable.
A typical run reconstructs m = 1000 samples in about 30 minutes on a Tesla V-100 32GB GPU for 100,000 epochs.
The reconstruction scheme uses a learning rate, σx initialization scale, backward-pass ReLU smoothing, and λmin to encourage samples toward a margin.

B.3 Post-Processing of Reconstructed Samples

Reconstructed samples are post-processed by scaling, nearest-neighbour matching, voting, and SSIM-based sorting to pair outputs with training samples.

Each reconstructed sample is linearly transformed to fit within the range [0, 1].
For every training sample, distances to all reconstructed outputs are computed using NCC to search for nearest neighbours.
Training samples are paired with the mean of reconstructed outputs whose NCC scores exceed 0.9 of the closest-neighbour distance.
Each training-sample/reconstruction pair is scored with SSIM and sorted in descending order.

C.1 Results for Models in Figure 4

Across the models shown in Figure 4, all models achieve perfect training accuracy; more training samples improve test accuracy, whereas more layers leave it approximately unchanged.

All models shown in Figure 4 achieve 100% train accuracy.
Adding more training samples improves test accuracy.
Adding more layers keeps test accuracy approximately the same.

C.2 All Comparisons for Subsection 5.4

The comparisons present reconstruction outputs alongside original samples across model architectures and training-set sizes, while contrasting the method with model inversion and first-layer weight visualizations.

Model-inversion comparisons on CIFAR10 and MNIST optimize model output with respect to randomly initialized inputs, using both maximization and minimization.
Figures 6–11 display reconstructed and original samples for architectures ranging from d-100-100-1 to d-1000-1000-1 and sample counts from 100 to 1000.
The reconstruction examples use odd rows for reconstructions and even rows for the corresponding original data.
First-layer weights provide some recognizable input indications, but the reconstruction scheme recovers more samples and with better quality than those weights represent.

C.3 Stretching the Theoretical Limitations

Experiments extend the theoretical analysis by testing standard initialization scales and comparing reconstruction quality using SSIM.

C.3 Stretching the Theoretical Limitations: The experiments in this section examine results beyond the theoretical limitations stated for Theorem 3.1.
C.3.1 Standard Initialization Scale: Standard Kaiming initialization still produced reconstructions that correlated strongly with some CIFAR10 training samples.The reconstruction quality was lower than with a small-scale first-layer initialization.
C.3.1 Standard Initialization Scale: Lower SSIM scores under standard initialization reflected lower reconstruction quality than in the other experiments.The comparison concerns experiments shown in Figure 16(a,b) and earlier results in Figure 4.

C.3.2 Less Epochs

The reconstruction method remained effective after substantially fewer training epochs, with similar quality on CIFAR10 and MNIST and performance beyond the theorem’s stated guarantees.

C.3.2 Less Epochs: 104 epochs produced reconstruction quality very similar to training for more epochs on 500-sample CIFAR10 and MNIST models.The result may indicate convergence sufficiently close to a KKT point even after substantially less training.
C.3.2 Less Epochs: A large portion of the data was reconstructed with high quality even when the MNIST model did not achieve zero training error.The experiments also reconstructed samples appearing approximately on the margin for both MNIST and CIFAR.
C.3.2 Less Epochs: Mini-batch SGD with batch size 50 was also evaluated on 500 CIFAR10 images using the d-1000-1000-1 architecture.The original experiments used full-batch gradient descent to align with the theoretical assumptions.
C.3.1 Standard Initialization Scale: For standard-initialization models, the highest SSIM scores were lower than in other experiments, indicating lower-quality reconstructions.