PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Reconstruction of Face Images from Deep Face Templates

Guangcan Mai, Kai Cao, Pong C. Yuen, Anil K. Jain

arXiv:1703.00832v4cs.CV

TL;DR

The paper asks whether deep face templates can be inverted into face images, an important question for face-recognition security and privacy. It proposes NbNet reconstruction models trained with public and DCGAN-synthesized faces, achieving strong verification and identification attack rates. The findings demonstrate that deep templates require protection.

  • Problem

    The study examines the extent to which deep-network face templates can be inverted to obtain original face images, addressing their security and privacy vulnerability.

  • Method

    The paper proposes NbNet and trains it with augmented public face datasets containing DCGAN-synthesized images, using perceptual loss to reconstruct faces from deep templates.

  • Results

    95.20% (58.05%) TAR was achieved on LFW for type-I (type-II) attacks at FAR=0.1%, alongside 96.58% (92.84%) FERET identification accuracy for fa (fb) reconstructions.

  • Takeaways & Limitations

    The results demonstrate the need to secure deep templates in face recognition systems.

  • Takeaways & Limitations

    Training the approximation optimally would require a sufficiently large face-image database, whose collection is impractical because of its time and cost.

Abstract

from arXiv · show

State-of-the-art face recognition systems are based on deep (convolutional) neural networks. Therefore, it is imperative to determine to what extent face templates derived from deep networks can be inverted to obtain the original face image. In this paper, we study the vulnerabilities of a state-of-the-art face recognition system based on template reconstruction attack. We propose a neighborly de-convolutional neural network (\textit{NbNet}) to reconstruct face images from their deep templates. In our experiments, we assumed that no knowledge about the target subject and the deep network are available. To train the \textit{NbNet} reconstruction models, we augmented two benchmark face datasets (VGG-Face and Multi-PIE) with a large collection of images synthesized using a face generator. The proposed reconstruction was evaluated using type-I (comparing the reconstructed images against the original face images used to generate the deep template) and type-II (comparing the reconstructed images against a different face image of the same subject) attacks. Given the images reconstructed from \textit{NbNets}, we show that for verification, we achieve TAR of 95.20\% (58.05\%) on LFW under type-I (type-II) attacks @ FAR of 0.1\%. Besides, 96.58\% (92.84\%) of the images reconstruction from templates of partition \textit{fa} (\textit{fb}) can be identified from partition \textit{fa} in color FERET. Our study demonstrates the need to secure deep templates in face recognition systems.

1 INTRODUCTION

The paper investigates whether deep face templates can be inverted to reconstruct face images and evaluates the resulting security and privacy risks. It proposes NbNet, trained with public and synthesized face images, and reports high attack success in verification and identification.

  • Motivation: Deep face templates are investigated for invertibility because face recognition systems require both strong performance and resistance to template reconstruction attacks.The study targets whether reconstructed images can be used to access privileges.
  • Proposed approach: NbNet reduces de-convolution channels and learns additional neighboring channels to limit repetition while increasing reconstructed-image detail.The framework is built from neighborly de-convolution blocks inspired by DenseNet and MemNet.
  • Training: Public face datasets were augmented with images synthesized by a trained DCGAN to supply the large training collection required by reconstruction models.The models also explored pixel-difference and perceptual losses.
  • Results: 95.20% TAR was achieved on LFW for type-I attacks at FAR=0.1%, while type-II attacks achieved 58.05% TAR under the same FAR.Type-I compares against the image used to create the template; type-II uses a different image of the same subject.
  • Results: 96.58% and 92.84% rank-one identification accuracy were achieved on color FERET for reconstructions from partitions fa and fb, respectively.Partition fa was used for type-I evaluation and fb for type-II evaluation.

2 RELATED WORK

Prior reconstruction methods include shallow MDS and RBF approaches and deep methods requiring stronger access assumptions. The paper also reviews GANs as a source of synthesized training faces while describing unresolved template-protection challenges.

  • Template protection: Face templates cannot be simply protected with standard ciphers because intra-subject variation complicates matching after encryption.Decryption also creates key-management and exposure challenges.
  • Template protection: Template protection remains an open challenge, and reliance on specialized hardware limits the range of biometric applications.The paper notes that vendors either neglect template security or use hardware-protected templates and keys.
  • Template reconstruction: Existing reconstruction algorithms include MDS and RBF regression, but their evaluations were limited to shallow templates or had unknown generalizability.Earlier studies often trained and tested on subsets of the same dataset.
  • Template reconstruction: Some deep reconstruction methods require gradient information from the target template extractor, conflicting with a black-box extractor assumption.Other methods estimate landmarks and textures before combining them.
  • GAN-based generation: GANs generate face images from randomly sampled vectors using a generator and discriminator, making them useful for synthesizing training data.The paper employs DCGAN as an efficient face-image generation method.

3 PROPOSED TEMPLATE SECURITY STUDY

The security study models deep-template reconstruction as an attack and presents the proposed NbNet, its training strategy, and implementation as the main components of the analysis.

  • Study overview: The study overview separates normal face-recognition processing from template-reconstruction attack flows and then introduces NbNet and its training strategy.The section frames the investigation within an adversarial machine-learning approach.

3.1 Template Reconstruction Attack

The attack model assumes access to target templates and a black-box feature extractor, but no target-subject information or enrollment images. The attacker learns a reconstruction model from public face images and injects reconstructed queries into the system.

  • Attack model: The adversarial framework characterizes the attack through the adversary’s goal, knowledge, capability, and strategy.The attacker seeks to impersonate a target subject and compromise system integrity.
  • Adversary knowledge: The attacker is assumed to possess leaked target templates and the target system’s black-box feature extractor, but not target-subject information or enrollment environments.The extractor may be obtained by purchasing the system SDK.
  • Adversary capability: The simplified capability model lets the attacker inject reconstructed face images directly into the feature extractor, although actual attacks may present photographs or masks to the sensor.The attacker also receives the identity decision for each query.
  • Attack strategy: The attacker estimates a face image as x_t = g_θ(y_t) from the target template and learns g_θ using public-domain face images.The reconstructed image is inserted as a query to access the target system.

3.2 NbNet for Face Image Reconstruction

NbNet reconstructs face images from deep templates using stacked neighborly de-convolution blocks that reduce repeated channels and enrich details through within-block channel learning.

  • 3.2 NbNet for Face Image Reconstruction: NbNet cascades de-convolution blocks with a final convolution operator to expand abstract deep-template signals into high-resolution face-image channels.Stacked blocks progressively up-sample and add detail before the output convolution summarizes the channels.
  • 3.2 NbNet for Face Image Reconstruction: NbBlock reduces channels learned simultaneously from previous blocks, then generates additional channels from neighboring channels within the same block.This design targets noisy, repeated channels and insufficient channel details in typical de-convolution blocks.
  • 3.2 NbNet for Face Image Reconstruction: NbNet-A and NbNet-B use different choices of neighboring outputs for constructing their convolutional channels.Both variants contain one de-convolution operator and multiple convolution operators, with their distinction determined by the selected neighboring outputs.
  • 3.2 NbNet for Face Image Reconstruction: In the 32-channel examples, the first two channel rows come from de-convolution, while later rows are produced by convolution operators learning from earlier rows.NbNet-A’s fourth row uses the third row, whereas NbNet-B’s fourth row uses the first three rows.

3.3 Reconstruction Loss

The reconstruction model is trained by comparing generated images with originals using pixel-wise or perceptual losses, with perceptual loss targeting robust image representations.

  • 3.3 Reconstruction Loss: Reconstruction loss compares an input face image x with its reconstruction x′ generated from the extracted deep template.The template extractor is treated as a black-box component in this formulation.
  • 3.3 Reconstruction Loss: Pixel difference uses a pixel-wise Minkowski distance between the original and reconstructed images.The metric order is k, and M denotes the number of pixels.
  • 3.3 Reconstruction Loss: Perceptual loss guides reconstructions toward the original images’ representations using a VGG-19 feature map.The selected representation is the ReLU3_2 activation-layer feature map, chosen empirically for face matching accuracy.

3.4 Generating Face Images for Training

Because generalizable reconstruction requires many subjects and large datasets are costly to collect, the paper augments public face datasets with generated images for NbNet training.

  • 3.4 Generating Face Images for Training: Training NbNet without target-subject knowledge requires accurate reconstruction from templates of faces belonging to different subjects.The training objective is expressed over the probability distribution of face images.
  • 3.4 Generating Face Images for Training: The empirical training loss averages reconstruction losses over N collected face images, with xi denoting the i-th training image.This approximation is optimal only when N is sufficiently large.
  • 3.4 Generating Face Images for Training: Large-scale collection is impractical because obtaining a sufficiently large database entails substantial time and cost.The limitation motivates augmenting existing public datasets rather than relying solely on newly collected data.
  • 3.4 Generating Face Images for Training: The paper augments publicly available datasets by sampling generated faces from a latent vector through a face generator.This avoids directly modeling the joint distribution of the very large number of face-image pixels.
  • 3.4 Generating Face Images for Training: A DCGAN generates face images from uniformly distributed latent vectors, approximating the distribution of real face images and producing unseen subjects with varied intra-subject appearances.The network details table specifies the de-convolution and convolution operators, channel counts, kernels, strides, normalization, and activations.

3.5 Differences with DenseNet

NbNet adapts the DenseNet-inspired idea of learning from previous channels to de-convolutional face-image reconstruction, with variants differing in channel connectivity.

  • NbNet uses de-convolution layers to reconstruct face images from deep templates, whereas DenseNet uses convolution layers for object recognition.
  • NbBlocks learn output channels from previous blocks and neighboring channels within each block, reducing repeated noisy channels and retaining more reconstruction detail.
  • NbNet-A uses skip-connection-like connectivity, while NbNet-B uses DenseNet-like connectivity across all previous channels within each block.
  • NbNet-A can achieve comparable performance to NbNet-B with roughly 67% of the parameters and 54% of the running time.

3.6 Implementation Details

The implementation augments face datasets with generated images, revises DCGAN training to improve generated-image quality, and trains NbNet on template–image pairs.

  • NbNet architectures differ in whether de-convolution blocks use nearest previous channels or all previous channels within the same block.
  • The revised DCGAN replaces batch normalization and ReLU with SeLU activation in both generator and discriminator.
  • Soft labels assign real images values in [0.7, 1.2] and generated images values in [0, 0.3] to smooth the discriminator and avoid model collapse.
  • The generator learning rate is set to 2 × 10^-4, exceeding the discriminator learning rate of 5 × 10^-5.
  • The training pipeline first uses a DCGAN to generate face images, then extracts corresponding templates for NbNet training.
  • NbNet is updated with generated images and templates using batch gradient descent to minimize an approximation of the stated loss function.

4 EXPERIMENTAL RESULTS

Experiments train reconstruction models on VGG-Face and Multi-PIE and test them on verification and identification benchmarks under type-I and type-II attacks. NbNets generally outperform D-CNN and RBF-based reconstruction, with strong attack success rates.

  • 4.1 Database and Experimental Setting: The study trains on VGG-Face and Multi-PIE, then evaluates verification on LFW and FRGC v2.0 and identification on color FERET.
  • 4.1 Database and Experimental Setting: The comparison includes D-CNN, NbNet-A, NbNet-B, pixel-difference and perceptual-loss variants, mixed and raw training data, and an RBF regression baseline.
  • 4.2.1 Performance on LFW: 95.20% TAR is achieved by VGG-NbA-P in the type-I LFW attack at FAR=0.1%.
  • 4.2 Verification Results: NbNets generally outperform D-CNN and RBF regression across type-I and type-II attacks, while MPIE-NbA-P is an exception against MPIE-Dn-P.
  • 4.2.2 Performance on FRGC v2.0: 73.76% TAR is achieved by MPIE-NbB-P in the type-I FRGC v2.0 attack at FAR=0.1%.

5 CONCLUSIONS AND FUTURE WORK

The study finds that deep templates can support face-image reconstruction with high attack success rates, revealing security and privacy issues from template leakage. Future work targets template protection and stronger reconstruction methods.

  • Conclusions: The NbNet-based method outperformed RBF-regression-based reconstruction in attack success rates.
  • Conclusions: 95.20% and 58.05% TAR were achieved on LFW under type-I and type-II attacks at FAR=0.1%.
  • Conclusions: Color FERET identification reached 96.58% from partition fa templates and 92.84% from partition fb templates using partition fa.
  • Future Work: Future work includes user-specific randomness for secure templates, stronger anti-spoofing, improved NbNets, holistic face content, and cross-system attacks.

APPENDIX A PROOF OF THE EXISTENCE OF A FACE IMAGE GENERATOR

The appendix proves that a face-image generator can be constructed by transforming a multivariate uniform vector through intermediate Gaussian and marginal-distribution transformations. The resulting generated representation matches the component distributions and approximately preserves covariance.

  • A face image is represented as a low-dimensional real vector b whose components follow distributions F_bi and covariance matrix Σ_b.The image dimensions h × w × c are much larger than the manifold dimension k.
  • The generator begins with a k-dimensional multivariate uniform vector z and constructs an intermediate Gaussian vector a.The proof describes a ∈ N(0, Σ_a) as the intermediate representation.
  • NORTA transforms the Gaussian vector into b′ with component distributions matching F_bi and covariance approximately matching Σ_b.The covariance of the Gaussian input is adjusted to achieve Σ_b′ ≈ Σ_b.
  • A feasible covariance matrix must be symmetric, positive semi-definite, and have unit diagonal elements; otherwise, Σ_a is approximated by solving an equation.These conditions are necessary but insufficient for feasibility.
  • The proof concludes that the generator function exists and can be constructed using these transformations.
Loading 1703.00832v4…