Source-linked AI summary
Privacy-Preserving Object Detection for Vision Transformer-Based Models
Homare Sueyoshi, Kiyoshi Nishikawa, Hitoshi Kiya
TL;DR
Visual-information protection studies have primarily focused on image classification, while object detection remains insufficiently addressed. The paper extends perceptual encryption to ViT-based object detection by encrypting test images and the model’s patch embedding with a key-based permutation. Experiments with ViTdet report nearly the same detection accuracy as non-encrypted models while protecting visual information.
Problem
Previous visual-information protection studies focus on image classification rather than object detection.
Source: §I · p. 1Method
The method encrypts ViTdet test images and its patch embedding using a secret-key-generated permutation so inference occurs in the encrypted domain.
Source: A. Overview of propoesd method · p. 2, D. Model Encryption · p. 3Results
The proposed method achieves almost the same object-detection accuracy as non-encrypted models while providing visual protection.
Source: B. Experimental Results · p. 3, §V · p. 4Takeaways & Limitations
Block-wise encryption can protect visual information while maintaining high detection accuracy in ViT-based object detection.
Source: E. Properties of proposed method · p. 3, §V · p. 4
Abstract
from arXiv · showhide
We propose a novel object detection method that enables us to protect sensitive visual information of test images. Previous studies considering visual information protection focus on image classification tasks. This paper proposes an object detection method using perceptual encryption for the first time. The proposed method can achieve almost the same accuracy as that of models without any protection by utilizing the embedding structure of the Vision Transformer (ViT) and a domain adaptation technique with keys. In experiments, the effectiveness of the proposed method is verified in terms of accuracy and visual protection under the use of ViTdet, which is a ViT-based object detection model.
I. INTRODUCTION
The paper extends perceptual encryption from image classification to object detection by using ViTdet and focusing on its patch-embedding structure. ViTdet processes image patches with positional embeddings and Transformer attention before constructing detection features.
I. INTRODUCTION
- The proposed method is the first perceptual-encryption approach for object detection, addressing prior work focused on image classification.It uses ViTdet as the representative ViT-based detector and evaluates detection accuracy and visual protection.Source: §I · p. 1
A. ViTDeT
- ViTdet uses a plain ViT backbone rather than hierarchical backbones and Feature Pyramid Networks.Source: A. ViTDeT · p. 1
- Image patches are linearly projected into an embedding space, with positional embeddings and a class token added before Transformer encoding.Source: A. ViTDeT · p. 1
- Window Attention reduces computation for high-resolution inputs, while selected Global Attention blocks propagate information across windows.Source: A. ViTDeT · p. 1
- The method focuses on the patch embedding matrix E within ViTdet’s input structure.Source: A. ViTDeT · p. 1
III. PROPOSED METHOD
The proposed framework encrypts both the ViTdet model’s patch embedding and test images with a shared secret-key-derived permutation. Encrypted models and images are processed by a cloud server without access to plain images or the secret key.
III. PROPOSED METHOD
- A model creator generates a random permutation matrix Eenc from a secret key k.Source: A. Overview of propoesd method · p. 1
- The framework encrypts the patch embedding E to create an encrypted model, then encrypts test images using the same Eenc.Source: A. Overview of propoesd method · p. 2
- The cloud server receives the encrypted model and encrypted test image and returns an inference result without plain images or the secret key.Source: A. Overview of propoesd method · p. 2
B. Generation of Random Permutation Matrix
The method constructs Eenc as a random permutation matrix from a vector containing distinct indices. Its orthogonality provides a stated algebraic property used by the encryption scheme.
B. Generation of Random Permutation Matrix
- The permutation vector l contains L elements, with every pair of elements having different values.Source: B. Generation of Random Permutation… · p. 2
- L equals p^2c, where p is the patch size and c is the number of image channels.Source: B. Generation of Random Permutation… · p. 2
- Each matrix component k(i,j) is 1 when l(i)=j and 0 otherwise, defining the random permutation matrix Eenc.Source: B. Generation of Random Permutation… · p. 2
- The paper states that Eenc is orthogonal.Source: B. Generation of Random Permutation… · p. 2
- For L = 4 and l = [3, 4, 2, 1], the paper gives a concrete Eenc construction example.Source: B. Generation of Random Permutation… · p. 2
C. Image Encryption
Image encryption divides an input into non-overlapping p × p blocks, flattens each block, and applies Eenc before reconstructing the encrypted image. The block size matches the ViT patch size.
C. Image Encryption
- The input image is divided into N non-overlapping blocks of size p × p.Source: C. Image Encryption · p. 2
- The encryption block size p is set equal to the Vision Transformer’s patch size.Source: C. Image Encryption · p. 2
- Each block Bi is flattened into a vector bi and encrypted using Eenc.Source: C. Image Encryption · p. 2
- Encrypted vectors are reshaped into encrypted blocks and concatenated to form the encrypted image x̂.Source: C. Image Encryption · p. 2
D. Model Encryption
The trained model is encrypted by transforming its patch embedding matrix with an encryption operation, so inference proceeds in the encrypted domain.
- The trained model is encrypted using an equation involving the patch embedding matrix E.Source: D. Model Encryption · p. 3
- The embedding matrix E has shape E ∈ R^D×L, where D is the Vision Transformer’s embedding dimension.Source: D. Model Encryption · p. 3
- Inference is consequently carried out in the encrypted domain.Source: D. Model Encryption · p. 3
E. Properties of proposed method
The method uses an encryption matrix to transform patch embeddings and encrypted image patches, yielding an input sequence equivalent to the original sequence.
- The analysis introduces an encryption matrix E_enc satisfying an orthogonality condition.Source: E. Properties of proposed method · p. 3
- The encrypted embedding matrix and encrypted image patches are defined using E_enc.Source: E. Properties of proposed method · p. 3
- Substituting the encrypted definitions derives the projection of each encrypted patch.Source: E. Properties of proposed method · p. 3
- The resulting encrypted input sequence is constructed from the class token, encrypted patch projections, and positional embeddings.Source: A. ViTDeT · p. 1, E. Properties of proposed method · p. 3
- The derivation shows that ẑ_0 = z_0, so model encryption cancels the influence of image encryption during Transformer processing.Source: E. Properties of proposed method · p. 3
A. Setup
The evaluation uses COCO and LVIS object-detection datasets with ViTdet-based Mask R-CNN models and standard AP metrics across object scales.
- COCO test images contain 80 object categories, and validation uses the val2017 subset.Source: A. Setup · p. 3
- LVIS uses the same images as COCO but has different splits, richer annotations, and 1,203 object categories.Source: A. Setup · p. 3
- ViTdet with the Mask R-CNN framework is used for the LVIS detection evaluation.Source: A. Setup · p. 3
- Average Precision ranges from 0 to 100, with AP_s, AP_m, and AP_l measuring small, medium, and large objects respectively.Source: A. Setup · p. 3
B. Experimental Results
Experiments compare encrypted and non-encrypted object detection under multiple ViTdet backbones on COCO and LVIS datasets.
- Table I compares Mask R-CNN object-detection AP on the COCO dataset.Source: B. Experimental Results · p. 3, Table I · p. 4
- The evaluation uses ViT-B, ViT-L, and ViT-H ViTdet backbones.Source: B. Experimental Results · p. 3
- Baseline uses non-encrypted models and images, while Without model encryption encrypts only images and Proposed encrypts both models and images.Source: B. Experimental Results · p. 3
- The proposed method achieves almost the same accuracy as non-encrypted models, with slight differences attributed to JPEG compression in COCO.Source: B. Experimental Results · p. 3
- Table II compares Mask R-CNN object-detection AP on the LVIS dataset.Source: Table II · p. 4
V. CONCLUSION
The proposed method protects visual information during object detection by exploiting ViT embedding structure. With ViTdet, block-wise encryption preserves detection accuracy while image-only encryption substantially reduces it.
- The method uses Vision Transformer embedding structure for privacy-preserving object detection.Source: §V · p. 4
- Block-wise encryption protects visual information while maintaining detection accuracy equivalent to non-encrypted models.Source: §V · p. 4
- Image-only encryption causes a significant accuracy decrease, demonstrating the necessity of encrypting the model’s embedding layer.Source: §V · p. 4