PaperlayerView exact revision on arXiv

Source-linked AI summary

Digital Watermarking for Deep Neural Networks

Yuki Nagai, Yusuke Uchida, Shigeyuki Sakazawa, Shin'ichi Satoh

arXiv:1802.02601v1cs.CV

TL;DR

Training and sharing deep neural networks creates a need to protect trained models treated as intellectual property. The paper embeds watermarks through a parameter regularizer during training, fine-tuning, or distillation. Experiments show that watermarking does not impair performance and remains detectable after fine-tuning and substantial parameter pruning.

  • Problem

    Trained models are valuable intellectual property, but the paper addresses the need for technical copyright protection as models are shared and reused.

  • Method

    The paper embeds watermarks in model parameters with a parameter regularizer while training, fine-tuning, or distilling neural networks.

  • Results

    The framework embeds watermarks without impairing network performance, retains them after fine-tuning, and remains sufficiently robust after 65% of parameters are pruned.

  • Takeaways & Limitations

    Digital watermarking can serve as a framework for protecting ownership rights in trained deep neural-network models.

  • Takeaways & Limitations

    Overwriting destroys the original watermark, while black-box client-server use prevents detection because the method assumes visible weight values.

Abstract

from arXiv · show

Although deep neural networks have made tremendous progress in the area of multimedia representation, training neural models requires a large amount of data and time. It is well-known that utilizing trained models as initial weights often achieves lower training error than neural networks that are not pre-trained. A fine-tuning step helps to reduce both the computational cost and improve performance. Therefore, sharing trained models has been very important for the rapid progress of research and development. In addition, trained models could be important assets for the owner(s) who trained them, hence we regard trained models as intellectual property. In this paper, we propose a digital watermarking technology for ownership authorization of deep neural networks. First, we formulate a new problem: embedding watermarks into deep neural networks. We also define requirements, embedding situations, and attack types on watermarking in deep neural networks. Second, we propose a general framework for embedding a watermark in model parameters, using a parameter regularizer. Our approach does not impair the performance of networks into which a watermark is placed because the watermark is embedded while training the host network. Finally, we perform comprehensive experiments to reveal the potential of watermarking deep neural networks as the basis of this new research effort. We show that our framework can embed a watermark during the training of a deep neural network from scratch, and during fine-tuning and distilling, without impairing its performance. The embedded watermark does not disappear even after fine-tuning or parameter pruning; the watermark remains complete even after 65% of parameters are pruned.

1 Introduction

Deep neural networks require substantial data and training time, making trained models valuable for research, development, and their owners. The paper proposes digital watermarking to protect ownership of these trained models.

  • Training deep neural networks requires large amounts of data and time, including several weeks for very deep ResNet training on ImageNet.
  • Sharing trained models through repositories such as Model Zoo supports trying models and reproducing research without retraining them.
  • Fine-tuning pretrained models can lower training error, reduce retraining time, and improve performance, motivating widespread sharing of trained models.
  • Trained models can be important assets because their weights and performance depend on data quality, quantity, and computational resources.
  • The paper treats trained models as intellectual property and focuses on technically protecting their copyrights.
  • The proposed digital watermarking technology aims to protect ownership and detect intellectual-property infringement of trained neural-network models.

2 Problem Formulation

The paper formulates watermark embedding for neural-network parameters, defines requirements and embedding situations, and identifies attacks against which watermarks should remain robust. It distinguishes train-to-embed, fine-tune-to-embed, and distill-to-embed settings, with fine-tuning, compression, and overwriting as expected attacks.

  • 2 Problem Formulation: A watermark is defined as a T-bit vector embedded into parameters of one or more neural-network layers; the modified model is the host network.The host network continues performing its original task while carrying the embedded watermark.
  • 2.1 Requirements: Effective watermarking requires preserving network performance rather than parameter fidelity, while maintaining detectability after fine-tuning or other model modifications.This contrasts neural-network watermarking with image watermarking, where perceptual host quality is central.
  • 2.2 Embedding Situations: The framework considers three embedding situations: train-to-embed, fine-tune-to-embed, and distill-to-embed.These situations differ in parameter initialization and label availability, as summarized in Table 2.
  • 2.2 Embedding Situations: Train-to-embed embeds a watermark during scratch training with labeled data, whereas fine-tune-to-embed starts from a pretrained network and may adapt the output layer to another task.Fine-tune-to-embed can also support individually watermarked distributed model instances for tracking.
  • 2.1 Requirements: Table 1 organizes requirements for effective watermarking across image and neural-network domains, while the neural-network setting prioritizes task performance and robustness to model changes.The domain comparison motivates why parameter fidelity is not the primary objective for neural-network watermarking.
  • 2.2 Embedding Situations: Distill-to-embed embeds a watermark into a trained network without labels by using predictions from a trained model as labels during fine-tuning.The distillation framework normally trains a smaller network from a larger network’s predicted labels to compress the model.
  • 2.3 Expected Attack Types: The expected attacks are fine-tuning, model compression, and watermark overwriting, all of which may alter parameters or replace an existing watermark.Fine-tuning is especially feasible because pretrained initialization can reduce training error and computational cost while improving performance; compression reduces memory or computation for deployment.
  • 2.3 Expected Attack Types: Watermark overwriting is treated as a severe attack in which an attacker embeds a different watermark into the same model to destroy the existing one.The stated ideal is robustness against this replacement attempt.

3 Proposed Framework

The framework embeds watermarks into neural-network parameters during training by adding an embedding regularizer to the original objective. It supports multiple embedding-parameter designs and is intended to preserve host-task performance while enabling watermark extraction.

  • Embedding targets: The framework embeds a T-bit watermark into parameters of one or more layers of a host neural network.Although the discussion focuses on DCNNs, the framework is described as applicable to MLPs, RNNs, and LSTMs.
  • Embedding regularizer: Embedding during training adds a parameter regularizer to the original task cost instead of directly modifying a trained network.The regularizer is intended to guide parameters toward a watermark-bearing solution without impairing the original task.
  • Embedding regularizer: The embedding regularizer imposes a statistical bias on parameters, unlike standard L1 or L2 regularization used for overfitting control.Its embedding loss is formulated through a binary-classification view using cross entropy.
  • Watermark extraction: Watermark extraction projects the parameter vector with a secret embedding parameter and thresholds the result at zero.The embedding parameter is used in both watermark embedding and extraction, so its design affects performance.
  • Embedding-parameter designs: The framework compares direct, difference-based, and random embedding parameters for encoding watermark bits.Direct parameters assign bits to individual weights, difference parameters encode weight differences, and random parameters distribute each bit across weights with random coefficients.

4 Experiments

The experiments evaluate whether the embedding regularizer preserves host-network performance and whether embedded watermarks remain robust under attacks.

  • Experimental goals: The experiments test watermark embedding without impairing host-network performance and robustness against multiple attacks.The implementation of the embedding regularizer is publicly available.

4.1 Evaluation Settings

Experiments use CIFAR-10 and Caltech-101 with a wide residual network, standardized optimization settings, and watermark insertion into selected convolutional groups.

  • Datasets: The experiments use CIFAR-10 and Caltech-101, with Caltech-101 images resized to 32 × 32 for fine-tuning.CIFAR-10 contains 60,000 images in 10 classes, while Caltech-101 contains objects from 101 categories.
  • Host network: The host network is a wide residual network, with N denoting the number of blocks and k denoting the widening factor.The network is described as an efficient variant of the residual network.
  • Training settings: Training uses SGD with Nesterov momentum, cross-entropy loss, initial learning rate 0.1, weight decay 5.0×10^-4, momentum 0.9, and minibatch size 64.The learning rate is multiplied by 0.2 at 60, 120, and 160 epochs during 200 total epochs.

4.2 Embedding Results

The embedding experiments show successful detection of watermarks and favor the random embedding-parameter design because it combines low embedding loss and low host-task error without altering parameter distributions.

  • Train-to-embed: A 256-bit watermark is embedded into the conv 2 group while training the host network from scratch on CIFAR-10.The experiment compares training with and without watermark embedding.
  • Watermark detection: All embedded watermarks are correctly detected after binarizing the sigmoid projections at a threshold of 0.5.The comparison includes direct, difference, and random embedding parameters.
  • Watermark detection: Random watermarks in non-embedded cases can be distinguished because their sigmoid-projection distribution differs from embedded cases.This distinction addresses false detection for random watermark candidates.
  • Parameter effects: Direct and difference embedding significantly alter the embedded-layer parameter distribution, whereas random embedding does not.The comparison examines parameters from the layer containing the watermark.
  • Parameter-design comparison: The random approach is selected for later experiments because it combines low embedding loss, low original-task test error, and no parameter-distribution alteration.The choice is based on the reported comparison among the three embedding-parameter designs.

4.3 Fidelity

The parameter-regularizer approach embeds watermarks while preserving host-network performance across training from scratch, fine-tuning, and distilling. Direct post-training embedding can fail because watermark recovery and original-task performance impose conflicting requirements.

  • Embedding without Training: Directly modifying trained parameters fails when λ ≤ 1 because BER remains nonzero, while λ > 1 makes the original task’s test error too large.The method minimizes embedding loss while constraining deviation from the original parameters, but this trade-off is ineffective without considering the original task.
  • Train-to-embed: Embedding with direct and diff parameters increases training loss relative to the unembedded case because embedding loss dominates the objective.The comparison uses CIFAR-10 training curves across unembedded, direct, diff, and random regularizers.
  • Train-to-embed: Random embedding achieves almost the same test error as the unembedded network while producing an extremely low embedding loss.Direct and diff embedding produce slightly larger test errors and higher embedding losses than random embedding.
  • Watermark detection: All watermarks are successfully detected by thresholding the watermark response at 0.5, and random embedding is identifiable from its histogram.The histogram comparison concerns watermark responses before thresholding and includes direct, diff, and random parameter choices.
  • Fine-tune-to-embed: Fine-tune-to-embed achieves almost the same test error as unembedded fine-tuning and very low embedding loss in both same-domain and different-domain experiments.The experiments fine-tune a pretrained host on CIFAR-10, including a host initially trained on Caltech-101.
  • Distill-to-embed: Distill-to-embed achieves low test error and embedding loss when the second training uses predicted values as soft targets rather than labels.The host is first trained on CIFAR-10, then further trained with or without embedding using soft targets.

4.4 Robustness of Embedded Watermarks

The watermark remains detectable after fine-tuning and substantial parameter pruning, but distillation from randomly initialized weights does not transfer it. Overwriting can erase the original watermark when the competing watermark is large relative to the available parameters.

  • Fine-tuning: Fine-tuning increased embedding loss slightly, but the detected watermark had zero bit error in both same-domain and cross-domain settings.The experiments evaluated CIFAR-10 → CIFAR-10 and Caltech-101 → CIFAR-10 fine-tuning.
  • Parameter pruning: After pruning rates of 80% and 95%, many detected watermark values remained close to one in the embedded case, supporting watermark-existence confidence scoring.At a 95% pruning rate, the detected-watermark histogram was also compared with a host network without an embedded watermark.
  • Distillation: The watermark could not be detected after distillation because the distilled model used randomly initialized weights rather than the embedded model's parameters.The distilled model was trained from the first network's predicted values without embedding a watermark.
  • Watermark overwriting: Overwriting erased the original watermark in some cases when the number of newly embedded bits was large relative to the number of parameters.A bit error rate near 0.5 indicates complete erasure of the original watermark.

4.5 Capacity of Watermark.

Watermark capacity is constrained by the number of parameters in the embedding layer. With a single-layer perceptron regularizer, embedding more bits than parameters increases embedding loss or test error.

  • Capacity constraint: Embedding loss or test error became high when the number of embedded bits exceeded the number of parameters in the selected group.The paper gives 2,048 bits in conv 3 as an example of an overdetermined embedding problem.
  • Capacity constraint: The single-layer perceptron embedding method requires the number of embedded bits to remain smaller than the number of parameters.The paper states that a multi-layer perceptron in the embedding regularizer could resolve this limitation.

5 Discussion

The proposed watermarking approach maintains original-task performance and retains the complete watermark after fine-tuning, including across domains. Discussion identifies capacity behavior and limitations involving distillation, overwriting, black-box access, network morphism, and steganalysis.

  • Robustness: The method maintained original-task performance while retaining the complete watermark after fine-tuning in both the same and a different domain.Fine-tuning was treated as the most feasible and significant attack.
  • Robustness: Embedding the watermark near the input layer can resist fine-tuning because those weights change less and the nearby configuration may remain unchanged across tasks.
  • Capacity: Watermark capacity is strongly related to the number of host weights rather than watermark length, and a multilayer perceptron in the regularizer may increase capacity.
  • Limitations: Distillation requires data very similar to the original training inputs to maintain fidelity, limiting its practical use as an attack.
  • Limitations: Overwriting destroys the original watermark when the embedding location is known, while targeting multiple layers would incur substantially greater computational cost.
  • Limitations: The approach cannot detect abuse in black-box client-server settings, and major network morphism may make an embedded watermark undetectable.

6 Conclusions

The paper presents a parameter-regularizer framework for watermarking deep neural networks to protect rights in trained models. Experiments show watermarking without impaired performance and complete retention after fine-tuning or pruning 65% of parameters.

  • The paper formulates watermark embedding in deep neural networks and defines requirements, embedding situations, and vulnerable attack types.
  • A parameter regularizer embeds watermarks in model parameters without impairing the network's performance.
  • 65% of parameters could be pruned while the entire watermark remained embedded.
Loading 1802.02601v1…