PaperlayerView exact revision on arXiv

Source-linked AI summary

REFIT: A Unified Watermark Removal Framework For Deep Learning Systems With Limited Data

Xinyun Chen, Wenxiao Wang, Chris Bender, Yiming Ding, Ruoxi Jia, Bo Li, Dawn Song

arXiv:1911.07205v3cs.CRcs.LG

TL;DR

Training DNNs from scratch is costly, while existing watermark-removal attacks have often assumed scheme knowledge or abundant data, leaving their practicality under limited data unclear. REFIT uses fine-tuning with EWC and unlabeled data augmentation to remove diverse watermarks. The evaluation reports that these techniques reduce the labeled data needed to preserve performance while removing watermarks.

  • Problem

    Prior watermark-removal work often assumes knowledge of the embedding scheme or full training data, leaving the practicality of attacks with limited data unclear.

    Source: §1 · p. 2

  • Method

    REFIT fine-tunes watermarked models using an adapted EWC technique and auxiliary unlabeled data annotated by the model.

    Source: §1 · p. 2

  • Results

    With EWC and AU, REFIT reduces the in-distribution labeled samples needed to preserve performance while removing watermarks, and AU data need not match the evaluation distribution.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that limited-data fine-tuning attacks threaten existing watermark schemes and motivate further study of more robust embedding approaches.

    Source: §1 · p. 2

    Approximating the EWC Fisher matrix with fine-tuning data assumes that data come from a distribution similar to the pre-training data.

    Source: §3 · p. 5

Abstract

from arXiv · show

Training deep neural networks from scratch could be computationally expensive and requires a lot of training data. Recent work has explored different watermarking techniques to protect the pre-trained deep neural networks from potential copyright infringements. However, these techniques could be vulnerable to watermark removal attacks. In this work, we propose REFIT, a unified watermark removal framework based on fine-tuning, which does not rely on the knowledge of the watermarks, and is effective against a wide range of watermarking schemes. In particular, we conduct a comprehensive study of a realistic attack scenario where the adversary has limited training data, which has not been emphasized in prior work on attacks against watermarking schemes. To effectively remove the watermarks without compromising the model functionality under this weak threat model, we propose two techniques that are incorporated into our fine-tuning framework: (1) an adaption of the elastic weight consolidation (EWC) algorithm, which is originally proposed for mitigating the catastrophic forgetting phenomenon; and (2) unlabeled data augmentation (AU), where we leverage auxiliary unlabeled data from other sources. Our extensive evaluation shows the effectiveness of REFIT against diverse watermark embedding schemes. In particular, both EWC and AU significantly decrease the amount of labeled training data needed for effective watermark removal, and the unlabeled data samples used for AU do not necessarily need to be drawn from the same distribution as the benign data for model evaluation. The experimental results demonstrate that our fine-tuning based watermark removal attacks could pose real threats to the copyright of pre-trained models, and thus highlight the importance of further investigating the watermarking problem and proposing more robust watermark embedding schemes against the attacks.

1 INTRODUCTION

Because training DNNs from scratch is costly, watermarking is used to protect pre-trained models, but watermark-removal attacks raise concerns about whether existing protections withstand limited-data adversaries. REFIT uses fine-tuning, EWC, and unlabeled data augmentation to remove diverse watermarks while preserving model performance under limited-data settings.

1 INTRODUCTION

  • Watermarking protects pre-trained DNNs by embedding specially designed samples whose predictions can later be checked to verify ownership.Training DNNs from scratch is computationally intensive and requires many high-quality annotated samples.Source: §1 · p. 1
  • Prior attacks include detection and removal; removal methods often assume watermark-scheme knowledge or full training data, or significantly hamper model accuracy.These limitations leave the practicality of watermark removal with limited data unclear.Source: §1 · pp. 1–2
  • REFIT uses a tuned fine-tuning learning-rate schedule and adapts EWC to limit updates to weights important for task knowledge while modifying weights used more for watermark memorization.The EWC adaptation is intended to mitigate catastrophic forgetting.Source: §1 · p. 2
  • Unlabeled data augmentation (AU) uses the watermarked model to label auxiliary unlabeled samples and adds them to the fine-tuning data.The authors note that unlabeled data may be cheaper to obtain than labeled data.Source: §1 · p. 2
  • With EWC and AU, REFIT removes watermarks while reducing the in-distribution labeled samples needed to preserve performance; AU data may come from a very different distribution.In transfer learning, REFIT consistently surpasses the pre-trained watermarked models’ test performance after watermark removal.Source: §1 · p. 2
  • REFIT is presented as an embedding-scheme-agnostic framework and the first comprehensive study of data efficiency in watermark-removal attacks, with results indicating threats to existing schemes.The study evaluates diverse watermarking approaches under limited-data conditions.Source: §1 · p. 2

2 MODEL WATERMARKING

The section describes watermarking as a way to verify ownership of a model and sets out a removal threat model in which the adversary lacks watermark knowledge and has limited fine-tuning data. It emphasizes that prior removal defenses may not generalize across watermark types.

2 MODEL WATERMARKING

  • A valid watermarking scheme should preserve task functionality and make watermark predictions more likely for the watermarked model than for other models; a threshold γ can determine whether a model is considered watermarked.The owner checks predictions on watermark samples to verify ownership.Source: §2 · p. 2
  • Pattern-based watermarks blend a shared pattern into images, while defenses often depend on assumptions such as pattern size or label distribution and may not generalize to other watermark types.Source: §2 · pp. 2–3

2.1 Threat Model for Watermark Removal

  • The threat model assumes the adversary has no knowledge of the watermarks or embedding scheme, but has computation resources for fine-tuning and only limited fine-tuning data.Unlike many prior studies, it does not assume access to as much benign task data as the model owner.Source: §2.1 · p. 3
  • The study focuses on adversaries with much smaller fine-tuning datasets than the pre-training dataset, a setting where training from scratch would typically yield inferior performance.The authors present this as a more realistic threat model for watermark removal.Source: §2.1 · p. 3

3 REFIT: REMOVING WATERMARKS VIA FINE-TUNING

REFIT uses fine-tuning to remove watermarks, with a learning-rate schedule and two additions—EWC and unlabeled data augmentation—to address utility loss when labeled data are limited.

3 REFIT: REMOVING WATERMARKS VIA FINE-TUNING

  • Watermark accuracy decreases substantially only at appropriately large fine-tuning learning rates, while training and test accuracies recover after initial drops.The learning rate is doubled every 20 epochs; the passage reports negligible performance change around 0.001 and noticeable change around 0.005.Source: §3 · p. 4, Figure 3 · p. 4
  • With 20% of CIFAR-100 training data, basic fine-tuning could reduce test accuracy by 5% when removing watermarks.The authors attribute the utility loss to catastrophic forgetting of normal training samples.Source: §3 · p. 4
  • REFIT adapts EWC to remove watermark behavior while preserving main-task performance, penalizing changes to parameters with large diagonal Fisher entries.The EWC loss adds a regularization term to the new-task loss; its strength is controlled by λ.Source: §3 · pp. 4–5
  • For transfer learning, REFIT first fine-tunes for the new task, then uses that model’s parameters as θ* for EWC or its predictions to label AU data.The two extensions are applied after an initial fine-tuning step that does not consider watermark removal.Source: §3 · p. 5
  • EWC approximates the previous-task Fisher matrix using fine-tuning data, assuming that data comes from a distribution similar to the pre-training data.The authors note this approximation may be imprecise; it is intended to limit updates to weights important for test accuracy while allowing watermark-related weights to change.Source: §3 · p. 5
  • AU augments fine-tuning with unlabeled samples labeled by the pre-trained model; even noisy labels from a different distribution can help preserve performance during watermark removal.The authors report that AU decreases the in-distribution labeled samples needed for effective removal while preserving model performance.Source: §3 · p. 5

4 EVALUATION SETUP

The evaluation covers four image-classification benchmarks, four watermarking approaches, and both transfer and non-transfer attack scenarios. Implementation choices set up comparisons of watermark removal and test accuracy under limited-data conditions.

4 EVALUATION SETUP

  • The evaluation setup introduces the benchmarks, watermark schemes, and experimental configurations used to assess REFIT.Unless otherwise specified, experiments use ResNet-18; fine-tuning starts at a larger learning rate than the cited prior setup, with the basic REFIT version omitting EWC and AU.Source: §4 · p. 5, §4.4 · p. 7

4.1 Datasets

  • The evaluation uses CIFAR-10, CIFAR-100, STL-10, and ImageNet32, spanning 10- to 1,000-class image-classification benchmarks.STL-10 additionally provides 100,000 unlabeled images from a similar but broader distribution.Source: §4.1 · p. 5

4.2 Watermarking Techniques

  • The study evaluates pattern-based, out-of-distribution, exponential weighting, and adversarial frontier stitching watermarks across its benchmarks.Pattern watermarks blend a key pattern and target label; OOD watermarks use external images with random labels; EW relabels training samples and applies exponential parameter weighting; ADV uses perturbed images labeled with their original classes.Source: §4.2 · pp. 5–6

4.3 Attack Scenarios

  • Non-transfer attacks use a watermarked model for the target task, while transfer attacks adapt a model trained on CIFAR-10 or ImageNet32 to STL-10.Non-transfer experiments vary the available labeled-data proportion; AU also uses unlabeled STL-10 or ImageNet32 images, which may differ substantially from the labeled data.Source: §4.3 · pp. 6–7

4.4 Implementation Details

  • Evaluation measures test accuracy alongside watermark accuracy, requiring removal below scheme-specific thresholds while preserving accuracy on the normal test set.For transfer learning, watermark accuracy is measured by replacing the fine-tuned model's output layer with the pre-trained model's original output layer.Source: §4.4 · pp. 7–8

5 EVALUATION

Across transfer and non-transfer settings, REFIT removes watermarks while EWC and AU help preserve test performance with limited labeled data. AU can also work with unlabeled data from a different distribution, and REFIT compares favorably with alternative removal methods.

5.1 Evaluation of transfer learning

  • For transfer learning, basic REFIT preserves STL-10 test performance after CIFAR-10 pretraining; EWC or AU can surpass watermarked models, while ImageNet32 pretraining requires AU for comparable performance.Combining EWC and AU yields the strongest performance among the reported REFIT variants and watermarked models after ImageNet32 pretraining.Source: §5.1 · p. 8, Table 1 · p. 9, Table 2 · p. 9

5.2 Evaluation of non-transfer learning

  • At 80% of the training set, basic REFIT exceeds pretrained-model test accuracy across evaluated watermark schemes while removing the watermarks.This result is reported for non-transfer learning on CIFAR-10 and CIFAR-100.Source: §5.2 · p. 8, Table 3 · p. 9, Table 4 · p. 9

5 EVALUATION

  • Across transfer and non-transfer settings, REFIT removes watermarks while EWC and AU reduce labeled-data needs and help preserve test performance.AU also works with unlabeled data from a different distribution than the evaluation task.Source: §5.2 · pp. 8, 10, §5.1 · p. 8

5.2 Evaluation of non-transfer learning

  • With limited labels, accuracy drops about 2% on CIFAR-10 and up to 5% on CIFAR-100; EWC typically improves accuracy nearly 1% and up to 3%, respectively, for schemes other than ADV.AU reaches the pretrained models’ test-performance level with 20%–30% of the full labeled training set.Source: §5.2 · p. 8
  • AU can fine-tune with no labeled data and keep accuracy within 1% of the pretrained model on CIFAR-10 and CIFAR-100, despite using unlabeled data from different distributions.When integrated into REFIT, the choice between STL-10 and ImageNet32 unlabeled data does not consistently favor one source.Source: §5.2 · p. 8
  • EWC becomes competitive with AU at about 30% labeled data on CIFAR-10 and 40% on CIFAR-100, as the estimated Fisher matrix better captures important parameters.Source: §5.2 · p. 10
  • On ImageNet32, combined EWC and AU match the pretrained model’s performance with 50% of labeled data, and combining them further improves performance over either component alone.At 20% labeled data, EWC raises accuracy around 1%–3% over basic REFIT; AU generally performs better until EWC becomes more competitive at 50%.Source: §5.2 · p. 10, Table 6 · p. 10

5.3 Comparison with alternative watermark removal techniques

  • Distillation does not improve performance in the reported comparison, while REFIT achieves better test performance than fine-pruning under the same learning-rate schedule.Pruning also makes watermark accuracy tightly associated with test accuracy, complicating selection of a pruning rate that preserves performance while removing watermarks.Source: §5.3 · p. 11

6 RELATED WORK

Related work covers attacks on machine-learning models, watermarking and watermark removal, and catastrophic forgetting. The authors adapt EWC because its original form requires old-task data unavailable in their removal setting.

6 RELATED WORK

  • Prior work addresses attacks that manipulate predictions or reveal sensitive information, alongside watermarking approaches for protecting DNN intellectual property.Source: §6 · p. 11
  • Existing watermark-removal methods often target pattern-based schemes, while general-purpose approaches may require substantial training data; some defenses rely on assumptions that limit generalizability.Source: §6 · p. 11
  • Catastrophic forgetting occurs when sequential training degrades old-task performance as important weights change; the original EWC requires old-task data, so the authors adapt it for watermark removal.Source: §6 · p. 11

7 CONCLUSION

REFIT uses fine-tuning with an adapted EWC approach and unlabeled data augmentation to remove watermarks. Under a limited-data adversary setting, both techniques remove diverse watermarks with little degradation in model performance.

7 CONCLUSION

  • EWC and AU enable watermark removal with little model-performance degradation under a weak adversary with limited training data.The evaluation covers several watermarking schemes of different types.Source: §7 · p. 11
  • REFIT is a unified framework that removes watermarks via fine-tuning, with a learning-rate schedule designed for effective removal.Source: §7 · p. 11
  • The study highlights the vulnerability of existing watermarking techniques and identifies more robust watermarking as future work.Source: §7 · p. 11

A MORE DISCUSSION ON EXPERIMENTAL DETAILS

The evaluation selects hyperparameters to maximize test accuracy after watermark removal, while using a common initial learning rate that worked relatively well across settings.

A MORE DISCUSSION ON EXPERIMENTAL DETAILS

B DISCUSSION OF SAMPLE EFFICIENCY

The sample-efficiency discussion reports gains from both EWC and AU, including when AU uses unlabeled samples from a different dataset distribution. It also notes a trade-off between EWC’s Fisher-sample count and computation overhead.

B DISCUSSION OF SAMPLE EFFICIENCY

  • With M = 100 Fisher-approximation samples, EWC raises test accuracy by around 1% over basic fine-tuning on CIFAR-100 with pattern-based watermarks.Larger M may improve results further, but can add computation overhead without significant performance gain.Source: B DISCUSSION OF SAMPLE EFFICIENCY · pp. 13–14
  • The EWC default M balances performance improvement against the computation overhead of using more Fisher-approximation samples.Source: B DISCUSSION OF SAMPLE EFFICIENCY · p. 13
  • Figure 8 plots fine-tuning learning-rate training curves with the model trained on the entire training set; Table 7 reports best hyperparameter ranges, including EWC λ and AU batch count m.Source: Figure 8 · p. 13, Table 7 · p. 14
  • AU with 5K unlabeled STL-10 samples yields considerable performance gain on CIFAR-100 despite the datasets’ distribution difference.Test accuracy continues to increase with more unlabeled samples.Source: Figure 7 · p. 13, B DISCUSSION OF SAMPLE EFFICIENCY · p. 14

C MORE DISCUSSION ON PRUNING

The pruning comparison finds that pruning does not reliably separate watermark removal from loss of test performance. REFIT with AU compares favorably, including without in-distribution labeled data.

C MORE DISCUSSION ON PRUNING

  • The comparison follows a prior pruning approach by pruning last-layer convolutional neurons in increasing order of validation-set activation magnitude.VGG-16 is also evaluated because ResNet skip connections can keep accuracy from becoming low even when pruning rates approach 1.Source: C MORE DISCUSSION ON PRUNING · p. 14
  • For both evaluated models, watermark accuracy is tightly associated with test accuracy, making it difficult to preserve test performance while removing watermarks through pruning.Source: C MORE DISCUSSION ON PRUNING · p. 14
  • On CIFAR-10, pruning at 90.72% test accuracy leaves 65% watermark accuracy; on CIFAR-100, pruning leaves 71% watermarks at 53.34% test accuracy, while REFIT reaches 66.79% accuracy.On CIFAR-10, REFIT with AU and no in-distribution labeled data achieves the pruning method’s test-performance level with the watermarks removed.Source: C MORE DISCUSSION ON PRUNING · p. 14

D MORE DISCUSSION ON FINE-PRUNING

The fine-pruning comparison finds that pruning is not necessary with a properly designed fine-tuning learning-rate schedule, and full REFIT outperforms fine-pruning.

D MORE DISCUSSION ON FINE-PRUNING

  • Fine-pruning uses REFIT’s learning-rate schedule, which is described as crucial for preserving test performance while removing watermarks.Pruning rates increase stepwise until model-performance degradation becomes observable.Source: D MORE DISCUSSION ON FINE-PRUNING · p. 14
  • Figure 9 illustrates neuron-pruning effects for OOD-watermarked models on CIFAR-10 and CIFAR-100.The caption identifies the datasets but does not state the outcome of the comparison.Source: Figure 9 · p. 15
  • Across the evaluated datasets and architectures, full REFIT outperforms fine-pruning, while the results are roughly similar.The evaluation covers CIFAR-10 and CIFAR-100 with ResNet-18 and VGG-16.Source: Table 8 · p. 15, D MORE DISCUSSION ON FINE-PRUNING · p. 15
  • Table 8 compares fine-pruning with basic REFIT without EWC or AU on models carrying OOD watermarks.It reports the labeled-training-set proportion and the pruned-neuron ratios for CIFAR-10 and CIFAR-100.Source: D MORE DISCUSSION ON FINE-PRUNING · p. 14, Table 8 · p. 15
Loading 1911.07205v3…