Source-linked AI summary
DVMark: A Deep Multiscale Framework for Video Watermarking
Xiyang Luo, Yinxiao Li, Huiwen Chang, Ce Liu, Peyman Milanfar, Feng Yang
TL;DR
Video watermarking must balance imperceptibility, robustness, and payload, while existing approaches can struggle with diverse distortions and video’s temporal correlations. DVMark uses multiscale message embedding and a differentiable proxy for video compression; the authors report greater robustness across a wide range of distortions than traditional watermarking and deep image-watermarking methods. They also test the framework in a video-editing application.
Problem
Video watermarking must balance imperceptibility, robustness, and payload, while handcrafted methods often do not cover a broad range of distortions and image models do not fully use video’s temporal correlations.
Source: §1 · pp. 1–2Method
DVMark is an end-to-end trainable video-watermarking framework that embeds messages across spatial-temporal scales and models video compression with a differentiable proxy.
Source: §1 · p. 2Results
DVMark is reported to be more robust than deep image-watermarking and traditional methods across a wide range of challenging video distortions while maintaining good perceptual quality.
Source: §5 · p. 8Takeaways & Limitations
The authors report practical evaluations across multiple video resolutions and lengths and in a video-editing application, where a detector locates watermarked frames among unwatermarked content.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Video watermarking embeds a message into a cover video in an imperceptible manner, which can be retrieved even if the video undergoes certain modifications or distortions. Traditional watermarking methods are often manually designed for particular types of distortions and thus cannot simultaneously handle a broad spectrum of distortions. To this end, we propose a robust deep learning-based solution for video watermarking that is end-to-end trainable. Our model consists of a novel multiscale design where the watermarks are distributed across multiple spatial-temporal scales. It gains robustness against various distortions through a differentiable distortion layer, whereas non-differentiable distortions, such as popular video compression standards, are modeled by a differentiable proxy. Extensive evaluations on a wide variety of distortions show that our method outperforms traditional video watermarking methods as well as deep image watermarking models by a large margin. We further demonstrate the practicality of our method on a realistic video-editing application.
1. Introduction
Video watermarking must balance imperceptibility, robustness, and payload, while existing handcrafted and image-based approaches struggle to cover diverse distortions and use video’s temporal information. DVMark addresses these challenges with multiscale embedding, a differentiable compression proxy, and a detector for locating watermarked frames in edited videos.
1. Introduction
- Imperceptibility, robustness, and payload are in tension: improving robustness can lower visual quality, while increasing payload can affect decoding accuracy and quality.Source: §1 · p. 1
- Handcrafted methods often target particular distortions, and image watermarking models do not fully exploit video’s spatial-temporal information, limiting their robustness-quality-payload trade-off.Video compression is also non-differentiable, complicating end-to-end training.Source: §1 · pp. 1–2
- DVMark is reported to outperform previous baselines across robustness, quality, capacity, and video distortions, with practicality tested across resolutions, lengths, and a video-editing application.The paper also describes a differentiable proxy for improving robustness to video compression.Source: §1 · p. 2
- DVMark embeds messages across multiple spatial-temporal scales in its encoder and decoder, a design reported to improve robustness over a single-scale network.Source: §1 · p. 2
- A differentiable proxy is proposed to improve robustness to video compression within the deep video watermarking framework.Source: §1 · p. 2
- A watermark detector is added to locate watermarked frames precisely when edited videos contain a mixture of watermarked and unwatermarked content.Source: §1 · p. 2
2. Related Work
Video watermarking research includes methods that modify pixels, work in a codec’s compressed domain, or transform the video before embedding. Deep learning approaches offer another direction, but image watermarking models do not use temporal correlations between video frames.
2. Related Work
- Traditional video watermarking is classified into spatial-, compressed-, and transform-domain methods according to the domain in which each operates.Source: §2 · p. 2
- Spatial methods modify pixel values, while compressed-domain methods can be codec-specific; for example, an H.264-compatible DCT method does not support conversion using another codec.Source: §2 · p. 2
- Transform-domain methods use techniques including DFT, DCT, DWT, and DT-CWT, with perceptual masking used to improve perceptual quality in some traditional methods.Source: §2 · p. 2
- Deep image-watermarking methods include HiDDeN, RedMark, and models of camera-capture or screen-display distortions, but applying image methods to video leaves temporal correlations unused.Source: §2 · pp. 2–3
- VStegNet addresses video steganography rather than video distortions, which can affect both watermark robustness and perceptual quality.Source: §2 · p. 3
- GAN-based training is another technique for perceptual quality in image and video processing, with cited examples including ESRGAN, TecoGAN, and MOCOGAN.Source: §2 · p. 3
3. Proposed Method
DVMark is an end-to-end video watermarking framework with multiscale encoder and decoder components, trained with distortions and a video discriminator. Its training objective combines video quality, message decoding, and discriminator losses.
3. Proposed Method
- The proposed framework comprises an encoder, decoder, distortion layer, and video discriminator for end-to-end embedding of a binary message into video frames.The message has length m and is embedded across n video frames.Source: §3 · p. 3
3.1. Encoder
- The encoder embeds messages at two spatial-temporal scales, using transformed video features and a learned residual added to the cover video.The residual is scaled by α, which controls the trade-off between quality and robustness; the transform layer learns a transformation before message fusion.Source: §3.1 · p. 3
3.2. Decoder
- The decoder uses multiple scale-specific heads and per-video WeightNet weights to combine decoded messages, plus a head that detects watermarked frames.Its transform layer maps the distorted video to features shared by the decoder and detector heads.Source: §3.2 · pp. 3–4, Figure 3 · p. 4
3.3. Distortion Layer
- Training randomly selects among temporal and spatial distortions and a differentiable video-compression proxy to improve robustness across distortions.CompressionNet is trained to mimic H.264 at CRF 25, reaches 33.3 dB PSNR against real compressed output, and is then fixed during watermark-model training.Source: §3.3 · pp. 4–5
3.4. Video Discriminator
- A multiscale discriminator uses four 3D residual networks at different spatial and temporal resolutions; 3D-convolution models show less flickering than models using only 2D convolutions.The discriminator optimizes its own hinge loss, while the generator loss is added to the encoder-decoder objective.Source: §3.4 · p. 5, Figure 4 · p. 5
3.5. Training
- DVMark jointly trains its encoder and decoder with pixel-wise video loss, sigmoid cross-entropy message loss, and a video-discriminator loss.The detector is trained separately on a pretrained, fixed encoder and decoder, with watermarking randomly applied to clips at probability 0.5.Source: §3.5 · p. 5
4. Experiments
Experiments compare DVMark with traditional 3D-DWT and deep image watermarking HiDDeN, then test quality, robustness–quality–payload trade-offs, larger-video performance, and detector-assisted editing. Across these tests, DVMark performs strongly across distortions while maintaining visual quality and supporting localization of watermarked content.
4.2. Method Evaluation
- DVMark outperforms 3D-DWT and HiDDeN by a large margin on almost all tested distortions, with gains consistent across distortion strengths.The comparison covers 500 random videos and common distortions; DVMark matches HiDDeN through a 50% frame-drop rate.Source: §4.2 · p. 6, Table 1 · p. 7, Figure 6 · p. 7
- DVMark has superior visual quality to HiDDeN and 3D-DWT under the reported quality metrics and user study.Quality is assessed using PSNR, MSSIM, LPIPS, temporal consistency (tLP), and a mean-opinion score from raters.Source: §4.2 · p. 6, Table 2 · p. 7
4.3. Robustness-Quality-Payload Trade-off
- DVMark is superior to both baselines across robustness–quality and robustness–payload curves.The first comparison fixes payload at m = 96 while varying watermark strength; the second targets an average PSNR of around 37 dB across separately trained payloads.Source: §4.3 · p. 7, Figure 7 · p. 7, Figure 8 · p. 7
4.4. Performance on Larger Videos
- Across larger video resolutions and lengths, DVMark shows no general performance degradation in average decoding accuracy over four distortions.The reported averages cover H.264, crop, frame drop, and Gaussian noise.Source: §4.4 · pp. 7–8, Table 3 · p. 8
4.5. Detector in a Video Editing Application
- The trained detector distinguishes watermarked from unwatermarked sequences with high accuracy despite good perceptual quality.In the editing test, it localizes the watermarked segment, and detector pre-filtering improves decoding compared with passing the full video to the decoder.Source: §4.5 · p. 8, Table 4 · p. 8, Table 5 · p. 8
- The editing test inserts 16 frames of cropped, saturation-adjusted watermarked source into a much longer unwatermarked background video.The detector filters frames below a confidence score of 0.3 before decoding.Source: §4.5 · p. 8, Table 5 · p. 8
5. Conclusion
The conclusion presents DVMark as an end-to-end trainable video watermarking framework that distributes messages across spatial-temporal scales. It reports stronger robustness across challenging video distortions than traditional and deep-image methods while maintaining perceptual quality, and describes detector-based localization and practical evaluations.
5. Conclusion
- DVMark combines multiscale message embedding, broad distortion robustness, and good perceptual quality in an end-to-end trainable framework.The framework also adapts to other distortion requirements by adding a distortion to the end-to-end pipeline.Source: §5 · p. 8
- The framework adds a watermark detector to locate watermarked video segments among unwatermarked content.Source: §5 · p. 8
- Evaluations on varied distortions and applications support the authors’ conclusion that DVMark is a practical and reliable video-watermarking solution.Source: §5 · p. 8
6. Additional Design and Experiment Details
The additional details describe training distortions and settings, the user-study procedure, and architectures for WeightNet, the video GAN discriminator, and CompressionNet.
6.1. Training Details
- Training samples ten differentiable distortions with equal probability, trains the encoder-decoder for 3M steps, and uses symmetric padding to reduce boundary artifacts and improve temporal stability.The detector is trained for 1M steps on 8 × 128 × 128 clips; the user study asks raters to judge side-by-side clips as identical or different after repeated viewing if needed.Source: §6.1 · p. 11, §6.2 · p. 11
6.3. WeightNet Architecture
- WeightNet uses three stride-2 3D convolutions, global averaging, a fully connected layer, and row-normalized softmax to produce its weight matrix.Its convolution channels are 32, 64, and 128, and the fully connected layer has m × 2 units with m = 96.Source: §6.3 · p. 11
6.4. Video GAN Architecture
- The video GAN discriminator has four scale-specific residual 3D-convolution subnetworks, each followed by global pooling and a scalar-output fully connected layer.Each subnetwork contains four residual layers.Source: §6.4 · p. 11, Table 6 · p. 11
6.5. CompressionNet Architecture
- CompressionNet is a seven-layer 3D-convolution proxy trained to mimic H.264 output at CRF 25 on 8 × 128 × 128 clips.Each layer uses a kernel size of 3, and a residual connection links the input to the final layer.Source: §6.5 · pp. 11–12, Figure 10 · p. 11
7. Additional Detailed Results on Robustness, Quality, Payload
The supplementary results provide detailed values for robustness–quality and robustness–payload trade-offs, then examine how video resolution and length affect decoding accuracy under several distortions.
7.1. Robustness-Quality Tradeoff
- Tables 7–9 provide PSNR and decoding-accuracy values for DVMark, HiDDeN, and 3D-DWT across four distortions in the robustness–quality analysis.Source: §7.1 · p. 12, Table 7 · p. 12, Table 8 · p. 12
7.2. Robustness-Payload Tradeoff
- Tables 10–12 report decoding accuracy across message lengths and four distortions for HiDDeN, 3D-DWT, and DVMark.Source: §7.2 · p. 12, Table 10 · p. 12, Table 11 · p. 12
7.3. Performance on Larger Videos
- Increasing spatial resolution generally improves decoding accuracy for crop and frame drop, is neutral for Gaussian noise, and slightly reduces it at the largest resolution for H.264; longer videos improve accuracy across resolutions and distortions.The resolution effect reflects a trade-off between added spatial redundancy and fewer high-frequency regions.Source: §7.3 · pp. 12–13, Table 14 · p. 12, Table 13 · p. 12
8. Ablation Studies
The ablations examine clip-dependent decoding, multiscale network design, loss weighting, and compression-specific robustness. They connect decoding variation to video content and design choices.
8.1. A Study of Per-video Robustness
- Across 200 eight-frame clips, most decoding accuracy exceeded 90%, while clips with less high-frequency content and less motion performed worse.Gaussian noise with standard deviation 0.06 was applied to make decoding more challenging; the low-accuracy examples are shown in Figure 12.Source: §8.1 · p. 13, Figure 12 · p. 13
8.2. Network Architecture
- The multiscale components improve overall robustness across distortions compared with the network-design variations.Table 17 reports decoding accuracy for the design choices, including versions with one decoder head removed and both heads summed without weighting.Source: §8.2 · p. 13, Table 17 · p. 13
8.3. Loss Function Weights
- The message-loss weight c1 controls the quality–robustness trade-off; Table 18 compares decoding accuracy and PSNR across c1 settings.Accuracy is averaged over Identity, H.264, and Gaussian Noise distortions.Source: §8.3 · p. 13, Table 18 · p. 14
- Lower GAN-loss weight c2 tends to produce noisier, less aesthetically pleasing watermarks.Figure 13 shows that frames trained without GAN loss (c2 = 0.0) are much noisier visually.Source: §8.3 · p. 13, Figure 13 · p. 14
8.4. CompressionNet
- Gaussian blurring, frequency truncation, and differentiable JPEG provide some robustness to H.264 compression, but are less effective than CompressionNet.The compared alternatives include 2D and 3D Gaussian blur and differentiable JPEG with quality factor 50 and chroma subsampling.Source: §8.4 · p. 14, Table 19 · p. 14
9. Visual Examples
Visual examples compare original and watermarked frames across methods, with zoomed views and difference maps offered as aids for inspection.
9. Visual Examples
- The displayed difference images are scaled to 0–1, so their magnitudes do not represent absolute differences between frames.The difference maps indicate where information is embedded in the signal.Source: §9 · p. 14, Figure 14 · p. 15
- The difference maps are visualizations of where watermark information is embedded, not measures of its absolute magnitude.The examples cover original and watermarked frames from multiple methods.Source: §9 · p. 14, Figure 15 · p. 15, Figure 16 · p. 16
- Figures 14–19 show original and watermarked frames from several methods, with a zoomed-in view in the second row.The figures provide six sets of visual samples.Source: Figure 14 · p. 15, Figure 15 · p. 15, Figure 16 · p. 16