PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Seal: Open and Efficient Video Watermarking

Pierre Fernandez, Hady Elsahar, I. Zeki Yalniz, Alexandre Mourachko

arXiv:2412.09492v1cs.MMcs.AIcs.CV

TL;DR

Existing video watermarking research lacks publicly released models, hindering fair comparison and reproducibility, while generated content raises concerns about misinformation and misuse. Video Seal combines temporal watermark propagation with multistage training; it outperforms strong image baselines on basic geometric transformations and reports training gains in bit accuracy, particularly at higher compression rates.

  • Problem

    Existing video watermarking models have not been publicly released, hindering fair comparisons and reproducibility amid growing concerns about generative content.

    Source: §1 · p. 2

  • Method

    Video Seal combines temporal watermark propagation, which avoids watermarking every frame, with multistage training and differentiable augmentations including H.264 compression.

    Source: §1 · p. 2

  • Results

    Video Seal outperforms strong image baselines in robustness under basic geometric transformations, while multistage training improves bit accuracy particularly at higher compression rates.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Video Seal provides an open foundation for researchers and practitioners to test and iterate on video watermarking.

    Source: §7 · p. 19

    Video imperceptibility remains difficult to evaluate: existing metrics can be high even when Video Seal artifacts may annoy viewers.

    Source: §4.2 · p. 12

Abstract

from arXiv · show

The proliferation of AI-generated content and sophisticated video editing tools has made it both important and challenging to moderate digital platforms. Video watermarking addresses these challenges by embedding imperceptible signals into videos, allowing for identification. However, the rare open tools and methods often fall short on efficiency, robustness, and flexibility. To reduce these gaps, this paper introduces Video Seal, a comprehensive framework for neural video watermarking and a competitive open-sourced model. Our approach jointly trains an embedder and an extractor, while ensuring the watermark robustness by applying transformations in-between, e.g., video codecs. This training is multistage and includes image pre-training, hybrid post-training and extractor fine-tuning. We also introduce temporal watermark propagation, a technique to convert any image watermarking model to an efficient video watermarking model without the need to watermark every high-resolution frame. We present experimental results demonstrating the effectiveness of the approach in terms of speed, imperceptibility, and robustness. Video Seal achieves higher robustness compared to strong baselines especially under challenging distortions combining geometric transformations and video compression. Additionally, we provide new insights such as the impact of video compression during training, and how to compare methods operating on different payloads. Contributions in this work - including the codebase, models, and a public demo - are open-sourced under permissive licenses to foster further research and development in the field.

1 Introduction

AI-generated media and routine editing or compression increase the need for efficient, robust video watermarking, while frame-by-frame image methods face computational and codec-related limits. The paper introduces Video Seal, combining temporal watermark propagation with multistage training and reporting stronger robustness than image baselines under basic geometric transformations.

1 Introduction

  • Generative image and video models raise concerns about misinformation and misuse, while regulators and hardware providers encourage watermarking for generated content and real data.The authors identify fast embedding and detection as part of the need for robust, efficient video watermarking.Source: §1 · p. 2
  • Watermarking every frame is computationally costly for high-resolution, high-frame-rate video, while codecs and editing tools can weaken watermark signals and challenge image watermarking models.Chunking can aid parallelization, but large clips exceed memory limits and smaller clips introduce synchronization issues.Source: §1 · p. 2
  • Prior neural video watermarking approaches include compression simulation and image-model adaptation, but most existing models target low-resolution video or short clips, and their lack of public release hinders reproducibility.The passage gives 128×128 resolution and 64-frame clips as examples of these constraints.Source: §1 · p. 2
  • Video Seal uses temporal watermark propagation to avoid watermarking every frame, alongside image pre-training, hybrid post-training, extractor fine-tuning, and differentiable augmentations.The augmentations include video codecs such as H.264.Source: §1 · pp. 2–3
  • Video Seal outperforms MBRS, TrustMark, and WAM in robustness under basic geometric transformations, although MBRS and TrustMark have higher message capacities.The cited passage identifies cropping, small rotations, and perspective changes as the transformations; it reports capacities of 256 bits for MBRS and 100 bits for TrustMark.Source: §1 · p. 2
  • Extractor fine-tuning improves bit accuracy and robustness without compromising PSNR, while image pre-training followed by video training with differentiable compression is especially effective at higher compression rates.The authors report these findings from ablations of training components.Source: §1 · p. 3
  • The authors release model checkpoints, training and evaluation code, and a demo under a permissive license to support research and testing.They describe the codebase as comprehensive and easy to use.Source: §1 · p. 3

2 Method

Video Seal extends an image embedder–extractor into video using streamable 2D processing, temporal watermark propagation, and multistage training.

2 Method

  • The framework extends an image embedder–extractor to video, operating in 2D for streamability, simpler extraction, and flexibility across resolutions and video lengths.The same mechanism supports both images and videos.Source: §2 · p. 3

2.1 Embedder & extractor architectures

  • A 16M-parameter U-Net embeds 96 bits through a bottleneck lookup table, while a 24M-parameter ViT extractor maps frame features to 96 soft bits.The lookup table maps each binary bit to an embedding vector; the extractor’s outputs can be thresholded to recover a hard message.Source: §2.1 · p. 3, §2.1.1 · pp. 3–4, §2.1.2 · p. 4

2.2.1 High-resolution and scaling factor

  • For higher-resolution video, frames are downscaled to 256 × 256 for embedding, then the watermark distortion is upscaled and added to the original frame.The scaling factor controls watermark strength and can trade quality for robustness at inference.Source: §2.2.1 · p. 5, §2.2 · p. 5

2.2.2 Temporal watermark propagation

  • Temporal watermark propagation embeds every k-th frame and copies its distortion to neighboring frames, reducing per-frame computation while larger k can add extraction noise and motion-related shadow artifacts.The paper uses k = 4 and notes that propagation is fully differentiable.Source: §2.2.2 · pp. 5–6

2.2.3 Extraction

  • Extraction processes frames independently, averages their soft messages, and thresholds the result; using fewer frames can speed extraction, while using more may increase robustness.The averaging scheme is selected for simplicity and speed.Source: §2.2.3 · p. 6

2.3.1 Training objectives

  • Joint training combines message-reconstruction loss, image MSE, and adversarial quality loss, while a separate dual-hinge update trains the discriminator against original and watermarked images.The hinge function penalizes incorrect classifications.Source: §2.3.1 · pp. 6–8

2.3.2 Multistage training

  • Training first pre-trains on images, then alternates images and videos; afterward, the extractor is fine-tuned with the embedder frozen to improve robustness without changing watermark imperceptibility.The hybrid schedule is described as more stable and as improving bit accuracy and robustness at higher compression rates.Source: §2.3.2 · p. 8

2.3.3 Transformations

  • Training spans frame-level value and geometric distortions plus H.264/H.265 codecs; a straight-through estimator approximates gradients through non-differentiable transformations.The estimator approximates the transformation’s gradient with the identity function.Source: §2.3.3 · p. 9

3 Experimental Setup and Implementation Details

The setup defines quality and robustness metrics, describes image and video data and staged training, and specifies baseline adaptation and evaluated transformations.

3.1 Metrics

  • Video quality is assessed with PSNR, SSIM, and LPIPS, while VMAF is designed to assess video quality with temporal consistency in view.SSIM and LPIPS are computed frame-wise and averaged over each video.Source: §3.1 · p. 9
  • Bit accuracy alone ignores payload size: the paper contrasts 0.9 accuracy at 128 bits with 0.99 at 64 bits as having greater total capacity.The comparison is presented in information-theoretic terms.Source: §3.1 · p. 10
  • Robustness uses bit accuracy, supplemented by log10(p) to express the chance-level significance of an observed accuracy and set detection thresholds.For FPR < 10^-6, the stated threshold is log10(p) < -6.Source: §3.1 · pp. 9–10

3.2 Datasets

  • Image training uses 500k SA-1B images at 256 × 256; video training samples 1.3-second, 32-frame clips from SA-V.Evaluation uses 1k images at original resolution and the first 5 seconds of videos unless stated otherwise.Source: §3.2 · p. 10

3.3 Training

  • Training uses 800 image-only epochs followed by 300 hybrid epochs; at epoch 250, the embedder is frozen and only the extractor is optimized.The schedule uses SA-1B images first, then adds SA-V video clips.Source: §3.3 · p. 10

3.4 Baselines

  • Five image-watermarking models are adapted as video baselines; their payloads range from 30 to 256 bits, versus Video Seal's 96 bits.Video evaluation applies the same every-fourth-frame embedding and propagation strategy, with soft extraction predictions averaged across frames.Source: §3.4 · p. 11

3.5 Evaluated transformations

  • Evaluated transformations are grouped as none, geometric, valuemetric, compression, and combined; combined examples apply compression, cropping, and brightness changes sequentially.Geometric examples include rotations, crops, and perspective changes; valuemetric examples include brightness, contrast, and saturation changes.Source: §3.5 · p. 11

4 Results

Video Seal is most robust under combined geometric transformations and video compression, while imperceptibility and latency vary across methods. Its robustness advantage persists in the imperceptibility–robustness trade-off, especially at high quality levels.

4.1 Robustness

  • Image watermarking models often provide strong video baselines, but MBRS and CIN are especially weak against geometric transformations.Their training augmentations do not change image geometry, while their robustness to valuemetric changes and video codecs is strong.Source: §4.1 · pp. 11–12
  • Under H.264 compression, brightness adjustment, and cropping, Video Seal achieves log10(p) = −6.1 on average, corresponding to detection at false-positive rates below 10−6.The robustness evaluation also compares bit accuracy and log10(p) across models and transformations.Source: Table 4 · p. 12, §4.1 · p. 12

4.2 Imperceptibility

  • Video Seal achieves the highest PSNR and SSIM, while MBRS leads VMAF and TrustMark leads LPIPS, closely followed by Video Seal.Video imperceptibility is not fully captured by these examples and metrics; artifacts can remain visually annoying and depend on the cover video.Source: §4.2 · p. 12, Table 5 · p. 13

4.3 Latency

  • Embedding requires 10–43 GFlops across models, while extraction requires 3–69 GFlops; WAM is slowest at GPU embedding.Other models take around 0.5–2 seconds on CPU, and CPU-to-GPU transfer and high-resolution CPU operations appear to bottleneck GPU timing.Source: Table 6 · p. 13, §4.3 · p. 14

4.4 Imperceptibility/Robustness trade-off

  • Video Seal is more robust under combined geometric transformations and video compression, particularly at PSNR above 50 dB or VMAF above 94.MBRS and TrustMark obtain higher −log10(p) over a good VMAF range, partly because they hide more bits.Source: §4.4 · p. 14, Figure 6 · p. 14

5 Ablation Studies

The ablations favor image pre-training followed by video training with compression augmentation, and show that extractor fine-tuning can improve robustness without compromising PSNR. Extraction aggregation methods have similar performance, while propagation step size and extraction frame count affect speed and accuracy differently.

5.1 Video training

  • Image pre-training followed by video training with compression augmentation raises bit accuracy rapidly, especially at stronger compression, without reducing PSNR; video-only training yields very low bit accuracy.The improvement is especially noted for CRF 40 and 50.Source: §5.1 · pp. 15–16, Figure 7 · p. 15

5.2 Extractor fine-tuning

  • Extractor fine-tuning increases average bit accuracy without compromising PSNR, and results suggest starting with all augmentations, including compression.The extractor is fine-tuned after the generator is frozen.Source: §5.2 · pp. 16–17, Figure 8 · p. 16

5.3 Video inference parameters

  • Temporal propagation step size has little effect on robustness but can increase embedding speed, while the number of extraction frames more strongly affects both extraction speed and performance.A larger step size sometimes introduces shadow or glitter artifacts.Source: Figure 9 · p. 17
  • Aggregation methods do not significantly change extraction performance, although L1 average shows a small, non-significant improvement.The comparison uses bit accuracy and log10(p) on 3-second SA-V videos.Source: Table 7 · p. 18, §5.3 · p. 18

6 Related Work

Related work spans traditional codec-based watermarking, deep video methods, and image watermarking architectures adapted to video. The paper situates Video Seal alongside these approaches and notes reproducibility and efficiency concerns in prior methods.

6 Related Work

  • Traditional video watermarking embeds data by modifying reversible variable-length codes or motion vectors in codec-based systems.Motion-vector methods alter vectors, typically those with larger magnitudes, while RVLC methods aim to restore the original video after extraction.Source: §6 · p. 18
  • Deep video watermarking methods vary in robustness and efficiency: VStegNet lacks distortion and compression robustness, while RivaGan's 4D tensors raise usability concerns.RivaGan uses a DCT noise layer to mimic H.264 compression, but its trained models are not released.Source: §6 · p. 18
  • DVMark and VHNet use trainable compression simulators, while ItoV adapts image models by merging time with channels but still processes whole videos at once.ItoV reports H.264 robustness at CRF=22, but lacks reproducibility assets; other methods include frequency-domain and camera-distortion approaches.Source: §6 · pp. 18–19
  • Image watermarking research includes deep-learning methods and AI-generation-specific approaches; Video Seal is post-hoc, requiring watermarking after generation and offering more flexibility but less security.The paper also reviews spatial- and frequency-domain image watermarking methods.Source: §6 · p. 19

7 Conclusion

Video Seal offers an open foundation for video-watermarking research while identifying unresolved challenges in evaluation and watermark design.

7 Conclusion

  • Video Seal is presented as a comprehensive, efficient video-watermarking framework and an open foundation for researchers and practitioners.The conclusion says it highlights open challenges in the field.Source: §7 · p. 19
  • The conclusion identifies better imperceptibility metrics and training objectives as open challenges.It also names visual consistency across watermarked frames and embedding in video-compression-suited domains such as YUV or YCbCr as future-work directions.Source: §7 · p. 19
  • Future work could also increase payload and watermark robustness and explore the framework's security.Source: §7 · p. 19

A.1 Comparing at different payloads

The section proposes expected capacity and a binomial-based p-value for comparing watermark methods at different payload sizes. Its plotted examples show how bit accuracy and payload jointly shape these measures.

A.1 Comparing at different payloads

  • Bit accuracy alone omits payload size, motivating two comparison metrics for models operating at different payloads.Source: §A.1 · p. 24
  • Expected capacity treats each bit as a binary symmetric channel and estimates theoretically transmittable bits from bit accuracy and payload.The section defines each bit's error probability as 1 minus the observed bit accuracy.Source: §A.1 · p. 24
  • The p-value is the probability, under the null hypothesis of independent fair output bits, of observing bit accuracy at least as extreme as measured.It is expressed using the binomial cumulative distribution function and the regularized incomplete Beta function.Source: §A.1 · p. 24
  • A 0.9 bit accuracy at 64 bits is approximately equivalent to 0.8 at 128 bits in expected capacity or p-value.The figure plots expected capacity and log2 p-value against bit count and bit accuracy; both measures follow the same trend, with p-value discontinuities from the discrete binomial distribution.Source: §A.1 · p. 24, Figure 10 · p. 24
  • The section's p-value and capacity are conceptual tools for binary-message transmission, not results of a statistical hypothesis test.Source: §A.1 · p. 25

B.1 More qualitative results

The appendix adds qualitative examples of watermarked images and video frames at their original resolution.

B.1 More qualitative results

  • The cited section presents additional examples of watermarked images from SA-1b in Fig. 11; it also begins a reference to watermarked material in Fig. 12.Source: §B.1 · p. 25
  • It also presents frames from videos from SA-V, extending the results shown in Fig. 5.Source: §B.1 · p. 25
  • These examples extend the qualitative results shown in Figure 5.Source: §B.1 · p. 25

B.2 Full robustness results

The appendix provides full robustness results across three datasets and describes the metrics and transformation groups used to report them.

B.2 Full robustness results

  • Robustness results cover SA-1b, COCO, and SA-V, with tables providing full extraction results for each dataset.Source: §B.2 · p. 25, Table 8 · p. 28, Table 9 · p. 29
  • Results report bit accuracy and log10(p) against PSNR for each transformation type.The reported categories are identity, value-metric changes, compression, geometric transforms, and combined compression, crop, and brightness changes.Source: §B.2 · p. 25
  • Additional qualitative figures show SA-1b images and SA-V video frames at approximately 2k × 1k original resolution.Source: Figure 11 · p. 26, Figure 12 · p. 27

Related papers

Loading 2412.09492v1…