PaperlayerView exact revision on arXiv

Source-linked AI summary

Safe-Sora: Safe Text-to-Video Generation via Graphical Watermarking

Zihan Su, Xuerui Qiu, Hongbin Xu, Tangyu Jiang, Junhao Zhuang, Chun Yuan, Ming Li, Shengfeng He, Fei Richard Yu

arXiv:2505.12667v2cs.CV

TL;DR

Video-generation watermarking remains relatively unexplored, particularly for graphical marks. Safe-Sora embeds graphical watermark patches into video through similarity-based matching and a 3D wavelet-enhanced Mamba architecture; the authors report state-of-the-art performance across video quality, watermark fidelity, and robustness.

  • Problem

    Graphical watermarking within video-generation pipelines is underexplored, despite the need for copyright protection of generated video content.

    Source: §1 · p. 2

  • Method

    Safe-Sora matches watermark patches to visually similar video frames and regions, then uses a 3D wavelet-enhanced Mamba architecture for spatiotemporal embedding and retrieval.

    Source: §1 · p. 2

  • Results

    Safe-Sora achieves state-of-the-art video quality, watermark fidelity, and robustness, with a Fréchet Video Distance of 3.77 versus the second-best baseline’s 154.35.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Safe-Sora establishes a foundation for copyright protection in generative video and opens avenues for applying state space models to digital watermarking.

    Source: §5 · p. 9

    The method currently supports static image-based watermarks such as logos or icons; embedding animated or temporally dynamic patterns remains a challenge.

    Source: E Limitations · p. 16

Abstract

from arXiv · show

The explosive growth of generative video models has amplified the demand for reliable copyright preservation of AI-generated content. Despite its popularity in image synthesis, invisible generative watermarking remains largely underexplored in video generation. To address this gap, we propose Safe-Sora, the first framework to embed graphical watermarks directly into the video generation process. Motivated by the observation that watermarking performance is closely tied to the visual similarity between the watermark and cover content, we introduce a hierarchical coarse-to-fine adaptive matching mechanism. Specifically, the watermark image is divided into patches, each assigned to the most visually similar video frame, and further localized to the optimal spatial region for seamless embedding. To enable spatiotemporal fusion of watermark patches across video frames, we develop a 3D wavelet transform-enhanced Mamba architecture with a novel spatiotemporal local scanning strategy, effectively modeling long-range dependencies during watermark embedding and retrieval. To the best of our knowledge, this is the first attempt to apply state space models to watermarking, opening new avenues for efficient and robust watermark protection. Extensive experiments demonstrate that Safe-Sora achieves state-of-the-art performance in terms of video quality, watermark fidelity, and robustness, which is largely attributed to our proposals. Code is publicly available at https://github.com/Sugewud/Safe-Sora

1 Introduction

Safe-Sora is presented as the first framework to embed graphical watermarks directly into video generation. It matches watermark patches to visually similar video frames and regions, then uses a 3D wavelet-enhanced Mamba architecture for spatiotemporal watermark processing.

1 Introduction

  • A 1,000-pair image-hiding study measures watermark-cover similarity with 1-LPIPS and watermarked-image and extracted-watermark quality with PSNR.The study uses Balujanet with Logo-2k watermarks and ImageNet cover images; higher PSNR and lower LPIPS indicate improved performance.Source: Figure 1 · p. 2
  • The framework assigns watermark patches to visually similar video frames and then localizes each patch within its selected frame.The authors motivate this coarse-to-fine matching by an observed relationship between watermark-cover similarity and watermarking performance.Source: §1 · p. 2
  • A 3D wavelet transform-enhanced Mamba architecture with tailored scanning models long-range dependencies across space and time for watermark embedding and retrieval.The authors describe this as the first application of state space models to generative watermarking.Source: §1 · p. 2
  • Safe-Sora achieves state-of-the-art video quality, watermark fidelity, and robustness, with a Fréchet Video Distance of 3.77 versus the second-best baseline’s 154.35.The authors attribute the overall performance in part to their proposals and report ablations supporting their effectiveness.Source: §1 · p. 2

2 Related Work

Prior work covers latent video diffusion, video watermarking, and state space models, but leaves graphical watermarking in video diffusion and state space models for watermarking unexplored.

2.1 Video Diffusion Models

  • Safe-Sora’s overview comprises patch matching, watermark embedding with 2D and 3D SFMamba blocks, and extraction with a distortion layer and position recovery.The figure caption says Mamba block types differ in their scanning strategies.Source: Figure 2 · p. 3
  • Latent video diffusion models perform diffusion in latent space for computational efficiency, while prior video diffusion frameworks had not explored integrating graphical watermarks.Examples discussed include VideoCrafter2 and Open-Sora.Source: §2.1 · pp. 2–3

2.2 Generative Video Watermarking

  • VideoShield and LVMark embed low-capacity binary strings, leaving richer graphical watermarks unaddressed in generative video watermarking.VideoShield modifies latent noise; LVMark uses importance-based weight modulation to limit visual quality degradation.Source: §2.2 · p. 3

2.3 State Space Models

  • Mamba uses selective state space modeling with data-dependent parameters, but its potential for watermarking had remained unexplored.The passage describes state space models as efficient alternatives to transformers for sequence modeling.Source: §2.3 · p. 3

3 Graphical Watermarking for Video Generation

Safe-Sora embeds graphical watermarks in the video-generation pipeline by matching watermark patches to similar video frames and regions, then fusing them with video features. Its pipeline uses spatial-frequency Mamba blocks for fusion and extraction, with separate video and watermark reconstruction losses.

3.1 Coarse-to-Fine Adaptive Patch Matching

  • Patches are matched coarse-to-fine to similar frames and regions, with frame-capacity redirection and binary position channels supporting later patch localization.The method computes similarities from patch and latent-frame features; using a single convolution for feature extraction reduces computational overhead. During training, correct positions are supplied, while testing uses embedded position channels to predict patch arrangement.Source: §3.1 · p. 4

3.2 Spatial-Frequency Mamba for Spatial Fusion

  • The 2D SFMamba block combines spatial and wavelet-frequency branches, adds a residual connection to the spatial output, then concatenates and projects both branches.The frequency branch applies a 2D wavelet transform and processes its subbands before inverse transformation and element-wise fusion.Source: §3.2 · pp. 4–5

3.3 3D Frequency Scanning for Spatiotemporal Interaction

  • 3D SFMamba uses spatial and frequency branches; its 3D wavelet frequency branch scans subbands in both low-to-high and high-to-low orders, spatially then temporally.The strategy rearranges eight subbands to restore video resolution, divides them into eight parts, and scans each part; the 3D spatial branch scans temporal, height, and width dimensions.Source: §3.3 · pp. 5–6, Figure 3 · p. 5

3.4 Training Objectives

  • Training combines video reconstruction loss and watermark reconstruction loss, using correct patch positions during training and embedded position channels to predict patch arrangement during testing.The video loss uses mean squared error to keep the watermarked video close to the original, and the watermark weighting hyperparameter balances video quality against watermark fidelity.Source: §3.4 · p. 6

4 Experiments

Experiments compare Safe-Sora with retrained watermarking baselines, evaluate quality under distortions, and ablate its adaptive matching and spatiotemporal scanning components. The reported results show strong benchmark performance and contributions from both components.

4.1 Experimental Setting

  • Training uses 10,000 Panda-70M videos and Logo-2K watermarks; evaluation uses 100 VidProm prompts and five retrained baselines.VideoCrafter2 is the backbone; videos are generated at 320 × 512 resolution, and training uses eight sampled frames per video.Source: §4.1 · p. 6

4.2 Comparison with State-of-the-art Methods

  • Safe-Sora produces watermarked videos with minimal visual differences and recovered watermarks that closely resemble the originals.The qualitative comparisons also report visible artifacts, stripe-like distortions, color shifts, or degradation for several baselines.Source: §4.2 · p. 7, Figure 4 · p. 7, Figure 5 · p. 8
  • FVD 3.77 significantly outperforms all baselines, and Safe-Sora achieves state-of-the-art results across the reported evaluation metrics.The evaluation covers watermark fidelity, video quality, and temporal consistency.Source: §4.2 · pp. 7–8, Table 1 · p. 8

4.3 Robustness

  • Across erasing, blur, noise, rotation, and H.264 attacks, Safe-Sora performs best; under H.264, it maintains high watermark quality as baselines decline.The evaluation uses PSNR, SSIM, and LPIPS to assess reconstruction quality.Source: §4.3 · p. 8, Figure 6 · p. 9

4.4 Ablation Study

  • CFAPM improves watermark fidelity and video quality by matching patches using video latents and refining their spatial locations.The authors attribute more accurate matching to compressed semantic information in the latent space.Source: §4.4 · pp. 8–9, Table 2 · p. 9
  • SLS improves watermark and video quality; temporal-first scanning slightly improves tLP but underperforms on watermark-fidelity metrics.The proposed traversal orders 3D-wavelet subbands hierarchically and scans spatially before temporally within each subband.Source: Table 2 · p. 9, §4.4 · p. 9

5 Conclusion

Safe-Sora introduces direct graphical watermark embedding into generated video, combining adaptive patch matching with a wavelet-enhanced Mamba architecture. Experiments report superior video quality, watermark fidelity, and robustness, positioning the work as a foundation for generative-video copyright protection.

5 Conclusion

  • Safe-Sora is presented as the first framework to embed graphical watermarks directly into generated video.Source: §5 · p. 9
  • Its adaptive matching strategy maps watermark patches to visually similar video frames and spatial regions.Source: §5 · p. 9
  • A 3D wavelet-enhanced Mamba architecture with spatiotemporal local scanning models dependencies for watermark embedding and retrieval.The paper describes this as a pioneering application of state space models to watermarking.Source: §5 · p. 9
  • Experiments report superior video quality, watermark fidelity, and robustness, establishing a foundation for copyright protection in generative video.Source: §5 · p. 9

Technical Appendices

The appendix describes Safe-Sora’s watermark-recovery scenario and outlines latent video diffusion, which generates videos by denoising video latents conditioned on text prompts.

Technical Appendices

  • Safe-Sora embeds a model owner’s graphical watermark during video generation and can recover it after distortion to verify authenticity and protect copyright.Source: Figure 7 · p. 13

A.1 Latent Video Diffusion Models

  • Latent video diffusion models operate in compressed latent space to improve computational efficiency while maintaining generation quality.Source: §A.1 · p. 13
  • A video encoder maps the input video x to a latent z with fewer spatial dimensions, while retaining the frame dimension.Source: §A.1 · p. 13
  • The diffusion process adds noise to latents, then a trained denoiser reverses it from Gaussian noise; a text encoder conditions denoising on the prompt.Source: §A.1 · p. 13

A.2 State Space Models

This section introduces state-space and Mamba sequence modeling, then describes confidence-guided recovery of watermark patch positions.

A.2 State Space Models

  • State-space models are presented as continuous dynamical systems, with x(t), h(t), and y(t) denoting input, hidden state, and output, and A, B, C, D as system parameters.Source: §A.2 · p. 13
  • For discrete sequence modeling, the continuous equations are discretized, with Ā and B̄ identified as discretized versions of A and B.Source: §A.2 · p. 13
  • Watermark position recovery decodes patch positions and confidence, assigns conflicts to the higher-confidence patch, then places unassigned patches at nearest vacant positions in confidence order.Source: §A.2 · p. 14
  • Mamba uses input-dependent parameters to adapt processing to input content, with its state update and output defined by h_t = Ā ⊙ h_{t−1} + B̄ ⊙ x_t and y_t = C h_t.Source: §A.2 · p. 14

A.3 Wavelet Transforms

The section describes multiscale wavelet representations for video watermarking and a confidence-guided algorithm for recovering patch positions when predictions conflict.

A.3 Wavelet Transforms

  • Wavelet transforms provide localized time and frequency information, and the 3D DWT extends image decomposition into the temporal domain for video processing.Source: §A.3 · p. 14
  • The 3D DWT decomposes video into eight frequency sub-bands at half resolution in frame, height, and width, supporting spatial-temporal representation and frequency-band watermarking.For images, the 2D DWT produces approximation and horizontal, vertical, and diagonal detail bands.Source: §A.3 · p. 14

B Robust Watermark Position Recovery Algorithm

  • For patches mapped to the same position after distortion or attack, the recovery algorithm prioritizes higher-confidence assignments and greedily places remaining patches in nearest vacant positions.The authors report that it handles noisy or partial position corruption and significantly improves watermark-extraction robustness.Source: B Robust Watermark Position Recovery… · p. 15

C More Backbones

Safe-Sora is evaluated on Open-Sora as well as VideoCrafter2, with comparable performance across the two backbones and a quality–fidelity difference on Open-Sora.

C More Backbones

  • Open-Sora achieves performance comparable to VideoCrafter2, with higher visual quality but slightly lower watermark fidelity, supporting effectiveness across different video generation models.Source: C More Backbones · p. 15
  • Table 4 presents additional ablation studies of Multi-Scale Feature Injection.Source: Table 4 · p. 16
  • After Multi-Scale Feature Injection, watermarked-original difference maps (×5) are significantly reduced, indicating improved video quality.Source: Figure 9 · p. 16

D Additional Ablation Studies

The extended ablation studies examine Multi-Scale Feature Injection and report improved visual quality when the VAE’s inherent multi-scale features are incorporated.

D Additional Ablation Studies

  • Multi-Scale Feature Injection improves generated-video visual quality by incorporating the VAE’s inherent multi-scale features.Quantitative results are reported in Tab. 4, with qualitative comparisons in Fig. 9.Source: D Additional Ablation Studies · p. 16

E Limitations

Safe-Sora is currently limited to static image-based watermarks, while the authors describe potential benefits for copyright protection alongside risks of misuse.

E Limitations

  • Safe-Sora supports static image-based watermarks such as logos or icons; embedding animated or temporally dynamic patterns remains challenging.The authors identify more complex, information-rich video watermarks as a limitation.Source: E Limitations · p. 16

F Societal Impact

  • Generative watermarking can help creators assert rights and trace misuse, fostering accountability and transparency in digital media.The authors frame this as a potential benefit as synthetic content becomes more widespread.Source: F Societal Impact · p. 16
  • The method could be misused to falsely claim ownership of public material or embed unauthorized logos in generated videos.The authors recommend research into verifying watermark authenticity and preventing abuse.Source: F Societal Impact · p. 16
  • Figure 10 shows additional VideoCrafter2 examples with difference maps for watermarked versus original videos and recovered versus original watermarks.The caption recommends viewing the figure with zoom.Source: Figure 10 · p. 17

Related papers

Loading 2505.12667v2…