DeOverlay: A Benchmark and Source-Anchored Flow for Video Overlay Removal

• Persistent overlays can leave content unobserved in every frame, requiring both use of available observations and inference of missing content.Source: §1 · p. 1
• DeOverlay learns source-to-clean corrections from paired latent paths and applies successive updates with fixed source guidance.Source: §1 · p. 2
• Visibility-normalized aggregation selects unmasked local cues, complemented by global scene context and an edit-region embedding added to the editing tokens.Source: §1 · p. 2
• DVTE-Bench provides 1,631 paired videos across eight text-overlay types, with clean targets and aligned pixel masks for common regional evaluation.Source: §1 · p. 2
• DeOverlay achieves the best results among five methods on all five reconstruction and perceptual metrics across all 1,631 released clips.Source: §1 · p. 2

• ProPainter propagates image content and features, while CLEAR targets subtitle removal and DECO targets visible watermark removal.Source: §2 · p. 3
• SEDiT concatenates fixed source-reference tokens with noise-initialized tokens, whereas DiffST combines input-derived initialization, motion-based aggregation, and video-prompt guidance.Source: §2 · p. 3
• Restoration and editing flow methods use differing initialization or prediction strategies; DeOverlay combines source initialization, fixed guidance, and regional weighting in a pretrained video transformer.Source: §2 · p. 3
• Existing evaluation resources address separate settings, including CLEAR’s subtitle benchmark, VSR-Bench-400, and synthesized stationary or moving watermarks for DECO.Source: §2 · p. 3

3 DVTE-Bench: Diverse Video Text Erasure

• DVTE-Bench pairs each overlaid video with a clean target and frame-aligned masks for common evaluation across layouts, temporal behaviors, and appearances.Source: §3 · p. 3

3.1 Challenge Design

• The eight overlay types vary in spatial layout, temporal behavior, and appearance, with category comparisons reflecting interacting conditions rather than isolated factors.Source: §3.1 · p. 3

3.2 Paired Data and Target Quality

• The release contains 1,631 clips; OCR screening rejects candidates with confirmed text in target frames, removing 569 of 2,200 candidates and retaining 1,631.Source: §3.2 · p. 4

3.3 Regional Evaluation Protocol

• All five methods use the same 1,631 clips and pixel-accurate ground-truth masks; perceptual metrics focus on overlay regions, while temporal diagnostics complement target-referenced fidelity.Source: §3.3 · p. 4

4 DeOverlay: Source-Anchored Flow

• The pipeline encodes an overlaid video as the initial editing state, predicts corrections with fixed local and global source guidance, then decodes the updated latent.The input also includes a removal instruction and an edit mask.Source: §4 · p. 5

4.1 Source-Initialized Continuous Flow

• Paired latent paths connect the source and clean videos, with inference starting directly from the source latent after training on intermediate path states.The training path perturbs the source endpoint; the clean endpoint is at τ = 0 and the source endpoint at τ = 1.Source: §4.1 · p. 5

4.2 Persistent Source Guidance

• The patchified state, local and global source features, and edit-mask embedding are added at transformer width, preserving the number of video tokens.With probability 0.2, missing-mask training zeros local guidance and the edit embedding while retaining global guidance; the annotated mask still sets loss weights.Source: §4.2 · pp. 5–6, §4.3 · p. 6
• Local source aggregation excludes masked positions in a five-frame temporal window, while global conditioning summarizes the source across temporal and spatial positions.The local mean is projected and gated to the edit region; the global summary complements local features when persistent coverage leaves local observations unavailable.Source: §4.2 · p. 6

4.3 Learning the Velocity Field

• Covered latent-grid elements receive six times the loss weight of exterior elements, while the denominator counts supervised elements without weighting.The mask is resized to the latent grid and dilated by one cell; overlap-conditioning elements are excluded.Source: §4.3 · p. 6

4.4 Shared-Weight Flow Updates

• Inference starts at the source latent and applies successive shared-weight updates as state and time change, with source and mask fixed.A single checkpoint supports multiple step counts; six steps require 12 network evaluations under classifier-free guidance at scale g = 2.Source: §4.4 · p. 6

5.1 Experimental Setup

• Training uses 350,000 aligned overlaid/clean clip pairs, while full-release evaluation uses the 40K checkpoint, K = 6, guidance scale g = 2, and 12-pixel seamless paste-back.Ablations use 200 randomly sampled benchmark videos; comparisons include four baselines, with SEDiT assessed conceptually because no executable model was available for matched evaluation.Source: §5.1 · p. 7, §5.3 · p. 9

5.2 Reconstruction across Diverse Overlays

• DeOverlay leads all five reconstruction and perceptual metrics on 1,631 videos, reducing Mask LPIPS and Mask VFID by 58.3% and 70.3% versus MiniMax-Remover.It also raises Mask PSNR from 26.17 to 30.80 dB and SSIM from 0.8044 to 0.8822.Source: §5.2 · p. 7
• Across eight overlay types, DeOverlay leads PSNR, SSIM, LPIPS, and DISTS, while artistic titles and vertical text stress different metric families.Artistic titles yield its lowest PSNR/SSIM (26.24 dB/0.8409), whereas vertical text yields its highest LPIPS/DISTS (0.0975/0.1015).Source: §5.2 · p. 7
• The temporal diagnostics report TWE Pred 9.3110 and TC 4.2498 for DeOverlay, versus ProPainter’s 8.1428 and 4.0349, against clean-target values of 9.3228.ProPainter and MiniMax-Remover score below the clean-target reference despite weaker reconstruction, so the diagnostic is interpreted alongside clean-target fidelity; DeOverlay’s aggregate TWE Gap is 0.0118 versus 0.6133 for MiniMax-Remover and 1.1801 for ProPainter.Source: §5.2 · pp. 7–8
• In paired examples, DeOverlay more closely preserves target appearance, including car and road alignment; additional crops show subtitles, titles, watermarks, a sticker, and a television logo.The examples also expose residual text, blurred structures, color shifts, and damaged background boundaries in outputs.Source: §5.2 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

5.3 Ablation Studies

• Replacing source initialization and additive guidance with noise initialization and concatenated source tokens lowers Mask PSNR by 2.861 dB; removing VSA or GSC lowers it by 0.091 or 0.171 dB.An ordinary temporal mean instead of visibility-normalized aggregation lowers PSNR by 0.101 dB and raises MAE to 4.360.Source: §5.3 · p. 9
• With a fixed checkpoint, four flow steps improve Mask PSNR from 32.410 to 32.461 dB over two steps, while eight steps add only 0.007 dB over four at 1.50× latency.The four-step flow also exceeds a separately trained direct-residual reference by 1.161 dB in Mask PSNR; each update recomputes velocity from the current state and time with fixed source cues.Source: §5.3 · p. 9

5.4 Efficiency

• Adding a source-token stream increases median time by 2.485× and incremental peak memory by 1.995×; additive guidance retains the video sequence length.These measurements isolate transformer cost, whereas end-to-end timing includes video-processing overhead.Source: §5.4 · p. 10

6 Limitations

• The benchmark's synthetic-text focus leaves graphic logos, stickers, and preservation of natural scene text for broader paired evaluation.When no neighboring frame reveals a fully covered background, reconstruction depends on the video prior; application examples also include residual graphics and local texture artifacts.Source: §6 · p. 10

7 Conclusion

• DVTE-Bench unifies eight overlay categories in 1,631 paired videos with clean targets, aligned pixel masks, and a common evaluation protocol.The protocol supports comparisons across overlay layouts, temporal behavior, and appearance.Source: §7 · p. 10
• DeOverlay combines source initialization, visibility-normalized local aggregation, and global source guidance with shared-weight flow updates and one video-token sequence.The conclusion reports that it outperforms four baselines on all five reconstruction and perceptual metrics.Source: §7 · p. 10
• On the 200-video subset, four-step flow exceeds a separately trained direct-residual reference by 1.161 dB in Mask PSNR.Component ablations support both guidance branches, while category-level evaluation finds artistic titles and vertical text stress different metrics.Source: §7 · p. 10

A Implementation Details

• The training corpus contains 350,000 paired samples: 250,000 base, 50,000 stylized-text, and 50,000 mirrored-text samples, each with an overlaid input, clean target, and mask.Training samples are separate from the 1,631 DVTE-Bench clips; subset sampling weights of 4 : 2 : 1 specify frequencies, not corpus sizes.Source: A Implementation Details · p. 12
• Local conditioning uses patch-grid source features and a max-pooled mask, while global conditioning averages the original latent across temporal and spatial positions.The local branch uses temporal patch size one, and local mask dilation and loss dilation operate at their respective resolutions.Source: A Implementation Details · p. 12
• The local projection uses two 1 × 1 × 1 convolutions with hidden width 512, and the global source branch uses a two-layer MLP with hidden width 512.Guidance output projections and the mask embedding start at zero; the VAE and text encoder remain fixed, and LoRA of rank 256 adapts eligible linear layers.Source: A Implementation Details · p. 13
• At guidance scale g = 2, inference requires two network evaluations per Euler step, then decodes the final latent; long videos use overlapping chunks with prior outputs as boundary conditions.Deployment derives text masks from OCR, while non-text overlays require a mask covering the designated graphic region.Source: A Implementation Details · p. 13
• Configured buckets use 41 frames at resolution 768 for long sides at least 1920, 65 frames at 768 for sides at least 1280, and 121 frames at 608 for smaller inputs.The decoder samples a random start with unit frame stride.Source: A Implementation Details · p. 13

B Euler Error and Endpoint Prediction

• Euler integration error is O(h) for a fixed interval when the learned field is Lipschitz and sufficiently smooth along the exact solution.Here h is the maximum step size; reducing it tightens the integration-error bound under these assumptions.Source: B Euler Error and Endpoint Prediction · p. 13
• At inference, the state and time evolve while the source, mask, and weights remain fixed, and the final latent is decoded as the output.The training procedure samples a noisy source endpoint and interpolates toward it from the clean latent; inference starts at the source and applies guided velocity updates along a descending schedule.Source: B Euler Error and Endpoint Prediction · p. 14
• The derived iterated bound assumes constant L and B and concerns approximation of the learned ODE, not distance from its endpoint to the clean target.For L = 0, the stated limit is Bh/2.Source: B Euler Error and Endpoint Prediction · p. 14
• The learned predictor can change across updates because its estimate depends on the observed intermediate state and time, even when a paired training displacement is constant.The weighted-loss minimizer is a weighted conditional mean over displacements; this does not ensure faithful recovery of multimodal hidden backgrounds.Source: B Euler Error and Endpoint Prediction · pp. 13–14

C Metric Definitions and Reporting Protocol

• Frame or frame-pair measurements are averaged within videos and then equally across videos, while VFID is computed globally and finite PSNR omits infinite-valued video scores.The evaluation uses all 1,631 videos and pixel-accurate packed ground-truth masks.Source: C Metric Definitions and Reporting… · p. 15
• Mask PSNR uses errors over ground-truth mask pixels, and Mask SSIM averages the spatial SSIM map over those pixels rather than the enclosing rectangle.Input masks can differ from scoring masks: bounding boxes may be supplied while outputs are scored on the common pixel mask.Source: C Metric Definitions and Reporting… · p. 15
• LPIPS and DISTS use target-composited predictions and a shared crop around the video-wide union of mask locations, while VFID compares global RGB-I3D feature distributions.The perceptual constructions focus on overlay-region errors and do not measure exterior preservation.Source: C Metric Definitions and Reporting… · p. 15
• Temporal reporting includes mask-restricted TWE and auxiliary TC and flow statistics; the released TWE gap is computed after aggregation against a clean-target reference of 9.3228.The aggregate absolute gap can hide opposite signed deviations through cancellation.Source: C Metric Definitions and Reporting… · p. 15

D Overlay Categories and Evaluation Coverage

• Overlay function and appearance or layout are distinct: a vertical name caption combines a text function with a spatial layout, although each clip receives one challenge type.The release distribution figure counts clips by assigned type, language, orientation, and rendered overlay elements; element count is not pixel coverage.Source: D Overlay Categories and Evaluation… · pp. 15–16, Figure 6 · p. 16
• DVTE-Bench focuses on synthetic text overlays; graphic logos, stickers, and preservation of natural scene text remain outside its paired evaluation scope.Its clean-target OCR policy excludes confirmed text, so natural scene-text preservation requires a separate evaluation setting.Source: D Overlay Categories and Evaluation… · p. 16
• The release has 1,631 clips, but rare language labels do not support reliable per-language comparisons.The listed language metadata includes Chinese, English, Korean, Japanese, Thai, and several labels represented by only one or two clips.Source: D Overlay Categories and Evaluation… · p. 16
• DeOverlay leads the five evaluated methods on Mask PSNR, SSIM, LPIPS, and DISTS in every overlay type, with video-equal rather than type-equal aggregate weighting.The two category tables together cover the complete 1,631-video release under the common pixel-mask protocol.Source: D Overlay Categories and Evaluation… · p. 16
• All methods are scored on released pixel masks, while mask provenance separately distinguishes known edit regions from estimated detection masks.For overlays containing graphics, coverage must include the full designated overlay rather than recognized characters alone.Source: D Overlay Categories and Evaluation… · p. 17

E Qualitative Exports and Case Analysis

• The selected exports cover varied overlays, and the broader collection includes occasional subtitle remnants, local texture artifacts, and residual geometric watermark elements.The television-logo example uses a logo-only mask that leaves other program graphics untouched; these exports are not a controlled step-count comparison.Source: E Qualitative Exports and Case Analysis · p. 17
• The seven benchmark-context cases use matched frames and crops, but illustrate restoration behavior rather than estimating aggregate rankings or comparing inference depths.The cases include 81-frame clips, with the four-step outputs drawn from the same-A100 timing package.Source: E Qualitative Exports and Case Analysis · p. 17, Figure 7 · p. 18, Figure 8 · p. 19
• Matched multilingual examples show outputs across six language settings, including bilingual overlays, changing subtitle content, and different scenes.Some language pairs share scene content, making glyph and line-layout differences visible against matching backgrounds.Source: E Qualitative Exports and Case Analysis · p. 17, Figure 12 · p. 23

F.1 Training Formulation and Initialization

• Four-step flow improves Mask PSNR from 31.300 to 32.461 dB, reduces MAE from 4.520 to 4.299, raises Crop SSIM from 0.91120 to 0.92765, and reduces TWE absolute gap from 0.195 to 0.180 versus a separately trained one-step residual model.The comparison uses independently trained models at their respective inference settings.Source: §F.1 · p. 25
• The Gaussian-start model's 2.861 dB gap reflects both its starting state and how source evidence enters the network, so the comparison does not isolate either factor.The independent token microbenchmark evaluates the computational effect of the second stream.Source: §F.1 · p. 25

F.2 Source Conditioning

• The full model's Outside MAE is 2.541, versus 2.552 without both source branches, 2.543 without VSA, and 2.550 without GSC.Replacing VSA with an ordinary temporal mean yields Outside MAE 2.542 and TWE absolute gap 0.183, versus 0.180 for the full model.Source: §F.2 · p. 25

F.3 Transformer Token Cost

• The token-cost microbenchmark uses real 32K weights and four transformer forwards; its measurements exclude VAE encoding, decoding, and video processing.The measured latent corresponds to an 81-frame input with mirror padding, and memory is incremental peak allocation above the loaded runtime.Source: Table 9 · p. 25

G Timing and Optimization Protocol

• The timing protocol treats end-to-end time as the primary cross-method comparison because it includes decoding, inference, postprocessing, and encoding.Per-video measurements exclude model loading; timings use eight selected clips totaling 650 frames on one otherwise idle A100.Source: G Timing and Optimization Protocol · pp. 25–26