Source-linked AI summary
Layered Neural Atlases for Consistent Video Editing
Yoni Kasten, Dolev Ofri, Oliver Wang, Tali Dekel
TL;DR
Video editing lacks a representation that is both intuitive and temporally consistent, especially for complex dynamic scenes. The paper addresses this with jointly optimized layered neural 2D atlases and demonstrates consistent editing across several applications while preserving scene effects. The method remains limited by time-varying appearance, long videos, topology, and extreme deformation.
Problem
Video editing must preserve temporal consistency across frames while providing an intuitive interface for manipulating spatio-temporal content.
Method
The method jointly optimizes coordinate-based MLPs for layered 2D atlas mappings, colors, and alpha values using reconstruction and regularization losses.
Results
The framework supports texture mapping, style transfer, image-to-video texture transfer, and segmentation propagation while preserving reflections, shadows, and occlusions.
Takeaways & Limitations
A self-supervised interpretable atlas bottleneck enables accessible, temporally consistent editing of complex real-world videos through 2D manipulation.
Takeaways & Limitations
The atlases are fixed in time, quality decreases for videos longer than 100 frames, and single foreground layers struggle with topology, self-occlusion, and extreme deformation.
Abstract
from arXiv · showhide
We present a method that decomposes, or "unwraps", an input video into a set of layered 2D atlases, each providing a unified representation of the appearance of an object (or background) over the video. For each pixel in the video, our method estimates its corresponding 2D coordinate in each of the atlases, giving us a consistent parameterization of the video, along with an associated alpha (opacity) value. Importantly, we design our atlases to be interpretable and semantic, which facilitates easy and intuitive editing in the atlas domain, with minimal manual work required. Edits applied to a single 2D atlas (or input video frame) are automatically and consistently mapped back to the original video frames, while preserving occlusions, deformation, and other complex scene effects such as shadows and reflections. Our method employs a coordinate-based Multilayer Perceptron (MLP) representation for mappings, atlases, and alphas, which are jointly optimized on a per-video basis, using a combination of video reconstruction and regularization losses. By operating purely in 2D, our method does not require any prior 3D knowledge about scene geometry or camera poses, and can handle complex dynamic real world videos. We demonstrate various video editing applications, including texture mapping, video style transfer, image-to-video texture transfer, and segmentation/labeling propagation, all automatically produced by editing a single 2D atlas image.
1 INTRODUCTION
Video editing must maintain temporal consistency across frames while offering an intuitive way to manipulate spatio-temporal content. The paper proposes layered neural 2D atlases as an interpretable representation for consistent editing of complex natural videos without requiring 3D scene knowledge.
- Challenges: Video editing requires edits to remain temporally consistent, including correct motion and occlusion relationships across frames.The paper identifies temporal consistency and intuitive interaction with spatio-temporal content as two central challenges.
- Prior limitations: Existing 2D propagation methods can suffer from drift and occlusion failures, while 3D methods typically rely on strong geometric priors or static scenes.The limitations affect both editing reliability and the range of videos that can be handled.
- Proposed representation: The method represents each editable object and the remaining background with layered neural 2D atlases that unify appearance over the video.Per-video pixels are mapped to atlas coordinates and visibility is represented with alpha values, enabling self-supervised training from the input video.
- Optimization: Coordinate-based MLPs jointly model mappings, atlas content, and alphas using video reconstruction and regularization losses.Regularization encourages locally rigid and temporally consistent mappings so the atlases remain semantic and interpretable.
- Design advantages: Continuous deformable atlas representations support complex geometry, parallax, pose changes, and direct compositing with original frames.These design choices extend applicability beyond static-camera or static-background assumptions and tolerate difficult scene reconstruction cases.
- Applications: The framework enables 2D atlas manipulation for consistent video editing on real-world videos with deformation, reflections, and occlusion.The paper demonstrates multiple editing applications, including texture mapping, style transfer, and propagation.
2 PRIOR WORK
Prior work includes video mosaics, layered decompositions, propagation methods, and continuous neural representations. The paper distinguishes its approach by combining layered editable atlases with learned mappings and alpha values for temporally consistent editing.
- Video mosaics: Video mosaics summarize or composite dynamic content into images, whereas this work reconstructs atlases specifically to enable temporally consistent editing.The paper’s motivation differs from panorama construction, video indexing, and animated panorama textures.
- Video mosaics: Unwrap Mosaics also maps video content to editable 2D mosaics, but relies on precomputed binary masks and therefore cannot represent semi-transparent effects.The proposed method refines rough masks through reconstruction into time-varying alpha mattes.
- Continuous representations: The pipeline uses coordinate-based MLPs to map pixels into atlas coordinates, predict atlas colors and alpha values, and reconstruct video through compositing.This connects continuous neural representations with the paper’s layered atlas formulation.
- Video layer decomposition: Per-frame RGBA layer methods support frame-level manipulation but cannot consistently propagate edits across time.The paper’s primary distinction is a unified atlas representation rather than separate layers for each frame.
- Video propagation approaches: Video propagation methods transfer edits between frames using bilateral spaces, phase-based operations, or optical flow.The paper compares against this category and reports that its own parameterization is smoother over time for video editing.
3 ESTIMATING NEURAL ATLASES FROM VIDEO
The method estimates layered neural atlases, mappings, and opacities from video using jointly optimized coordinate-based MLPs. Reconstruction and regularization losses encourage interpretable, temporally consistent atlas representations that support editing and preserve scene effects.
- The system estimates background and foreground 2D atlases, pixel-to-atlas mappings, and opacity values from a natural video and coarse object mask.
- Coordinate-based MLPs represent the mappings, atlas appearance, and alpha values continuously, with atlas images rendered to a grid for user editing.Edits to the rendered atlas are mapped back to video frames through the mapping networks.
- The framework is trained end-to-end with reconstruction, rigidity, consistency, sparsity, and bootstrapping losses.The reconstruction objective recovers the input video, while regularizers shape the decomposition and mapping.
- Loss terms: Rigidity regularization encourages locally rigid video-to-atlas mappings so the atlas remains less distorted and easier to edit.The Jacobian is approximated using one-pixel horizontal and vertical offsets, and rigidity encourages both singular values to equal 1.
- Loss terms: Consistency regularization maps corresponding pixels in neighboring frames to the same atlas point and masks unreliable optical-flow correspondences.Correspondences are obtained from precomputed optical flow and filtered with a forward-backward consistency check.
- Loss terms: Sparsity regularization discourages duplicate foreground representations by encouraging foreground atlas content to be zero where the foreground is occluded.This encourages scene points to map many-to-one while retaining only content needed for reconstruction.
- Qualitative results: A checkerboard placed in the foreground atlas maps back to input frames, with its deformation revealing continuity of the atlas mapping over time.
4 RESULTS
The method produces interpretable, temporally consistent layered atlases and supports diverse edits on complex videos. Ablations and comparisons show that the complete design improves editability, texture fidelity, and temporal consistency, while remaining subject to several scope limitations.
- Evaluation setup: The qualitative evaluation covers DAVIS and custom videos with moving objects and background parallax, using editing demonstrations, loss ablations, and propagation comparisons.Because the task targets consistent video editing, the authors report no reasonable quantitative baselines.
- Atlas quality: The atlases are interpretable and temporally consistent: features are not duplicated, and checkerboard patterns remain fixed on object surfaces across frames.The visualization includes background and foreground atlas layers for DAVIS examples.
- Editing applications: The method supports texture transfer, foreground or background stylization, and still-image transfer into moving backgrounds while preserving reflections, shadows, and occlusions.Edits can be applied to atlases or directly to a video frame and are mapped consistently across the original frames.
- Editing applications: Nonprofessional users completed the demonstrated edits in minutes by drawing texture elements or placing image overlays directly on atlases.The workflow uses editing software such as Photoshop and also supports projecting a frame edit onto the atlas and back into the video.
- Loss ablation: Removing rigidity loss distorts atlases until foreground objects are difficult to recognize, while removing optical-flow loss can duplicate and split background content.The complete model preserves comparable overall PSNR despite being more constrained.
- Architecture ablation: The coordinate-based MLP produces cleaner, higher-quality atlases than a fixed-resolution Grid atlas, which can fail to converge to a useful decomposition on complex videos.Grid atlas reconstructions may still achieve high PSNR by using alpha channels to reproduce colors, even when mappings are inconsistent and regions are not editable.
- Limitations: The framework does not model temporal appearance changes in atlases, degrades for videos longer than 100 frames, and can yield different solutions from different random initializations.It is also limited to disk-topology objects and struggles with complex geometry, self-occlusions, and extreme deformations in a single foreground layer.
5 CONCLUSION
The paper introduces neural video unwrapping through an end-to-end optimized atlas representation, enabling simple and temporally consistent editing across varied input videos.
- The method introduces an end-to-end optimized atlas-based representation for neural video unwrapping.
- The frame/atlas editing framework is intended to provide accessibility for beginners and creative control for professionals.
- The authors demonstrate the approach on multiple downstream applications involving simple, temporally consistent video editing.
A NETWORK ARCHITECTURE
The implementation uses coordinate-based MLPs for atlas components and includes an identity-mapping calibration phase to stabilize learned mappings.
- The background mapping MLP has 4 layers with 256 channels, while the foreground mapping MLP has 6 layers with 256 channels.
- The alpha and atlas MLPs each have 8 layers with 256 channels, and the atlas network uses skip connections at layers 4 and 7.
- The mapping networks first train an identity mapping for 100 iterations to reduce flipping and retain point order.
B RECONSTRUCTION PSNR
The reconstruction evaluation reports PSNR results for videos using the paper’s two-atlas approach.
- Table 1 reports PSNR results for video reconstruction.
- The reported reconstruction quality is evaluated across different videos.
- The evaluation specifically uses the two-atlas approach for video reconstruction.