Source-linked AI summary
Painting Change Map as guidance for generating painting processes
Markus Pobitzer, Chang Liu, Niccolò Biondi, Nicu Sebe
TL;DR
Existing painting-video controls rely on isolated object-level masks or fixed linear progression, failing to represent layered composition, user intent, and temporal flexibility. PCM jointly encodes spatial location and temporal buildup, and a two-stage pipeline generates structural painting actions before color-based rendering. The framework achieves strong evaluation results across global unpainting benchmarks and localized instance-level control tasks, establishing a high-performance upper bound.
Problem
Existing painting-video controls rely on isolated object-level masks or fixed linear progression, failing to represent layered composition, user intent, and temporal flexibility.
Source: §1 · p. 2Method
PCM jointly encodes spatial location and temporal buildup, and a two-stage pipeline generates structural painting actions before color-based rendering.
Source: §1 · p. 2, §3 · p. 3, §3.3 · p. 3Results
The framework achieves strong evaluation results across global unpainting benchmarks and localized instance-level control tasks, establishing a high-performance upper bound.
Source: §5 · p. 8Takeaways & Limitations
For real-world painting video, combining temporal progression with explicit lineart boundaries provides a geometric anchor that enables cross-frame alignment.
Source: §5 · p. 8Physical canvas movements can desynchronize past strokes from current frame geometry over extended periods, making explicit tracking relevant for real-world data.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Digital art generation has undergone a massive paradigm shift, moving from static image synthesis to dynamic video generation that can mimic the human painting process. While existing models can synthesize visually stunning time-lapses of paintings being created or "unpainted," they largely treat the creative process as a black box. Current conditioning mechanisms are either too rigid, relying on static geometric cues like Canny edges or depth maps, or too holistic, utilizing simple scalar progress variables that offer no localized control. Consequently, users lack the agency to steer how a painting evolves over time. In this work, we argue that true control over the video generation of artwork requires a joint understanding of two fundamental dimensions: where an artist acts, and when those actions take place. To achieve this, we introduce the Painting Change Map, a novel spatio-temporal control mechanism that explicitly tracks normalized structural changes across a 1,000-timestep continuum, a resolution sufficient to capture fine-grained stroke-level changes while remaining computationally tractable. By mapping localized visual differences into an encoded RGB space, our representation enables fine-grained guidance for both the forward painting and reverse unpainting processes. Extensive benchmarking demonstrates the efficacy of our approach. We outperform existing control baselines and achieve competitive or superior results compared to state-of-the-art painting methods. Furthermore, under oracle conditions, our method surpasses all baselines by a significant margin. Code is available at https://github.com/Markus-Pobitzer/painting-change-map.
1. Introduction
Existing painting-video controls simplify the process in ways that limit spatial or temporal steering. The paper introduces a joint spatial-temporal control map and a pipeline for using it to recreate painting processes.
Problem
- Existing approaches rely on object masks, global progress scalars, or fixed progression, limiting layered composition, local spatial control, or control over action pacing and order.The paper identifies these as three restrictive paradigms in painting-video synthesis.Source: §1 · p. 2
Contribution
- The Painting Change Map (PCM) jointly encodes where and when painting changes occur, abstracting from stroke color and physical paint interactions.The representation is intended to support fine-grained, near-stroke-level process control.Source: §1 · p. 2
- Interpolation & Lineart integrates PCM with RGB video-diffusion controls, alongside grayscale interpolation, 24-bit packaging, and historical channel tracking alternatives.The hybrid combines temporal control with lineart guidance.Source: §1 · p. 2
- A two-stage pipeline first generates structural painting actions and then color-based rendering, while the benchmark reports superior localized steering against spatial controls and specialized painting architectures.The compared controls include Canny Edges, Depth, Paints Undo, and Loomis Painter.Source: §1 · p. 2
2. Related work
Prior work synthesizes painting processes through stroke-based and pixel-based approaches, including newer diffusion and video-diffusion systems. The paper positions PCM as explicit guidance for where and when a model generates.
Painting process generation
- Neural painting methods infer stroke sequences or parameters, while pixel-based methods synthesize coherent sequences and include iterative masking with diffusion inpainting.The cited examples include neural painting, Paint Transformer, convolutional approaches, and Inverse Painting [7].Source: §2 · p. 2
- Recent diffusion systems generate painting steps as images or video, and Loomis Painter generates the inverse painting process.PaintsUndo recreates digital-art workflows, while PaintsAlter adapts a video-diffusion model to the task.Source: §2 · p. 2
Controlling video generation
- Despite realistic workflows, existing approaches lack active guidance; PCM is presented as explicit control over where and when generation occurs.The related-work discussion also describes video-to-video and image-to-video approaches and pixel-level video controls.Source: §2 · pp. 2–3
3. Method
The method represents painting as spatially localized changes over time, filters motion artifacts, and projects the resulting temporal map into RGB control signals with structural lineart guidance.
3. Method
- The method models where and when geometric canvas changes occur, leaving complex micro-level physical stroke interactions to the generative backbone.The mapping abstracts from variables such as medium viscosity, underpainting, and tool dynamics.Source: §3 · p. 3
3.1. Spatial Painting Change Map
- Adjacent-frame differences are reduced to per-pixel L2 magnitudes and thresholded into binary change maps, assuming occlusion-free forward or reverse painting sequences.The threshold filters noise and subtle lighting changes.Source: §3.1 · p. 3
3.2. Robustness and Global Motion Filtering
- A frame-wide altered-pixel threshold nullifies changes when canvas tremors or camera motion trigger widespread detections.At θ = 5%, the filter is reported to remove motion artifacts while preserving large valid sweeping strokes.Source: §3.2 · p. 3
3.3. Temporal Encoding and Representation Modes
- The method encodes each location’s latest change as a cumulative temporal signal, then projects it into RGB; the hybrid uses interpolation for temporal values and lineart in the green and blue channels.Figure 2 also compares interpolation, Pack 24-bit, History, and hybrid modalities.Source: §3.3 · pp. 3–4, Figure 2 · p. 4
4. Experiments
Experiments evaluate PCM on global reconstruction, localized control, in-the-wild examples, threshold ablations, and comparisons with painting-generation methods. The combined Interpolation & Lineart control performs strongly across these settings, while results also show that performance depends on filtering thresholds.
4. Experiments
- The experiments extend the Loomis Painter dataset by sampling 1,000 frames per video and train reverse-chronological 81-frame sequences randomly sampled from those frames.The acrylic subset contains 132 training, 16 test, and 13 validation videos; experiments use ground-truth control maps for upper-bound comparisons.Source: §4 · p. 4, §4.2 · p. 5
4.1. Baselines
- Baselines include depth, Canny, and Lineart structural controls, evaluated under identical training configurations; Table 1 reports global and localized control metrics.Table 1 marks the overall best and best baseline.Source: §4.1 · pp. 4–5, Table 1 · p. 5
4.2. Reconstructing the painting process
- Interpolation & Lineart outperforms all baselines across evaluated global metrics, reducing MSE from 592.81 to 449.68.Standalone temporal maps underperform Canny and Lineart; the combined approach pairs stroke progression with static structural outlines.Source: §4.2 · p. 5, Table 1 · p. 5
4.3. Local Instance Control
- For masked instance regions, Interpolation & Lineart achieves the best FID, MSE, and DINOv2 results, tying Lineart on LPIPS.Evaluation covers seven instance categories, with metric computation confined to the targeted masks.Source: §4.3 · p. 5, Table 1 · p. 5
4.4. Qualitative Results
- Qualitative examples show reconstruction from ground-truth change maps, zero-shot mountain-only unpainting, and in-the-wild reconstruction using model-generated maps.For in-the-wild paintings, the change map is generated first, then used with the input painting to reconstruct the process.Source: §4.4 · p. 6, Figure 3 · p. 6, Figure 4 · p. 7
4.5. Ablation of Spatiotemporal Thresholds
- Threshold performance is robust within a reasonable range but worsens at extremes: τ = 0.05 raises LPIPS to 0.214 and MSE to 494.96, while τ = 0.13 is optimal.Pairing τ = 0.13 with θ ∈[4.0, 5.0]% yields the best equilibrium; θ ≤3.0% can discard authentic sweeping strokes.Source: Table 2 · p. 5, §4.5 · pp. 6–7
4.6. Comparison to Painting Generation Methods
- Our pipeline nearly matches Loomis Painter on CLIP (0.87 vs. 0.88) and LPIPS (0.45 vs. 0.44), beats its MSE (3127.13 vs. 3152.96), and the oracle variant beats all baselines across metrics.The pipeline reaches 0.62 FPS; its sequential stages peak at 25.84 GB and 17.04 GB VRAM, respectively.Source: §4.6 · pp. 7–8, Table 3 · p. 8
5. Conclusion
The framework uses the Painting Change Map (PCM) and a two-stage pipeline to separate spatiotemporal painting actions from physical media properties. It reports strong results across global unpainting and localized control, while real-world canvas movement and limited dataset diversity constrain aspects of the approach.
5. Conclusion
- The PCM and decoupled two-stage pipeline achieve strong results across global unpainting and localized instance-level control tasks, establishing a high performance upper bound.The framework isolates spatiotemporal brushstroke progression from physical interactions such as color mixing, paint viscosity, and canvas textures.Source: §5 · p. 8
- Fusing temporal progression with lineart boundaries provides a geometric anchor that enables the diffusion model to maintain cross-frame alignment.The authors note that real-world canvas displacements and camera tremors can make isolated strokes struggle to capture global trajectories.Source: §5 · p. 8
- Physical canvas movement can cause past strokes to desynchronize from current frame geometry over extended periods, even with global motion filtering.The authors identify automated canvas alignment or keypoint rectification as a future direction.Source: §5 · p. 8
- Scarcity of sufficiently diverse, large-scale datasets constrains further analysis of information carried by change maps, including possible clues to medium, style, or painter.The authors pose these as questions for future analysis.Source: §5 · p. 8
7. Supplementary
The supplementary section describes alternative ways to project temporal change information into RGB representations. These include grayscale interpolation, 24-bit packing, history tracking, and a hybrid that combines interpolation with lineart.
7.1. Representation Modes Change Map
- Linear interpolation between two reference boundary colors encodes temporal values; black-to-white endpoints yield grayscale, with brighter values marking more recent interventions.The interpolation uses the temporal value to blend the start and end colors.Source: §7.1 · p. 10
- 24-bit packing maps normalized temporal values to integers and partitions them across RGB channels, supporting up to 16.7 million steps but introducing chromatic discontinuities that can complicate pattern recognition.The integer values are partitioned into distinct 8-bit color channels.Source: §7.1 · p. 10
- History mode maps the three most recent sequential modifications at each pixel to the RGB channels, exposing local canvas evolution in one tensor.It tracks a sliding history window rather than a single event.Source: §7.1 · p. 10
- Interpolation & Lineart replaces the interpolation map's green and blue channels with extracted lineart to combine temporal change with structural outlines.The lineart is extracted from the targeted frame.Source: §7.1 · p. 10