PaperlayerView exact revision on arXiv

Source-linked AI summary

Loomis Painter: Reconstructing the Painting Process

Markus Pobitzer, Chang Liu, Chenyi Zhuang, Teng Long, Bin Ren, Nicu Sebe

arXiv:2511.17344v2cs.CV

TL;DR

Painting-process models face gaps in cross-media generalization and temporal or structural continuity. Loomis Painter combines semantic media conditioning, cross-media training, reverse-painting learning, and a dataset of real processes; its evaluations report strong metric performance and a PDP progression resembling human artistic stages.

  • Problem

    Generative models can synthesize artistic images but struggle to generalize across media and produce temporally or structurally inconsistent painting sequences.

    Source: §1 · p. 2

  • Method

    The framework combines semantic media conditioning and cross-media style training with reverse-painting learning and a dataset of real painting processes.

    Source: §1 · p. 2, §3.2.2 · p. 4, §3.3 · p. 5

  • Results

    The authors report strong LPIPS, DINO, and CLIP performance, and say PDP reveals progression resembling composition, color blocking, and detail refinement.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PDP compares generated and ground-truth perceptual-distance profiles over normalized time to assess painting-process progression beyond frame-based metrics.

    Source: §5.2.1 · p. 7

    The dataset is pencil-heavy with limited artist and subject diversity, and the base model struggles with portraits and hand-shadow artifacts.

    Source: §7 · p. 8, §A.3 · p. 11

Abstract

from arXiv · show

Step-by-step painting tutorials are vital for learning artistic techniques, but existing video resources (e.g., YouTube) lack interactivity and personalization. While recent generative models have advanced artistic image synthesis, they struggle to generalize across media and often show temporal or structural inconsistencies, hindering faithful reproduction of human creative workflows. To address this, we propose a unified framework for multi-media painting process generation with a semantics-driven style control mechanism that embeds multiple media into a diffusion models conditional space and uses cross-medium style augmentation. This enables consistent texture evolution and process transfer across styles. A reverse-painting training strategy further ensures smooth, human-aligned generation. We also build a large-scale dataset of real painting processes and evaluate cross-media consistency, temporal coherence, and final-image fidelity, achieving strong results on LPIPS, DINO, and CLIP metrics. Finally, our Perceptual Distance Profile (PDP) curve quantitatively models the creative sequence, i.e., composition, color blocking, and detail refinement, mirroring human artistic progression.

1. Introduction

Existing tutorials are passive, while generative models can produce discontinuous painting sequences and generalize poorly across media. Loomis Painter combines multi-media conditioning, reverse-painting training, and a dataset with occlusion removal to model painting processes and evaluate their progression.

1. Introduction

  • Artists must reproduce not only a scene but also the process of rendering it with chosen tools, including medium-specific steps such as layering oil color and reconstructing proportions.The Loomis Method is presented as a structural approach to drawing a head with correct proportions from a reference photo.Source: §1 · p. 2
  • The framework embeds artistic media in a diffusion model’s conditional space and uses cross-media style augmentation to support consistent, controllable transfer of painting procedures.Reverse-painting training learns from completed artwork back to a blank canvas; the first frame is grounded by the input image.Source: §1 · p. 2
  • The dataset pipeline removes hand occlusions and other visual clutter so the model can learn accurate stroke-level transformations.Source: §1 · p. 2
  • The evaluation covers cross-media consistency, temporal coherence, and final-image fidelity, and the proposed PDP tracks progression from composition through color blocking to detail refinement.The authors report strong performance on LPIPS, DINO, and CLIP metrics.Source: §1 · p. 2

2. Related Work

Prior neural painting and pixel-based methods model painting through parametric strokes or narrower training data, while Loomis Painter learns from real painting sessions and uses a video diffusion framework. The method also builds on pretrained diffusion models and outlines a process-generation and media-transfer pipeline.

2. Related Work

  • Neural painting methods such as Paint Transformer generate parametric stroke sequences, which the paper contrasts with authentic artistic processes learned from real-world sessions.The related-work discussion says synthesized stroke sequences can diverge from how artists build compositions in tutorials.Source: §2 · p. 2
  • The overview depicts curated videos reversed for LoRA-tuned WAN 2.1 [31] to reconstruct painting steps, alongside a media-transfer model for acrylic, oil, or pencil styles.For transfer, image-editing models generate reference-image variations used to train reconstruction of the original painting process.Source: Figure 2 · p. 3
  • Prior pixel-based approaches use synthetic or narrowly scoped datasets, whereas Loomis Painter learns painting workflows from real sessions and is described as generalizing across artistic media.The paper positions its video diffusion model as producing temporally coherent progressions from blank canvas to completed painting.Source: §2 · p. 3
  • The paper situates its approach within diffusion models that use latent representations, extend conditioning to video, and can be fine-tuned with lightweight methods such as LoRA.It notes that newer models use Flow Matching [13] and video latents contain multiple frames with temporal compression.Source: §2 · p. 3

3. Methods

The framework generates painting sequences through medium-aware conditioning and reverse-painting supervision, modeling medium-specific workflows and progressive structure-to-detail evolution.

3. Methods

  • The framework combines cross-media conditioning with reverse-painting learning to generate temporally coherent sequences that mimic artistic workflows and support transfer across media.It constructs a semantic condition, evolves a temporal latent representation, and decodes a painting sequence.Source: §3 · pp. 3–4

3.1. Video Diffusion

  • The video diffusion model trains a DiT to predict the velocity of a noisy video latent, while image-to-video conditioning concatenates an encoded input-image latent with the noise latent.The flow-matching loss is the mean squared error between predicted and target velocity.Source: §3.1 · p. 4

3.2.1. Medium-Aware Semantic Embedding

  • A pretrained text encoder maps a combined medium-and-scene prompt to a conditioning vector that influences temporal latent features through cross-attention.The model can learn medium-specific behaviors such as oil-paint color layering and pencil-sketch hatching.Source: §3.2.1 · p. 4

3.2.2. Cross-Media Structural Alignment

  • Cross-media training style-transforms the input while retaining the target video latent, exposing the model to consistent shapes and spatial relationships across styles.This teaches the model to render objects progressively as a painting trajectory in the specified medium.Source: §3.2.2 · p. 4

3.3. Reverse-Painting Learning Strategy

  • The curation pipeline detects process boundaries and the canvas, samples video segments, removes hand and tool occlusions, and cleans logos and text overlays.Masked medians use the preceding median frame as a reference when removing transient obstructions.Source: Figure 4 · p. 5
  • Reverse-painting supervision addresses the mismatch between a completed-image first frame and the intended progressive trajectory by training the model on reversed sequences.The reversal yields monotonic detail removal: textures fade, color regions simplify, and composition becomes more prominent.Source: §3.3 · p. 5

4. Dataset Curation Pipeline

The pipeline curates painting workflows from tutorial videos through canvas localization, frame sampling, occlusion removal, and manual quality review.

4. Dataset Curation Pipeline

  • The pipeline trims videos to the painting process, localizes the canvas, samples 30 frames at 3 fps from each 10-second segment, and removes occlusions and overlays.It uses masked medians, incorporating prior frames to fill persistent occlusions.Source: §4 · p. 5
  • The YouTube tutorial dataset prioritizes static camera views and minimal canvas movement, and the processed results are manually reviewed to filter poor-quality examples.A unified fine-tuning dataset merges Acrylic, Loomis Portrait, Oil, and Pencil subsets.Source: §4 · p. 5
  • The authors will release code and configuration files, but not the dataset itself because of licensing constraints.Each dataset entry includes a finished-artwork reference frame and progressive painting states.Source: §4 · p. 6

5. Experiments

The experiments compare painting-process reconstruction against representative methods, introduce PDP for temporal evaluation, and assess reverse-order tuning and qualitative performance. The authors report stronger metrics and a progression resembling human painting workflows.

5. Experiments

  • Fine-tuning extends Wan 2.1 14B 480p I2V with LoRA adapters; training takes 24 hours on four H100 GPUs for 14 epochs.The separate media-transfer LoRA starts from the base model after seven epochs and trains for seven more on the art-media-transfer dataset.Source: §5 · p. 6

5.1. Baselines

  • The baselines include Inverse Painting, ProcessPainter, and PaintsUndo, which use autoregressive inpainting, a few real examples, and fine-grained progression control, respectively.The supplied description of Inverse Painting is truncated.Source: §5.1 · p. 6

5.2. Quantitative Comparison

  • Our method achieves the best performance across LPIPS, CLIP, DINOv2, and FID in the reported comparison.Metrics are averaged across frames within each test video and then across test videos.Source: Table 2 · p. 6, §5.2 · pp. 6–7

5.2.1. Perceptual Distance Profile

  • PDP compares generated and ground-truth perceptual-distance profiles over normalized time; Table 3 reports PDP variants and final-frame reconstruction distance.The final-frame distance should approach zero; the average LPIPS PDP curve has a Time = 1 gap because methods cannot perfectly reconstruct the input.Source: Table 3 · p. 7, §5.2.1 · p. 7, Figure 6 · p. 7

5.3. Ablation

  • Reverse-order LoRA tuning reconstructs the input more fully than the equally trained painting-order ablation, whose bottom-right details remain incomplete.Both models were trained for seven epochs.Source: Figure 7 · p. 7, §5.3 · p. 7

5.4. Qualitative Results

  • The base model reconstructs varied inputs and extends face-trained Loomis drawing to rabbit heads; media transfer preserves objects while changing color and process to match the requested medium.The media-transfer model distinguishes pencil-based Loomis rendering from other artistic formats.Source: §5.4 · pp. 7–8

5.5. Qualitative Comparison

  • Against baselines, Inverse Painting may stall on backgrounds, ProcessPainter yields synthetic eight-frame steps, and PaintsUndo can rush early, while ours follows GT’s sketch-to-detail progression.Figure 8 presents the method comparison; the described progression moves from structural sketch through layered color to added detail.Source: Figure 8 · p. 8, §5.5 · p. 8

6. Conclusion

The conclusion presents video diffusion as capable of recreating painting processes and describes a traditional-media dataset and PDP evaluation. The model exhibits recognizable drawing and painting techniques.

6. Conclusion

  • The work demonstrates that a video diffusion model can recreate a given image’s painting process, supported by a curated dataset focused mainly on traditional media.The authors developed a data-collection pipeline for that dataset.Source: §6 · p. 8
  • The trained model sketches outlines, uses pencil hatching and layering, and depicts object depth and light-shadow interactions; PDP evaluates the painting process.Source: §6 · p. 8

7. Limitation & Future Work

The authors identify data artifacts and training-scope boundaries for the base and media-transfer models, and call for tutorials that convey practical material and tool use.

7. Limitation & Future Work

  • The occlusion detector misses hand shadows, producing dark training-data artifacts that are mainly visible in pencil generations, typically at bottom right.Source: §7 · p. 8
  • The base model struggles with portraits because it was not trained on them; the media-transfer model can fail on content-medium combinations unseen during training.Examples include Loomis on non-portrait drawings and acrylic on human portraits.Source: §7 · p. 8
  • Future tutorial support would need to indicate color selection and mixing, tool choice, and how to apply materials to the canvas.Source: §7 · p. 8

Supplementary Material

The supplementary material expands the dataset-curation description with a painting-video extraction pipeline.

A. Dataset Curation Pipeline

  • The pipeline trims tutorials from the first hand appearance to the last, then localizes the canvas using object detection or, for Loomis portraits, an image-gradient split.The portrait split separates the reference photo from the canvas region.Source: A. Dataset Curation Pipeline · p. 11, §A.1 · p. 11

A.1. Painting Video Extraction Pipeline

  • It samples 30 frames per segment, masks detected hands or brushes, combines median frames to fill persistent occlusions, and removes logos or text by inpainting.Processing 640×360 videos takes approximately the video duration on an NVIDIA RTX A4000 GPU.Source: §A.1 · p. 11

A.2. Data Collecting

The dataset comprises 767 painting-tutorial videos selected from YouTube, with manual review used to exclude poor processing results. Its tutorials span several media and include Loomis portrait instruction.

A.2. Data Collecting

  • The curated dataset contains 767 videos, selected from tutorials with static cameras and minimal canvas movement and manually screened to exclude poor outputs.The selection criteria were intended to simplify temporal alignment.Source: §A.2 · p. 11
  • It includes 81 acrylic, 151 oil, 270 pencil, 28 colored-pencil, and 207 Loomis portrait tutorials.The oil subset includes 142 impressionist landscapes and 9 photorealistic paintings; Loomis tutorials typically show a reference photo beside the canvas.Source: §A.2 · p. 11

A.3. Limitations.

The dataset has limited color-workflow and artist diversity, while tutorial footage can disrupt temporal consistency. The paper describes synthetic reference-frame variations for media transfer and introduces PDP to compare the progression of generated and ground-truth painting videos.

A.3. Limitations.

  • The dataset is pencil-heavy, has limited artist diversity, and includes tutorials with camera movements, zooms, and occlusions that reduce temporal consistency.Its subjects are limited to landscape and portrait drawings using the Loomis method, though the dataset is presented as a foundation for acrylic, oil, and pencil process modeling.Source: §A.3 · p. 11

A.4. Painting Media Transfer Dataset

  • Loomis tutorials require real portrait photographs as references, so the transfer dataset synthesizes aligned reference-frame variations across media.ControlNet generates realistic photos and color paintings from pencil sketches and Loomis portraits; Qwen Image Edit generates pencil sketches and drawing styles from acrylic and oil paintings.Source: §A.4 · p. 11
  • ControlNet preserves reference outlines but loses color, while image-editing models retain color but can alter composition, motivating different tools for pencil and color references.The authors use ControlNet for pencil sketches and Qwen Image Edit for color paintings.Source: §A.4 · pp. 11–12
  • The transfer training combines captions generated for original frames with art-medium labels, then replaces original frames with generated variations during fine-tuning.ControlNet uses the LineArt processor, and its image-generation model is a fine-tuned version of Stable Diffusion 1.5.Source: §A.4 · p. 12

B. Perceptual Distance Profile

  • PDP profiles each video by measuring every frame’s perceptual distance from its final painting; ground-truth profiles form a smooth curve from steep early change to fine-detail refinement.The paper presents this curve as a characteristic painting progression.Source: B. Perceptual Distance Profile · p. 12
  • PDP interpolates profiles to a shared normalized time axis and compares them with L2 distance, allowing videos of different lengths to be compared.A separately normalized score focuses on process speed while accommodating differences in starting frames or final reconstructions; lower scores indicate closer process similarity.Source: B. Perceptual Distance Profile · p. 12

C. Qualitative results

The qualitative comparisons show varied results across painting methods and inputs: the proposed approach has an advantage on a portrait example, while its Loomis art-media variant does not produce a coherent castle result. The figures also compare generated art-media sequences with base-model outputs for castle and Mona Lisa inputs.

C. Qualitative results

  • On a portrait photo, the base model struggles to reconstruct the subject’s head and shows noticeable movement, artifacts absent in the art-media model.The authors attribute the art-media model’s benefit to fine-tuning on portrait photos using the Loomis method.Source: C. Qualitative results · p. 12
  • For the castle example, the base model adds a white background in its final frames, while the Loomis art-media variant fails to produce a coherent result.The authors note that training data primarily uses colored paper or canvas, making completely white backgrounds challenging for the model.Source: C. Qualitative results · p. 12, Figure C · p. 16
  • In the visual comparison, Inverse Painting does not converge to the reference, while PaintsUndo struggles with the abstract oil painting’s content.The figure also shows Process Painter, the proposed method, and the ground-truth process.Source: C. Qualitative results · p. 14
  • The Mona Lisa figure compares linearly sampled art-media outputs against the base-model output, whose final frame closely matches the input image.The figure presents the input image rather than the base model’s final frame.Source: Figure D · p. 17

Related papers

Loading 2511.17344v2…