PaperlayerView exact revision on arXiv

Source-linked AI summary

LociArt: A Unified Paradigm for Free-Style Local Video Stylization

Yuxuan Chen, Feng Xie, Jiagao Hu, Fuhao Li, Zepeng Wang, Fei Wang, Daiguo Zhou

arXiv:2610.36456v1cs.CVcs.AIcs.MM

TL;DR

Local video stylization requires control over both the edited region and its style while preserving other content. LociArt combines paired-data construction, GRNStyle region–style conditioning, and a regional benchmark; GRNStyle achieves the best reported SPS for foreground and background stylization and outperforms evaluated baselines on the benchmark.

  • Problem

    Local stylization needs to specify both the edited region and its style, while mask representations and existing supervision and evaluation have limitations for spatial fidelity and regional assessment.

    Source: §1 · p. 2

  • Method

    LociArt constructs 100K local-stylization pairs and uses GRNStyle’s equal-norm polar style-aware masks and representation–geometry conditioning, evaluated with a benchmark for regional style, preservation, and temporal consistency.

    Source: §1 · pp. 2–3

  • Results

    GRNStyle achieves the best SPS on the LociArt Benchmark for foreground stylization (0.407) and background stylization (0.705), and outperforms evaluated baselines overall.

    Source: §5.2 · p. 8, §6 · p. 11

  • Takeaways & Limitations

    The framework supports zero-shot multi-region compositions from single-region, single-style supervision, reducing the need for exhaustive paired data.

    Source: §6 · p. 11

    Temporal consistency can degrade for fast-moving objects, especially near motion boundaries, where stable region alignment and coherent stylization are difficult to maintain.

    Source: H Limitations and Future Work · p. 29

Abstract

from arXiv · show

Video stylization typically transforms entire scenes, while local artistic editing requires precise region--style association, spatial control, and temporal consistency. We introduce LociArt, a unified paradigm for free-style local video stylization that explicitly specifies where to stylize and what style to apply while preserving non-target content. At its core, GRNStyle builds on Generative Refinement Networks (GRN) and introduces a polar style-aware mask that embeds style identities as equal-norm directions within selected regions. Complementary representation--geometry conditioning combines encoded mask features with raw-mask spatial support, integrating spatiotemporal context with geometric control. To provide paired supervision, we develop an automated pipeline combining global stylization, video object removal, matting, and compositing, and construct the LociArt-100K Dataset with 100K local stylization pairs. We further establish the LociArt Benchmark, comprising 80 source videos and 84 target masks covering 168 foreground/background stylization cases, with a Style Prototype Similarity (SPS) evaluation scheme for regional style quality and complementary measures of non-target preservation and temporal consistency. Experiments show that GRNStyle outperforms existing methods in regional style quality and spatial control while maintaining strong temporal consistency. Qualitative examples further illustrate joint stylization of multiple regions with the same or different learned styles, despite training only on single-region, single-style pairs. Together, these components provide a unified foundation for learning and evaluating controllable local video stylization. Project page: https://higherhu.github.io/lociart/.

1 Introduction

LociArt frames local video stylization as specifying both the region and style while preserving non-target content. It combines paired-data construction, region–style conditioning, and region-aware evaluation to address the task’s modeling and supervision needs.

1 Introduction

  • Binary masks define editing extent but not style identity, leaving the model to associate a separately supplied style with the selected region.The paper also identifies a trade-off between learned spatiotemporal context and spatial fidelity in mask representations.Source: §1 · p. 2
  • Global stylization pairs and broad video-editing datasets do not directly provide paired supervision for local artistic changes, non-target preservation, and natural moving-boundary transitions.Global or video-averaged evaluation can also conceal weak regional style, changes outside the target, and transient failures.Source: §1 · p. 2
  • The automated pipeline combines global stylization, video object removal, matting, and compositing to construct 100K curated local-stylization pairs.Each example pairs a source video, editing mask, style instruction, and locally stylized target.Source: §1 · pp. 2–3
  • GRNStyle binds style identities to selected regions with equal-norm polar directions and combines VAE-encoded mask features with raw-mask support for spatiotemporal context and geometric control.Source evidence guides content preservation during refinement.Source: §1 · p. 3
  • The LociArt Benchmark contains 80 source videos and 84 object–mask units evaluated in 168 foreground and background stylization cases, using SPS plus preservation and temporal-consistency measures.The contributions also describe qualitative multi-region, multi-style compositions learned from single-region, single-style supervision.Source: §1 · p. 3
  • LociArt introduces a unified paradigm for specifying where to stylize and what style to apply while preserving non-target content and temporal consistency.Its contributions include GRNStyle, the LociArt-100K Dataset, and the LociArt Benchmark.Source: §1 · pp. 2–3

2 Related Work

Prior work spans image style transfer, video stylization, and broader video editing, but the paper positions local artistic video stylization as requiring explicit spatial control and paired supervision. LociArt contributes aligned local-stylization data and region-aware evaluation.

Image style transfer

  • Image style-transfer methods progress from per-image optimization to feed-forward arbitrary transfer and latent-diffusion approaches, while localized editing requires explicit spatial control.StyleShot separates style from content, and TeleStyle [25] extends curated-triplet transfer to video.Source: §2 · p. 3

Localized stylization

  • Localized image and video methods use masks, attention, semantic-region matching, or spatiotemporal alignment, while UniVST combines propagated masks with latent and attention manipulation and optical-flow smoothing.The paper distinguishes its focus as paired-supervision local artistic video stylization with fixed equal-norm polar codes binding style identity to editable regions.Source: §2 · p. 3

Video stylization and generative editing

  • Video stylization and editing methods use style adapters, stylization triplets, masks, or context-conditioned inpainting, while GRNStyle extends source-evidence editing with encoded mask features and raw-mask geometry.The passage describes VISTA [7], MoStyle, DreamStyle [20], VACE, VideoPainter, GRN, and GRNEdit [23].Source: §2 · pp. 3–4

Data and evaluation

  • LociArt-100K aligns content, regions, styles, and targets, and its benchmark measures regional style, non-target preservation, temporal consistency, and VLM-based perceptual quality.The related work contrasts these with existing image and video quality, style-similarity, and temporal-consistency evaluations.Source: §2 · p. 4

3 LociArt-100K Dataset Construction

LociArt-100K constructs paired local-stylization targets by combining globally stylized source videos with stylized object-removed backgrounds. Matting supports localized compositing, and human verification filters the resulting examples.

3 LociArt-100K Dataset Construction

  • The dataset pipeline globally stylizes source and object-removed videos, then recombines their components into complementary foreground- and background-stylized targets.Each example contains a source video, binary editing mask, style instruction, and locally stylized target.Source: §3 · p. 4
  • Salient objects are identified and tracked masks are checked before guiding object removal, which reconstructs background content exposed by stylization-induced contour changes.The reconstructed background helps avoid original-object remnants during compositing.Source: §3 · p. 4
  • Candidate stylizations of source and reconstructed-background videos pass cascaded checks of style quality, content preservation, and temporal quality before compositing.The pipeline runs Ditto and a fine-tuned VACE stylizer on both videos.Source: §3 · p. 5
  • Separate alpha mattes follow the original and stylized object contours through time and support composites that stylize either foreground or background.The directional mask is the object mask for foreground stylization and its complement for background stylization.Source: §3 · p. 5
  • Human verification checks content preservation, boundary quality, and temporal consistency, retaining 100K video pairs across 18 style categories.The pipeline retains only samples that annotators qualify.Source: §3 · p. 5

4 GRNStyle: Free-Style Local Video Stylization

GRNStyle uses source evidence and region–style masks to guide local video stylization. Its conditioning combines equal-norm style codes with anchored mask evidence and geometric support, and supports multi-region compositions after single-region, single-style training.

4.1 Problem Formulation and Binary Evidence

  • GRNStyle formulates local stylization as predicting a target video from source video, editing mask, and text instruction while transforming the selected region and preserving other content.It builds on GRN binary-state refinement and GRNEdit source evidence for retaining or changing target bits.Source: §4.1 · p. 5

4.2 Equal-Norm Style–Space Conditioning

  • The polar style-aware mask assigns each style a fixed unit-radius direction and embeds its coordinates in the selected region, binding style identity to spatial position.Styles differ by direction rather than magnitude, and unedited positions remain zero.Source: §4.2 · p. 6

4.3 Representation–Geometry Mask Conditioning

  • Representation–geometry conditioning combines encoded mask features with binary-occupancy support to retain spatiotemporal context while localizing mask evidence.Subtracting the encoding of an empty mask gives zero mask evidence for an absent edit; pooled occupancy support localizes projected mask injection.Source: §4.3 · pp. 6–7

4.4 Evidence-Guided Generation and Compositional Inference

  • GRNStyle projects source and anchored mask evidence into a shared space, gates it using visual evidence and text, and injects gated residuals into model blocks.Training uses full-video bitwise cross-entropy with style-specific and generic mask-following instructions; inference iteratively refines random target bits.Source: §4.4 · p. 7
  • Disjoint regions can be composed in one refinement trajectory under a generic mask-following instruction, without extra channels or conditioning tokens.Qualitative examples demonstrate compositions of learned styles despite training only on single-region, single-style pairs.Source: §4.4 · p. 7

5 Experiments

Across the LociArt Benchmark, GRNStyle leads on regional style quality while maintaining strong preservation and temporal consistency. User ratings and ablations support its localization and conditioning choices, while additional experiments examine replay-free style expansion and data construction.

5.1 Experimental Setup

  • The benchmark evaluates 80 source videos and 84 object-mask units under foreground and complementary-background stylization, totaling 168 cases, with SPS, CLIP-T, PSNR, TC, and separate Style and Preserve scores.Training uses the 2B GRNStyle model, fully fine-tuned on LociArt-100K with 8 × NVIDIA H200 GPUs.Source: §5.1 · p. 8

5.2 Comparison with Existing Methods

  • GRNStyle achieves the best foreground and background SPS (0.407 and 0.705), best background CLIPT (0.199), and strong preservation and temporal consistency.It has second-best background PSNR (26.15 dB), near-best TC (0.983), and the highest Style score (4.62) with a high Preserve score (4.64).Source: §5.2 · pp. 8–9

5.3 User Study

  • In the user study, GRNStyle achieves the highest mean ratings for style fidelity (4.34) and background preservation (4.47), outperforming CoinVE-Edit on both criteria across all ten evaluated styles.It improves preservation over Seedance 2.5 for every style and style fidelity for seven styles.Source: §5.3 · p. 9

5.4 Ablation of Mask Conditioning

  • Representation–geometry conditioning improves preservation without materially changing SPS or CLIPT, while polar encoding improves SPS; combining them yields the strongest style-conditioning and spatial-control trade-off.The ablation compares the RG and polar components; Figure 6 illustrates the localization improvement over raw encoding.Source: §5.4 · pp. 9–10, Figure 6 · p. 10, Table 2 · p. 10

5.5 Learning New Styles without Replay

  • Expanding from 10 to 15 styles without replay reduces old-class SPS by 0.022 with polar conditioning, versus 0.102 with binary masks; new-class scores are slightly higher across metrics.The expansion uses only the five new classes and evaluates retention and acquisition across 1,680 original-style and 840 new-style cases.Source: §5.5 · p. 10, Table 3 · p. 11

5.6 Effect of the Data Construction Pipeline

  • Object removal eliminates ghost curls and head outlines in background edits, while matting follows actual foreground contours to preserve fine boundaries and avoid clipped stylized features.Figure 7 compares direct compositing, removal without matting, and the complete pipeline.Source: §5.6 · p. 11, Figure 7 · p. 11

6 Conclusion

LociArt unifies local video stylization through GRNStyle, paired-data construction, and a region-aware benchmark. The conclusion reports that GRNStyle outperforms evaluated baselines and supports zero-shot multi-region compositions from single-region, single-style training.

6 Conclusion

  • GRNStyle combines polar style codes and representation–geometry conditioning, outperforms evaluated benchmark baselines, and supports zero-shot multi-region compositions from single-region, single-style supervision.The conclusion describes this as reducing the need for exhaustive paired data.Source: §6 · p. 11

A Detailed Data Pipeline

  • The automated pipeline aligns source, removed, stylized, and alpha videos, then uses object removal, separate mattes, and compositing to produce natural boundary transitions.Object removal supplies clean background content for exposed areas, while mattes for original and stylized objects guide compositing along their respective contours.Source: A Detailed Data Pipeline · p. 16

A.1 Object Inventory and Mask Filtering Details

  • Object inventory and tracked instance masks are filtered for semantic correctness and temporal coverage; accepted masks must be nonempty in over 40% of frames and occupy at most 85% of any frame.The pipeline rejects missing, flickering, or incorrectly assigned masks.Source: §A.1 · pp. 16–17

A.2 Global Stylization Quality Filtering Details

  • Global-stylization candidates pass two cascaded VLM gates, each scoring style adherence, content preservation, structure, style consistency, and temporal quality against set thresholds.The source-video gate is evaluated first; candidates proceed only if it passes, and both source and object-removed pairs must pass before matting and compositing.Source: §A.2 · p. 17

A.3 Human Verification Details

  • Human review retains composites only when style, semantic content, non-target preservation, mask boundaries, and temporal consistency meet the stated requirements.Style-consistent changes in color, texture, detail, and object contours are permitted; typical rejection cases include style errors, unintended content changes, incomplete masks, and temporal artifacts.Source: §A.3 · pp. 17, 19
  • The pipeline yields 100K qualified pairs across 18 style categories, with 52.4% passing global-stylization filtering and 65% passing subsequent human verification.Figure 9 summarizes style distribution and foreground-object mask areas, not complementary background editing-mask areas.Source: §A.3 · p. 19, Figure 9 · p. 19

B.1 Codebook Assignment and Input Normalization

  • The fixed 100-slot circular codebook assigns styles distinct unit directions, encoding style by angle while reserving the third channel for shared binary occupancy.The mapping remains fixed across training and inference; after input normalization, the first two channels encode direction and the third retains occupancy.Source: §B.1 · p. 20

B.2 Why Circular Rather Than Spherical Codes?

  • Circular codes keep styles farther from the empty condition than the spherical alternative, while the shared occupancy cue avoids class-dependent edit/keep inference.The 100-slot VAE/HBQ validation found no exact token collisions.Source: §B.2 · p. 20

B.3 Interaction Error and Exact Local Readout

An additive readout cannot generally represent the interaction between a region mask and style, whereas polar encoding supplies that interaction explicitly. The evaluation protocol also assesses local stylization across complementary regional, temporal, preservation, and full-video axes.

B.3 Interaction Error and Exact Local Readout

  • The best additive readout has positive minimum squared instruction error when at least two distinct styles have positive probability, bounded above by β(1 − β) ≤ 1/4.The error depends on mask occupancy and the variance of style directions; it reaches 1/4 when β = 1/2 and μ = 0.Source: §B.3 · p. 21
  • Polar encoding computes the region–style interaction before video encoding, unlike separate additive conditions; this readout distinction is not an error bound on nonlinear generators.A nonlinear network may learn the interaction from separate inputs, while the representation supplies it before encoding.Source: §B.3 · pp. 20–21

C Details of the Evaluation Protocol

  • The evaluation protocol measures regional style, temporal consistency, non-target preservation, and full-video perceptual quality with complementary metrics.It names SPS and regional CLIPT, TC and regional PSNR, and full-video VLM Style and Preserve scores.Source: C Details of the Evaluation Protocol · p. 21

C.1 Benchmark Construction and Evaluation Units

The LociArt Benchmark evaluates foreground and background stylization across aligned source videos and object-mask units. Its protocol localizes regional style scores and pairs them with temporal, preservation, and full-video assessments.

C.1 Benchmark Construction and Evaluation Units

  • The benchmark comprises 80 source videos and 84 object-mask units, each with 81 aligned frames, evaluated for both foreground and background stylization.The complete ten-style comparison grid contains 1,680 results across 168 cases.Source: §C.1 · pp. 21–22

C.2.2 Style Prototype Similarity (SPS)

  • 93.61% macro-averaged accuracy is achieved in leave-one-source-out validation of SPS style classification; Figure 10 shows the prototype visualization and confusion matrix.Samples from each held-out source are excluded during construction of its reference prototypes, and frame predictions are aggregated by video-level majority vote.Source: Figure 10 · p. 22, §C.2.2 · p. 23

C.2.1 Localized Region Construction

  • SPS and regional CLIPT use shared localization: alpha-guided crops for foreground edits and full frames with foreground tokens excluded for background edits.The output alpha allows localization to follow stylization-induced contour changes.Source: §C.2.1 · p. 22

C.2.2 Style Prototype Similarity (SPS)

  • SPS compares an edited region with a class-specific bank of visual style prototypes rather than measuring text–image similarity.It is a prototype-based scheme built on Contrastive Style Descriptors.Source: §C.2.2 · p. 22
  • The prototype bank contains 80 centroids across 18 styles, selected by silhouette score from k-means partitions with three or five candidates per style.Multiple prototypes represent within-style variation.Source: §C.2.2 · p. 23

C.2.3 Regional CLIPT

  • Regional CLIPT complements SPS by comparing localized image embeddings with the canonical style phrase, using the same region selection but different scoring targets.The alpha video defines region selection; it is not the RGB image input to the encoder.Source: §C.2.3 · p. 24

C.3.1 Temporal Consistency; C.3.2 Non-Target Preservation

  • TC measures adjacent-frame feature consistency globally, while regional PSNR measures source fidelity outside the intended edit using alpha-derived preserved-region cores.The anti-aliased transition band is excluded from the PSNR valid region.Source: §C.3.1 · p. 24, §C.3.2 · p. 24

C.4 VLM Evaluation

  • The VLM evaluation reports separate 0–5 Style and Preserve scores, so stylization quality and non-target preservation are assessed independently.Style combines correctness, edit-region coverage, and cross-frame stability; Preserve assesses source fidelity and temporal stability outside the edit.Source: §C.4 · pp. 24–25

D.1 Training Setup

GRNStyle-2B is trained for local control and evaluated with region-wise validation measures. The supplied comparisons also report its inference cost and performance when applied to global stylization without retraining.

D.1 Training Setup

  • The model uses 100K paired examples across 18 styles, fine-tunes the GRN-2B backbone and conditioning branches, and keeps the video tokenizer and T5 encoder frozen.With probability 0.50, the style-specific instruction is replaced by a generic mask-following instruction; source/mask residuals enter at transformer blocks 1, 13, and 25.Source: §D.1 · pp. 26–27
  • From update 2,000 to 16,000, mean SPS rises from 0.512 to 0.554 and mean PSNR from 23.01 to 26.00 dB, with slower changes after update 10,000.SPS measures the edited region and PSNR the preserved region; the plotted checkpoint lines are not smoothed.Source: §D.1 · p. 27, Figure 11 · p. 27, Figure 12 · p. 27

E Inference Efficiency

  • GRNStyle-2B requires 34.5 s and 27.4 GB per inference, versus 39.2 s and 23.7 GB for fine-tuned VACE-1.3B.The reported comparison is a 12.0% latency reduction with 15.6% more memory; Table 6 summarizes inference cost.Source: E Inference Efficiency · p. 27, Table 6 · p. 27

F Global Stylization Comparison

  • On the same 80 benchmark videos and ten-style subset, GRNStyle (2B backbone) achieves highest CSD (0.683), second-highest VLM Style (4.67), matches EchoStyle in CLIPT (0.190), and remains competitive in TC (0.977).Source: F Global Stylization Comparison · p. 28, Table 7 · p. 28

G Robustness to Imperfect Masks

The robustness evaluation tests GRNStyle with five types of imperfect masks at light and strong severity, without retraining. Light perturbations leave metrics near baseline, while stronger erosion, dropout, and dilation affect measures tied to their geometric changes.

G Robustness to Imperfect Masks

  • The evaluation uses 400 video–style cases per variant from a 40-case benchmark subset and ten styles, keeping prompts, seeds, guidance, and sampling fixed.The perturbed mask replaces the original in both polar style encoding and geometric support, with no retraining or parameter update.Source: G Robustness to Imperfect Masks · p. 29
  • Five perturbation types are tested at light and strong severity: boundary noise, dilation, erosion, temporal jitter, and frame dropout.Boundary flips occur within a six-pixel contour band; dilation and erosion alter edit-region radii, jitter varies radius by frame, and dropout clears entire masks in some frames.Source: G Robustness to Imperfect Masks · p. 29
  • Light perturbations keep absolute SPS and CLIPT changes within 0.006 and PSNR changes within 0.49 dB, while TC remains 0.972.Under strong perturbations, erosion and dropout reduce stylization quality, while dilation lowers background PSNR from 26.3 to 22.2 dB as it modifies preserved content.Source: G Robustness to Imperfect Masks · p. 29

H Limitations and Future Work

The paper identifies limitations in temporal consistency for fast-moving objects, shadow preservation in constructed training targets, and evaluation of boundary adherence. It describes the conditions behind each limitation and corresponding future-work directions.

H Limitations and Future Work

  • Temporal consistency can degrade for fast-moving objects, especially near motion boundaries, causing local flicker or inconsistent boundaries.The authors identify motion-aware conditioning and boundary tracking as future-work directions.Source: H Limitations and Future Work · p. 29
  • The object-removal model may remove cast shadows, leaving some paired targets without scene-consistent shadows and potentially encouraging the model to erase them.Future work should improve shadow-aware data construction and add explicit supervision for preserving scene-consistent shadows.Source: H Limitations and Future Work · p. 29
  • The evaluation suite lacks a boundary-adherence metric, and styles may intentionally expand or shrink objects, so no single fixed boundary fits every output.The authors call for style-aware evaluation that distinguishes intended shape deformation from mask leakage and inadvertent boundary violations.Source: H Limitations and Future Work · pp. 29–30

I Additional Qualitative Results

Additional qualitative results show GRNStyle applied to foreground and background regions with diverse target styles. The displayed frames illustrate regional stylization and preservation of surrounding content over time.

I Additional Qualitative Results

Related papers

Loading 2610.36456v1…