PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens

Yiying Yang, Wei Cheng, Sijin Chen, Honghao Fu, Xianfang Zeng, Yujun Cai, Gang Yu, Xingjun Ma

arXiv:2603.02138v1cs.CV

TL;DR

Vector animation generation is hindered by inefficient raw Lottie JSON representations and limited standardized evaluation. OmniLottie addresses these issues with a Lottie tokenizer, a pretrained VLM, the MMLottie-2M dataset, and MMLottie-Bench, producing high-quality, semantically aligned animations while retaining challenges in sequence validity and generalization.

  • Problem

    Raw Lottie JSON contains extensive structural metadata and formatting tokens, while vector animation generation lacks mature standardized benchmarks and metrics.

  • Method

    OmniLottie converts Lottie JSON into compact command–parameter token sequences and trains a pretrained VLM on MMLottie-2M for multimodal autoregressive generation.

  • Results

    OmniLottie produces high-quality vector animations with strong visual fidelity and semantic alignment across multimodal tasks.

  • Takeaways & Limitations

    The framework unifies text-to-, text-image-to-, and video-to-Lottie generation using compact token sequences and multimodal inputs.

  • Takeaways & Limitations

    Autoregressive decoding can yield invalid sequences, while diverse-scenario generalization and context length remain challenges for complex animations.

Abstract

from arXiv · show

OmniLottie is a versatile framework that generates high quality vector animations from multi-modal instructions. For flexible motion and visual content control, we focus on Lottie, a light weight JSON formatting for both shapes and animation behaviors representation. However, the raw Lottie JSON files contain extensive invariant structural metadata and formatting tokens, posing significant challenges for learning vector animation generation. Therefore, we introduce a well designed Lottie tokenizer that transforms JSON files into structured sequences of commands and parameters representing shapes, animation functions and control parameters. Such tokenizer enables us to build OmniLottie upon pretrained vision language models to follow multi-modal interleaved instructions and generate high quality vector animations. To further advance research in vector animation generation, we curate MMLottie-2M, a large scale dataset of professionally designed vector animations paired with textual and visual annotations. With extensive experiments, we validate that OmniLottie can produce vivid and semantically aligned vector animations that adhere closely to multi modal human instructions.

1 Fudan University 2 StepFun 3 HKU MMLab 4 University of Queensland

OmniLottie is presented as a versatile autoregressive model for high-quality Lottie animation generation. The paper identifies project and corresponding authorship roles.

  • OmniLottie supports text-to-Lottie, text-image-to-Lottie, and video-to-Lottie generation from interleaved multimodal instructions.These capabilities target creative and design-oriented tasks.
  • The framework is designed for high-quality Lottie animation generation.
  • The paper identifies a project lead and corresponding authors.

1. Introduction

OmniLottie targets multimodal vector-animation generation using Lottie, a structured and widely adopted representation of shapes, effects, and motion. It introduces a tokenizer, dataset, and end-to-end framework to improve generation efficiency and quality.

  • Vector animations are lightweight, editing-friendly, and resolution-independent, supporting appearance and motion manipulation in design workflows.Autonomous multimodal generation could lower creation barriers and accelerate professional design pipelines.
  • Lottie stores shape, effect, and motion parameters in one hierarchical JSON representation with layers and inheritance.Its cross-platform compatibility and unified representation motivate its use for multimodal generation.
  • Raw Lottie JSON contains extensive formatting tokens and invariant metadata that occupy space without representing meaningful geometric or motion cues.This makes direct generation inefficient and motivates a more compact representation.
  • The Lottie tokenizer converts raw JSON into compact sequences of shape, effect, and animation commands with associated parameters.The serialization supports end-to-end autoregressive training using VLM training practices.
  • MMLottie-2M pairs two million Lottie animations with text descriptions, reference images, and video demonstrations for multimodal generation research.The dataset supports Text-to-, Text-Image-to-, and Video-to-Lottie tasks.
  • Experiments report that OmniLottie outperforms strong baselines in visual fidelity and semantic alignment across diverse multimodal prompts.The framework is presented as generating high-quality vector animations that follow multimodal instructions.

2. Related Work

Prior vector-animation methods often rely on simple interpolation or external motion priors, whereas OmniLottie jointly synthesizes appearance, effects, and motion in a unified autoregressive framework.

  • Early SVG-based methods relied on simple interpolation, which limited motion expressiveness.
  • Later approaches used motion priors from reference GIFs or text-to-video models, while commercial tools automated keyframe generation.
  • Lottie unifies shapes, effects, and animations in one JSON representation, allowing VLMs to apply instruction-following and multimodal reasoning.
  • The dataset pipeline converts SVG assets into static Lottie files, applies randomized animation effects, and gathers professionally created animations from five platforms.
  • Each animation is normalized spatio-temporally, rendered as video, sampled for random keyframes, and annotated at multiple granularities.Annotations emphasize geometric structure, color attributes, and motion characteristics.
  • OmniLottie jointly synthesizes appearance, effects, and motion within a unified autoregressive framework.Autoregressive visual generation has also been applied to images, videos, vector graphics, and 3D content.

3. The MMLottie-2M Dataset

MMLottie-2M is a large-scale curated dataset for multimodal Lottie generation, paired with annotations and supported by the standardized MMLottie-Bench evaluation protocol. The benchmark measures visual quality and semantic alignment on real and synthetic evaluation subsets.

  • Dataset scope: MMLottie-2M is the first large-scale Lottie dataset, supporting multimodal generation tasks and standardized evaluation through MMLottie-Bench.The tasks are Text-to-Lottie, Text-Image-to-Lottie, and Video-to-Lottie.
  • Data curation: The dataset collects animations from LottieFiles, IconScout, Flaticon, Iconfont, and Icons8.
  • Data curation: Curation removes base64 image layers, audio and camera layers, After Effects expressions, and files containing non-parameterizable layers.The filtering aims to retain clean, fully parameterizable data.
  • Data curation: Canonical motion templates are applied to static SVG-derived Lottie files through automated keyframe injection, broadening motion and animated-layer coverage.The process reduces the path distribution gap without additional manual animation efforts.
  • Annotation: Annotations use coarse-to-fine VLM descriptions covering subjects, objects, motion, color, style, and finer temporal details across frames.
  • Benchmark: MMLottie-Bench addresses immature evaluation standards by measuring visual quality and semantic alignment with multimodal inputs.
  • Benchmark: The Real Subset contains 450 disjoint samples from artist-designed animations, with 150 samples assigned to each of the three generation tasks.It is intended to test genuinely unseen, real-world content.
  • Benchmark: A complementary Synthetic Subset uses instruction-based synthesis to improve benchmark robustness against contamination from future models trained on similar data.The full synthesis process is recorded and released for transparency.

4. Proposed Method: OmniLottie

OmniLottie combines a Lottie tokenizer with a pretrained vision-language model to generate vector animations from interleaved text, image, and video instructions. The tokenizer converts verbose hierarchical JSON into compact, structured command and parameter sequences while preserving the animation hierarchy and generative flexibility.

  • Motivation: Raw Lottie JSON contains extensive structural metadata and formatting tokens that distract models from learning shapes, effects, and temporal dynamics.The proposed tokenizer addresses this redundancy by focusing representation on animation-relevant content.
  • Lottie Tokenizer: The tokenizer abstracts animations into compact command sequences and control parameters while supporting precomposition, solid, null, shape, and text layers.Each layer type receives specialized parsing to preserve structural integrity and temporal accuracy.
  • Lottie Tokenizer: The encoding process serializes metadata, layer types, and type-specific attributes into sequential tokens, then decodes generated tokens back into complete Lottie JSON.Low-level JSON formatting is ignored while the scene hierarchy and parent-child relationships are retained.
  • Token Representation: An offset-based vocabulary assigns distinct discrete ranges to temporal, spatial, transformation, and style parameters to avoid token conflicts.Continuous parameters are mapped using type-specific scale factors and vocabulary offsets.
  • Model Architecture: OmniLottie adds Lottie vocabulary embeddings to Qwen2.5-VL and trains it autoregressively to generate Lottie tokens conditioned on multimodal instructions and prior tokens.The generated token sequence is detokenized into a vector animation.

5. Experiments

Experiments evaluate OmniLottie against language, vision-language, optimization-based, and commercial baselines across text, text-image, and video generation tasks. OmniLottie achieves the strongest reported performance and qualitative results, while ablations attribute gains to moderate SVG mixing and the Lottie tokenizer.

  • Evaluation Setup: OmniLottie is evaluated against task-specific baselines across Text-to-Lottie, Text-Image-to-Lottie, and Video-to-Lottie using quality, alignment, reliability, and efficiency metrics.Efficiency includes generated-token length and average generation time per sample.
  • Quantitative Evaluation: OmniLottie consistently outperforms baselines across tasks, with near-perfect Text-to-Lottie success rates and strong FVD and motion alignment.Recraft is competitive on object alignment and CLIP but trails in token efficiency and motion fidelity; open-source language models have very low or zero success rates.
  • Quantitative Evaluation: OmniLottie ranks first in Text-Image-to-Lottie FVD, object alignment, and motion alignment while maintaining high reliability.AniClipart and Livesketch can achieve reasonable CLIP scores when successful but have low success rates and longer runtimes.
  • Quantitative Evaluation: In Video-to-Lottie, OmniLottie achieves the best FVD, PSNR, SSIM, and DINO scores and produces richer token sequences than all baselines.The richer sequences support more expressive and detailed vector animation generation.
  • Qualitative Evaluation: Qualitative comparisons show stronger prompt, image, and video alignment from OmniLottie, while competing methods exhibit failures, weak motion, raster-to-vector dependence, or low-quality outputs.For Text-Image-to-Lottie, OmniLottie produces native vector animations with shorter latency and high success rates.
  • Ablation Studies: Moderate Lottie-SVG mixing yields the best Text-to-Lottie and Text-Image-to-Lottie performance by balancing geometric richness with motion complexity.Excessive SVG biases the model toward simplistic motion and reduces Motion Alignment scores.
  • Ablation Studies: The Lottie tokenizer consistently improves performance across tasks and metrics compared with pretrained and raw-JSON fine-tuned Qwen2.5-VL configurations.The structured compact representation improves generation quality and computational efficiency.

6. Conclusions and Limitations

OmniLottie is presented as a unified framework for multimodal vector-animation generation, supported by a tokenizer, pretrained VLM, and MMLottie-2M. Its main limitation is that autoregressive decoding can still produce invalid sequences and may struggle with complex animations.

  • Contributions: OmniLottie handles Text-to-, Text-Image-to-, and Video-to-Lottie generation using compact token sequences, a pretrained VLM, and MMLottie-2M.The framework combines the Lottie tokenizer with a pretrained VLM and a large-scale dataset.
  • Evaluation: The paper includes qualitative comparisons and ablations covering Video-to-Lottie generation, data composition, and the Lottie tokenizer.These studies are referenced through Figures 6–8 and Tables 2–3.
  • Ablations: 30% SVG data achieves optimal performance by improving geometric understanding while maintaining motion complexity.The result comes from the SVG-data mixing ablation.
  • Limitations: Autoregressive decoding can still yield invalid sequences despite higher success rates than direct JSON learning.The limitation motivates constrained decoding, renderability rewards, and integration with professional tools as future directions.

7. Disclaimer

The supplied disclaimer states that MMLottie-2M is provided for research and non-commercial purposes and that the appendix contains additional evaluation material. It also describes supplementary qualitative results and failure analyses.

  • Dataset disclaimer: MMLottie-2M is provided for research and non-commercial purposes only.The dataset is supplied without warranties, and users are responsible for compliance with applicable laws and third-party rights.
  • Appendix scope: The appendix includes additional qualitative results and systematic failure-mode analysis for OmniLottie and baseline methods.The analysis uses a unified five-level failure taxonomy.
  • Additional results: Additional examples cover Text-to-Lottie, Text-Image-to-Lottie, and Video-to-Lottie across varied complexity, object types, and motion patterns.These examples are reported in Figure 7.

A.2.2. Failure Analysis of OmniLottie

OmniLottie’s failures are concentrated in structural generation and rendering rather than specification compliance. Failure rates rise as inputs become more complex, with rendering errors forming the largest category.

  • Failure rates: 2.7%, 8.0%, and 9.3% are OmniLottie’s failure rates for Text-to-Lottie, Text-Image-to-Lottie, and Video-to-Lottie, respectively.The ordering indicates increasing failures with input complexity.
  • Structural failures: Approximately 35% of failures are structural generation failures involving valid JSON with an empty layers array.These failures often occur with ambiguous prompts or complex visual inputs that are difficult to decompose into vector primitives.
  • Rendering failures: Approximately 65% of failures are rendering-level failures despite generated layers.Subcategories include missing styles, temporal visibility errors, opacity or scale errors, and off-canvas positioning.
  • Task-specific patterns: Video-to-Lottie is more susceptible to temporal errors, while Text-Image-to-Lottie shows more style-related failures.The distinction reflects different difficulties in motion extraction and matching textual semantics with visual appearance.

A.2.3. Failure Analysis of LLM/VLM Baselines

Baseline methods fail at different stages, ranging from schema incompatibility to rendering, pipeline, and input-dependency problems. OmniLottie achieves higher success and lower successful-generation cost in the supplied comparisons.

  • LLM/VLM failures: 0.0% success across all tasks is reported for Qwen2.5-VL because of schema hallucination and incompatible JSON conventions.Examples include using “version” instead of “v”, “frames” instead of keyframe interpolation, and “duration” instead of “op”.
  • Rendering and pipeline failures: GPT-5 and Gemini produce schema-valid JSON but exhibit rendering errors, while Recraft reaches 100% Lottie-generation success but fails in 22.7% of Lottie-to-video renderings.Reported rendering problems include geometric degeneration, transform-stack errors, and keyframe discontinuities.
  • Optimization-based failures: AniClipart and LiveSketch primarily fail through input dependency and optimization non-convergence, with failure rates of 92.7% and 52%, respectively.AniClipart depends on semantic keypoints that may be absent in abstract shapes or geometric patterns.
  • Comparative success: OmniLottie achieves 90.7–97.3% success, compared with 0% for Qwen2.5-VL, 7.3% for AniClipart, 48.0% for LiveSketch, and 77.3% for Recraft.The comparative summary associates these outcomes with different failure levels across methods.
  • Cost-benefit analysis: 31s per successful animation is reported for OmniLottie, yielding a 52× speedup over LiveSketch and 530× over AniClipart.The comparison uses 28.57s runtime and 92.0% success for OmniLottie.
  • Interpretation: The authors attribute OmniLottie’s relative robustness to Lottie-specific training and direct generation, while identifying rendering-level failures as remaining improvement targets.They point to enhanced training-data coverage and numerical precision as possible remedies.

B.3. Correlation Analysis

Automated Object Alignment and Motion Alignment metrics correlate positively with human rankings, while the tokenizer organizes Lottie parameters into structured, recoverable representations. The framework separates numerical, textual, and structural information to support efficient vector-animation generation.

  • Correlation Analysis: Motion Alignment correlates strongly with human judgments, reaching Pearson r = 0.4823 with p < 0.001.
  • Correlation Analysis: Object Alignment and Motion Alignment both show positive Pearson, Spearman, and Kendall correlations with human rankings.The reported p-values are highly significant at p < 0.001.
  • Tokenizer Design: The tokenizer maps parameter categories into distinct vocabulary regions, preventing semantic overlap while preserving within-category numerical relationships.The offset ranges are selected using dataset-wide parameter distributions and outlier filtering.
  • Sequence to Token Conversion: Continuous, text-based, and structural Lottie parameters receive specialized tokenization procedures within a unified vocabulary.Text attributes use the Qwen2.5-VL tokenizer, while numerical values use type-specific scaling and offsets.
  • Token-to-Sequence Reconstruction: Detokenization parses command boundaries, recovers numerical values, decodes text, and reconstructs valid renderable Lottie JSON.The process reverses parameter transformations and restores padding markers and hierarchical relationships.

D.2. Dataset Statistics

MMLottie-2M combines web-crawled and SVG-derived animations with varied temporal and spatial characteristics. Normalization provides consistent training inputs while retaining source diversity and aspect ratios.

  • Source Distribution: SVG-derived animations comprise 52.5% of MMLottie-2M, while web-crawled samples come from five major online platforms.LottieFiles contributes 42.3% of the web-crawled portion, followed by IconScout, Flaticon, Iconfont, and Icons8.
  • Temporal Statistics: Web-crawled animations average 3.2 seconds in duration, compared with 2.8 seconds for SVG-derived animations.The web-crawled distribution is long-tailed, whereas SVG-derived durations are more concentrated between 2 and 4 seconds.
  • Temporal Statistics: Temporal normalization maps animations to the 0 to 60 range, producing an average normalized duration of approximately 90 frames at 30 fps.This preserves a consistent temporal resolution for model training.
  • Spatial Statistics: Square resolutions account for approximately 55% of web-crawled animations, with 512 × 512 the most common resolution at 18.47%.All animations are normalized to 512 × 512 while preserving original aspect ratios through center alignment.

D.3. Caption Statistics and Semantic Analysis

MMLottie-2M annotations provide concise and detailed descriptions of visual content and motion, while semantic analyses show broad coverage across object categories and animation types. The benchmark evaluates object and motion alignment using multimodal prompts and automated scoring.

  • Caption Length Distribution: Overview captions average 86 words, while detailed temporal descriptions average 114 words.The two-level annotation strategy spans concise summaries and frame-by-frame accounts of animation progression.
  • Vocabulary Analysis: Annotation vocabulary frequently covers geometric shapes, objects, colors, motion verbs, and directional modifiers.Terms such as rotate, scale, fade, slide, bounce, clockwise, left, and down capture dynamic attributes.
  • Semantic Category Distribution: User interface elements form the largest semantic category, followed by abstract patterns, decorative motifs, and entertainment-related content.Characters, nature, business, technology, shopping, communication, and other domains make up additional categories and the long tail.
  • Motion Type Analysis: Translation is the most prevalent motion type, alongside widespread rotation and scaling transformations.The analyzed motions include both straight and Bézier-path translations, continuous and bounded rotations, and varied scaling operations.
  • Evaluation: MMLottie-Bench samples held-out data for multimodal generation tasks and uses Claude-3.5-Sonnet to score Object Alignment and Motion Alignment.Object Alignment scoring considers object presence, type, quantity, visual characteristics, and spatial relationships.
Loading 2603.02138v1…