Source-linked AI summary
TokenDial: Continuous Attribute Control in Text-to-Video via Spatiotemporal Token Offsets
Zhixuan Liu, Peter Schaldenbrand, Yijun Li, Long Mai, Aniruddha Mahapatra, Cusuh Ham, Jean Oh, Jui-Hsien Wang
TL;DR
TokenDial addresses limited slider-style control over attribute strength in text-to-video generation, especially for motion dynamics and explicit spatiotemporal localization. It learns additive semantic offsets in visual patch-token space, producing smooth, predictable edits for appearance and motion; experiments report stronger controllability and editing quality than state-of-the-art baselines.
Problem
Text-to-video systems provide weak control over attribute strength and lack reliable motion-dynamics sliders or explicit specification of where and when edits apply.
Method
TokenDial learns additive attribute-specific offsets in intermediate spatiotemporal visual patch-token space, using pretrained understanding signals to control appearance and motion without retraining the backbone.
Results
Quantitative benchmarks and human evaluations report superior controllability and editing quality versus state-of-the-art baselines across diverse appearance and motion attributes, while balancing modification with content preservation.
Takeaways & Limitations
Scaling offsets enables smooth, monotonic attribute changes with largely preserved identity, background, and temporal coherence, while token-level addressing supports explicit localized edits.
Takeaways & Limitations
The method relies on pretrained video-understanding models to define semantic directions for appearance control, making its effectiveness dependent on that understanding space.
Abstract
from arXiv · showhide
We present TokenDial, a framework for continuous, slider-style attribute control in pretrained text-to-video generation models. While modern generators produce strong holistic videos, they offer limited control over how much an attribute changes (e.g., effect intensity or motion magnitude) without drifting identity, background, or temporal coherence. TokenDial is built on the observation: additive offsets in the intermediate spatiotemporal visual patch-token space form a semantic control direction, where adjusting the offset magnitude yields coherent, predictable edits for both appearance and motion dynamics. We learn attribute-specific token offsets without retraining the backbone, using pretrained understanding signals: semantic direction matching for appearance and motion-magnitude scaling for motion. We demonstrate TokenDial's effectiveness on diverse attributes and prompts, achieving stronger controllability and higher-quality edits than state-of-the-art baselines, supported by extensive quantitative evaluation and human studies.
1. Introduction
Text-to-video models generate high-quality videos but provide weak control over continuous attribute strength, especially motion dynamics and explicit spatial-temporal localization. TokenDial addresses this gap with semantic token offsets that enable controllable edits while preserving unrelated video content.
- Creators need continuous control over attribute strength while preserving identity, background, and temporal coherence.
- Existing video sliders mainly target appearance, leaving motion intensity, rhythm, speed, and explicit where-and-when editing unreliable.
- Progressive prompts and V2V models lack continuous, monotonic attribute-strength control, while finetuning-based sliders can be training-intensive, difficult to disentangle, and prone to degrading general capability or overfitting.
- Finetuned models also make localized edits difficult, such as aging only the left person rather than the right.
- TokenDial learns additive offsets in pretrained T2V visual patch-token space, using offset magnitude to strengthen or weaken semantic attributes without retraining the backbone.
- Experiments across appearance and motion attributes, supported by quantitative benchmarks and human evaluations, report superior controllability and editing quality over state-of-the-art baselines while preserving content.
2. Related Work
Prior work has expanded video control and editing, but typically offers discrete or coarse interfaces rather than continuous, monotonic attribute-strength modulation. TokenDial is positioned as a continuous control approach using pretrained understanding signals instead of substantial edited supervision or synthetic training data.
- Controllable video-generation methods commonly address discrete or coarse choices such as content, placement, or camera trajectory rather than continuously dialing attribute strength.
- Instruction- and example-based video editing achieves open-domain edits but usually specifies strength through text or a few predefined levels.
- These editing methods often rely on substantial edited supervision or synthetic training data, which limits scalability.
- FreeSliders uses training-free inference-time concept steering, whereas Text Slider learns reusable lightweight LoRA directions in a shared text encoder.
- Research on semantic latent spaces motivates controllable generation through linear directions, semantic bottlenecks, and denoising representations aligned with pretrained understanding models.
3. Method
TokenDial learns additive attribute offsets in intermediate spatiotemporal visual patch-token space while keeping the pretrained video generator frozen. It combines semantic and motion-specific supervision with explicit spatiotemporal masking and vector-field-level strength control.
- Token offset space: Patchification maps each visual token to a specific spatial and temporal region, while the offset space depends only on hidden dimension d and transfers across video resolutions and lengths.The model applies additive offsets Δ∈R^d to token sequences T∈R^L×d, where L changes with resolution and video length.
- Token offset space: A soft spatiotemporal mask s gates offsets so users can specify which regions and frames are affected, while scaling Δ controls edit strength.Layer-dependent offsets add learnable degrees of freedom without modifying backbone weights.
- Training stabilization: Posterior refinement runs additional reverse-denoising steps before supervision, stabilizing gradients from understanding models applied to noisy intermediate predictions.Gradients are stopped through the refinement unroll, and experiments use a small K such as K=4 at high-noise timesteps.
- Appearance control: Appearance offsets are trained by aligning predicted attribute directions in InternVideo2 embedding space with target directions, while LPIPS regularization preserves identity and scene content.Target directions come from paired text prompts or exemplar videos.
- Motion control: Motion offsets use DINOv2 patch embeddings and Lucas–Kanade optical flow to scale motion magnitude, with first-frame feature consistency preserving identity and scene content.The objective uses a self-supervised target and regularization strength λm.
- Inference control: At inference, compositional flow guidance scales only the edit-induced differential velocity, preserving the text-consistent base trajectory while providing continuous slider control.This avoids distortions that can arise from directly scaling Δ at large magnitudes.
4. Applications
TokenDial supports continuous appearance and motion control, localized edits, composition, and transfer across video lengths, resolutions, and architectures.
- TokenDial provides smooth, monotonic control over appearance and motion attributes while largely preserving identity, background, and temporal coherence.
- Attention-derived or user-provided masks specify where and when offsets apply, confining edits to target trajectories and protecting unrelated regions.
- Independent offset gating enables simultaneous edits to multiple regions and attributes without interference.
- Offsets transfer zero-shot across video lengths and resolutions because they depend only on hidden dimension, not token count.
- TokenDial transfers to the Wan 2.1 backbone for continuous appearance sliders such as “more kitten” and “more furry.”
- Debiasing projects out biased principal directions so “older” edits do not unintentionally co-vary with body weight.
5. Experimental Setup
The evaluation defines localized appearance and motion slider tasks across diverse concepts and attributes, using real or synthetic training videos and a frozen backbone.
- Slider tasks test progressive, monotonic attribute changes under the same prompt and seed while preserving scene layout, identity, or underlying motion pattern.
- The benchmark covers 12 concepts spanning particle systems, volumetric phenomena, fluids, and articulated subjects, with five attributes per concept.
- Appearance offsets use small concept-specific video–text collections without paired edits, while motion offsets use a few hundred green-screen clips.
- Semantic debiasing removes coupling between the “older” attribute and body weight in InternVideo2-based supervision.
- Offsets are optimized on a frozen pretrained text-to-video DiT for 300 AdamW steps, using 32-frame videos at 24 FPS and 320 × 176 resolution.
6. Results
TokenDial provides continuous appearance and motion-slider control with stronger edits and better preservation than prior methods. Quantitative, human, and ablation studies support its controllability, quality, and localized-editing design.
- Qualitative Comparison: TokenDial produces smooth, continuous appearance edits while better preserving identity and context than prior slider-based methods.
- Qualitative Comparison: TokenDial scales motion dynamics, such as walking to running, while maintaining coherent structure and producing larger pose, clothing, and hair displacements.
- Quantitative Comparison: TokenDial achieves the best overall slider score, 0.982, compared with I2I+I2V at 0.808 and Text Slider at 0.742.
- Quantitative Comparison: TokenDial obtains the highest editing-quality, identity-preservation, and continuity scores while maintaining comparable video quality and text alignment.
- Human Evaluation: Human evaluations prefer TokenDial for motion edits, with the highest perceived edit quality and strong preservation; appearance preferences align with quantitative results.
- Ablation Study: Ablations show that V+-space offsets enable localized appearance edits, posterior refinement stabilizes training, and attention masking reduces background drift.
7. Conclusion
TokenDial introduces continuous slider control by learning additive offsets in an intermediate spatiotemporal patch-token space without retraining the backbone. The paper concludes that this enables smooth, predictable control of appearance and motion while noting entanglement and bias limitations in pretrained understanding spaces.
- TokenDial learns additive token offsets in an intermediate spatiotemporal patch-token space to control appearance and motion without retraining the backbone.
- Scaling a learned token-space direction yields smooth, predictable changes in attribute strength for controllable video generation.
- Limitations: The method relies on pretrained video understanding models whose semantic spaces may contain difficult-to-disentangle biases and low-level attribute entanglements.
A. Human Evaluation Study
The human study evaluates progressive appearance and motion edits through randomized comparisons, Likert ratings, and quadrant plots. Participants generally favor TokenDial for strong, progressive edits with preservation, while conservative baselines sometimes score better on preservation alone.
- Study Design: Participants rate edit quality, progression, content preservation, and preference using Likert-scale questions aligned with the VLM evaluation.
- Study Design: The study compares progressively stronger edits across 64 directions, including 32 appearance and 32 temporal or dynamic tasks.
- Results: TokenDial is preferred for motion edits, achieving the highest perceived edit quality while maintaining strong preservation.
- Results: TokenDial consistently receives more positive responses across most questions, although Text Slider and Kontinuous Kontext + Senorita sometimes match or exceed it on content preservation.
- Baseline Analysis: Text Slider and Kontinuous Kontext + Senorita often make minimal edits, preserving identity and background but limiting noticeable and controllable changes.
- Baseline Analysis: Kontinuous Kontext + Senorita can fail when the target object or region is absent from the first frame, propagating weak edits through the video.
C. Dataset Details
The supplementary evaluation constructs lightly supervised concept–attribute datasets, uses green-screen videos for speed training, and tests unseen prompts across multiple edit scales. It also analyzes posterior refinement and DINOv2-feature optical flow used for motion supervision.
- Dataset Construction: Training pairs are collected by keyword filtering text–video data, followed by manual cleaning and deduplication, with target concepts usually visually centered.
- Dataset Construction: Speed training additionally uses a few hundred green-screen videos to reduce background noise and focus supervision on foreground motion changes.
- Evaluation Protocol: Testing uses 16 unseen base prompts per concept–attribute pair, two random seeds, and edit scales 1–4, with scale 0 representing the original video.
- Posterior Refinement: Posterior refinement keeps intermediate predictions closer to ground truth, producing sharper structures, better local contrast, and more stable spatial layouts.
- Motion Representation: Motion supervision uses Lucas–Kanade optical flow computed on DINOv2 patch features, producing a patch-level flow field from spatiotemporal feature changes.
F. Implementation Details on Wan 2.1
On Wan 2.1, TokenDial injects learned additive offsets into the pretrained video DiT while keeping the backbone unchanged. The self-attention residual branch provides the best reported trade-off between editability and preservation.
- Injection location: Self-attention residual injection consistently provides the best trade-off between editability and preservation among the tested locations.Alternative locations included positions around modulation layers, the cross-attention branch, and after the full DiT block.
- Wan 2.1 integration: TokenDial adapts to Wan 2.1 by injecting learned additive offsets into intermediate video patch tokens without modifying backbone weights.The implementation places the intervention in each DiT block’s self-attention residual branch.
G. VLM Prompt for Slider Evaluation
The VLM evaluation prompt assesses five-scale slider sequences using four criteria: prompt following, identity preservation, background consistency, and progressive intensity. It defines the scales, evaluation target, scoring range, and strict preservation requirements.
- Evaluation setup: The evaluation compares five frames from the same timestamp, with Scale 1 as the original and Scales 2–5 as progressively stronger edits.The sequence is judged as a whole rather than as independent frames.
- Evaluation setup: The target is to make the same concept progressively more aligned with the specified attribute from Scale 2 to Scale 5.
- Scoring criteria: Each sequence receives integer scores from 0 to 5 for prompt following, identity preservation, background consistency, and progressive intensity.The requested output includes a short justification for each criterion.
- Scoring criteria: Identity preservation requires the same subject and category across all scales, with replacement or category change treated as a severe violation.
- Scoring criteria: Background consistency evaluates whether unedited regions, scene layout, and camera viewpoint remain stable across scales.
- Scoring criteria: Progressive intensity requires a smooth, monotonic increase in edit strength from Scale 1 to Scale 5.
H. More Results
The supplementary results include additional appearance-slider examples and further appearance- and motion-slider results in the video presentation and project-page index file.
- Additional results: Further appearance-slider and motion-slider results are available in the attached video presentation and the project page’s index.html file.
- Additional results: Figures 16–19 provide additional results for the appearance slider.