PaperlayerView exact revision on arXiv

Source-linked AI summary

TrailBlazer: Trajectory Control for Diffusion-Based Video Generation

Wan-Duo Kurt Ma, J. P. Lewis, W. Bastiaan Kleijn

arXiv:2401.00896v2cs.CV

TL;DR

Text prompts and labor-intensive low-level controls leave object layout and trajectories difficult to steer. TrailBlazer edits spatial and temporal attention using interpolated bounding-box and prompt keyframes; reported outputs follow trajectories and show perspective effects as boxes change size.

  • Problem

    Text prompts alone provide weak control over object layout and trajectories, while producing low-level per-frame guidance can be labor-intensive.

    Source: §1 · p. 2

  • Method

    TrailBlazer edits spatial and temporal attention in a pre-trained video model, using interpolated bounding-box and prompt keyframes to guide subject position and appearance.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 4

  • Results

    Subjects generally follow the bounding boxes, and changing box size produces perspective effects, including motion toward or away from the virtual camera.

    Source: §4.1 · p. 8

  • Takeaways & Limitations

    Keyframed boxes and prompts offer casual users a way to guide subject trajectories and coarse behavior without detailed masks or frame-by-frame control signals.

    Source: §1 · p. 2, §5 · p. 10

    TrailBlazer inherits underlying diffusion-model failure cases, and clip length remains restricted to the pre-trained model’s output length.

    Source: §8 · p. 14

Abstract

from arXiv · show

Within recent approaches to text-to-video (T2V) generation, achieving controllability in the synthesized video is often a challenge. Typically, this issue is addressed by providing low-level per-frame guidance in the form of edge maps, depth maps, or an existing video to be altered. However, the process of obtaining such guidance can be labor-intensive. This paper focuses on enhancing controllability in video synthesis by employing straightforward bounding boxes to guide the subject in various ways, all without the need for neural network training, finetuning, optimization at inference time, or the use of pre-existing videos. Our algorithm, TrailBlazer, is constructed upon a pre-trained (T2V) model, and easy to implement. The subject is directed by a bounding box through the proposed spatial and temporal attention map editing. Moreover, we introduce the concept of keyframing, allowing the subject trajectory and overall appearance to be guided by both a moving bounding box and corresponding prompts, without the need to provide a detailed mask. The method is efficient, with negligible additional computation relative to the underlying pre-trained model. Despite the simplicity of the bounding box guidance, the resulting motion is surprisingly natural, with emergent effects including perspective and movement toward the virtual camera as the box size increases.

1 Introduction

Text-to-video systems have advanced, but prompts alone provide weak control over object layout and trajectories, while low-level guidance can be laborious to create. TrailBlazer uses bounding boxes and keyframes to guide subject position, size, and prompts by editing spatial and temporal attention.

1 Introduction

  • Text prompts provide limited control over object layout and trajectories, while producing frame-by-frame edge maps or tracked skeletons can be difficult or tedious.The paper motivates trajectory control as important for understandable story narration.Source: §1 · p. 2
  • Users specify bounding boxes and prompts at keyframes, which are interpolated to guide smooth changes in subject motion, size, and appearance or behavior.Different prompts can also be interpolated to produce a morphing effect.Source: §1 · p. 2
  • TrailBlazer edits spatial and temporal attention maps during initial denoising steps to concentrate activation at the desired object location.The approach leverages the observation that object position is established early in diffusion denoising and does not disrupt learned text-image associations.Source: §1 · p. 2
  • The method operates on a pre-trained ZeroScope model and requires no inference-time optimization; its contribution targets video trajectories rather than image-position control.The paper distinguishes its temporal cross-frame attention problem from closely related image-generation work.Source: §1 · p. 2
  • TrailBlazer guides subjects with high-level bounding boxes, avoiding frame-by-frame positioning signals that can be difficult to draw or derive from existing videos.Existing-video processing restricts available motion to copies of those sources.Source: §1 · p. 2
  • Bounding-box size can be keyframed to control perspective effects, while keyframed prompts influence the subject’s behavior and identity over time.The paper illustrates size control in Figs. 1 and 6.Source: §1 · p. 2

2 Related Work

Related work addresses controllability in both image and video generation, but video methods generally face temporal-consistency demands and often lack direct position control. The paper situates TrailBlazer among coarse-box image controls and video guidance approaches based on temporal modeling or low-level conditioning.

2.1 Text-to-Image (T2I)

  • Stable Diffusion performs text-to-image diffusion in a variational autoencoder’s latent space, while ControlNet and T2I-Adapter add fine-tuned conditioning for edges or rigging skeletons.The cited literature also identifies difficulties with multiple objects and prompt-attribute assignment.Source: §2.1 · p. 3
  • Directed Diffusion, BoxDiff, and MultiDiffusion use coarse bounding boxes to control subject position through spatial latents and text-embedding cross-attention maps.These methods address layout-to-image generation using few-shot learning.Source: §2.1 · p. 3

2.2 Text-to-Video (T2V)

  • Video generation is harder than image generation because of temporal consistency and paired-data demands; prior methods add temporal layers, manipulate cross-frame attention, or correlate diffusion noise.The reviewed methods generally lack position control in synthesized videos.Source: §2.2 · p. 3
  • Recent video-control methods use edge or depth conditioning, while T2V-Zero provides indirect position control through first-frame initialization and translation offsets.The paper describes its attention-injection approach as an alternative to optimization and reports that it is generally robust to different random seeds.Source: §2.2 · p. 3
  • TrailBlazer and concurrent Peekaboo both guide video subjects by manipulating attention, but the paper reports that Peekaboo often produces backgrounds with missing detail.The authors state that several very recent layout-to-video preprints were not compared because their sources were unavailable at writing time.Source: §2.2 · p. 3

3 Method

TrailBlazer uses a pretrained ZeroScope model and interpolated bbox-and-prompt keyframes to steer subject motion and appearance, editing spatial and temporal attention early in denoising. For multiple subjects, it combines subject-specific latents with a composed prompt.

3 Method

  • TrailBlazer uses pretrained ZeroScope without training, finetuning, or inference-time optimization, and requires only prompts and approximate subject bounding boxes.Keyframed boxes and prompts are interpolated to control motion and prompt content.Source: §3 · pp. 3–4

3.1 Pipeline

  • Users specify at least two bbox-and-prompt keyframes, which the system linearly interpolates to control subject position and appearance between frames.A larger set of trailing attention maps can increase controllability but risks failed reconstruction.Source: §3.1 · p. 5

3.2 Spatial Cross Attention Guidance

  • Spatial attention editing strengthens selected prompt-word and trailing-map attention inside the interpolated bbox while attenuating it outside.The bbox injection uses a Gaussian window sized to the box.Source: §3.2 · p. 6

3.3 Temporal Cross-Frame Attention Guidance

  • Temporal editing weights bbox activation by normalized frame distance, increasing it for nearby frames and reducing it for distant pairs.This editing is applied during the initial NM denoising steps.Source: §3.3 · p. 6

3.4 Scene compositing

  • For multiple subjects, TrailBlazer composites subject-specific latents into an overall latent under a composed prompt, gradually shifting priority from local subject latents to the composed latent.The transition occurs within each subject's bbox and ends after NC denoising steps.Source: §3.4 · p. 7

4 Experiments

The experiments compare TrailBlazer with Peekaboo and T2V-Zero on single-subject trajectory control, dynamic bounding boxes, and two-subject composition. Objective metrics do not give a clear overall method ordering, while the reported results highlight TrailBlazer’s movement control and higher mIoU scores.

4.1 Main result

  • With identical prompts, TrailBlazer’s subjects fit the moving boxes more accurately and move with plausible anatomy and orientation than the compared methods.T2V-Zero translates subjects without reorienting them, while Peekaboo’s subjects may follow motion but fit boxes less well; the paper also reports better background quality for TrailBlazer.Source: §4.1 · pp. 7–8, Figure 5 · p. 8
  • Changing box size over time yields perspective-like motion, with subjects naturally facing toward the virtual camera as boxes grow or shrink.The tiger examples show movement toward or away from the camera; Peekaboo’s tiger does not show these effects with identical boxes.Source: §4.1 · p. 8, Figure 6 · p. 9
  • TrailBlazer handles rapid back-and-forth cat motion better than Peekaboo in the five-keyframe example.The box alternates between the left and right sides of the frame across a 24-frame clip.Source: §4.1 · p. 8, Figure 6 · p. 9
  • Two-subject cat-and-dog compositions remain plausible across varied environment prompts, though some inherited artifacts such as extra limbs appear.The reported examples include plausible reflections, splashes, and consistent shadows; the authors note that multi-subject synthesis is challenging, particularly beyond two objects.Source: §4.1 · p. 8, Figure 7 · p. 9

4.2 Quantitative evaluation

  • The quantitative comparison uses 24-frame videos and static or dynamic randomly keyframed boxes, with TrailBlazer and Peekaboo sharing boxes.T2V-Zero instead uses motion vectors, with settings depending on the task; the paper reports the best of ten random-seed trials for its visual main comparison.Source: §4.1 · p. 7, §4.2 · pp. 9–10
  • Tables 1–2 do not give a clear overall method ordering, although TrailBlazer shows significantly better mIoU scores.The static-bbox results are roughly equivalent across metrics except for lower FVD than both baselines; for dynamic boxes, TrailBlazer has better FID than Peekaboo, while Peekaboo has better FVD.Source: Table 1 · p. 10, Table 2 · p. 10, §4.2 · p. 10

5 Conclusion

TrailBlazer guides subject motion in a pretrained text-to-video model using bounding boxes and prompts keyframed over time. The paper reports natural motion and scene interactions without finetuning, training, or online optimization.

5 Conclusion

  • TrailBlazer edits spatial and temporal attention in pretrained ZeroScope, using simple boxes and keyframed prompts to guide subject trajectories and coarse behavior.The conclusion reports natural motion, plausible orientation, perspective effects, and object–environment interactions arising automatically.Source: §5 · p. 10

6 Implementation

The implementation adapts pretrained ZeroScope through the Diffusers video pipeline. It uses fixed defaults for most attention-editing parameters and a 24-frame output sequence.

6 Implementation

  • TrailBlazer overrides Diffusers’ TextToVideoSDPipeline and adapts pretrained ZeroScope, which was fine-tuned from ModelScope using nearly ten thousand 24-frame clips.The implementation uses PyTorch 2.01 and Diffusers 0.21.4; synthesized sequences are set to 24 frames following the model’s recommended practice.Source: §6 · p. 11
  • Most results use five spatial and temporal editing steps, coefficients cw = 0.001 and cs = 0.1, and 15 trailing attention maps.The authors report that 10–20 trailing maps generally work satisfactorily and identify that count as the parameter requiring tuning.Source: §6 · p. 11
  • Spatial attention editing is applied at several UNet resolutions, while temporal attention editing uses one layer because a multi-resolution approach was unnecessary and unpredictable.The spatial module list and the single temporal layer are specified in the implementation passages.Source: §6 · p. 11

7 Ablations

The ablations vary the number of trailing attention maps and temporal editing steps. They report poor results at extremes and a useful intermediate balance.

7 Ablations

  • With no trailing maps, the astronaut remains centered despite box motion; too many maps can produce a flag instead of the astronaut.The authors identify 10–20 edited trailing maps as a good range.Source: §7 · p. 12, Figure 8 · p. 13
  • Temporal editing with zero steps produces a moving red blob, while ten steps yields an astronaut but leaves the background missing along the box path.Five temporal steps provide the reported desired result of an astronaut moving over a moon background.Source: §7 · p. 12, Figure 9 · p. 13
  • The ablations indicate that spatial trailing maps and temporal editing need a reasonable balance, since extreme values of either produce poor results.The illustrated temporal ablation holds trailing maps at 10 while varying temporal steps.Source: §7 · p. 12, Figure 9 · p. 13

8 Limitations

TrailBlazer inherits failures of its underlying diffusion model and provides approximate rather than exact bbox placement. Its generated clip length is bounded by the pre-trained model, such as 24 frames with ZeroScope.

8 Limitations

  • The method shares underlying-model failures, including deformed objects, difficulty generating multiple objects and assigning attributes, and occasional implausible outputs.Examples include a red jeep sinking into a muddy road, camera motion instead of panda motion, and a deformed car.Source: §8 · p. 14
  • BBox guidance directs object placement but does not constrain the subject to lie exactly within the box.The authors describe this as potentially advantageous for casual users, who otherwise would need to specify perspective-dependent aspect ratios.Source: §8 · p. 14
  • Generated clip length is limited to the duration produced by the pre-trained model, for example, 24 images with ZeroScope.Source: §8 · p. 14

9 Further comparison with Peekaboo

TrailBlazer is compared with Peekaboo under extreme bbox and motion conditions. The reported metrics favor TrailBlazer on mIoU and FID, while the authors attribute its worse FVD to a mismatch between the evaluation dataset and the experiments’ extreme motions.

9 Further comparison with Peekaboo

  • TrailBlazer succeeds across extreme cases—including delayed fast motion, size changes, zigzags, dense keyframes, and tiny boxes—where these successes are generally not evident with Peekaboo.The examples include an elephant stationary for 75% of the clip before running, a dog following eight keyframes in 24 frames, and a clownfish fitting a tiny bbox.Source: §9 · p. 15
  • TrailBlazer’s mIoU and FID surpass Peekaboo’s, and its mIoU is approximately twice Peekaboo’s.The metrics are derived from the Figure 11 experiments using AnimalKingdom.Source: §9 · p. 15
  • TrailBlazer’s FVD is worse than Peekaboo’s in this section; the authors believe the AnimalKingdom dataset lacks the varied, extreme motion used in their experiments.Source: §9 · p. 15

10 Subject Morphing

TrailBlazer uses prompt interpolation to morph subject identity across video frames while maintaining the subject’s motion. The examples show smooth identity transitions without significant artifacts, and the authors present morphing as a possible route to new content combinations.

10 Subject Morphing

  • TrailBlazer demonstrates video subject morphing by manipulating prompts, which the authors identify as the first such demonstration in the video diffusion domain.Earlier related concepts were shown for image generation in MagicMix Liew et al. (2022).Source: §10 · p. 16
  • Prompt embedding combinations may offer an entry point to new content, such as a fish-like cat, that the authors say may be difficult to generate with a single prompt given CLIP limitations.Source: §10 · p. 16
  • Linear interpolation of prompt embeddings across frames produces smooth identity transitions without significant artifacts, while subjects maintain the same motion across frames.Examples include cat-to-dog and cat-walking-to-fish-swimming transitions.Source: §10 · p. 16, Figure 12 · p. 16

11 Comprehensive ablations

The ablations examine how bbox placement, size, and keyframe choices affect synthesis. They show that bbox size and trajectory can shape perspective and speed, while extreme conditions and difficult turns can produce artifacts or poor alignment.

11 Comprehensive ablations

  • Without trajectory control, ZeroScope produces plausible, temporally coherent slow-motion videos, but subjects may remain near the image center regardless of depicted motion, with occasional anatomical or other implausible artifacts.Source: §11 · p. 17, Figure 13 · p. 17

11.1 Exploration and Ablation: Varied static bbox sizes

  • Static bbox size strongly affects results: very small boxes can yield unexpected entities or attribute leakage, while overly large boxes can produce broken results; the authors recommend 30%–60% for reconstruction quality.Very small or large boxes may still be used for particular frames rather than an entire sequence.Source: §11.1 · p. 18, Figure 14 · p. 18

11.2 Exploration and Ablation: Varied dynamic bbox sizes

  • Changing bbox size dynamically can create perspective-like motion, but prompt and shape affect aspect-ratio fit, and the subject only approximately follows the box.A fish prompt poorly matches a short-wide box, while a sardine prompt conforms more closely; both do well with a tall-thin box.Source: §11.2 · p. 19, Figure 15 · p. 19

11.3 Exploration and Ablation: Speed control with multiple keys

  • Varying keyframe count controls speed and can yield relatively natural motion, though faster movement introduces motion blur that is hard to eliminate with negative prompts.Source: §11.3 · p. 20, Figure 16 · p. 20

11.4 Exploration and Ablation: Controlling speed with different placement of a single keyframe

  • Moving a middle keyframe’s timing can extend a stationary phase before faster motion; high speeds may also produce motion blur and artifacts.Source: §11.4 · p. 21, Figure 17 · p. 21

11.5 Exploration and Ablation: Irregular trajectory

  • Irregular trajectories can produce perspective-like size changes, but fast turns may cause artifacts; realistic results depend on consistency between the prompt and keyframe timing and locations.The authors note that tight, high-speed turns are difficult for horses to depict naturally.Source: §11.5 · p. 22, Figure 18 · p. 22
Loading 2401.00896v2…