PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Specification Guidance for Text-to-Video Generation

Avishree Khare, Hideki Okamoto, Bardh Hoxha, Georgios Fainekos, Rajeev Alur

arXiv:2610.20822v1cs.AIcs.CV

TL;DR

Text-to-video models can depict requested concepts while failing their required ordering, timing, and persistence. TSG turns temporal specifications into executable sampling guidance using attention traces and differentiable scores. Across five models, it improves specification satisfaction accuracy over CFG on a 220-prompt benchmark.

  • Problem

    Text-to-video models often include requested visual concepts but fail to satisfy their intended ordering, timing, and persistence requirements.

    Source: §1 · p. 1

  • Method

    TSG uses video-text attention traces, differentiable Metric Temporal Logic scores, and gradient-derived attention biases to add specification-guided directions during sampling.

    Source: §1 · pp. 1–2

  • Results

    9.5–20 percentage points: TSG improves specification satisfaction accuracy over CFG-only sampling across five models on a 220-prompt benchmark spanning 11 temporal-constraint categories.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support specification-guided attention steering as an effective, memory-efficient approach to temporal control in video generation.

    Source: §1 · p. 2

    TSG is limited by models' latent temporal resolution, constraining its current steering to that grid rather than finer-grained temporal constraints.

    Source: §8 · p. 9

Abstract

from arXiv · show

Text-to-video models often generate the visual concepts requested in natural language prompts but fail to satisfy their intended ordering, timing, and persistence requirements. We propose representing these requirements as specifications in a formal Temporal Logic, and introduce Temporal Specification Guidance (TSG), a training-free inference-time framework that turns temporal specifications into executable objectives for guiding video generation during sampling. TSG scores traces of visual concept evidence using differentiable quantitative semantics and uses the score's gradients to construct a specification-guided denoising direction. We instantiate this framework with video-text attention traces, efficient differentiable scores for Metric Temporal Logic, and gradient-derived attention biases. This instantiation requires neither an external concept classifier nor backpropagation through the video transformer. Its attention-steered forward pass yields a denoising direction that augments classifier-free guidance (CFG). On a benchmark of 220 prompts spanning 11 categories of temporal constraints, including ordered object entry and state transitions, this instantiation improves specification satisfaction accuracy over CFG by 9.5-20 percentage points across five models, with minimal memory overhead. These results demonstrate the potential of temporal specifications and differentiable satisfaction objectives as a flexible interface for controlling video generation.

1 Introduction

Text-to-video models can include the requested concepts yet miss their intended order, timing, and persistence. The paper proposes formal temporal specifications and TSG, which uses attention traces and differentiable scores to guide sampling; it reports higher specification satisfaction than CFG across five models.

1 Introduction

  • The motivating examples show CFG capturing broad semantic concepts while missing required timing, ordering, or requested entities.In the traffic-light example, CFG keeps amber for the first two-thirds instead of following the expected yellow–red–green progression.Source: §1 · p. 1, Figure 1 · p. 2
  • Temporal requirements can be expressed in a Metric Temporal Logic specification language that represents persistence and ordering and allows different degrees of timing precision.The language can also express prior timed binary masks and untimed temporal-logic formulae.Source: §1 · p. 1
  • TSG is a training-free, inference-time framework that turns temporal specifications into executable objectives for guiding transformer-based video generation during sampling.It scores video-text attention traces and uses gradients of differentiable specification scores to steer attention, avoiding backpropagation through the video transformer.Source: §1 · pp. 1–2
  • TSG uses differentiable quantitative semantics for Metric Temporal Logic, with log-domain scores that aggregate support for temporal witnesses and alternatives.For a fixed specification, the score can be evaluated in time linear in trace length.Source: §1 · p. 2
  • TSG improves specification satisfaction accuracy by 9.5–20 percentage points over CFG-only sampling across five models on 220 prompts spanning 11 temporal-constraint categories.The benchmark includes timed state transitions, entity appearance and disappearance, persistence, and choice.Source: §1 · p. 2

2 Temporal constraints as formal specifications

The paper represents frame-level visual concepts and their temporal requirements with a finite-trace Metric Temporal Logic language. Normalized time intervals let specifications express untimed, interval-timed, or strictly timed constraints, including constraints that allow multiple valid traces.

2 Temporal constraints as formal specifications

  • The setup treats a generated video as a finite sequence of frames and visual concepts as frame-level labels indicating presence or absence.Examples of concepts include entity presence, entity states, and setting properties.Source: §2 · pp. 2–3
  • A finite-trace Metric Temporal Logic specification is built from visual-concept propositions, Boolean operators, and interval-bounded temporal operators.The language includes Next, Until, and Always; Eventually can be derived from Until.Source: §2 · p. 3
  • Intervals over normalized time [0, 1] express untimed, interval-timed, and strictly timed requirements for visual concepts.The paper gives a drone-persistence constraint with at least one of two rescue cues appearing in the second half as an example.Source: §2 · p. 3
  • The language can encode untimed temporal-logic formulae and framewise binary temporal masks as specifications.A mask corresponds to a single valid trace, whereas these specifications can admit multiple valid traces, including choices between concepts and object disappearance within an interval.Source: §2 · p. 3
  • A video's satisfaction label for a specification can be computed deterministically from frame-level concept labels using standard Boolean semantics for finite-trace Metric Temporal Logic.Specifications may be supplied with the prompt or extracted from it using an LLM.Source: §2 · p. 3

3 Video Generation and Classifier Free Guidance

The paper considers latent text-to-video models that iteratively update a compressed spatiotemporal representation and decode the final latent into a video. Classifier-free guidance combines unconditional or reference and prompt-conditional predictions to strengthen prompt conditioning during sampling.

3 Video Generation and Classifier Free Guidance

  • Latent text-to-video generation starts from Gaussian noise and iteratively uses a transformer to predict denoising directions in compressed spatiotemporal latent space.A VAE decodes the final latent into the output video.Source: §3 · p. 3
  • Classifier-free guidance strengthens prompt conditioning by combining reference-branch and prompt-conditional transformer predictions during sampling.The reference embedding can represent an empty or negative prompt, and the CFG guidance scale controls the extrapolation.Source: §3 · pp. 3–4

4 TSG: Specification Guidance for Text-to-Video Generation

TSG augments classifier-free guidance with a specification-guided prediction computed by steering transformer attention. It derives visual-concept traces from video-text attention, scores them with differentiable Metric Temporal Logic semantics, and applies score-gradient biases during an additional forward pass.

4 TSG: Specification Guidance for Text-to-Video Generation

  • TSG computes a specification-guided prediction from attention traces and combines it with the reference and prompt-conditional predictions to form the sampling update.Its relative strength is controlled with respect to CFG, and TSG is applied for an initial number of update steps before CFG alone is used.Source: §4 · p. 4
  • Each attention head's video-text attention supplies temporal traces of evidence for visual concepts named in the specification.Concept presence scores are derived from attention over latent frames using a sigmoid scaled by the trace's temporal mean and population standard deviation.Source: §4 · p. 4
  • TSG limits steering to the model's latent temporal resolution and maps normalized specification intervals onto that latent grid.A latent frame does not correspond one-to-one with a video frame.Source: §4 · p. 4
  • TSG scores concept traces with interval-aware, log-space quantitative semantics for Metric Temporal Logic, aggregating support for required conditions and admissible alternatives or temporal witnesses.The score requires O(|φ|Tz) arithmetic operations and is non-decreasing in independent literal-support inputs, yielding predictable gradient signs under consistent polarity.Source: §4 · pp. 4–5
  • At each selected layer and sampling step, score gradients are normalized into attention biases, broadcast to spatial queries and concept tokens, then centered and clipped before modifying attention logits.Negligible gradients receive zero bias, and the resulting attention-steered pass yields the specification-guided direction without gradients to transformer weights.Source: §4 · p. 5, Figure 2 · p. 5

5 Experimental Evaluation

Across five models, TSG improves temporal-specification accuracy over CFG on interval- and strictly-timed benchmarks, with low memory overhead but added generation time and modest visual-quality changes.

5.1 Setup and Benchmarks

  • The interval-timed benchmark contains 220 prompts across 11 categories, with timing, ordering, persistence, and choice variations.Prompts use halves and thirds of 5–6-second videos; Figure 3 depicts permitted timing intervals, absence, and mutually exclusive outcomes.Source: §5.1 · pp. 5–6, Figure 3 · p. 6
  • Evaluation covers five CogVideoX and Wan models against CFG, measuring satisfaction accuracy, visual quality, generation latency, and peak GPU memory.The study also compares TSG with TempoControl on strictly-timed prompts and evaluates concept presence on 21 decoded frames.Source: §5.1 · p. 6

5.2 RQ1. Does TSG improve adherence to temporal specifications?

  • TSG raises satisfaction accuracy by 9.5–20 percentage points across five models and temporal accuracy by 17.6–22.0 points over CFG across TempoControl splits.On TempoControl, TSG also exceeds full TempoControl by 2.97 and 0.58 percentage points on the one-object and two-object splits, respectively.Source: Table 1 · p. 6, §5.2 · p. 7, Table 2 · p. 7

5.3 RQ2. What are the computational and quality costs of TSG?

  • TSG matches CFG-only peak GPU memory, while using 30.52 GiB versus TempoControl’s 79.43 GiB, approximately 62% less.TempoControl’s latent optimization requires a gradient graph through the video transformer.Source: §5.3 · p. 7, Table 2 · p. 7
  • TSG adds generation latency, from around 1 extra minute with Wan2.1 1.3B to 11 extra minutes with Wan2.2 A14B.Full TempoControl takes approximately 4.9× and 3.6× as long as TSG on the one-object and two-object splits, respectively.Source: §5.3 · p. 7, Table 2 · p. 7
  • TSG modestly impacts visual quality: it lowers imaging quality by up to 3.0 points on four models, while improving it by 3.2 points on Wan2.2 A14B.Source: §5.3 · p. 7

6 Analysis and Discussion

TSG’s category gains are strongest for state transitions, while multi-entity coordination remains difficult; guidance-strength and CFG–TSG ablations support the tested combined guidance design.

6 Analysis and Discussion

  • TSG’s mean accuracy gains span approximately 2–43 percentage points across 11 categories, with the largest gains in alternative final states, two-state transitions, and reversible returns.Gains are 23% to 66%, 35% to 71%, and 21% to 43%, respectively.Source: §6 · p. 7, Figure 4 · p. 8
  • Handover with overlap, triggered responses, and three ordered entries remain challenging, with mean accuracies of approximately 9%, 17%, and 12%.Gains in these categories are only about 2 percentage points; performance also varies substantially across models.Source: §6 · p. 7
  • Figures 1, 2, and 5 illustrate TSG improving adherence over CFG-only across temporal requirements from different categories.Figure 4 presents category-wise mean satisfaction accuracy and the guidance-strength and CFG–TSG ablations.Source: §6 · p. 8, Figure 4 · p. 8
  • On the tested subset, guidance strength ρ = 1 gives the highest satisfaction accuracy: 32% for Wan2.1 1.3B and 36% for Wan2.2 5B.Accuracy is non-increasing above ρ = 1 and at ρ = 5 matches or falls below CFG-only.Source: §6 · p. 8, Figure 4 · p. 8
  • Combined CFG + TSG reaches 32% and 36% accuracy on Wan2.1 1.3B and Wan2.2 5B, exceeding doubled CFG strength at 24% and 20%.Replacing CFG with TSG has mixed effects across models.Source: §6 · p. 8, Figure 4 · p. 8

7 Related work

The related work distinguishes TSG’s temporal-concept control from methods for spatial layouts, latent optimization, post-sampling editing, and learned control, while noting complementary benchmarks and constraints.

7 Related work

  • Unlike TempoControl’s repeated latent optimization, TSG handles more expressive temporal constraints and outperforms it on strict-timed traces with lower computational overhead.TSG uses frozen parameters without additional training data; unlike NeuS-E, it does not require a VLM or image-editing model, while NeuS-E can target black-box models.Source: §7 · p. 8
  • TSG supports richer temporal compositions than VideoTetris but does not directly control spatial positions of concepts.VideoTetris composes attention maps across space and time to control concept positions.Source: §7 · p. 8
  • Figure 5 compares CFG-only and TSG examples for ordered entry, three-state transition, and overlapping handover.The first two examples use Wan2.1 T2V-1.3B; the handover example uses Wan2.2 TI2V-5B in text-to-video mode.Source: Figure 5 · p. 9
  • The TSG benchmark adds interval-timed constraints and alternate outcomes through choice, complementing benchmarks for untimed transitions and exact object-appearance timing.VBench and T2V-CompBench assess broader video qualities and compositional capabilities.Source: §7 · p. 9

8 Conclusion and Future Work

The conclusion presents TSG as a training-free way to turn temporal specifications into sampling objectives, reporting improved adherence across five models and temporal accuracy comparable to TempoControl at lower computational cost.

8 Conclusion and Future Work

  • TSG improves specification adherence across five models on a 220-prompt, 11-category benchmark and achieves temporal accuracy comparable to TempoControl at lower computational cost.The conclusion describes a training-free framework using attention-derived concept traces, differentiable Metric Temporal Logic scores, and attention interventions without backpropagation through the video transformer.Source: §8 · p. 9
  • TSG is limited by models’ latent temporal resolution, leaving finer-grained temporal constraints and richer specification languages as directions for future work.The authors also suggest exploring concept-evidence traces beyond video-text attention maps and other scoring functions.Source: §8 · p. 9

AI use statement

  • The authors used generative AI to help construct the benchmark and prepare the manuscript, reviewed the assisted work, and take responsibility for the final content.Source: AI use statement · p. 10

Reproducibility statement

  • The appendices provide benchmark construction details, model settings, evaluation protocols, TSG pseudocode, and quantitative semantics.Source: Reproducibility statement · p. 10

A Boolean MTL semantics and examples

B Quantitative semantics for MTL

B.1 Score definition and evaluation

  • For steering, the scalar objective is the log score Sφ[1] at the first frame; the Until recurrence aggregates valid witnesses within its permitted window.Source: §B.1 · pp. 12–13

B.2 Linear-time evaluation

The paper develops linear-time evaluation for temporal specification scores, establishes how those scores respond to literal support, and relates score gradients to steering directions. The TSG algorithm uses those gradients in an online attention-steering pass.

B.2 Linear-time evaluation

  • Associative block summaries and a two-stack aggregate queue compute all sliding-window aggregates in linear time while preserving temporal order.Each element enters and leaves the queue at most once; the Until operation’s noncommutativity makes the order of combination essential.Source: §B.2 · pp. 13–14
  • The sum–product log score and its reverse-mode derivatives use O(|φ|N) arithmetic operations and storage, but this arithmetic bound does not establish GPU wall-clock speedup.The linear-time procedure handles nested temporal operators; the stated complexity counts scalar-score operations as constant time.Source: §B.2 · p. 15

B.3 Monotonicity and its scope

  • The score is nondecreasing in independent literal-support inputs; a concept–time coordinate has a predictable direction when it occurs with only one polarity.With both polarities at the same frame, the derivative’s sign need not be fixed.Source: §B.3 · pp. 15–16

B.4 Examples and implications for gradients

  • For dog U ¬dog, increasing absence evidence cannot decrease the score, but the formula permits absence throughout unless initial presence is explicitly conjoined.The example shows that more absence does not always help when the specification also requires persistence.Source: §B.4 · pp. 16–17
  • For nested departure-and-return specifications, each admissible departure–return pair contributes gradients favoring presence before departure, absence until return, and presence at return.The log-score gradient combines these patterns with weights proportional to their score contributions; replacing the final dog proposition with Always dog requires persistence after return.Source: §B.4 · p. 17

C The TSG algorithm

  • TSG applies attention steering during selected sampling steps in one additional forward pass, differentiating detached traces, their calibration, and the specification score rather than the transformer.Later selected layers use representations produced by earlier steered layers.Source: C The TSG algorithm · p. 17

D The TSG benchmark

The benchmark covers 11 categories of temporal constraints through prompt–specification pairs and evaluates phase adherence across video-generation models. The section also specifies the steering settings and a controlled comparison with TempoControl.

D The TSG benchmark

  • The benchmark defines 11 temporal-constraint categories with 20 examples each, covering state transitions, entity appearance and disappearance, and choice.The category descriptions summarize temporal structure; individual specifications also encode timing, absence, and persistence requirements.Source: Table 3 · p. 18, D The TSG benchmark · p. 18
  • The templates allow transitions anywhere within designated half- or third-video intervals, while encoding category-specific ordering and persistence requirements.The benchmark’s entity inventory contains 50 object and setting entries with associated states, manually reviewed for plausibility.Source: D The TSG benchmark · pp. 17, 19
  • Evaluation requires each phase to occur in its designated interval and persist for at least 0.5 seconds, while leaving transitions unconstrained unless the specification adds constraints.The criterion does not require a phase to occupy its entire interval or restrict behavior outside required witness runs.Source: D The TSG benchmark · pp. 21–22

E.1 Models and sampling settings

  • The main benchmark evaluates five named Wan and CogVideoX checkpoints, generating one video per prompt and replaying the same initial latent for each paired TSG run.Prompts and negative prompts are held identical within each paired comparison.Source: §E.1 · p. 22

E.2 TSG implementation and hyperparameters

  • Main-benchmark TSG uses online attention steering for the first 15 denoising steps, with norm ratio ρ = 1, fixed calibration statistics, and clipped attention-logit biases.The implementation steers all attention heads in all transformer layers, using a bias step size of 1 and clipping to [−1.0, 1.0].Source: §E.2 · pp. 22–23

E.3 TempoControl comparison

  • The TempoControl comparison uses Wan 2.1 at matched generation settings and replays the full TempoControl run’s saved initial latent across CFG-only, TempoControl, one-round TempoControl, and TSG.The one-round variant limits optimization updates to one while retaining the same early-stopping check and 11 guided steps.Source: §E.3 · p. 23

E.4 Ablation protocol and parameter selection

The ablation protocol varies guidance strength and CFG interaction while retaining the stated base settings and paired inputs.

E.4 Ablation protocol and parameter selection

  • The guidance-strength sweep compares ρ values 2, 3, 4, and 5 with base and ρ = 1 runs on the same 25 samples, changing only ρ.Each configuration replays its model’s saved base latent and retains the specified layers, guided steps, calibration, and clipping settings.Source: §E.4 · p. 23
  • CFG-interaction ablations double CFG strength for selected runs or use TSG-only guidance across all 50 steps in place of CFG’s conditional direction.TSG-only retains CFG’s guidance scale while replacing its conditional direction with the TSG-predicted specification-guided direction.Source: §E.4 · p. 23

E.5 Compute and measurement protocol

The compute protocol records synchronized per-video latency and allocated-memory peaks, with explicit measurement boundaries and hardware qualifications.

E.5 Compute and measurement protocol

  • Generation latency includes noise preparation or replay, prompt encoding, denoising and guidance, and VAE decoding, but excludes model loading, export, and evaluation.Measurements use CUDA synchronization at boundaries, with allocator-cache clearing and peak-statistic resets before each video; no separate warm-up video is discarded.Source: §E.5 · p. 23
  • Reported peak memory is per-video PyTorch allocated memory, including resident model weights, rather than device-wide usage or allocator-reserved memory.Mean ± standard deviation uses sample standard deviation across videos, not a confidence interval or repeated-seed variation.Source: §E.5 · pp. 23–24
  • Cross-model latency comparisons also reflect differing hardware because Wan 2.2 A14B runs on H200 GPUs while the other listed models run on B200 GPUs.The exact attention kernel selected at runtime is not recorded in per-video profiles.Source: §E.5 · pp. 23–24

E.6 Evaluation metrics

Evaluation samples decoded videos at 21 uniformly spaced frames and measures prompt-derived specification satisfaction, with separate metrics for object timing and visual quality. A per-frame vision-language evaluator labels concepts without seeing the requested temporal schedule or neighboring frames.

Evaluation metrics

  • Specification satisfaction accuracy evaluates prompt-derived specifications under Boolean semantics using binary concept traces from 21 uniformly spaced decoded frames, including the first and last frame.The evaluator is asked to inspect each sampled frame independently and report only visibly supported evidence, not the desired temporal phase.Source: §E.6 · p. 24
  • Visual quality is measured with MUSIQ-SPAQ on the sampled frames, averaged first within each video and then across videos, with larger raw MUSIQ values indicating better perceptual frame quality.Frames are resized so their longer side is at most 512 pixels using VBench preprocessing.Source: §E.6 · p. 24
  • The vision-language evaluator labels object presence as present, absent, or unclear and can return multiple visibly supported states; explicit unclear labels and omitted fields map to false in the binary trace.The parser normalizes labels, ignores unknown fields, rejects invalid known fields, and permits two schema-correction retries on the same frame.Source: §E.6 · p. 24
  • 0.94% of frames had at least one unclear proposition; conservative treatment left reported accuracies unchanged, while optimistic treatment increased CFG accuracy by 0.73 percentage points and TSG accuracy by 1.36 percentage points.The evaluator returned unclear labels in 433 of 46,200 individual frames; conservatively, no currently passing video failed.Source: §E.6 · p. 24
  • Object temporal accuracy is the mean per-video fraction of correct sampled frames, requiring the target presence mask for one-object clips and both objects in the appropriate phases for two-object clips.This frame-wise metric differs from the whole-video specification accuracy used on the TSG benchmark.Source: §E.6 · pp. 24–25

Related papers

Loading 2610.20822v1…