Source-linked AI summary
TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Models
Chao Zhou, Yiling Chen, Qi Chu, Tao Gong, Nenghai Yu, Tianyi We
TL;DR
Pretrained joint audio-visual diffusion models control generated content but lack explicit control over utterance timing, motivating inference-time speech scheduling. TimeSteer localizes source spans and remaps coupled audio-visual latent content to user-specified intervals, and experiments across two backbones show improved interval controllability with competitive generation quality.
Problem
Joint audio-visual generators lack an explicit mechanism for placing utterances within user-specified intervals, while existing benchmarks do not evaluate interval-level speech controllability.
Method
TimeSteer uses attention-based Source Span Localization and Region-Aware Latent Remapping during denoising, while SpeechShift benchmarks interval-level scheduling.
Results
TimeSteer substantially improves interval controllability over training-free baselines across two backbones while maintaining competitive generation quality.
Takeaways & Limitations
Inference-time scheduling can redirect model-generated speech together with its visual articulation without retraining the generator or regenerating their content.
Takeaways & Limitations
The scheduling operator assumes a pretrained joint audio-visual diffusion model with frozen parameters and must preserve its generation quality.
Abstract
from arXiv · showhide
Although pretrained joint audio-visual diffusion models offer rich control over \emph{what} to generate, they provide no explicit control over \emph{when} an utterance should occur. To address this, we study \emph{inference-time speech scheduling}, a novel task that places coupled speech and visual articulation within user-specified begin--end intervals without finetuning the backbone model. We uncover two intrinsic properties of the denoising process that enable this task. First, a timing-sensitive text-to-audio cross-attention head exposes each utterance's model-implied source span along the latent timeline. Second, the predicted clean latent already organizes coupled speech and visual articulation, allowing their temporal placement to be edited without regenerating the content. Building on these discoveries, we propose \textbf{TimeSteer}, a training-free framework that localizes each utterance's source span through \textbf{Source Span Localization} and transfers the associated audio-visual latent content from the source interval to the specified target interval through \textbf{Region-Aware Latent Remapping}. We further introduce \textbf{SpeechShift}, the first benchmark for interval-level speech scheduling in joint audio-visual generation. Experiments across two representative backbones show that TimeSteer substantially improves interval controllability over training-free baselines while maintaining competitive overall generation quality.
Introduction
TimeSteer introduces inference-time speech scheduling for placing coupled utterances and visual articulation within user-specified intervals without retraining. It localizes model-implied source spans and remaps their latent content, while SpeechShift evaluates interval control and generation quality.
- Inference-time speech scheduling places each generated utterance and its coupled visual articulation within a user-specified begin–end interval without retraining.
- A timing-sensitive text-to-audio cross-attention head exposes each utterance’s source span, while the predicted clean latent contains its coupled audio-visual articulation.
- TimeSteer uses Source Span Localization followed by Region-Aware Latent Remapping to transfer joint audio-visual latent content into target intervals without regenerating it.
- Minimal-Distortion Remapping adapts speech duration with an affine map, while Curvature Bridging smoothly transitions across non-speech gaps.
- SpeechShift contains 400 prompts, 600 utterance-level targets, and 102 scenes, measuring boundary timing, temporal overlap, and preserved generation quality.
- Across two backbones, TimeSteer substantially improves interval controllability over training-free baselines while maintaining competitive generation quality.
Related Work
Prior work controls timing within a single modality or inherits timing from externally provided audio or video. TimeSteer instead relocates model-generated speech together with visual articulation inside a pretrained joint audio-visual diffusion model at inference time.
- Joint audio-visual diffusion models synthesize sound and video through increasingly integrated architectures, including separate-stream cross-attention and unified modality-token attention.
- Recent conditioning, alignment, and preference-optimization methods broaden control over generated content but do not expose when individual utterances should occur.
- Text-to-audio methods place acoustic events using timelines, timestamps, compositional controls, or inference-time attention manipulation.
- Audio-conditioned video, video-to-audio, and talking-face systems obtain timing from a pre-generated modality rather than jointly scheduling generated speech and articulation.
- TimeSteer relocates model-generated speech and visual articulation within a pretrained joint model using user-specified intervals, without external speech or additional training.
Method and Benchmark
TimeSteer schedules coupled audio-visual speech by estimating utterance source spans from cross-attention and remapping predicted clean latent content to user-specified intervals. SpeechShift provides a benchmark with scene prompts, quoted utterances, speaker roles, target intervals, and challenge metadata.
- TimeSteer: TimeSteer applies a training-free scheduling operator to the predicted clean latent during denoising, leaving the pretrained model parameters unchanged.The scheduled clean latent is converted back into a velocity prediction so denoising proceeds unchanged.
- Region-Aware Latent Remapping: Region-Aware Latent Remapping constructs a destination-to-source read map that relocates joint audio-visual content from each localized source span to its target interval.The map samples source positions at destination positions, aligns interval endpoints, fixes clip boundaries, and linearly interpolates between discrete latent indices.
- Source Span Localization: Source Span Localization estimates each utterance’s model-implied source span from a timing-sensitive text-to-audio cross-attention response before decoding.Attention is aggregated over the utterance’s quoted tokens to produce correspondence scores over audio latent positions and a thresholded span estimate.
- Region-Aware Latent Remapping: Minimal-Distortion Remapping uses affine maps within speech intervals, where the constant slope represents the speaking-rate ratio and applies equally to associated visual articulation.The method targets minimal deviation from unit slope while satisfying source–destination endpoint constraints.
- Region-Aware Latent Remapping: Curvature Bridging uses a cubic read map across non-speech gaps to smoothly connect neighboring speech-interval maps while minimizing temporal-rate variation.The cubic coefficients are uniquely determined by four boundary constraints, including inherited slopes from adjacent affine speech maps.
- SpeechShift Benchmark: SpeechShift represents each benchmark sample with a scene prompt, ordered quoted utterances, speaker roles, target intervals, and metadata describing scene and challenge conditions.It is designed as a benchmark for interval-level speech scheduling in text-to-audio-visual generation.
Experiments
Experiments evaluate TimeSteer on two joint audio-visual diffusion backbones, comparing localization and remapping choices against training-free baselines. TimeSteer improves interval controllability while preserving competitive generation quality.
- Main results: On LTX-2, HR0.2 increases from 0.21 to 0.73 and IoU from 0.63 to 0.87 over Uncontrolled sampling.
- Main results: On daVinci-MagiHuman, HR0.2 increases from 0.09 to 0.53 and IoU from 0.63 to 0.79 over Uncontrolled sampling.
- Main results: TimeSteer achieves the lowest onset and offset errors on both architectures while maintaining generation-quality metrics comparable to uncontrolled sampling.
- Source Span Localization: The clean-latent attention strategy achieves the highest interval IoU and lowest onset and offset errors among four source-span localization strategies.The noisy-latent attention strategy has less reliable offsets, while decode-based variants perform substantially worse because intermediate speech activity is unstable.
- Remapping-space ablation: Remapping the predicted clean audio-visual latent achieves the highest HR0.2 and lowest WER among four intervention spaces.Direct remapping of velocity or noisy latents disrupts denoising, while post-hoc remapping cannot reliably relocate content.
- Read-map design: The region-aware read-map design achieves a 67.9% Span Win versus 19.4% for Global PCHIP, while also attaining the highest HR0.2 and temporal IoU.It assigns linear remapping to speech intervals and smooth cubic bridging to non-speech gaps.
Conclusion
The paper shows that pretrained joint audio-visual diffusion models’ implicit timing can be exposed and redirected during generation. TimeSteer and SpeechShift provide a training-free scheduling method and benchmark, with improved controllability across two backbones and competitive quality.
- A timing-sensitive cross-attention head reveals each utterance’s source span, while the predicted clean latent contains coupled speech and visual content for relocation.
- TimeSteer transfers localized content to user-specified target intervals during denoising through a region-aware read map without retraining the generator.
- SpeechShift is introduced as the first benchmark for interval-level speech scheduling in joint audio-visual generation.
- Across LTX-2 and daVinci-MagiHuman, TimeSteer improves interval controllability over training-free baselines while retaining competitive generation quality.
Appendix
The appendix presents the complete TimeSteer sampling procedure, derives region-aware latent remapping, and documents experimental setup, implementations, component selection, and expanded results.
- The appendix covers the complete TimeSteer sampling procedure, region-aware latent remapping for speech intervals and non-speech gaps, and experimental details.
TimeSteer Procedure
TimeSteer integrates source-span localization and region-aware latent remapping into standard flow-matching sampling. At each denoising step, it edits the predicted clean latent before the original sampler update.
- At each denoising step, the model predicts velocity and recovers the corresponding clean latent.
- TimeSteer remaps the clean latent before converting it back into the velocity used by the original flow-matching update.
- The procedure loops over the noise schedule and decodes the final latent into audio and video outputs.
- Source Span Localization estimates each utterance’s source span, while Region-Aware Latent Remapping constructs a destination-to-source read map.
- The same neighboring indices and interpolation weights are applied to audio and video branches, preserving temporal correspondence.
Remapping
Region-Aware Latent Remapping preserves speech structure within utterances while smoothly connecting neighboring utterances across non-speech gaps. It uses affine and cubic maps to control speaking-rate distortion and read-map-slope variation.
- Region-Aware Latent Remapping preserves internal speech structure and smoothly connects neighboring utterances across non-speech gaps.
- The destination-to-source read-map slope determines the local speaking-rate ratio.
- The method derives an affine speech map minimizing speaking-rate distortion and a cubic gap map minimizing read-map-slope variation.
Speaking-Rate Interpretation of the Read-Map Slope
The read map relates source and destination speech timelines, with its slope determining local speaking-rate changes. Applying the same map to audio and video latents temporally scales the associated visual articulation consistently.
- The source and destination cumulative speech-content functions define corresponding local speaking rates through their derivatives.
- The read map h specifies which source coordinate is sampled at each destination coordinate.
- The chain-rule relationship connects the read-map slope to source and destination speaking rates.
- h′(t) gives the local multiplicative change from original to output speaking rate, while h′(t) = 1 preserves the original rate.
- Applying the same read map to audio and video latents gives visual articulation the same temporal scaling.
Minimal-Distortion Remapping for Speech Intervals
Within each speech interval, the method minimizes deviation from the original speaking rate under source–destination endpoint constraints. Euler–Lagrange analysis yields affine stationary solutions, and strict convexity establishes the selected affine map as the unique global minimizer.
- Minimal-Distortion Remapping for Speech Intervals: Within each speech interval, the objective minimizes accumulated deviation from h′(t) = 1 under source–destination endpoint constraints.
- Minimal-Distortion Remapping for Speech Intervals: The Euler–Lagrange equation supplies the stationary condition for the remapping functional.
- Minimal-Distortion Remapping for Speech Intervals: Integrating the stationary condition twice yields an affine solution whose constants are determined by endpoint constraints.
- Minimal-Distortion Remapping for Speech Intervals: The integrand is strictly convex in h′, making the stationary affine map the unique global minimizer of the complete objective.
Curvature Bridging for Non-speech Gaps
Curvature Bridging smoothly connects remapped speech intervals across non-speech gaps by matching boundary slopes, yielding a continuously differentiable read map. TimeSteer then relocates coupled audio-visual latent content while preserving competitive generation quality.
- Curvature Bridging: Curvature Bridging smoothly transitions between neighboring speech-rate ratios across non-speech gaps by minimizing accumulated squared curvature under boundary constraints.The objective penalizes rapid changes in the local speaking-rate ratio h′(t).
- Curvature Bridging: The resulting cubic-polynomial bridge matches endpoint values and slopes, making the read map continuously differentiable across speech intervals and non-speech gaps.Both h and h′ match the adjacent affine speech maps at the gap boundaries.
- Curvature Bridging: The cubic solution is the unique global minimizer because every nonzero admissible perturbation has strictly positive second variation.The squared second-derivative objective is strictly positive for every nonzero admissible perturbation.
- TimeSteer: TimeSteer uses affine speech segments and cubic bridges to relocate joint audio-visual latent content without retraining the backbone.The remapped clean latent returns to the original sampler for subsequent denoising refinement.
- Evaluation: Across two backbones, TimeSteer improves interval controllability while preserving competitive transcription, semantic-fidelity, perceptual-quality, and cross-modal scores.It also remains close to uncontrolled generation in amortized per-step latency.