PaperlayerView exact revision on arXiv

Source-linked AI summary

AVControl: Efficient Framework for Training Audio-Visual Controls

Matan Ben-Yosef, Tavi Halperin, Naomi Ken Korem, Mohammad Salama, Harel Cain, Asaf Joseph, Anthony Chen, Urska Jelercic, Ofir Bibi

arXiv:2603.24793v1cs.CVcs.MMcs.SD

TL;DR

Existing audio-visual generation controls span diverse modalities, but monolithic or costly-to-adapt systems limit flexible extension. AVControl uses independently trained LoRAs with parallel-canvas conditioning on LTX-2, outperforming evaluated VACE Benchmark baselines while requiring a small, modular training budget.

  • Problem

    Audio-visual generation needs diverse controls, but existing systems are monolithic or costly to adapt across modalities.

  • Method

    AVControl trains independent LoRA adapters on a frozen LTX-2 backbone, conditioning generation through reference tokens on a parallel canvas.

  • Results

    AVControl outperforms all evaluated VACE Benchmark baselines while the 13 modalities require approximately 55K total training steps.

  • Takeaways & Limitations

    The framework supports practical extension to new audio-visual controls through small datasets and short training runs without retraining existing modalities.

  • Takeaways & Limitations

    AVControl does not address reference image conditioning, which is better handled by a separate complementary mechanism.

Abstract

from arXiv · show

Controlling video and audio generation requires diverse modalities, from depth and pose to camera trajectories and audio transformations, yet existing approaches either train a single monolithic model for a fixed set of controls or introduce costly architectural changes for each new modality. We introduce AVControl, a lightweight, extendable framework built on LTX-2, a joint audio-visual foundation model, where each control modality is trained as a separate LoRA on a parallel canvas that provides the reference signal as additional tokens in the attention layers, requiring no architectural changes beyond the LoRA adapters themselves. We show that simply extending image-based in-context methods to video fails for structural control, and that our parallel canvas approach resolves this. On the VACE Benchmark, we outperform all evaluated baselines on depth- and pose-guided generation, inpainting, and outpainting, and show competitive results on camera control and audio-visual benchmarks. Our framework supports a diverse set of independently trained modalities: spatially-aligned controls such as depth, pose, and edges, camera trajectory with intrinsics, sparse motion control, video editing, and, to our knowledge, the first modular audio-visual controls for a joint generation model. Our method is both compute- and data-efficient: each modality requires only a small dataset and converges within a few hundred to a few thousand training steps, a fraction of the budget of monolithic alternatives. We publicly release our code and trained LoRA checkpoints.

1 Introduction

AVControl is a flexible framework that trains each video or audio control modality as an independent LoRA on a parallel canvas over the joint audio-visual LTX-2 backbone. This design supports diverse controls, faithful structural conditioning, efficient extension, and lower-latency inference for sparse modalities.

  • Framework design: AVControl replaces a monolithic all-controls system with independently trained lightweight LoRAs that can be extended to new modalities without retraining existing ones.New controls require only a small dataset and a short training run.
  • Backbone: AVControl builds on LTX-2, a joint audio-visual DiT that natively generates synchronized video and audio.This makes LTX-2 a natural backbone for multimodal control.
  • Control modalities: The framework covers spatial controls, camera trajectories, sparse motion, editing, and audio-visual applications including synchronized video and reference audio.Examples include depth, pose, Canny edges, camera-motion control, trajectory re-rendering, acoustic adaptation, and audio-conditioned synchronization.
  • Structural control: Parallel-canvas LoRAs provide faithful video structural control while enabling fine-grained inference-time strength modulation.The approach addresses the failure of image-based in-context conditioning to faithfully follow structural signals such as depth.
  • Inference efficiency: A small-to-large control grid reduces reference-canvas resolution for spatially sparse modalities, lowering latency without sacrificing control fidelity.The strategy is intended to accelerate inference for controls such as camera parameters.

2 Related Work

Prior work spans parameter-efficient LoRA adaptation, reference-conditioned generation, controllable and unified video frameworks, and emerging audio-visual controls. AVControl instead trains each modality as a separate LoRA on an unmodified joint audio-visual backbone without positional-encoding changes.

  • Parameter-efficient adaptation: LoRA enables parameter-efficient fine-tuning by injecting trainable low-rank matrices into frozen layers, supporting identity, style, motion, and multi-LoRA video control.This flexibility motivates modular control through independently trained adapters.
  • Reference-guided generation: Reference-guided generation commonly uses channel concatenation or additional attention tokens to incorporate spatial signals such as depth, pose, and masks.Attention-token methods enable richer interactions but increase the token budget.
  • Unified frameworks: Unified video frameworks combine diverse tasks or conditions, but VACE remains limited to its training-time control set.Other approaches use single token sequences, task-aware RoPE, or shared condition units for multimodal generation.
  • Camera trajectory control: Camera-control methods add architectural components, while AVControl camera LoRAs require only 3,000–10,000 steps and no backbone modifications.Earlier methods control extrinsics or use 4D geometric mechanisms, with BulletTime requiring 40K iterations at batch size 64 and VerseCrafter 380 GPU hours.
  • Audio-driven talking video: Audio-driven talking-video methods add audio cross-attention and Label RoPE, whereas AVControl models who-is-talking with a single LoRA on the unmodified joint audio-visual backbone.Its control signal is an abstract bounding-box activity signal.
  • Our approach: AVControl trains each control modality as a separate LoRA with no new layers, input projections, or positional-encoding changes, distinguishing reference from generation via LTX-2’s per-token timestep.This contrasts with monolithic or unified-token approaches and methods that introduce RoPE offsets or task-aware positional mechanisms.

3 Method

AVControl injects reference signals through a parallel canvas of latent tokens while training only lightweight LoRA adapters on a frozen audio-visual backbone. This design improves structural fidelity over spatial concatenation and supports efficient, modular control across video, audio, and joint modalities.

  • Parallel-canvas conditioning: AVControl places the reference signal on a parallel canvas, encodes it with the target’s VAE, and jointly processes reference and noisy target tokens through self-attention.Reference tokens use a clean timestep (t=0), while generation tokens retain the current noise level, distinguishing the two without positional-encoding changes.
  • Parameter-efficient training: Only a lightweight LoRA adapter is trainable, while the pretrained backbone remains frozen.The adapters target projection matrices and feed-forward layers, with the exact optimized modules selected per modality.
  • Training efficiency: Some modalities converge in a few hundred steps, while most spatially aligned controls require only a few thousand.The efficiency comes from leveraging pretrained self-attention layers to inject control; reference-target attention can also be directly reweighted.
  • Structural control: A concatenation-based depth LoRA captures scene semantics but fails to faithfully follow the depth signal’s spatial structure, motivating the parallel-canvas design.The failure is illustrated in Figure 3 and specifically concerns structural adherence rather than general scene understanding.
  • Modular multimodal control: Each LoRA can train on video, audio, or joint audio-visual pairs, enabling single-stream control while the base model generates the synchronized complementary stream.Multiple controls can be composited onto one canvas, and reference resolution is scaled with modality information density to reduce extra tokens, latency, and memory overhead.

4 Experiments

AVControl achieves leading results across VACE depth, pose, inpainting, and outpainting tasks while extending modular control to camera, audio-visual, and cross-modal applications. Independent LoRAs train efficiently, converge rapidly, and rely on parallel-canvas conditioning rather than spatial concatenation.

  • VACE Benchmark: AVControl achieves the highest average VBench score on all four VACE Benchmark tasks, outperforming VACE by 2.9 points on depth and 2.3 on pose.It maintains dynamic degree of 68.4 on depth and 84.2 on pose, avoiding ControlVideo’s over-constraining DD of 10–25.
  • VACE Benchmark: AVControl outperforms VACE by 3.8 points on inpainting and 2.3 points on outpainting, with inpainting gains of +8.4 in both AQ and IQ.The same inpainting LoRA is used for both tasks, and qualitative outputs show higher structural fidelity with natural motion and visual quality.
  • Modality breadth: The framework supports independently trained LoRAs for spatial controls, editing, composited controls, sparse trajectories, camera trajectories, and audio-visual modalities without retraining existing LoRAs.Camera control includes image-to-motion and video re-rendering modes with estimated extrinsics, intrinsics, and per-frame FOV, enabling focal-length changes such as dolly zoom.
  • Camera control: 99.13% CLIP-F is achieved on the ReCamMaster Benchmark, surpassing ReCamMaster at 98.74%, while COLMAP-based RotErr is 6.00° and SpatialTrackerV2 yields 3.55°.The evaluation covers 200 videos across 10 trajectory types; COLMAP SfM fails on 27% of the videos.
  • Audio-visual applications: Audio-visual controls include audio intensity, speech-to-ambient, and who-is-talking generation, with inference producing synchronized audio and video in one joint pass.On VGGSound, the intensity LoRA achieves the highest IS of 34.51; on HDTF, it achieves E-FID 0.18 and FID 12.31 while lagging dedicated methods on lip-sync.
  • Efficiency and ablations: ∼55,000 aggregate training steps cover all modalities, versus VACE’s 200,000; depth reaches VBench scores of 81.1 at 1,000 steps and 81.6 at 3,000 steps.Ranks 32, 64, and 128 yield depth averages of 80.9, 81.3, and 81.6, while spatial concatenation fails to follow depth signals under matched conditions.

5 Limitations

AVControl inherits limitations from its underlying base model and introduces failure modes in mask encoding, complex motion, and fast camera-controlled scenes. It also lacks VACE-level reference image conditioning, which is better addressed by a complementary mechanism.

  • Base-model limitations: AVControl inherits the base model’s limitations, so improvements in character motion, high-frequency detail, and audio quality would directly improve its results.The framework’s performance remains bounded by the capabilities of the underlying joint audio-visual model.
  • Design constraints: Mask encoding within the reference video is reliable but can fail rarely when scene colors resemble the designated inpainting color.Masks are not supplied as a separate input; they are encoded directly in the reference video.
  • Motion failures: Rapid, intricate character motion in depth or pose references can cause temporal jitter or implausible limb configurations.This failure mode arises when the reference signal contains complex character movements.
  • Camera-control failures: Fast non-rigid scene dynamics can create stretching or ghosting in camera-control reference canvases, which the LoRA may reproduce rather than correct.The artifacts result from per-frame point-cloud reprojection into the target viewpoint.
  • Capability gap: Reference image conditioning remains AVControl’s main capability gap relative to VACE because identity preservation requires a separate complementary mechanism.The framework targets spatial and temporal controls rather than identity preservation.

6 Conclusion

AVControl provides modular, faithful structural control for audio-visual foundation models through a parallel canvas and per-modality LoRA adapters, while remaining extensible to new control modalities. The conclusion identifies open work in evaluation, LoRA composition, and personalization.

  • Core mechanism: The parallel canvas disambiguates reference and generation tokens through per-token timesteps, enabling faithful structural control without positional-encoding changes.The mechanism also supports fine-grained control-strength adjustment at inference time.
  • Modularity: Per-modality LoRA adapters on a frozen backbone make AVControl modular and extendable for audio-visual foundation models.Independent control modalities are trained as separate LoRAs, with examples requiring 200–15,000 training steps.
  • Extensibility: Concurrent work applies AVControl to video dubbing, identity-driven audio-video personalization, and portrait video editing, demonstrating generalization beyond the authors’ trained modalities.The cited efforts are JUST-DUB-IT, ID-LoRA, and In-Context Sync-LoRA, respectively.
  • Open directions: Open directions include perceptual-metric and user-study evaluation of audio-visual modalities, lightweight inference-time LoRA combination, and user-specific controls from a handful of personal examples.These directions target evaluation, composition, and personalization capabilities not yet fully addressed.

7 Comprehensive Training Details

AVControl trains each modality as an independently selected LoRA using a shared, lightweight optimization setup, while constructing modality-specific reference canvases and targets. The resulting controls span spatial, camera, editing, motion, and audio-visual tasks with efficient convergence and reduced inference cost for sparse references.

  • Optimization setup: Each modality LoRA is trained independently on one NVIDIA H100 with AdamW and a linearly decaying learning rate from 1×10−4 to 1×10−5.Checkpoints are selected by manually ranking validation-set outputs by overall preference.
  • Optimization setup: The per-modality configurations specify optimization steps, dataset size, LoRA rank, and adapted transformer modules, with video-only and audio-only LoRAs omitting the other branch’s loss.Table 5 defines SA as self-attention, CA as cross-attention, and FF as feed-forward modules across video and audio blocks.
  • Spatial and editing data: Spatial controls derive depth, pose, and edge references from real videos, place them on half-resolution canvases, and target the original video.Depth uses Video Depth Anything, pose uses DWPose, and edges use OpenCV’s Canny detector with thresholds 100/200.
  • Efficiency: Training ranges from 200 steps for video detailing to 15,000 for cut-on-action, with approximately 3,000 steps for most spatial controls and ∼55K steps across 13 LoRAs.The combined budget is less than one-third of a 200K-step VACE run, and 4× reference downscaling yields 35–50% inference speedup for sparse controls.
  • Spatial and editing data: Inpainting and outpainting use random rectangular or free-form masks covering 10–60% of each frame, unified by a shared green-filled mask representation.The same LoRA handles interior masks for inpainting and border masks for outpainting.
  • Audio-visual data: Audio controls use task-specific references, including energy-envelope sonification for audio intensity, speech-only tracks for ambient reconstruction, and speaker activity layouts for who-is-talking.Speech-to-ambient training uses 2,600 samples in 5,000 steps, while who-is-talking is the only LoRA training video, audio, and cross-modal modules.

8 Small-to-Large Control Grid

AVControl reduces reference-canvas resolution according to each modality’s information density, lowering token usage while preserving spatial control for selected modalities. These reductions improve inference speed through lower quadratic attention cost.

  • Information Density: Reference-canvas resolution is matched to information density: sparse camera trajectories need fewer tokens, whereas pixel-transferring modalities require full detail.Attention computation decreases because its cost scales quadratically with the combined target and reference token count.
  • Resolution Selection: Camera-from-image uses a 4× reference-canvas reduction, while depth, pose, canny, and sparse tracks use 2× reductions.The default is a full-resolution 1× canvas; camera-from-image is inherently sparse, while the 2× grid remains sufficient for faithful spatial control.
  • Inference Efficiency: 25–35% inference speedup results from a 2× canvas reduction, while 4× reduction yields 35–50%.Measurements were made on an H100 GPU, with exact gains depending on output resolution.

9 Extended Ablations

Extended ablations show that depth-control quality saturates early, while inference-time strength and canvas resolution provide practical fidelity–diversity and latency trade-offs. Synthetic-data-trained LoRAs also transfer to diverse real-world videos without fine-tuning.

  • Training steps vs. quality: VBench average scores rise from 79.8 at 500 steps to 81.6 at 3,000 steps, with performance plateauing beyond 2,000 steps.The evaluated checkpoints are 500, 1,000, 2,000, and 3,000 steps; later checkpoints add less than 0.2 points beyond 3,000 steps.
  • Inference-time strength modulation: Global strength modulation continuously trades structural fidelity against generative diversity, from no reference influence at 0 to close depth-map adherence at 1.Intermediate strengths preserve coarse depth structure while allowing more creative freedom in fine details; the mechanism also extends to temporal and spatial modulation.
  • Control grid resolution: A 4× downscaled camera reference canvas reduces inference latency by 35–50% versus full resolution while maintaining comparable qualitative results.For dense spatial controls such as depth and pose, reducing beyond 2× downscale causes some structural-fidelity loss.
  • Generalization from synthetic data: Synthetic or generated-data-trained LoRAs generalize to diverse real-world videos without fine-tuning across cut-on-action, camera trajectory, and local-edit controls.Cut-on-action and camera trajectory use rendered multi-camera scenes, local edit uses ROSE synthetic scene pairs, and all transfer to diverse real-world content.
  • Generalization from synthetic data: Who-is-talking also generalizes to held-out speakers and scenes despite training data generated by the base model rather than an external renderer.Together, these results support the transferability of parallel-canvas controls beyond the data-generation setup used for training.

10 Extended Qualitative Gallery

The extended qualitative gallery demonstrates AVControl across spatial controls, video editing, camera trajectories, and audio-visual modalities. Results include controllable generation and editing behaviors, while audio-only quality is deferred to supplementary video because static frames cannot convey it.

  • Spatially-aligned controls: Canny edge and sparse track controls produce corresponding spatially guided video outputs.Figures 10–11 show edge-guided generation and point trajectories rendered as colored dots on a black canvas.
  • Video editing: Video editing results cover inpainting, outpainting, local editing, and video detailing.The local-edit LoRA propagates a first-frame edit consistently across the video.
  • Camera trajectory control: Camera control supports image-to-video, video-to-video, diverse viewpoints, and cut-on-action re-rendering.The gallery demonstrates target camera motion from a single image, trajectory changes from video while preserving scene motion, and substantially different camera angles.
  • Audio-visual modalities: Who-is-talking control generates multi-person talking video with synchronized lip motion and audio.An abstract layout of colored bounding boxes with voice activity serves as the reference control.
  • Audio-visual modalities: Audio-only modalities are best evaluated in the supplementary video because static frames cannot convey audio quality.The modalities include audio intensity and speech-to-ambient.
Loading 2603.24793v1…