PaperlayerView exact revision on arXiv

Source-linked AI summary

AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation

Kien T. Pham, I Chieh Chen, Qifeng Chen, Long Chen

arXiv:2606.30811v1cs.CVcs.MMcs.SDeess.AS

TL;DR

Existing audio-video generation methods often use separate modality-specific tokenizers, leaving representation differences and computational costs unresolved. AVTok jointly tokenizes audio and video into a compact one-dimensional representation and performs strongly in reconstruction and downstream generation tasks.

  • Problem

    Existing audio-video generation methods use separate modality-specific tokenizers, leaving representation differences and high computational costs unresolved for unified audio-video modeling.

  • Method

    AVTok uses a dual-stream transformer with shared encoder-decoder components, modal-specific queries, a unified codebook, and hierarchical Video-First-Audio-Later training.

  • Results

    AVTok demonstrates strong unified audio-video reconstruction and downstream performance in audio-to-video, video-to-audio, and class-conditional joint generation.

  • Takeaways & Limitations

    The results support the feasibility of unified one-dimensional audio-video tokenization for holistic generation.

  • Takeaways & Limitations

    The authors note that AVTok still has limitations, but the supplied passage does not specify them.

Abstract

from arXiv · show

Audio-video generation has recently gained unprecedented research attention, aiming to synthesize high-quality sounding video content with fine-grained synchronization and semantic alignment between the auditory and visual components. The preceding methods predominantly adopt a dual-branch design with separate tokenization and generation modules per modality, neglecting the representation gap while necessitating intensive computational resources for proper training. Inspired by recent advancements in one-dimensional visual tokenization, we present \textbf{AVTok}, a novel unified tokenizer designated for holistic audio-video generation. AVTok features a dual-stream transformer-based architecture with shared encoder-decoder and modal-specific learnable queries to efficiently and effectively encode an audio-video pair into a compact one-dimensional latent representation with a unified codebook. To cope with the heterogeneous information imbalance that hinders AVTok from exploiting aligned audio-visual information, we devise a hierarchical training strategy to progressively realize reconstruction capabilities for each modality. Extensive experiments demonstrate that AVTok excels both in audio-video reconstruction and when integrated into downstream pipelines for audio-to-video, video-to-audio, and class-conditional joint audio-video generation. AVTok paves the way for the challenge of joint audio-video tokenization and provides a potential direction to build unified large multimodal models for audio-video generation.

1 Introduction

AVTok addresses the representation gap created by separate audio and video tokenizers by jointly encoding sounding video into a single 1D latent representation. Its dual-stream query-based transformer and hierarchical training strategy target holistic AV reconstruction and downstream generation.

  • Motivation: Separate modality-specific tokenizers neglect the intrinsic representation difference between audio and video embedding spaces.This motivates a unified tokenizer that encodes both modalities into a shared latent space.
  • AVTok: AVTok introduces unified audio-video tokenization, jointly encoding auditory and visual components into a single latent representation.The representation is intended to support efficient and effective reconstruction and downstream generation.
  • AVTok: AVTok uses a dual-stream query-based transformer with shared encoder-decoder components and modality-specific learned queries for 1D unified tokenization.Dual-stream processing preserves modality-specific elements while implicitly fusing audio-visual information for reconstruction.
  • Training: VFAL progressively incorporates video and audio encoding and reconstruction through hierarchical training with representation alignment learning.The strategy addresses information-density imbalance and insufficient cross-modal interaction during training.
  • Results: Experiments show AVTok supports unified AV reconstruction and downstream audio-to-video, video-to-audio, and class-conditional joint AV generation.These tasks are identified as the principal downstream applications demonstrating AVTok’s capabilities.

2 Related Work

Prior work develops compact 1D tokenization for visual data and investigates neural audio coding, while audio-video generation increasingly targets synchronized and semantically aligned cross-modal synthesis. Joint audio-video methods mainly use separate modality-specific branches, motivating unified approaches.

  • 1D Visual Tokenization: 1D visual tokenization bridges vision-language representation gaps and reduces visual-processing costs for efficient LLM integration.TiTok encodes a 256 × 256 × 3 image with as few as 32 discrete tokens using learnable queries.
  • 1D Visual Tokenization: Video tokenizers extend query-based holistic encoding into 1D latent representations, with later work adapting temporal causality and token allocation.LARP also targets optimal token order for downstream autoregressive generation.
  • Audio Tokenization: Audio tokenization remains challenging because neural codecs must balance high-fidelity reconstruction against low-bitrate discrete representations for LLM integration.Early codecs such as EnCodec and DAC use residual vector quantization within their coding pipelines.
  • Audio-Video Generation: Audio-video generation spans A2V, V2A, and JAVG, with prior systems improving synchronization, semantic coherence, spatial alignment, temporal alignment, and audio fidelity.Representative methods include TempoTokens, Seeing-and-Hearing, SpA2V, SpecVQGAN, V-AURA, FoleyCrafter, VinTAGe, and MMAudio.
  • Audio-Video Generation: JAVG prioritizes high-fidelity generation in both modalities with cross-modal synchronization and semantic alignment, but recent methods mainly use separate VAE and DiT branches per modality.This dual-branch design separates tokenization and generation modules for audio and video and is described as heavy-weight.

3 Method

AVTok extends query-based 1D tokenization to a unified audio-video setting through a dual-stream transformer with shared encoder-decoder parameters and modality-specific components. Its training combines modality-appropriate reconstruction objectives with hierarchical optimization to address imbalance between visual and auditory learning.

  • Input representations: The video stream uses flattened spatiotemporal patch embeddings, while the audio stream uses normalized mel-spectrogram patches instead of raw 1D audio.The mel-spectrogram representation reduces computation complexity and provides frequency-time structure for audio tokenization.
  • Unified tokenizer: AVTok is the first extension of query-based 1D tokenization to unified audio-video modeling, jointly encoding and reconstructing both modalities.The unified setting concatenates audio and video patch embeddings for shared cross-modal processing.
  • Dual-stream architecture: The finalized tokenizer uses separate audio and video forward passes with shared encoder-decoder parameters, modality-specific holistic and patch queries, and separate normalization layers.This design preserves modality-specific information while enabling implicit audio-visual fusion through shared parameters.
  • Training objectives: AVTok trains video reconstruction with L1, LPIPS, GAN adversarial, and SVQ quantization losses, while audio uses Multi-Scale Mel-Spectrogram, two discriminator losses, and SVQ quantization.The audio objectives account for reconstruction through pretrained vocoders.
  • Hierarchical training: VFAL hierarchically trains AVTok because visual information dominates auditory information when both streams are trained from scratch, suppressing audio learning.Stage 1 begins with the more challenging video stream while discarding the audio stream.

4 Experiments

Experiments on TAVGBench and VGGSound evaluate AVTok for reconstruction and downstream audio-video generation. AVTok improves video reconstruction, remains competitive for audio, surpasses most selected generation baselines, and benefits from its dual-stream architecture and hierarchical training components.

  • Experimental Setup: Experiments use TAVGBench and VGGSound for reconstruction, while downstream generation uses only VGGSound because of time and resource constraints.All assessments use the VGGSound test set, with 16-frame 128 × 128 clips and 3.6fps video.
  • Baselines & Metrics: Because no open-source unified-tokenization baseline exists, comparisons use state-of-the-art unimodal video and audio tokenizers with PSNR and FVD metrics.Video baselines include OmniTokenizer, AdapTok, and LARP; audio baselines include WavTokenizer, UniCodec, and SpectralCodec.
  • Reconstruction Results: AVTok consistently outperforms the vanilla design and selected unimodal baselines in video reconstruction while maintaining competitive audio performance.These results support the feasibility of unified audio-video tokenization and suggest that cross-modal information can improve both modalities.
  • Downstream Generation: AR models using AVTok surpass the majority of selected baselines across audio-to-video, video-to-audio, and class-conditional joint audio-video generation tasks.Compared baselines include TempoTokens for A2V, MMAudio, VinTAGe, V-AURA, and SpecVQGAN for V2A, and JavisDiT and Ovi for cJAVG.
  • Ablation Study: Ablations show that the dual-stream architecture, hierarchical VFAL strategy, and representation-alignment objective improve reconstruction and downstream generation.The dual-stream design combines modality-specific queries with shared parameters for cross-modal interaction, while hierarchical training progressively optimizes each stream and improves semantic alignment.

5 Conclusion

AVTok is a unified audio-video tokenizer that jointly encodes audio-video pairs into compact one-dimensional latent representations using a unified codebook. Its shared dual-stream architecture and hierarchical VFAL training strategy support modality-specific reconstruction while implicitly fusing auditory and visual information.

  • 5 Conclusion: AVTok jointly encodes audio-video pairs into a single compact one-dimensional latent representation with a unified codebook.This establishes the paper’s central unified tokenization approach.
  • 5 Conclusion: A dual-stream transformer with shared encoder-decoder components and modal-specific holistic queries fuses auditory and visual information for efficient reconstruction.The modal-specific queries exploit modality-specific elements while the shared architecture enables implicit information fusion.
  • 5 Conclusion: Video-First-Audio-Later (VFAL) hierarchically trains AVTok to progressively develop reconstruction capability for each individual modality.VFAL addresses the training requirements of the unified tokenizer through staged modality development.

A Additional Experiment Details … AR Generative Models.

The experiments use VGGSound and TAVGBench for AVTok reconstruction, while downstream autoregressive generation models train only on VGGSound. Implementation details specify positional encodings, dual-stream transformer components, pretrained modules, and deterministic autoregressive tokenization.

  • Statistics.: VGGSound provides more than 210K clips across 310 classes, with approximately 200K usable audio-video pairs after corruption.Its train split contains 180K samples, and the remaining samples form the test split.
  • Composition.: 640K combined TAVGBench and VGGSound training samples train AVTok reconstruction, whereas downstream generative models use only 180K VGGSound samples.The VGGSound test set is used for all evaluations, with preprocessing following HiFi-GAN and LARP.
  • AVTok Tokenizer.: AVTok follows LARP by using fixed sin-cos positional encodings in both encoder and decoder stages.The encoder applies fixed 3D and 2D encodings to video and audio patches, while the decoder applies fixed 1D encodings to holistic tokens; learnable queries require none.
  • AVTok Tokenizer.: Each AVTok transformer layer combines multi-headed self-attention, layer normalization, multilayer perceptrons, and residual connections.The forward pass applies self-attention after LN1, adds the residual input, then applies the MLP after LN2 with another residual connection.
  • AVTok Tokenizer.: AVTok forms transformer inputs by concatenating learnable holistic or patch queries with input patches or holistic tokens.Separate normalization parameter sets for audio and video streams formulate the final dual-stream architecture.
  • AVTok Tokenizer.: HiFi-GAN, CAV-MAE Sync, and GPT-2 serve as the neural vocoder, audio-visual foundation model, and cross-modal autoregressive prior, respectively.Only the prior model and its small MLP projector are trained; the other modules remain frozen and are discarded during inference.
  • AR Generative Models.: AR generative models use Llama-like transformers with absolute learned positional encodings and a 0.1 dropout rate.Dropout is applied to token sequences, residual connections, and feedforward layers, while AVTok’s SVQ quantizer remains deterministic during training.

A.3 Training Details … Metrics.

AVTok training combines modality-specific reconstruction objectives with hierarchical VFAL optimization, while downstream autoregressive models and task-specific metrics support evaluation across reconstruction and generation tasks.

  • A.3 Training Details: Video reconstruction combines L1, LPIPS, and GAN losses, weighted 1.0, 1.0, and 0.3, respectively.The video GAN component uses a ViT-based discriminator.
  • A.3 Training Details: Audio reconstruction combines Multi-Scale Mel-Spectrogram, deep feature matching, and GAN losses, weighted 15.0, 2.0, and 1.0, respectively.Audio uses Multi-Scale Sub-Band CQT and Multi-Period discriminators for GAN and feature matching components.
  • A.3 Training Details: All AVTok stages use batch size 112, Adam with base lr = 0.0001, β1 = 0.9, β2 = 0.95, and a warm-up cosine schedule.Discriminators update once every five tokenizer iterations with a 70% lower learning rate and LeCam regularization.
  • A.3 Training Details: VFAL trains video reconstruction for 75 epochs, audio reconstruction for 35 epochs, and refinement for 10 epochs.The stages progressively target video modules, audio-specific parameters with shared components frozen, and decoder refinement.
  • Downstream Generation.: Downstream audio-to-video, video-to-audio, and class-conditional joint audio-video generation use autoregressive models trained on VGGSound for 75 epochs with batch size 128.Class-conditional joint generation uses CFG scale 1.25, while audio-to-video and video-to-audio omit CFG and all tasks omit top-k and top-p sampling.
  • Downstream Generation.: For audio-to-video and video-to-audio, conditioning inputs pass through the corresponding AVTok stream to produce holistic tokens before autoregressive prediction.The autoregressive models predict holistic tokens for the respective output modality conditioned on obtained holistic tokens.
  • Representative Baselines.: Because unified audio-video tokenization lacks an open-source direct baseline, reconstruction is compared with relevant unimodal 1D tokenizers, while downstream baselines are selected for similar relevance.The baseline selection applies separately to reconstruction and downstream generation tasks.
  • A.4 Evaluation Details: Evaluation uses PSNR/SI-SDR for accuracy, FVD/FAD for realism, LPIPS/MR-STFT for perceptual quality, and DeSync plus ImageBind Score for downstream synchronization and semantic alignment.For downstream generation, FVD/FAD are supplemented by DeSync and ImageBind Score.

B Additional Results … C Discussion

Additional experiments assess AVTok’s generation efficiency, scalability, external-model choices, and qualitative performance. These results support efficient downstream generation, identify tokenizer scaling and token-count trade-offs, and motivate the default external-model configuration.

  • Generation Latency.: The complete AVTok-plus-autoregressive generation pipeline is evaluated against baselines using TFLOPs and average per-sample latency for 100 samples with batch size 1.All methods are tested in the same environment, with baseline settings kept at their defaults where necessary.
  • Tokenizer Scalability.: Scaling from AVTok-S to AVTok-B lowers FVD by 6.18 and FAD by 2.85, while further scaling to the default model yields only minor improvements.AVTok-B and AVTok-S use the same latent-token count as the default and are trained under identical settings.
  • Tokenizer Scalability.: Halving the holistic-token count for one modality accelerates autoregressive generation but degrades reconstruction quality.The experiments keep the other modality’s token count unchanged to isolate each modality’s token-count effect.
  • External Models.: Replacing CAV-MAE Sync with CAV-MAE causes performance degradation despite similar model size.The comparison is part of the external-model ablation alongside vocoder alternatives.
  • External Models.: BigVGAN provides slight improvements over HiFi-GAN but has a significantly larger model size, leading to the default choice of CAV-MAE Sync and BigVGAN for efficiency and effectiveness.The external-model ablation covers both foundational audio-visual models and vocoders.
  • B.2 Visualization: Qualitative results show AVTok performing strongly in reconstruction, audio-to-video, video-to-audio, and class-conditional joint audio-video generation.The paper provides results in Fig. 7–10 and includes MP4 files for subjective inspection.
  • C Discussion: Across the additional experiments, AVTok is examined as both a tokenizer and a component of downstream autoregressive generation models.The reported evaluations span reconstruction and multiple audio-video generation tasks.

C.1 Potential Limitations

Although AVTok performs strongly in audio-video tokenization and downstream autoregressive generation, the authors acknowledge that limitations remain and motivate future exploration.

  • C.1 Potential Limitations: AVTok’s strong tokenization and downstream autoregressive-generation performance does not eliminate remaining limitations, which the authors identify as future research directions.The passage does not specify the individual limitations.

Data Scale and Complexity.

AVTok achieves efficacy and efficiency with relatively small training sets, but limited data scale and simple scenes constrain scalability and can produce artifacts in complex settings. Larger, more diverse, higher-quality audio-video datasets may improve robustness and generalizability.

  • Data Scale and Complexity.: AVTok and its autoregressive models are trained on approximately 640K and 180K data entries, respectively.These dataset sizes may constrain scalability compared with larger proprietary models and systems.
  • Data Scale and Complexity.: Simple scenes in the datasets can cause artifacts when generated samples depict particularly complex scenes.The limitation arises from the inherent simplicity of the included datasets.
  • Data Scale and Complexity.: Larger-scale training on more diverse and higher-quality audio-video datasets could enhance model robustness and generalizability.The passage presents expanded dataset scale and quality as a potential way to address these limitations.

Model Design and Training Resource. · Synchronization Modeling.

AVTok is constrained by fixed-resolution inputs and limited training resources, while its current architecture models audio-visual synchronization only implicitly. More explicit synchronization modeling is proposed to mitigate temporal misalignment and improve performance.

  • Model Design and Training Resource.: AVTok performs best with fixed-resolution audio and video because of positional encoding constraints.This limitation is shared with other transformer-based unimodal tokenizers.
  • Model Design and Training Resource.: Limited resources restricted training and evaluation to 16-frame 128×128×3 clips and roughly 4-second audios sampled at 22kHz.The passage frames these settings as short, low-resolution audio-video inputs.
  • Model Design and Training Resource.: With sufficient resources, scaling AVTok and autoregressive generative models could enable reconstructing longer or higher-resolution audio-video content.The supplied passage presents this as a contemplated direction rather than a demonstrated result.
  • Synchronization Modeling.: The current AVTok tokenizer and autoregressive generative-model setups exploit audio-visual synchronization only partially and implicitly.The limitation concerns the present architecture and training setup.
  • Synchronization Modeling.: Synchronization is encouraged through synchronized sounding-video inputs, shared cross-modal parameters, an AR prior, foundation models, and causal self-attention.These mechanisms apply across AVTok and the autoregressive generation pipeline as described in the passage.
  • Synchronization Modeling.: The additional qualitative results include an audio-to-video generation example.The supplied figure caption identifies the task but does not report a specific outcome.
  • Synchronization Modeling.: More explicit modeling of synchronization between modalities could mitigate temporal misalignment and improve final performance.This is presented as a proposed way to address the synchronization limitation.

End-to-end Training. … LLM Usage.

The paper discusses end-to-end AVTok training as a simpler but more sensitive and computationally costly alternative to VFAL, and states that its datasets, models, and LLM use raise no privacy or authorship concerns.

  • End-to-end Training.: End-to-end training could streamline AVTok optimization through a unified objective, avoiding complicated stage-wise tuning and cascading errors.The alternative may suffer from higher optimization sensitivity and computational cost.
  • End-to-end Training.: VFAL hierarchical training is effective but requires complicated stage-wise tuning and can produce cascading errors leading to suboptimal performance.
  • Ethics.: All datasets and models used in the work are publicly accessible online and contain no private or sensitive information.
  • Ethics.: Additional qualitative results are provided for the video-to-audio generation task.
  • LLM Usage.: Large Language Models were used solely to polish language and refine presentation, not to develop core concepts or research design.
  • LLM Usage.: Additional qualitative results are provided for the class-conditional joint generation task.
Loading 2606.30811v1…