PaperlayerView exact revision on arXiv

Source-linked AI summary

JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation

Kai Liu, Jungang Li, Yuchong Sun, Shengqiong Wu, Jianzhang Gao, Daoan Zhang, Wei Zhang, Sheng Jin, Sicheng Yu, Geng Zhan, Jiayi Ji, Fan Zhou, Liang Zheng, Shuicheng Yan, Hao Fei, Tat-Seng Chua

arXiv:2512.22905v2cs.CV

TL;DR

JavisGPT addresses the limited integration of audio and video in multimodal comprehension and generation by unifying them in one model with synchronized representations and generation conditioning. The authors report strong performance across JAV benchmarks and better joint-generation results than existing methods.

  • Problem

    Existing efforts often focus on image-text frameworks or treat audio and video as separate modalities rather than jointly modeling JAV tasks.

    Source: §1 · p. 2

  • Method

    JavisGPT combines an encoder-LLM-decoder, SyncFusion, hierarchical conditioning for a pretrained JAV-DiT generator, and a three-stage training pipeline.

    Source: §1 · p. 2

  • Results

    JavisGPT reports strong performance across JAV benchmark scenarios and significantly outperforms NExT-GPT and UnifiedIO-2 in AV quality, semantic consistency, and AV synchrony.

    Source: §1 · p. 3, §6.1 · p. 8

  • Takeaways & Limitations

    The authors describe JavisGPT as a strong foundation for future research in synchronized media generation.

    Source: §7 · p. 10

    Future work identifies higher-fidelity speech input/output, fine-grained controllable generation and editing, and complex instruction-based generation as directions needing further development.

    Source: §7 · p. 10

Abstract

from arXiv · show

This paper presents JavisGPT, the first unified multimodal large language model (MLLM) for joint audio-video (JAV) comprehension and generation. JavisGPT has a concise encoder-LLM-decoder architecture, which has a SyncFusion module for spatio-temporal audio-video fusion and synchrony-aware learnable queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation from existing vision-language models. For instruction tuning, we construct JavisInst-Omni, a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that cover diverse and multi-level comprehension and generation scenarios. On JAV comprehension and generation benchmarks, our experiments show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings.

1 Introduction

JavisGPT is presented as a unified model for joint audio-video comprehension and generation, combining synchronized multimodal processing with a three-stage training pipeline and a broad instruction dataset. The paper reports strong performance across JAV benchmark scenarios.

1 Introduction

  • JavisGPT uses an encoder-LLM-decoder architecture with SyncFusion for synchronized audio-video features and JavisCond tokens that condition a pretrained JAV-DiT generator.Hierarchical JavisQueries provide spatiotemporal priors for generation.Source: §1 · p. 2
  • A three-stage pipeline progressively adds audio input and generator alignment, strengthens synchronized audio-video tasks, then tunes multimodal instruction following and reasoning.The stages are MM-PreTrain, AV-FineTune, and MM-InstTune.Source: §1 · p. 2
  • JavisInst-Omni contains 200K interleaved audio-video-text dialogue trajectories, GPT-4o annotated and manually verified, spanning QA, captioning, generation, and multimodal reasoning.The dataset covers single- and multi-turn interactions.Source: §1 · pp. 2–3
  • JavisGPT performs impressively across JAV benchmark scenarios and significantly outperforms existing methods in audio-video joint generation.The authors report high-quality, diverse sounding videos.Source: §1 · p. 3
  • The paper presents JavisGPT as the first unified MLLM for sounding-video comprehension and generation.Its contributions include an audio-video synchronization mechanism, multi-stage tuning, and an instruction dataset.Source: §1 · pp. 2–3

2 Related Work

Prior multimodal systems often integrate audio and video as separate channels or focus on one modality, leaving limited treatment of their fine-grained interactions. Existing generation pipelines can also produce desynchronized outputs, motivating joint audio-video modeling.

2 Related Work

  • Many MLLMs process audio and video as independent inputs using concatenation or simple feature padding, overlooking fine-grained spatiotemporal interactions needed for synchronized tasks.Earlier multimodal work often combined text with static visual inputs, while later systems added dynamic vision and audio.Source: §2 · p. 3
  • Audio-video generation systems such as HuggingGPT, NExT-GPT, and UnifiedIO-2 execute separate generators sequentially, risking error propagation and desynchronized outputs.The passage attributes these issues to a lack of joint modeling.Source: §2 · p. 3
  • Prior unified systems often lack cross-modal alignment: some use separate audio and video encoders with little alignment, while others use separate decoders.Earlier talking-head and speech-driven dialogue systems were also limited by simple architectures or absent text input/output.Source: §2 · p. 3

3 Architecture

JavisGPT combines modality-specific encoders and an LLM with SyncFusion for synchronized audio-video comprehension, then bridges LLM states to a pretrained DiT generator. Semantic and spatiotemporal conditions support synchronized sounding-video generation.

3 Architecture

  • JavisGPT uses Qwen2.5-VL-7B-Instruct with SyncFusion and JavisQueries, integrating a pretrained JAV generator for joint comprehension and generation.Its encoder-LLM-decoder design is described as extending to audio perception and synchronized generation.Source: §3 · p. 3

3.1 Synchrony-aware Audio-Video Comprehension

  • SyncFusion aligns audio features to video-frame timing, injects audio cues into frame-wise visual features through shared cross-attention, and produces localized SyncAV event representations.The SyncAV sequence matches the extracted video representation length; Figure 3 illustrates the fusion mechanism.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 Spatio-Temporally Synchronized Audio-Video Generation

  • Learnable queries and an MLP project LLM hidden states into the DiT condition space; alignment to the frozen T5-XXL text encoder avoids fine-tuning the generator.The alignment design is described as simplifying training and reducing computational cost while retaining effective conditioning.Source: §3.2 · p. 4
  • Additional spatiotemporal-prior queries encode where and when events occur, and their alignment loss is combined with the denoising objective to strengthen generated audio-video synchrony.The semantic condition describes the overall event, while the spatiotemporal prior captures finer synchrony; Figure 4 depicts the generation design.Source: §3.2 · pp. 4–5, Figure 4 · p. 5

4 Training and Datasets

The training and dataset design progressively adapts a vision-language backbone for audio-video comprehension, synchronized generation, and interleaved instruction following. It combines staged pretraining and fine-tuning with JavisInst-Omni and joint tuning of newly introduced modules.

4 Training and Datasets

  • The three-stage pipeline adapts the visual-language backbone for joint comprehension and sounding-video generation.Table 1 names the stages multimodal pretraining, audio-video fine-tuning, and multimodal instruction-tuning.Source: §4 · p. 5, Table 1 · p. 5

4.1 Pre-training for Basic Video-Audio Comprehension and Generation

  • Pretraining learns an audio projection on about 600K audio-text pairs and aligns LLM conditions to frozen T5-XXL conditions using 1.5M sounding-video captions.The alignment phase is intended to provide semantically meaningful initialization for later fine-grained synchronization training.Source: §4.1 · pp. 5–6

4.2 Fine-tuning for Synchronized Audio-Video Comprehension and Generation

  • Fine-tuning trains SyncFusion on about 360K audio-video-text triplets for comprehension, then tunes semantic and spatiotemporal-prior queries and projectors for generation.The generation stage reuses the collected triplets and minimizes the joint loss defined in Equation (1).Source: §4.2 · p. 6

4.3 Instruction-tuning for Interleaved Comprehension and Generation

  • JavisInst-Omni combines sourced modality-specific and joint audio-video data with dedicated synchrony-aware comprehension and generation sets.Instruction tuning jointly updates the new projectors, queries, and LoRA adapter on its cross-modal instances.Source: §4.3 · pp. 6–7, Figure 5 · p. 6
  • Table 2 reports competitive performance across video, audio, and synchronized audio-video comprehension benchmarks, alongside adaptation-sample counts.The listed test sets include ActivityNet, Perception, MVbench, ClothoAQA, TUT2017, AVQA, MU-AVQA, and AVSD.Source: Table 2 · p. 7

5 Contribution Statement

The project combines established network components into a concise system for joint sounding-video understanding and generation, with a three-stage pipeline and a rich multimodal instruction dataset. The authors present these as a competitive baseline for future data-centric and model-centric work.

5 Contribution Statement

  • JavisGPT connects known network components into a concise, state-of-the-art system, a plan reached through substantial trials and errors.The authors describe the project as an open-source platform for joint understanding and generation of sounding video.Source: §5 · p. 7
  • The three-stage training pipeline has proven useful for gradually improving model performance.JavisInst-Omni mixes single- and multi-modality data, with most examples single-turn and the rest multi-turn.Source: §5 · p. 7
  • The architecture, training methods, and dataset are presented as a competitive baseline for future data-centric and model-centric studies.Source: §5 · p. 7

6 Experiments

JavisGPT performs strongly on audio-video comprehension and generation benchmarks, including interleaved tasks, and the analyses examine its synchronization module, training stages, and joint training. The reported results also include qualitative cases of instruction-followed, synchronized generation.

6 Experiments

  • The experiments use the three-stage training pipeline, with one epoch per stage, and zero-shot evaluation under official downstream-task protocols.Source: §6 · p. 7

6.1 Comparison with the State of the Art

  • JavisGPT tops AVQA and MU-AVQA and outperforms NExT-GPT and UnifiedIO-2 in audio-video generation quality, semantic consistency, and synchrony.It retains Qwen2.5-VL's Perception performance (70.2 vs. 70.5) and slightly exceeds Qwen2.5-Omni on MVBench and TUT2017.Source: §6.1 · pp. 7–8, Table 3 · p. 8
  • On 100 multi-turn interleaved-task dialogues, human evaluation finds JavisGPT consistently ahead of the two competing joint-capability methods by a large margin.NExT-GPT often refuses or generates noisy outputs, while UnifiedIO-2 struggles to connect user history with current instructions.Source: §6.1 · p. 8, Figure 6 · p. 8

6.2 Further Analysis

  • SyncFusion achieves strong audio-video understanding while reducing encoded-token count and inference latency; interleaving adds over 2× slowdown, while Q-former performance is lower and optimization unstable.Simple concatenation also outperforms the interleaving strategies in the reported comparison.Source: §6.2 · p. 8, Table 4 · p. 9
  • Combining multimodal pretraining with audio-video fine-tuning yields the best generation results; pretraining alone scores 0.069 JavisScore, while fine-tuning alone is unstable and requires a lower learning rate.The fine-tuning-only learning rate is reduced from 1e-4 to 1e-5, compromising final performance.Source: §6.2 · p. 9, Table 5 · p. 9
  • Joint training outperforms separate training on both understanding and generation, with a larger improvement on generation.Source: §6.2 · p. 9, Figure 7 · p. 9
  • As generation training samples increase, adding comprehension data steadily improves AV-Quality and Text-Consistency, while AV synchrony improves only marginally (+0.04).The authors suggest the synchrony result may point to a need for a more unified mechanism bridging comprehension and generation.Source: §6.2 · p. 9, Figure 8 · p. 9
  • In a two-round gunfire dialogue, JavisGPT grounds the sound in a video-game scene and generates coherent synchronized audio-visual content, unlike the compared models' errors or failure.Source: §6.2 · p. 9, Figure 9 · p. 10

7 Conclusion and Discussion

The paper presents JavisGPT as a unified system for synchronized audio-video comprehension and generation, combining a SyncFusion architecture, hierarchical generator conditioning, and three-stage training. It identifies extensions for higher fidelity and points to end-to-end autoregressive multimodal modeling as another promising framework direction.

7 Conclusion and Discussion

  • JavisGPT combines an encoder–LLM–decoder with SyncFusion and hierarchical condition embedding for a pretrained JavisDiT generator.The paper reports state-of-the-art performance on unimodal and synchronized audio-video tasks, with mutual enhancement between understanding and generation.Source: §7 · p. 10
  • The authors identify speech input/output, fine-grained controllable generation and editing, and complex instruction-based generation as future extensions for higher fidelity.The proposed directions involve replacing BEATs, adding JAV-DiT conditioning signals, and strengthening LLM reasoning and dense contextual conditioning.Source: §7 · p. 10
  • The discussion contrasts the encoder–LLM–decoder paradigm with end-to-end autoregressive multimodal token modeling, whose discrete and continuous representations show synergy.Extending the latter from text–image to text–audio–video modeling is described as having substantial potential for further capability gains.Source: §7 · p. 10

A.1 Limitation and Future Work

The paper identifies architectural, scalability, and post-training limitations, discusses societal risks, and describes its model configuration and staged training. Its current architecture uses separate comprehension and generation objectives and asymmetric input-output pathways.

A.1 Limitation and Future Work

  • The shared LLM receives inconsistent optimization signals from next-token prediction for comprehension and diffusion loss for generation, while generation queries do not feed back into comprehension.The paper identifies these as two architectural inconsistencies and suggests exploring unified audio-video representations.Source: §A.1 · p. 17
  • The 7B-backbone implementation and limited publicly available training data leave the approach’s scalability unexplored.Scaling to 70B or larger models and trillions of multimodal tokens is proposed as future work, not evaluated here.Source: §A.1 · p. 17
  • Post-training provides basic instruction-following and simple reasoning, while reinforcement learning is proposed to improve complex reasoning and generated-content quality, textual consistency, and synchrony.The paper leaves reinforcement learning for future exploration.Source: §A.1 · p. 17

A.2 Societal Impact

  • Synchronized audio-video generation raises risks of misinformation, impersonation, bias, and privacy violations, for which the paper recommends controlled access, watermarking, dataset audits, and detection tools.It calls for continued evaluation of broader impacts as the field progresses.Source: §A.2 · p. 17

B.1 Model Architecture

  • JavisGPT combines a Qwen2.5-based LLM with visual and audio encoders, SyncFusion for synchronized audio-video features, and a JavisDiT generator.The LLM backbone is Qwen2.5; visual features come from Qwen2.5-VL and audio features from BEATs.Source: §B.1 · p. 18

B.2 Training Configuration

  • Training progresses from audio understanding and caption-aligned generation, through synchronized audio-video fine-tuning, to multimodal instruction tuning for comprehension and generation.The instruction-tuning stage includes synchrony-aware QA, other modality-understanding data, generation instructions, and samples to retain generation capability.Source: §B.2 · pp. 18–19, Table A1 · p. 19

B.3 Evaluation Setup

The evaluation setup benchmarks comprehension across visual, audio, and synchronized audio-video tasks, and generation on text-to-audio-video samples. It uses quality, text consistency, audio-video semantic consistency, and spatiotemporal synchrony measures.

B.3 Evaluation Setup

  • Comprehension is evaluated zero-shot across visual, audio, and synchronized audio-video benchmarks using average accuracy, with greedy decoding and standardized video-frame and audio sampling.Open-ended responses are evaluated with GPT-3.5; multiple-choice accuracy uses lexical matching.Source: §B.3 · p. 19
  • Generation evaluation uses 1,000 JavisBench-mini text-to-audio-video samples, with 4-second, 240P, 24-fps videos and 16-kHz audio.The protocol follows JavisDiT.Source: §B.3 · p. 19
  • Generation metrics assess audio and video quality, text consistency, audio-video semantic consistency, and spatiotemporal synchrony.FVD, KVD, and FAD measure distribution distances; text consistency uses ImageBind, CLIP, and CLAP similarities; semantic consistency uses ImageBind audio-video similarity and AVHScore.Source: §B.3 · pp. 19–20
  • JavisScore averages ImageBind cross-modal similarity over sliding temporal segments to assess spatiotemporal synchrony; higher AVHScore indicates stronger audio-video semantic consistency.The paper omits CAVP-Score because its 0.798–0.801 range cannot capture semantic-consistency differences.Source: §B.3 · p. 20

C JavisInst-Omni Dataset Details

JavisInst-Und and JavisInst-Gen address gaps in sounding-video instruction data through synchrony-aware comprehension and synchronized audio-video generation examples. Together, their task designs span local-to-global understanding and varied generation scenarios, including multi-turn interaction.

C.1 Motivation

  • The dataset motivation is limited scale, capability coverage, and task diversity in existing sounding-video instruction data, particularly for synchrony-aware comprehension and synchronized generation.JavisInst-Und targets open-ended QA; JavisInst-Gen supports generation scenarios and multi-turn interaction.Source: §C.1 · pp. 20–21

C.2 JavisInst-Und

  • JavisInst-Und contains 110K QA samples and emphasizes audio-video synchrony, addressing limits in prior datasets’ scale, diversity, and support for event-level synchrony or multi-turn dialogue.Its taxonomy spans entity, relation, and global comprehension, including alignment, temporal relationships, emotion, and atmosphere.Source: §C.1 · p. 21, §C.2 · p. 21
  • JavisInst-Und organizes comprehension into entity, relation, and global levels, covering fine-grained event properties, relationships among events, and overall themes or atmosphere.The ten dimensions include existence, alignment, grounding, counting, spatial, temporal, and causal relationships, theme, emotion, and atmosphere.Source: §C.2 · p. 21

C.3 JavisInst-Gen

  • JavisInst-Gen includes text-conditioned instruction generation, generation conditioned on other modalities, and multi-turn tasks such as proactive, rethink, and understanding-to-generation interactions.Instruction-based instances combine GPT-4o prompt templates with TAVGBench captions; AV-edit examples use InstV2V pairs and FoleyCrafter-generated audio.Source: §C.3 · pp. 21–24
  • JavisInst-Gen contains 90K samples for synchronized audio-video generation, filling a gap in instruction data for generation tasks.Its task types include instruction-based generation, conditional generation, and multi-turn conversation.Source: §C.3 · pp. 21, 22, 24, §C.1 · p. 21

D Architecture of Joint Comprehension and Generation of Sounding Videos

The paper favors an LLM+DiT design for joint comprehension and generation, contrasting it with unified autoregressive-diffusion models and autoregressive encoder-decoder approaches. Within LLM+DiT, JavisGPT pairs learnable queries with a text encoder for instruction-guided generation.

D.1 Design Choices of Generative MLLM

  • MonoLLMs combine textual autoregression and multimodal diffusion in one LLM, but conflicting objectives and heterogeneous outputs can cause optimization difficulty and unstable generation.The paper contrasts this with JavisGPT’s modular LLM+DiT design.Source: §D.1 · p. 25, Figure A4 · p. 26
  • LLM+AR+Decoder designs add encoder-decoder training complexity and inference cost from sequentially predicting many multimodal tokens; the paper sees potential for future unified audio-video representations.At 512 tokens per frame, one minute of 24-fps video requires 737,280 predicted tokens, even without audio.Source: §D.1 · p. 26, Figure A4 · p. 26
  • The paper identifies LLM+DiT as the best current JAVG option, citing experimental verification and its balance of efficiency and generation quality.It separates LLM-based semantic understanding from diffusion-transformer decoding and uses learnable queries for flexible, temporally coherent generation.Source: §D.1 · pp. 25–26, Figure A4 · p. 26

D.2 Design Choices of LLM-DiT Combination

  • JavisGPT combines learnable query embeddings with a pretrained text encoder, which the paper associates with stable training and high-quality, diverse, instruction-aligned sounding-video generation.The design is contrasted with NExT-GPT’s fixed-token interface and MetaQuery’s removal of the text encoder.Source: Figure A5 · p. 26, §D.2 · p. 27

E.1 Details for Human Evaluation on Interleaved Conversation

The interleaved-conversation evaluation compares JavisGPT with two baselines, while follow-up analyses examine the LLM backbone and ST-Prior queries. Additional qualitative cases illustrate joint understanding and generation.

E.1 Details for Human Evaluation on Interleaved Conversation

  • Three annotators rated 100 dialogues—25 each across four scenarios—on a 0–5 scale for five dimensions, including instruction following, generation quality, and proactive thinking.The scenarios were Gen2Und, Und2Gen, Proactive, and Rethink.Source: §E.1 · p. 27
  • JavisGPT significantly outperforms both baselines in question answering and shows stronger contextual understanding and responsiveness.The authors associate its question-answering performance with JavisInst-Omni's scale and diversity.Source: §E.1 · pp. 27–28
  • NExT-GPT succeeded in only 33 out of 100 cases; both baselines had low generation quality, with fixed-token limits or keyframe stitching cited as constraints.The evaluation also reports weak context reasoning and proactive generation for both baselines, which the authors relate to a lack of training on preference reasoning and proactive dialogue.Source: §E.1 · p. 27

E.2 Investigation on Backbone LLM

  • Qwen2.5-VL consistently improves on Qwen2-VL across unimodal understanding, audio-visual comprehension, and generation, with the largest gain in AV-Gen.The comparison uses the same training data and procedure; Figure A6 presents the backbone investigation.Source: Figure A6 · p. 28, §E.2 · p. 28

E.3 Ablation on the ST-Prior Query

  • Removing ST-Prior queries minimally affects overall generation quality and consistency but slightly reduces audio-visual synchrony.Table A5 reports the ablation; the authors describe the queries as serving temporal and spatial alignment.Source: Table A5 · p. 28, §E.3 · p. 28

F Qualitative Case Studies of JavisGPT

Related papers

Loading 2512.22905v2…