Source-linked AI summary
Qwen-Audio-3.0-Gen-Preview Technical Report
Junyu Dai, Xiaoyue Duan, Xinyue Fan, Yihan Feng, Jingbei Li, Xiangang Li, Yunjia Li, Lejun Min, Yufei Shi, Xingchen Song, Yiran Wang, Cheng Wen, Menglin Wu, Bajian Xiang, Huaicheng Zhang, Han Zhao, Ruichen Zheng
TL;DR
Existing systems remain limited in directly organizing heterogeneous audio components, ambience, and multiple roles into long-form temporal scenes. Qwen-Audio-3.0-Gen-Preview uses a unified DiT-and-VAE generation path with structured textual conditions. Its clearest reported strengths span speaker similarity, cross-turn consistency, and temporal localization, while synthetic data retain source-context and acoustic-mismatch limitations.
Problem
Existing audio systems remain limited in organizing heterogeneous components, ambience, and multiple roles together in long-form temporal scenes.
Source: §1 · p. 1Method
A unified non-autoregressive DiT and shared VAE generate complete waveforms from structured textual conditions across standalone and mixed-scene audio.
Source: §1 · p. 2, §7 · p. 16Results
The model’s clearest strengths are speaker similarity on the public reference-conditioned benchmark, cross-turn consistency on multi-speaker benchmarks, and temporal localization on rich-timeline benchmarks.
Source: §6 · p. 11, §7 · p. 16Takeaways & Limitations
The results indicate the potential of unified generation for temporally controllable audio without task-specific branches.
Source: §7 · p. 16Synthetic scenes retain residual source-clip context and simulated-versus-real acoustic mismatch, and their downstream benefit must be established empirically.
Source: §4.3 · p. 7
Abstract
from arXiv · showhide
Existing single-domain and multi-task audio systems remain limited in directly organizing heterogeneous audio components, ambience, and multiple roles into long-form temporal scenes. We present Qwen-Audio-3.0-Gen-Preview, a unified non-autoregressive framework that uses a Diffusion Transformer (DiT) and a shared variational autoencoder (VAE) to generate the complete mixed waveform. Prompt enhancement converts free-form requests into structured temporal records that are rendered as textual conditions, while a two-stage data curriculum and semantic conditional views train the proposed model to use these conditions across standalone and mixed-scene audio. A shared continuous VAE compresses 48kHz stereo waveforms into 25Hz latent sequences and incorporates semantic supervision, providing one representation for heterogeneous audio. On the public reference-conditioned benchmark, speaker similarity is the proposed model's clearest strength across all three subsets. Across the multi-speaker and rich-timeline benchmarks, its clearest comparative strengths are cross-turn consistency in both languages and temporal localization, respectively. On AudioCaps, its advantages are concentrated in evaluations using large audio-language models and AudioBox. These results demonstrate the potential of unified generation for temporally structured audio without task-specific branches.
1 Introduction
Existing systems support multiple audio tasks, but often generate components separately rather than organizing them as interacting parts of one continuous scene. Qwen-Audio-3.0-Gen-Preview uses a unified generation path for heterogeneous audio and shows comparative strengths in speaker similarity, cross-turn consistency, and temporal localization.
1 Introduction
- Separate component-generation and manual editing make scene production time-consuming and complicate coherence and precise control.The process typically requires timeline placement, alignment, level adjustment, and mixing.Source: §1 · p. 1
- Existing unified models cover multiple audio tasks but often do not organize their components as connected parts of one scene.Complex scenes require control over event timing, overlap, relative levels, dialogue, ambience, and long-range background structure.Source: §1 · pp. 1–2
- Qwen-Audio-3.0-Gen-Preview uses one non-autoregressive DiT and shared VAE path for standalone audio, dialogue, and mixed scenes, with strengths in speaker similarity, cross-turn consistency, and temporal localization.It converts free-form requests into structured temporal conditions and uses a two-stage training curriculum.Source: §1 · p. 2
2 Related Work
Prior work has broadened audio generation across domains and tasks, and explored both coordinated pipelines and direct mixed-track generation. The remaining focus is organizing heterogeneous components into coherent scenes with character identity, dialogue, ambience, and long-range temporal relations.
2.1 From Domain-Specific Generation to Unified Audio Generation
- Domain-specific systems produce strong results across linguistic, environmental, and structured audio, while unified models share representations or parameters across multiple tasks.Task breadth alone does not ensure that components are organized together as one connected scene.Source: §2.1 · p. 2
2.2 Complex Audio Scene Generation
- Scene-generation approaches either coordinate specialized models through scripts and external mixing or generate a mixed track directly with one model.External coordination offers editable scripts but relies on mixing for consistency and increases pipeline complexity and inference cost; direct generation avoids track-by-track assembly but existing methods mainly target shorter clips or simpler combinations.Source: §2.2 · pp. 2–3
2.3 Temporal Control and Compositional Instruction Following
- Temporal-control work models event timing and compositional constraints, but complex scenes also require character identity, multi-turn dialogue, persistent ambience, and long-range relations.The paper’s focus is scene-level organization: modeling heterogeneous components as interacting parts and jointly rendering them into a complete mixed track.Source: §2.3 · p. 3
3 System Overview
The system uses a shared non-autoregressive path to generate standalone and mixed-scene audio. A DiT maps noise latents to continuous VAE latents under textual conditions, and the shared VAE decodes the complete waveform.
3 System Overview
- The two-stage training process first establishes acoustic generation and text–audio correspondence, then teaches composition across dialogue, ambience, localized events, and long-range structure.Replayed basic samples during rich-timeline fine-tuning mitigate loss of standalone competence.Source: §3 · p. 3
- Caption and text tokens condition a DiT that transforms noise latents into continuous VAE latents, which the shared VAE decodes into the complete waveform.The same interface, DiT, latent space, and VAE support heterogeneous standalone and mixed-scene audio.Source: §3 · p. 3, Figure 1 · p. 4
4 Data
The data pipeline moves from broad audio learning to structured scene composition, using compatible records for annotations, synthetic recipes, and prompt-enhanced requests. These records preserve source roles and temporal organization and are rendered as model-facing textual conditions.
4.1 Data Curriculum
- Pre-training establishes broad audio generation and text–audio correspondence; rich-timeline fine-tuning teaches how multiple sources coexist on a shared timeline.The curriculum covers linguistic, long-range structured, and sound-event audio, then emphasizes dialogue, ambience-rich mixtures, continuity, and localized events while replaying clean standalone samples.Source: §4.1 · p. 4, Table 1 · p. 5
4.2 Data Annotation Pipeline
- The annotation pipeline consolidates heterogeneous supervision into records separating global scene properties, source attributes, localized content, and primary audible content.It tracks roles across dialogue turns, preserves overlapping intervals, and checks role binding, alignment, event order, and temporal scope before admitting examples.Source: §4.2 · pp. 5–6
4.3 Synthetic Audio Data Construction
- Synthetic soundscapes represent independently addressable sources with explicit semantic roles, timing, levels, spatial and channel rendering, and controlled one-factor counterfactuals.Construction-time event boundaries need not match perceptually audible activity; residual source-clip context and simulated-versus-real acoustic mismatch remain limitations.Source: §4.3 · pp. 6–7
4.4 Prompt Enhancement and Condition Rendering
- Prompt enhancement maps free-form requests into structured scene, role, audible-content, spatial, and temporal fields, then renders compatible records as textual conditions.Validation checks preserved requirements, role-to-content binding, event order, and timeline constraints; rendering uses finite templates and rule-based omission of low-priority details under a token budget.Source: §4.4 · pp. 7–8
5 Conditional Training for Rich-Timeline Audio
Rich-timeline training uses interpretable semantic condition views and carefully structured dropout to teach the model conditional distinctions. A shared continuous VAE supplies the latent representation, while classifier-free guidance combines visible and unconditional conditions at inference.
5.1 Semantic Conditional Views
- Semantic views retain all conditions, dialogue and role information, persistent scene and event information, or no conditions, while dropout preserves interpretable event descriptions and localizations.The scene view removes dialogue and roles together; condition dropout does not alter the target waveform or true event order.Source: §5.1 · p. 8
5.2 Attribute Contrast and Role-Bundle Integrity
- Independently interpretable voice attributes are sometimes hidden for contrast, but role cards referenced by visible dialogue retain core anchors and remain bound to their turns.Optional personality or lengthy descriptive prose may be dropped; removing a role bundle also removes its linked turns.Source: §5.2 · pp. 8–9
5.3 Classifier-Free Guidance
- At inference, classifier-free guidance weights the difference between vector fields predicted from visible and empty conditions, without supplying information absent from the visible condition.The semantic views provide the visible and empty conditions used for guidance.Source: §5.3 · p. 9
5.4 Shared Continuous Audio VAE
- The shared VAE compresses 48 kHz stereo waveforms into 128-dimensional continuous latent sequences at 25 Hz, and uses posterior means for semantic supervision.The latent sample is used for waveform reconstruction, while the posterior mean is used for semantic supervision.Source: §5.4 · p. 9
- VAE training first optimizes acoustic reconstruction and then adds semantic supervision from a frozen language model, with a staged schedule designed to limit acoustic degradation.The objective combines reconstruction, variational, adversarial, feature-matching, and semantic terms; the language model is absent from downstream audio generation.Source: §5.4 · pp. 9–10
6 Evaluation
The evaluation covers reference-conditioned and text-conditioned generation, including in-house tests of multi-speaker consistency and temporal control. It also separately assesses VAE reconstruction and a controlled downstream representation probe.
6 Evaluation
- Qwen-Audio-3.0-Gen-Preview records the highest SIM on the EN, ZH, and Hard-ZH Seed-TTS-Eval subsets, but its WER/CER values are not the column minima.Speaker preservation is its clearest numerical strength in this test, not a uniform advantage in linguistic accuracy.Source: Table 2 · p. 10, §6 · p. 11
- Compared with Seed-Audio-1.01, Qwen-Audio-3.0-Gen-Preview records higher CONS in both languages, while the baseline leads on EN WER, ZH CER, and EN SIM.The consistent advantage is cross-turn speaker consistency, rather than a uniform lead in intelligibility or speaker fidelity.Source: §6 · p. 11, Table 3 · p. 11
- 0.8373 versus 0.8177 is the mean across three subset judges for Qwen-Audio-3.0-Gen-Preview and UniFlow-Audio; Qwen leads three LALM columns but none of the CLAP columns.The full-set and subset Qwen3 scores differ by 0.0018.Source: §6 · pp. 11–12, Table 4 · p. 12
- On AudioCaps, Qwen-Audio-3.0-Gen-Preview records the highest values across all four AudioBox dimensions.The reported dimensions are PQ, PC, CE, and CU.Source: §6 · p. 12, Table 5 · p. 12
- 43.73 and 43.12 are Qwen-Audio-3.0-Gen-Preview's mIoU scores under Gemini-3.1-Pro-Preview and Qwen3.5-Omni-Plus, versus 38.48 and 37.36 for Seed-Audio-1.0.Qwen has higher rates at both IoU thresholds, while Seed-Audio-1.0 has higher event recall; the authors report no uncertainty estimates.Source: §6 · pp. 12–13, Table 6 · p. 13
- With roughly an order of magnitude less data, Qwen-Audio-3.0-Gen-Preview remains numerically close across SongBench's seven components and leads in musicality, instrumental quality, and mixing.The comparison is uncontrolled, and the base-checkpoint evaluation uses a small sample without uncertainty estimates.Source: §6 · p. 13, Table 7 · p. 13
6.1 VAE Component Evaluation
- The acoustic VAE achieves the best Low-Rate Mel and STFT distances on Song Describer, MuChin, and AudioCaps, while semantic continuation improves ViSQOL on four datasets but modestly degrades spectral and stereo-coherence metrics.The acoustic VAE remains competitive on Seed-TTS EN and ZH, where HoliTok performs better on several metrics.Source: §6.1 · p. 14, Table 8 · p. 14, Table 9 · p. 15
- Under a fixed LibriSpeech generator, Acoustic VAE beats Mel and Stable Audio Open targets on WER, SIM, and UTMOS; Semantic-VAE and LoSATok beat Qwen VAE on WER and SIM.The probe holds generator architecture, training data, and optimization budget fixed, so these comparisons concern target representations rather than the complete proposed system.Source: §6.1 · pp. 14, 16, Table 11 · p. 15
7 Conclusion
Qwen-Audio-3.0-Gen-Preview unifies heterogeneous standalone and mixed-scene audio through one non-autoregressive generation path. Its clearest comparative strengths span speaker similarity, cross-turn consistency, temporal localization, and selected AudioCaps evaluations.
7 Conclusion
- Qwen-Audio-3.0-Gen-Preview uses one non-autoregressive path, with compatible structured records rendered as textual conditions for a DiT and shared VAE to produce complete mixed waveforms.The representation uses 48 kHz stereo waveforms compressed to 25 Hz latent sequences, with semantic views, role-bundle integrity, and classifier-free guidance providing complementary conditioning information.Source: §7 · p. 16
- Its clearest comparative strengths are speaker similarity on the reference-conditioned benchmark, cross-turn consistency on the multi-speaker benchmark, temporal localization on rich-timeline tasks, and LALM and AudioBox evaluations on AudioCaps.Reconstruction measurements and the LibriSpeech-PC-200 controlled probe also support the utility of the low-frame-rate VAE representation.Source: §7 · p. 16
- The results highlight the potential for temporally controllable audio generation without task-specific branches.Source: §7 · p. 16