PaperlayerView exact revision on arXiv

Source-linked AI summary

Salt++: Context-Aligned Post-Training for Few-Step Streaming Multimodal Generation

Xingtong Ge, Yutong Wang, Lunjie Zhu, Haitao Lin, Fangyu Lin, Yushi Huang, Xin Zhang, Yi Zhang, Yu Liu, Jun Zhang

arXiv:2609.36995v1cs.CVcs.AIeess.IV

TL;DR

Few-step streaming audio–video generation must combine causal modeling with step distillation, but existing approaches face challenges in contextual representation learning and matching generation and scoring contexts. Salt++ combines Causal Self-Flow with context-aligned AR DMD, improving visual and motion quality over OmniForcing at 480p and extending four-step generation to 1664 × 960.

  • Problem

    Streaming audio–video generation needs causal modeling and few-step sampling, while teacher forcing indirectly supervises predictive contextual representations and bidirectional scoring can mismatch causal generation contexts.

    Source: §1 · p. 1

  • Method

    Salt++ combines Causal Self-Flow, which aligns student and EMA-teacher features under different histories and the same noisy target, with AR DMD that shares causal masks and prefixes across generation and scoring.

    Source: §1 · p. 2

  • Results

    At 480p, Salt++ improves visual quality by 57% and motion quality by 45% over OmniForcing; at 1664 × 960, it outperforms bidirectional LTX-2 on six of seven metrics.

    Source: §1 · p. 2

  • Takeaways & Limitations

    With aligned conditional paths, distribution matching serves as the primary few-step objective, removing the separate consistency-distillation stage of prevailing recipes.

    Source: §5 · p. 10

Abstract

from arXiv · show

Few-step streaming audio--video generation requires both causal modeling and step distillation, yet standard training recipes face two context-related challenges. Teacher forcing pairs clean history with a noisy target, but supervises predictive contextual representations only indirectly through velocity prediction. Meanwhile, directly reusing bidirectional score models in causal Distribution Matching Distillation (DMD) creates a mismatch between generation and scoring contexts. We address these challenges with Salt++, a two-stage post-training framework comprising Causal Self-Flow (CSF) and context-aligned autoregressive DMD. CSF exploits contextual information asymmetry by varying the history while keeping the noisy target fixed: a noise-mixed-history student aligns its intermediate representations with those of a clean-history exponential-moving-average teacher. This self-supervised signal encourages the student to extract semantic information and improves cross-modal alignment. Context-aligned AR DMD shares the causal mask and prefix across generator sampling, fake-score training, and real-score evaluation to match generated and reference distributions under a block-conditional KL objective. With calibrated teacher guidance, it performs clean-prefix few-step distillation and then adapts to generated histories without switching objectives or requiring separate consistency distillation. At 480p, Salt++ improves visual and motion quality by 57% and 45% over OmniForcing on JavisBench under the same 4-step causal setting. A separate scale-wise post-training stage extends Salt++ to 4-step $1664\times960$ generation, outperforming bidirectional LTX-2 on six of seven reported metrics. Project page: https://xingtongge.github.io/Saltpp

1 INTRODUCTION

Salt++ addresses the context challenges of few-step streaming audio–video generation with Causal Self-Flow and context-aligned AR DMD. Its 4-step generator improves visual and motion quality over OmniForcing at 480p, with a scale-wise stage extending generation to 1664 × 960.

1 INTRODUCTION

  • Streaming generation requires causal block-wise modeling, while few-step distillation compresses sampling to a few evaluations; prevailing recipes pursue these goals through a multi-stage pipeline that changes objectives.Temporally bidirectional models generate a full clip before display, and multi-step ODE integration is slow and expensive.Source: §1 · pp. 1–2
  • The two challenges are indirectly supervised predictive contextual representations under teacher forcing and score–context mismatch when bidirectional models evaluate blocks generated under causal prefixes.The latter misaligns score estimates with the intended block-conditional objective.Source: §1 · p. 2
  • Causal Self-Flow aligns noise-mixed-history student features with clean-history EMA-teacher features under the same noisy target, while context-aligned AR DMD shares the causal mask and prefix across generation and scoring.With calibrated teacher guidance, AR DMD retains one objective from clean-prefix distillation through adaptation to generated histories.Source: §1 · p. 2
  • A separate scale-wise post-training stage distributes four generator evaluations across two spatial scales through a causal latent upsampler, reaching 1664 × 960 without additional calls.The two-stage core produces a 4-step causal audio–video generator.Source: §1 · p. 2
  • At 480p, Salt++ improves visual quality by 57% and motion quality by 45% over OmniForcing, and outperforms bidirectional LTX-2 on six of seven reported metrics at 1664 × 960.The high-resolution result uses scale-wise post-training.Source: §1 · p. 2

2 RELATED WORK

Salt++ builds on work in joint audio–video generation, representation alignment, few-step distillation, and autoregressive diffusion. It adapts EMA-feature alignment to causal audio–video history and distinguishes its approach from prior causal distillation recipes.

2 RELATED WORK

  • Unlike prior recipes that switch objectives and score causal generators with bidirectional models, Salt++ retains one AR DMD objective while shifting conditioning from clean context to generated rollouts.The framework also uses a separate scale-wise stage to reach 1664 × 960 with four generator calls.Source: Figure 1 · p. 3
  • CSF adapts Self-Flow’s cleaner EMA-feature alignment to causal audio–video history, aligning a noise-mixed-history student and clean-history teacher under the same noisy target.The paper builds on bidirectional LTX-2 and discusses REPA’s external visual-encoder alignment and Self-Flow’s EMA branch.Source: §2 · pp. 2–3
  • Compared with prior causal few-step methods, Salt++ isolates score-context effects, recalibrates teacher guidance for distribution matching, and obtains its AR teacher through representation alignment for joint audio–video generation.The related-work discussion contrasts it with methods that separate teacher-forced initialization from on-policy refinement and with concurrent causal DMD work on video-only distillation.Source: §2 · p. 3

3 METHOD

Salt++ addresses predictive-context learning and score–context mismatch with Causal Self-Flow and context-aligned AR DMD. A scale-wise post-training stage extends the four-step generator to high-resolution output.

3.1 PROBLEM SETUP

  • Causal generation needs predictive representations of past audio–video blocks, while block-conditional DMD requires generation and scoring to use aligned history content and temporal visibility.Standard flow matching supervises contextual representations indirectly through velocity prediction.Source: §3.1 · p. 4

3.2 CAUSAL SELF-FLOW

  • CSF pairs a noise-mixed-history student with a clean-history EMA teacher on the same noisy target, aligning intermediate representations alongside flow matching.The representation target comes from the clean-history teacher rather than matching final denoising predictions.Source: §3.2 · pp. 4–5

3.3 CONTEXT-ALIGNED AR DMD FOR FEW-STEP CAUSAL GENERATION

  • Context-aligned AR DMD shares the causal context across generation and both scores, then adapts from clean prefixes to generated histories without changing the distillation objective.Calibrated teacher guidance enables direct few-step distillation without a separate consistency-distillation stage.Source: §3.3 · pp. 5–6
  • AR–AR achieves the best overall performance among the tested score configurations by placing generation and both score models under the same causal context.The real and fake scores share the generator's block packing, causal mask, and prefix.Source: Figure 3 · p. 5, §3.3 · p. 6

3.4 SCALE-WISE HIGH-RESOLUTION POST-TRAINING

  • Scale-wise post-training extends the four-step streaming generator to 1664 × 960 output, using two low-resolution and two high-resolution evaluations per block.A causal latent upsampler spatially upsamples the video latent using the current block and a bounded history window.Source: §3.4 · pp. 6–7

4 EXPERIMENTS

Experiments evaluate Salt++ as a four-step causal audio–video generator, with AR–AR DMD leading the reported 480p comparisons and scale-wise post-training extending generation to 1664 × 960. Ablations examine causal representation learning, score-context alignment, guidance calibration, and adaptation to generated histories.

4.1 IMPLEMENTATION DETAILS

  • Evaluation uses shared official and LTX-2-enhanced JavisBench-mini prompts; both complete routes use 4-step causal inference with CFG = 1.At 480p, videos contain 121 frames at 832×480 and 24 FPS; high-resolution evaluation uses 1664 × 960 output.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS

  • AR–AR DMD leads on seven metrics among 4-step methods, improving visual and motion quality over OmniForcing by 57.1% and 44.5%, respectively.The TF-dCM route retains advantages in IB-AV, JavisScore, and DeSync, while OmniForcing leads in aesthetics.Source: §4.2 · p. 8
  • On 1,000 rewritten prompts, AR–AR DMD improves all seven reported metrics over OmniForcing, with VQ and MQ increasing by 59.0% and 64.6%.It leads TF-dCM on a majority of metrics, while TF-dCM retains higher audio–visual semantic alignment scores.Source: §4.2 · p. 8
  • At 1664 × 960, scale-wise post-training outperforms 40+3-step LTX-2 on six of seven metrics, with VQ/MQ at 2.730/0.957 versus 2.231/0.607.DeSync is the exception; the OmniForcing comparison is directly extrapolated to 960p, not a resolution-matched trained baseline.Source: §4.2 · p. 8

4.3 ABLATION STUDIES AND ANALYSIS

  • Before few-step distillation, CSF improves IB-AV, AVHScore, and JavisScore after initial optimization, with CLAP gains emerging later in training.Figure 5 compares CSF with clean teacher forcing over 19.2k iterations using 40-step inference on 1,000 JavisBench prompts; panels (a–c) cover audio–visual agreement and (d) audio–text agreement.Source: Figure 5 · p. 8, §4.3 · p. 9
  • AR–AR DMD reaches VQ/MQ of 3.147/1.351 and the best scores on five of seven metrics; against TF-dCM, it improves all seven reported metrics.The BI–AR configuration raises VQ/MQ from 0.837/0.140 in BI–BI to 1.131/0.157; Fig. 7 illustrates stronger scene structure and detail.Source: §4.3 · p. 9, Table 3 · p. 9, Figure 7 · p. 9
  • Changing teacher guidance from fixed 4.0 to independent sampling from U(1.0, 3.5) improves VQ by 91.5% on official prompts and 82.9% on rewritten prompts.MQ more than doubles in both settings; the adjustment combines guidance-range change and randomization.Source: §4.3 · p. 9
  • On-policy training follows clean-prefix distillation by adapting the generator to its own histories while retaining context-aligned DMD; qualitative inspection shows reduced overexposure.The complete-route results report performance after this continuation.Source: §4.3 · p. 9

5 CONCLUSION

Salt++ combines Causal Self-Flow and context-aligned AR DMD in a 4-step streaming audio–video generator, with scale-wise post-training extending generation to 1664 × 960.

5 CONCLUSION

  • At 480p, Salt++ improves visual quality by 57% and motion quality by 45% over OmniForcing.The resulting 4-step generator reaches these gains under the same budget.Source: §5 · p. 10
  • Salt++ uses Causal Self-Flow to strengthen contextual learning and context-aligned AR DMD to condition the generator and both score models on the same causal prefix.The objective remains the same as prefixes change from ground truth to generated rollouts.Source: §5 · p. 10
  • With aligned conditional paths, distribution matching serves as the primary few-step objective without a separate consistency-distillation stage.Source: §5 · p. 10
  • Scale-wise post-training reaches 1664 × 960 within the same budget.Source: §5 · p. 10

A DETAILED IMPLEMENTATION AND EVALUATION

Table 4 collects the symbols used in Sections 3.2–3.4 and the appendix.

A DETAILED IMPLEMENTATION AND EVALUATION

  • Table 4 collects the symbols used in Sections 3.2–3.4 and the appendix.Source: §A.1 · p. 14

A.2 EVALUATION PROTOCOL

The evaluation uses JavisBench-mini joint prompts and a shared prompt-rewrite track, with one generated sample per prompt under fixed seeds.

A.2 EVALUATION PROTOCOL

  • The official track evaluates joint text-to-audio–video generation on 1,000 JavisBench-mini prompts using the released prompts and evaluator.Source: §A.2 · p. 14
  • Each model generates one sample per prompt using seed 12,345 + i, with video and audio evaluated from H.264 MP4 and PCM WAV outputs.Source: §A.2 · p. 14
  • A fixed rewrite track applies the official LTX-2 prompt enhancer once per joint prompt with rewrite seed 42, sharing the resulting 1,000 longer prompts across models.Source: §A.2 · p. 14

A.3 METRIC DEFINITIONS

The appendix defines evaluation metrics and sampling setups, and cautions that DeSync can be misleading when video is nearly static.

A.3 METRIC DEFINITIONS

  • The official comparison reports visual, motion, and audio quality, text–video and audio–visual agreement, JavisScore, and DeSync, with lower DeSync better.Additional VBench consistency measurements concern evaluated clips, not extended rollouts.Source: §A.3 · pp. 14–15
  • BI–AR records the table’s best DeSync at 0.531, but only 1.131 VQ and 0.157 MQ, versus 3.147 VQ and 1.351 MQ for aligned AR–AR.DeSync is informative only among configurations producing comparable motion.Source: §A.3 · p. 15

A.4 MODELS AND SAMPLING

  • At 480p, videos use 121 frames at 24 FPS in five one-second blocks; the 4-step grid is [1000, 960, 889, 727, 0].Each block pairs three video latent frames with 25 audio latent frames, with the first block carrying each stream’s initial latent frame.Source: §A.4 · p. 16
  • At 960p, scale-wise generation uses two low-resolution and two high-resolution generator evaluations; LTX-2 uses a 40 + 3-step high-resolution path.OmniForcing at 960p is a resolution-extrapolation diagnostic, not a resolution-matched trained baseline.Source: §A.4 · p. 16

A.5 TRAINING CONFIGURATION

The configuration specifies separate CSF and context-aligned AR DMD training routes, followed by generated-history continuation and a scale-wise stage for high-resolution generation.

A.5 TRAINING CONFIGURATION

  • CSF aligns student layer 14 with EMA-teacher layer 34 while alternating CSF and clean teacher-forcing updates with equal probability.The target noise is shifted toward higher noise levels, and the history noise is drawn from U(0, 0.5).Source: §A.5 · p. 16
  • AR DMD initializes the generator and both scores from the same CSF checkpoint, sharing entry noise levels across blocks and modalities.Video and audio teacher guidance are sampled independently from U(1.0, 3.5), unless ablated.Source: §A.5 · p. 16
  • The TF-dCM comparison uses dense adjacent-pair consistency distillation with fixed guidance of 4.0 and no fake score or EMA target.Its loss is scaled by 100; the authors report unscaled training was substantially worse and attribute the difference to numerical behavior at small loss magnitudes.Source: §A.5 · pp. 16–17
  • On-policy continuation trains on generated histories, with AR–AR retaining its context-aligned objective and TF-dCM switching to bidirectional score models.AR–AR continues for 1.2k iterations with guidance sampled from U(1.0, 3.5); TF-dCM runs for 2.4k iterations with guidance fixed at 3.0.Source: §A.5 · p. 17

A.6 HIGH-RESOLUTION CONFIGURATION

  • Scale-wise post-training uses four generator calls across two spatial scales to produce 1664 × 960 videos, while separately warming up the generator and initializing its causal upsampler.The upsampler is initialized by matching windowed causal outputs to the released upscaler on online low-resolution rollouts, then updated during scale-wise training.Source: §A.6 · pp. 17–18

B.1 480P QUALITATIVE COMPARISONS

The appendix adds matched qualitative 480p examples and training-guidance comparisons, with figures organized around shared prompts, times, and method order.

B.1 480P QUALITATIVE COMPARISONS

  • The appendix comparisons show three frames per case with shared prompts and displayed times; LTX-2 Base uses 40 steps, while OmniForcing and both Salt++ routes use four.Frames are not retouched or color corrected.Source: §B.1 · p. 18
  • Figure 9 presents four additional 480p comparisons featuring a race car, a cat at a window, sunset rocks, and a garden cottage; method ordering follows Fig. 4.Source: Figure 9 · p. 19
  • The lower randomized guidance setting preserves more tonal and texture detail than fixed high guidance in the podcast and arcade examples.Figure 11 adds microphone-host and studio-speaker cases comparing fixed 4.0 with independently sampled U(1.0, 3.5) guidance.Source: Figure 10 · p. 20, Figure 11 · p. 20

B.2 ADDITIONAL GUIDANCE EXAMPLES AND SAMPLE PROVENANCE

These appendix materials document additional guidance and high-resolution visualizations, their display and checkpoint provenance, and the prompts used for selected comparisons.

B.2 ADDITIONAL GUIDANCE EXAMPLES AND SAMPLE PROVENANCE

  • The guidance examples come from a separate internal visualization set rather than the 1,000-prompt evaluation, and matched prompts and seeds do not imply pixel-aligned compositions.Figure 11 adds two speaking cases under the same settings as Figure 10.Source: §B.2 · p. 20, Figure 12 · p. 21
  • Figure 12 presents the Wellness host comparison with full frames above corresponding facial and hair detail windows, following Figure 6's four-method order.The examples use matched prompts, sample-index seeds, and times, with equal-sized detail windows.Source: Figure 12 · p. 21, §B.3 · p. 21

B.3 HIGH-RESOLUTION QUALITATIVE COMPARISONS

  • The high-resolution examples use a Stage 3 SALT BI–BI parent and Stage 4a scale-wise output, separate from the 1,000-prompt aggregate evaluation.Selected stills do not establish full-video temporal or audio quality; the 480p parent is bilinearly upsampled for display, whereas the native 960p model uses two low-resolution and two high-resolution calls.Source: §B.3 · p. 21

B.4 PROMPTS FOR MAIN-RESULT VISUALIZATIONS

  • The main-result visualization prompts are original generation inputs shared by all methods within each case, with sampling settings recorded in Appendices B.2 and B.3.The supplied cases include a JavisBench white-cliff beach prompt and prompts describing a diner scene and a fitness studio scene.Source: §B.4 · pp. 21–22
Loading 2609.36995v1…