Spend Where It Matters: Particle Search with MH Refinement for Budget-Constrained Audio-Visual Alignment

• Joint audio-visual models still struggle to align generated outputs with text prompts and to align audio with visual content.Source: §1 · p. 1
• Inference-time alignment is training-free and modular, allowing alignment objectives to change by swapping reward models without retraining the generator.Source: §1 · p. 1
• Finite particle populations can cause high estimation variance, while MH requires substantial compute for global exploration because of slow mixing.Source: §1 · p. 2
• SWIM uses online variance estimates to allocate particles across denoising checkpoints, then refines the highest-reward particle with local noise-and-denoise MH proposals.The two stages target coarse alignment followed by fine-grained alignment under a fixed compute budget.Source: §1 · p. 2
• At closely matched compute, SWIM’s mean relative gains across seven metrics are 30.7% over BoN and 14.1% over EvoSearch with JavisDiT++, and 9.2% over BoN and 17.0% over EvoSearch with LTX-2.Source: §1 · p. 2
• On SD1.5, SWIM improves AestheticScore by 2.2% and ImageReward by 6.0% over uniform allocation.Source: §1 · p. 2

• SyncDPO and SpongeBob improve audio-visual alignment by modifying the generative model through preference optimization or synchronization-aware training.SWIM instead operates at inference time on pretrained backbones without additional training.Source: §2 · p. 2
• ITS-AVGen uses multiple verifiers and Adaptive Reward Weighting to calibrate heterogeneous reward signals at test time; SWIM first establishes coarse alignment, then applies finer refinement.Source: §2 · p. 2
• SWIM builds on particle-based search by adaptively allocating particles across denoising checkpoints to reduce estimation variance, then uses MH to refine the candidate.Its allocation strategy is compatible with different resampling rules.Source: §2 · p. 3

• Diffusion models use a forward noising process and train a noise predictor, while flow matching learns a velocity field whose backward integration maps noise to data.The sampler is associated with the underlying model, and the denoised estimate can be decoded for reward evaluation.Source: §3 · p. 3
• SMC uses weighted particles and resampling to approximate a reward-tilted target, but choosing the terminal best particle is a search decision rather than a draw from that target.Source: §3 · p. 3
• The effective sample size measures particle-weight concentration, and its associated empirical squared coefficient of variation is used to guide compute across checkpoints.Source: §3 · p. 3
• The allocation method uses a tractable variance surrogate rather than the exact variance of the final output.Source: §3 · p. 3
• With an exact reverse process, the reward-only MH acceptance rule applies to the reward-tilted target; with a learned reverse model, that rule is approximate.A finite MH run also depends on its initialization and proposal scale.Source: §3 · p. 3

4 METHOD

• SWIM divides total compute between particle search and MH proposals, fixing the stage budget split while adapting computation within the particle stage.Source: §4 · p. 3

4.1 ADAPTIVE PARTICLE FUNNEL

• Online allocation uses observed weight dispersion to forecast future dispersion and adjust the next particle population against cumulative budget error.A pacing floor preserves a fraction of uniform allocation, while an affordability cap reserves compute for remaining segments.Source: §4.1 · p. 4
• The adaptive particle funnel formulates checkpoint population selection as minimizing a variance surrogate under a particle budget, with initial population K0 treated as a hyperparameter.The formulation defines particle counts at checkpoints and the denoising-segment costs used in the allocation.Source: §4.1 · p. 4

4.2 METROPOLIS-HASTINGS REFINEMENT

• The funnel returns its highest-reward terminal particle to initialize MH, which re-noises its latent and re-denoises both modalities along a fresh stochastic path.The proposal scale controls proposal cost, and the number of MH steps controls refinement length.Source: §4.2 · p. 5
• MH scores completed proposals with a fixed weighted reward and accepts them using a reward-difference rule that is exact with an exact reverse model but approximate with a learned one.The short chain makes its starting point consequential, motivating particle search for global exploration before local refinement.Source: §4.2 · p. 5

5.1 SETUP

• The study evaluates JavisDiT++ and LTX-2 on JAVISBENCH-MINI and Stable Diffusion v1.5 on DrawBench and Simple Animals, comparing matched-budget baselines with seven AV and two image metrics.The AV budgets are 360 and 560 NFE, each allocating 160 NFE to MH.Source: §5.1 · pp. 5–6

5.2 MAIN RESULTS

• At the lower, closely matched budget, SWIM’s mean relative gains across seven metrics are 30.7% over BoN and 14.1% over EvoSearch with JavisDiT++, and 9.2% over BoN and 17.0% over EvoSearch with LTX-2.Table 1 reports total NFE and identifies SWIM as strongest at lower budgets.Source: §5.2 · p. 6, Table 1 · p. 7
• Across AestheticScore and ImageReward, SWIM’s mean relative gains over fixed-population Uniform+MH are 4.1% with systematic, 4.6% with multinomial, and 3.3% with Fleming-Viot resampling.The overall mean gain is 4.0%.Source: Table 2 · p. 7, §5.2 · p. 7

5.3 HUMAN EVALUATION

• Across six alignment and aesthetic criteria, SWIM is preferred over both model baselines in human pairwise evaluation.The study covers 80 prompts and 62 participants, pooling 741 JavisDiT++ and 764 LTX-2 pairs.Source: Figure 2 · p. 7, §5.3 · pp. 7–8

5.4 ABLATIONS

• At matched NFE, SWIM has the highest mean JavisScore at every tested budget, with its largest advantage over particle-only search at the lowest budget.Particle-only search benefits from global exploration but retains finite-population estimation error; MH-only refinement mixes slowly from unguided starts.Source: Figure 3 · p. 8, §5.4 · p. 8
• MH noise levels at or below 0.3 yield small gains despite high acceptance, while levels at or above 0.8 can discard coarse alignment; smh = 0.7 balances useful changes and preservation of structure.The reported setting uses smh = 0.7.Source: Figure 5 · p. 8, §5.4 · p. 9
• With the same reward and MH schedule, a stronger funnel starts higher and peaks earlier, while a weaker funnel gains more and reaches comparable final reward with fewer particles.MH-only improves initially but plateaus below both funnel-initialized chains.Source: §5.4 · pp. 8–9, Figure 4 · p. 8
• Adaptive allocation yields higher TV-IB and lower estimator variance than uniform-K across tested budgets, with smaller score gains at budget extremes and a narrowing variance gap at the largest budget.The AV evaluation uses TV-IB separately from the VR-TA reward used to guide allocation.Source: Figure 6 · p. 9, §5.4 · p. 9

6 LIMITATIONS AND CONCLUSION

• SWIM uses variance-optimal particle allocation to find a candidate, then applies Metropolis-Hastings locally, with online ESS guiding allocation toward informative denoising stages.The method combines global exploration and local refinement within a limited budget, without offline calibration or reward-specific tuning.Source: §6 · p. 9
• At larger budgets, the smaller gains from adaptive allocation may not justify its extra schedule parameters, although large models still incur substantial inference cost.Source: §6 · p. 9
• The persistence forecast may be suboptimal when variance changes rapidly between adjacent checkpoints.Source: §6 · p. 9

AI USE STATEMENT

• Generative AI tools assisted with manuscript framing and editing, reference checking, mathematical claims and proofs, and feedback on methodology and interpretation; the authors take responsibility for the final content.Source: AI USE STATEMENT · p. 12

A.1 WHY THE ESS-BASED OBJECTIVE IS REASONABLE

• The ESS-based estimate is the squared coefficient of variation of checkpoint weights, and an independent-draw calculation motivates using σ²_m/K_m−1 in the allocation surrogate.Source: §A.1 · p. 13
• Resampled trajectories share ancestors, so the full SMC variance includes dependence and propagation effects; the surrogate is not the exact final-output variance.Source: §A.1 · p. 13
• Weight dispersion may separate particles without predicting final quality; the small-temperature reward effect is approximated locally by λ Cov(R₀, R_m), motivating delayed use of the text-audio term.This is a local approximation, not an exact finite-population resampling formula; the experiments assess whether the proxy helps final quality.Source: §A.1 · p. 13

A.2 OPTIMAL ALLOCATION

• The optimal-allocation derivation uses a Lagrange multiplier for the compute constraint, and convexity ensures the stationary values minimize the objective.Source: §A.2 · p. 13
• In the continuous limit, the optimal population schedule is K*(t) = B σ(t)/∥σ∥L1.Source: §A.2 · p. 13

A.3 UNIFORM PACING AS A MINIMAX SAFEGUARD

• Uniform population allocation minimizes worst-case risk; the continuous relaxation is implemented with rounding and an affordability cap.Source: §A.3 · pp. 13–14

A.4 BUDGET FEASIBILITY

• Budget feasibility assumes enough compute for at least one particle per segment and imposes a maximum population at each checkpoint.Source: §A.4 · p. 14

A.5 REWARD RELIABILITY UNDER GAUSSIAN ASSUMPTIONS

• In the Gaussian example, denoised reward becomes more correlated with terminal reward as t decreases, but this does not predict a universal particle-weight variance profile.The online scheduler uses observed ESS rather than assuming one.Source: §A.5 · p. 14

A.6 MH ACCEPTANCE FOR NOISING-DENOISING PROPOSALS

• With an exact reverse process, the noising-denoising bridge makes the reference-density factors cancel in the MH acceptance rule; with a learned reverse model, the reward-only rule is approximate.The resulting chain is not guaranteed to have the exact reward-tilted reference distribution as its stationary distribution.Source: §A.6 · pp. 14–15

A.7 IDEAL COMMON-TARGET MH REFINEMENT

• Under a shared target, MH cannot increase total-variation distance from that target, but this bound does not establish that selecting the highest-reward particle gives a closer initialization.In the practical method, particle and MH rewards differ, and the learned reverse proposal may not preserve the target; the bound is therefore a design motivation.Source: §A.7 · p. 15

B.1 STAGED REWARD ABLATIONS

• The staged-reward settings favor t_mix = 0.85 and β_mix = 0.3, balancing reliable estimates and search time while improving TA-IB with only a modest VR-TA reduction.Introducing text-audio reward too early or too late performs worse; larger weights offer diminishing TA-IB gains at VR-TA's expense.Source: §B.1 · p. 15, Figure 7 · p. 16

B.2 PACING FLOOR γ

• At γ = 0.6, pacing achieves the best reward by balancing adaptive allocation against a more uniform schedule.A small pacing floor risks pruning too aggressively, while a large one reduces the benefit of adaptation.Source: §B.2 · pp. 15–16, Figure 8 · p. 16

B.3 EFFECT OF DUAL STEP SIZE ην

• At η_ν = 10^-3, final reward is highest (0.215), versus the lowest reward (0.145) at η_ν = 10^-1.Larger step sizes prune more aggressively; moderate adaptation preserves particles until later, more informative denoising stages.Source: §B.3 · pp. 16–17, Figure 9 · p. 17

B.4 VARIANCE AND PERFORMANCE CURVES FOR STABLE DIFFUSION

• At NFE=400, SWIM's largest quality gains over uniform-K are 2.5% for TV-IB and 2.3% for AestheticScore; gains narrow to 1.3% and 0.9% at the respective highest budgets.The respective highest budgets are NFE=500 for TV-IB and NFE=1000 for AestheticScore.Source: §B.4 · pp. 17–18, Figure 11 · p. 18
• At NFE=200 and 400, SWIM reduces variance by 36.1% and 27.2% for TV-IB, and 14.9% and 10.6% for AestheticScore.At NFE=500, the TV-IB variance gap remains 28.7%; at NFE=1000, the AestheticScore gap narrows to 0.8%.Source: §B.4 · p. 18, Figure 11 · p. 18

B.5 EFFECT OF POPULATION GROWTH STRATEGY

• Rescue yields the largest VR-TA improvement over base generation (0.099), while uniform performs worst (0.068), despite similar population schedules across strategies.Rescue combines reward-guided resampling with lineage preservation and is used in subsequent experiments.Source: §B.5 · p. 18, Figure 12 · p. 19