Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding

1 Introduction

• Classifier guidance requires differentiable proxy models, limiting direct use of non-differentiable rewards and principled application to discrete diffusion, while fine-tuning can be computationally intensive.Examples of non-differentiable feedback include molecular or protein descriptors and physics-based simulations.Source: §1 · p. 2
• SVDD introduces soft value functions that look ahead to future rewards and selects the highest-valued candidate among multiple noisy states at each inference step.The value functions predict reward at time 0 from states at time t − 1.Source: §1 · p. 2, Figure 1 · p. 2
• SVDD avoids differentiable proxies and diffusion-model fine-tuning, enabling non-differentiable reward feedback and use with recent discrete diffusion models without modification.The method is presented as addressing the computational cost associated with fine-tuning.Source: §1 · pp. 2–3
• SVDD was demonstrated on image generation, molecule optimization, and DNA/RNA generation, including optimization of docking scores, QED, SA, and activity levels.The paper reports effectiveness across these diverse domains.Source: §1 · p. 3

Non-fine-tuning methods

• Classifier guidance incorporates a value-function derivative into the diffusion drift but requires differentiable models and does not directly extend in a principled way to discrete diffusion.A discrete-specific classifier-guidance method is noted, but its practical method still requires differentiable proxy models; SVDD applies to continuous and discrete diffusion in a unified manner.Source: §2 · p. 3
• Best-of-N generates multiple samples and selects the top ones by reward, whereas SVDD uses soft values to predict the future rewards of intermediate noisy samples.The paper characterizes Best-of-N as significantly less efficient than SVDD and says it validates this experimentally.Source: §2 · p. 3

Fine-tuning of diffusion models

• Fine-tuning approaches such as classifier-free guidance and RL-based fine-tuning can incur high computational costs and risk forgetting pre-trained models.SVDD is presented as an inference-time technique that does not fine-tune the generative model.Source: §2 · p. 3

Discrete diffusion models

• Discrete diffusion has been applied to natural-language and biological-sequence generation, but adding a continuous gradient to a discrete objective is not inherently valid for standard classifier guidance.The paper states that SVDD can be applied to discrete diffusion models.Source: §2 · p. 3

3.1 Diffusion Models

• A standard diffusion model pairs a fixed forward process with learned backward processes whose induced distributions match the forward-process marginals.The backward processes are parameterized and trained using a loss derived from the variational lower bound of negative log-likelihood.Source: §3.1 · p. 4
• Continuous diffusion typically uses a Gaussian forward process, while discrete diffusion can use categorical transitions toward a mask state and a neural prediction of the clean sample from a noisy state.For sequences, the backward probability is formed as a product over token-level probabilities.Source: §3.1 · p. 4

3.2 Objective: Generating Samples with High Rewards While Preserving Naturalness

• The objective is to use a pre-trained model to generate samples that optimize downstream rewards while maintaining the naturalness captured from its training data.The stated objective includes rewards such as QED and synthetic accessibility for molecule generation.Source: §3.2 · p. 4
• Among non-fine-tuning approaches, the paper targets classifier guidance's need for differentiable proxies and Best-of-N's inefficiency.The stated setup also covers conditional pre-trained diffusion models, including text-conditioned Stable Diffusion.Source: §3.2 · p. 5

4.1 Key Observation

• Soft value functions represent expected future reward from intermediate noisy states, and the paper uses a theorem connecting soft-optimal policies to sampling from the target distribution.The paper identifies unknown and unnormalized policies as challenges; importance sampling is applied at each time step, unlike Best-of-N selection over completed samples.Source: §4.1 · p. 5

4.2 Inference-Time Algorithm

• SVDD iteratively integrates soft values into pre-trained diffusion inference, with each step approximating sampling from a value-weighted policy.The paper recommends α = 0 in practice, which selects the candidate with the highest estimated value; candidate generation can be parallelized at additional memory cost.Source: §4.2 · p. 6

4.3 Learning Soft Value Functions

• SVDD-MC estimates soft values by regressing terminal rewards onto noisy states, and the authors recommend Monte Carlo approaches over soft-Q-learning for stability.The paper notes that directly regressing exp(r(x0)/α) can scale excessively when α is very small.Source: §4.3 · p. 6
• SVDD-PM estimates value by applying the reward to a posterior-mean denoised sample, requiring no additional training when the reward is available.The posterior mean approximation uses the denoised estimate x̂0(xt).Source: §4.3 · p. 7

5.1 Advantages

• SVDD requires no generative-model fine-tuning, and SVDD-PM needs no additional training when reward feedback r is available.Source: §5.1 · p. 7
• Because SVDD avoids derivative computations, it can use non-differentiable feedback and apply directly to discrete diffusion models.SVDD-MC still requires a value-function model, but that model need not be differentiable.Source: §5.1 · p. 7

5.2 Extensions

• For inverse problems or conditioning, likelihoods or classifiers can replace regression rewards.The paper relates this extension to Zhao et al. (2024), Chung et al. (2022), and Bansal et al. (2023).Source: §5.2 · p. 7

5.3 Potential Limitations

• Inference can require approximately M times more compute when sequential or memory when parallelized; policy distillation is proposed to mitigate this cost but left for future work.Source: §5.2 · p. 7, §5.3 · p. 8
• When substantial changes to pretrained models are desired, RL-based fine-tuning may be more effective in some scenarios, including image examples.The authors also describe SVDD's proximity to pretrained models as potentially robust to reward optimization.Source: §5.3 · p. 8

6 Comparison between SVDD and SMC-Based Methods

• SMC methods can be applied to reward maximization but were not designed specifically for it.Source: §6 · p. 8
• SMC resampling across the entire batch complicates parallelization, and its theoretical guarantees primarily hold for large batches.The paper notes that small batches are common with recent large diffusion models.Source: §6 · p. 8
• SVDD samples per example, which the authors say supports parallelization, performance at batch size 1, and diversity at larger batch sizes; both approaches avoid differentiable models.Source: §6 · p. 8

7.1 Settings

• The comparison includes pretrained sampling, Best-of-N, DPS, SMC, and SVDD-MC and SVDD-PM; typical settings use M = 20 for images and M = 10 elsewhere.The Best-of-N selection is chosen to make inference time approximately equivalent to SVDD.Source: §7.1 · p. 8
• Tasks span image compressibility and aesthetics, molecule QED, SA, and docking scores, and enhancer and 5′UTR activity rewards.The experiments use Stable Diffusion, GDSS, and a discrete diffusion model, respectively.Source: §7.1 · p. 9

7.2 Results

• SVDD consistently outperforms the baseline methods in generated-sample top-10 and top-50 quantiles, with higher values preferred.Table 2 reports 95% confidence intervals.Source: Table 2 · p. 9
• SVDD-MC and SVDD-PM have domain-dependent relative performance; SVDD-PM may be more robust because it directly uses reward feedback without additional learning.Figure 2 presents sample examples, including generated molecules shown with target proteins in panels (e)–(h).Source: §7.2 · p. 9, Figure 2 · p. 10
• As M increases, performance gradually plateaus across domains; the reported ablation also finds computational complexity rising linearly while memory complexity remains nearly constant.Source: §7.2 · p. 10

8 Conclusion

• SVDD optimizes downstream rewards in pretrained diffusion models at inference time without requiring differentiable proxy models.The conclusion proposes the method as an inference-time algorithm.Source: §8 · p. 10
• Protein-sequence optimization and 3D molecule generation are identified as future application areas.The cited passage presents both as future work.Source: §8 · p. 11

A Further Related Works

• Prior inference-time reward-decoding methods for autoregressive models had not, to the authors’ knowledge, been extended to diffusion models.The authors also say their approach leverages characteristics unique to diffusion models.Source: A Further Related Works · p. 17

B Difference Between SVDD and “Standard” SMC-Methods

• The compared SMC-based guidance methods were originally designed for conditioning, but can be converted to reward maximization.The passage names Wu et al. (2024), Trippe et al. (2022), Dou and Song (2024), Phillips et al. (2024), and Cardoso et al. (2023).Source: B Difference Between SVDD and… · p. 17
• SVDD is an instantiation of nested-IS SMC, whereas the compared methods use standard sequential Monte Carlo.The passage relates this distinction to the computational-statistics literature.Source: B Difference Between SVDD and… · p. 17
• The comparison first explains the reward-maximization conversion, then discusses differences between the methods.Source: B Difference Between SVDD and… · p. 17

B.1 SMC-Based Methods

• The standard SMC procedure alternates an importance-sampling step with a selection step that resamples particles.The procedure outputs the final sample x0.Source: §B.1 · p. 17
• At each iteration, the procedure starts with N particles and generates a new set of samples before calculating importance weights.The passage says the algorithm is iterative and considers N samples at time t.Source: §B.1 · p. 17
• The method uses the soft Bellman equation in its importance-weight calculation and resamples indices during selection.The appendix also notes that its SMC resampling is performed across the entire batch, unlike SVDD’s within-batch sampling.Source: §B.1 · p. 18

B.2 Comparison of SVDD with “Standard” SMC-Based Methods (SSM) for Reward Maximization

• SVDD reward performance depends on M rather than batch size G, and it is easier to parallelize; with small batches, SVDD-PM can retain high performance by increasing M.The standard SMC comparison also notes that approximating both numerator and denominator in its weight ratio could increase error propagation.Source: §B.2 · pp. 18–19
• With Stable Diffusion and an aesthetic-score objective, SVDD produced more diverse samples while retaining high rewards; SMC samples were often similar despite roughly maintained reward diversity.The reported setup used SMC batch size N = 60 and SVDD duplication size M = 20; the figure shows cat-prompt samples and reward histograms.Source: §B.2 · pp. 18–19, Figure 4 · p. 19
• SVDD applies across continuous and discrete domains; its importance sampling and resampling avoid computational overheads discussed for discrete guidance alternatives.For Nisonoff et al. (2024), the stated O(KL) cost motivates a Taylor approximation without a theoretical guarantee, while SVDD has an asymptotic guarantee as M grows; the DiGress comparison likewise describes a Taylor expansion and leaves detailed empirical comparison for future work.Source: C Comparison against DG (Nisonoff et… · pp. 19–20

D Extension with Arbitrary Proposal Distribution

• With an arbitrary proposal distribution, SVDD adjusts the importance weight and can use a differentiable proxy’s gradient to form the proposal.The value function used for weighting can remain non-differentiable, even when the proxy is not highly accurate.Source: D Extension with Arbitrary Proposal… · p. 20

E Soft Q-learning

• Soft Q-learning estimates soft value functions recursively by regressing v_{t-1}(x_{t-1}) onto x_t.The appendix presents this as an alternative to Monte Carlo regression and relates the recursion to the soft Bellman equation.Source: E Soft Q-learning · p. 21

F.1.1 Settings

• The image settings use Stable Diffusion and animal prompts, while enhancer and 5′UTR activity is measured using MPRA datasets.The enhancer sequences have length 200 and 5′UTR sequences length 50.Source: §F.1.1 · p. 22

F.1.2 Baselines and Proposals

• The baseline implementations use task-specific value and reward models, with several hyperparameter candidates evaluated for DPS and SMC.The SVDD-MC value models differ by task, including CNNs for images, GIN for molecules, Enformer for enhancers, and ConvGRU for 5′UTRs.Source: §F.1.2 · pp. 22–23

F.2 Software and Hardware

• Experiments used PyTorch on single NVIDIA RTX 2080Ti or RTX A6000 GPUs.Source: §F.2 · p. 23

• Value-function training curves report test-set Pearson correlation for Enhancer and 5’UTR, and test MSE for parp1 docking.Source: §F.3 · p. 23, Figure 6 · p. 25
• Molecule validity metrics cover chemical-rule validity, uniqueness, novelty, distribution similarity, and atom and molecule stability.Atom and molecule stability are calculated using RDKit-generated conformers optimized with UFF and MMFF.Source: §F.3 · pp. 23–24
• Reward-score histograms show generated-sample distributions, and the authors report consistently strong SVDD performance.The distributions are plotted for the reward functions discussed in the main text.Source: §F.3 · p. 23, Figure 5 · p. 24
• SVDD is comparable to pretrained GDSS across molecule metrics and outperforms it on molecule stability, FCD, SNN, and scaffold similarity.The comparison uses 512 molecules generated by GDSS and SVDD optimizing SA; validity, novelty, and uniqueness remain high.Source: Table 3 · p. 23, §F.3 · p. 24
• Additional ablations of M show results consistent with those reported in Figure 3.Source: §F.3 · p. 24, Figure 7 · p. 25
• Supplementary visualizations compare generated images and molecules across rewards and show SVDD ligands in protein-binding contexts, generally fitting snugly with minimal steric clashes.The docking visualizations cover parp1, 5ht1b, jak2, and braf; their scores quantify ligand–protein binding affinity.Source: §F.3 · pp. 24–25, Figure 8 · p. 26, Figure 9 · p. 26