Source-linked AI summary
ASSERT: Adaptive Stochastic Sampling for Robust Diffusion Models on Analog Compute-in-Memory Hardware
Yuannuo Feng, Yizhe Chen, Wenshuai Yao, Yuxin Xie, Ngai Wong, Wenyong Zhou, Wang Kang
TL;DR
Diffusion inference is computationally expensive and vulnerable to persistent spatial noise on analog CIM hardware, with temporal sensitivity underexplored. The paper calibrates a multichip noise model, analyzes error propagation, and introduces ASSERT, an adaptive stochastic DDIM sampler. ASSERT reports up to 2.58× lower FID on evaluated high-resolution datasets and 7.68× lower FID in the CIFAR-10 step-count study without changing model parameters or network evaluations.
Problem
Diffusion sampling requires many network evaluations, while persistent CIM weight perturbations can accumulate during sampling and the models' temporal sensitivity to hardware noise remains underexplored.
Method
The paper uses a noise model calibrated and validated on multiple physical CIM chips, derives a first-order error recursion, and introduces ASSERT's adaptive stochastic-to-deterministic DDIM schedule.
Results
Up to 2.58× lower FID than deterministic DDIM is achieved on evaluated high-resolution datasets, and 7.68× lower FID in the CIFAR-10 step-count study.
Takeaways & Limitations
Early high-noise denoising is more vulnerable than final refinement, and early stochastic updates reduce cross-step error alignment before deterministic refinement.
Takeaways & Limitations
The persistent-noise model fixes each spatial weight-perturbation map across all denoising steps within one deployment realization, while output noise is redrawn per MVM.
Abstract
from arXiv · showhide
Diffusion models achieve strong image generation quality but incur high iterative denoising costs. Analog compute-in-memory (CIM) can accelerate matrix-vector multiplications, yet spatial memory variations perturb weights and accumulate during sampling. Unlike conventional neural networks, diffusion models' temporal sensitivity to hardware noise remains underexplored. We investigate diffusion inference using a noise model calibrated and validated against measurements collected from multiple physical CIM chips. Our results show that the early, high-noise denoising stage is substantially more vulnerable than the final refinement stage. A first-order trajectory analysis attributes this behavior to the repeated propagation of correlated prediction errors induced by a fixed hardware mapping. Based on this observation, we propose ASSERT, a training-free sampler that uses higher stochasticity early and smoothly transitions to deterministic denoising. The injected stochasticity changes subsequent activation trajectories and thereby reduces their alignment with persistent spatial errors. Across the evaluated settings, ASSERT achieves up to 2.58$\times$ lower FID than deterministic DDIM on high-resolution datasets and 7.68$\times$ lower FID in the CIFAR-10 step-count study, without changing model parameters or the number of network evaluations.
1 Introduction
Diffusion models offer strong visual synthesis but require costly iterative denoising, while analog CIM acceleration introduces hardware noise that can severely degrade generation quality. ASSERT studies this vulnerability and uses adaptive stochastic sampling to improve robustness without retraining.
- Hundreds of neural-network evaluations make matrix-vector multiplications the dominant computational workload in iterative diffusion sampling.
- CIM accelerates diffusion by performing matrix-vector multiplications within memory arrays, reducing data movement between processing units and memory.
- Device variability, programming uncertainty, thermal fluctuations, and quantization noise perturb stored weights and can significantly degrade generation quality.
- FID increases rapidly with stronger weight perturbations, while CelebA-HQ samples degrade from facial structure to blurred texture and incoherent artifacts.
- Existing robustness methods often require retraining or fine-tuning and largely overlook diffusion sampling's iterative characteristics under analog hardware noise.
- ASSERT combines a multichip-calibrated noise study, temporal sensitivity analysis, and training-free adaptive stochastic sampling for diffusion inference.
- Up to 2.58× and 7.68× lower FID are reported in the high-resolution and CIFAR-10 step-count evaluations, respectively.
2 Preliminary
Diffusion models reverse a gradual noising process through iterative denoising, whose matrix-vector multiplications remain a major computational bottleneck. CIM offers efficient acceleration, but analog non-idealities perturb weights and activations, creating a robustness challenge that training-free methods aim to address.
- 2.1 Diffusion Models: Diffusion models synthesize data by learning to reverse a gradual noising process; DDPM uses a fixed Markov chain, while DDIM permits deterministic or stochastic trajectories.
- 2.2 CIM Noise on Neural Network Robustness: Iterative denoising remains computationally intensive because matrix-vector multiplications are the primary diffusion-model bottleneck.
- 2.2 CIM Noise on Neural Network Robustness: CIM architectures provide energy-efficient acceleration by executing matrix-vector multiplications within memory arrays and reducing costly data movement.
- 2.2 CIM Noise on Neural Network Robustness: Analog CIM non-idealities perturb stored weights and computed activations, leading to inference accuracy degradation.
- 2.2 CIM Noise on Neural Network Robustness: Reliable CIM deployment therefore requires model robustness under analog-induced noise.
- 2.2 CIM Noise on Neural Network Robustness: Robustness strategies for large-scale neural networks often require fine-tuning or hardware modifications, whereas the paper's approach changes only sampler coefficients.
3 Methodology
The methodology models measured CIM hardware noise, analyzes its propagation across denoising time, and uses that analysis to design ASSERT's adaptive stochasticity schedule. ASSERT adds stochastic exploration early and deterministic refinement later without retraining or additional network evaluations.
- Hardware Noise Modeling: Weight perturbations are calibrated from measurements across multiple physical chips, while each deployment keeps the spatial weight-noise map fixed across all denoising steps.Output computation noise is redrawn for every MVM, but persistent weight perturbations induce correlated prediction errors across steps.
- Temporal Sensitivity: Early high-noise denoising is substantially more vulnerable than final refinement under the same fixed hardware mapping.Activating noisy computation during steps 99–80 causes the largest FID degradation, whereas steps 20–0 have a much smaller effect.
- ASSERT Framework: The stochasticity parameter ηᵢ ranges from deterministic DDIM at 0 to the stochastic DDIM variance at 1, with the valid standard deviation determined by the selected timestep pair.ηᵢ is dimensionless and is not itself the injected standard deviation.
- Persistent-Noise Error Analysis: Persistent spatial errors can remain directionally aligned after propagation through later denoising transitions, increasing their end-to-end impact.The first-order recursion attributes this alignment to reuse of the same Δ𝜽 across steps; stochasticity changes Hᵢ through altered states and can reduce cross-step alignment.
- ASSERT Framework: ASSERT controls DDIM stochasticity with ηᵢ, using maximum stochasticity during an early plateau and smoothly decaying it to zero for deterministic refinement.The schedule is parameterized by ηmax and τtrans; S−1 normalization ensures the final update has ηS−1 = 0.
4 Experiments
Experiments use measured-chip-calibrated noise and evaluate ASSERT across datasets, samplers, noise sources, and step counts. ASSERT consistently improves noisy diffusion robustness, especially when stochasticity is concentrated early, while clean-quality trade-offs remain.
- Experiment Setup: Experiments evaluate public diffusion checkpoints across CIFAR-10, Butterfly, CelebA-HQ, LSUN-Church, and LSUN-Bedroom using BF16 inference and native schedules.The study uses complete end-to-end trajectories with mapped CIM layers and digital normalization, activation, residual, and sampler control.
- Ablation Study: Cosine decay achieves the lowest noisy CIFAR-10 FID, while reserving stochasticity for final refinement performs much worse than early-stochastic schedules.The comparison supports concentrating stochasticity in the vulnerable early denoising stage.
- Experiment Results: 2.39×, 2.58×, and 2.54× lower mean FID than deterministic DDIM are achieved by ASSERT on CelebA-HQ, Church, and Bedroom at noise level 0.08.Relative to the step schedule, the corresponding reductions are 1.36×, 1.22×, and 1.51×.
- Experiment Setup: The noise model is fitted to physical eFlash CIM measurements and validated on 100 chips, with fixed spatial weight maps persisting across denoising steps.Measured and modeled errors achieve R^2 = 0.993, while output noise is redrawn per MVM.
- Experiment Results: 7.68× lower FID is achieved at 500 steps, decreasing from 163.38 to 21.27; at 100 steps, FID decreases from 33.53 to 5.90, or 5.68×.ASSERT has the lowest noisy FID in every evaluated step-count row, but 20-step ASSERT has clean FID 13.98 versus 11.08 for DDIM.
- Experiment Results: ASSERT better preserves facial structure and scene texture than noisy DDIM in Bedroom, CelebA-HQ, and Church examples, complementing aggregate FID evaluation.The noisy DDIM examples show color distortion and detail loss.
- Ablation Study: A short, strong stochastic phase followed by gradual deterministic refinement performs best, with 5.95 observed FID at ηmax = 0.8 and onset 90.At ηmax = 0.6, extending the stochastic plateau lowers FID from 17.08 to 9.28.
5 Conclusion
The paper studies persistent spatial noise in diffusion inference using measurements from multiple CIM chips and explains robustness through error propagation. ASSERT uses early stochasticity followed by deterministic refinement, achieving lower FID without retraining or additional network evaluations.
- 5 Conclusion: Persistent hardware perturbations create correlated prediction errors whose early contributions propagate through later denoising updates.A first-order trajectory recursion motivates reducing cross-step error alignment.
- 5 Conclusion: ASSERT applies pair-dependent stochastic DDIM updates early and smoothly decays to deterministic refinement without retraining or additional network evaluations.Changing activation trajectories reduces positive cross-step alignment with persistent spatial errors.
- 5 Conclusion: 2.58× lower FID than deterministic DDIM is achieved on evaluated high-resolution datasets, and 7.68× lower FID in the CIFAR-10 step-count study.The results support adaptive stochasticity as a robustness mechanism for diffusion inference with measured CIM non-idealities.