Source-linked AI summary
Forward-Only Learning for Shared-Kernel Wavelet Neural Networks without Backpropagation
Yongfei Guo, Xuan Sun, Ang Li, Zhao Zhang, Yuanhao Gong
TL;DR
Backpropagation-free learning methods often remain layer-local, while whole-network search is difficult for large neural networks. This paper uses CMA-ES with structure-aware decomposition and channel-specific credit assignment to train compact shared-kernel wavelet networks using forward evaluations only. On BSDS500, the method improves test reconstruction metrics over Adam/BP, and ablations indicate that decomposition and aligned credit assignment matter more than simply increasing population size.
Problem
Backpropagation-free methods such as Forward-Forward remain predominantly layer-local, while whole-network population search is generally impractical for large neural networks.
Source: §1 · p. 1Method
The framework uses CMA-ES to optimize complete shared-kernel wavelet-network candidates through forward losses, with structure-aware parameter decomposition and channel-specific credit assignment.
Source: §1 · p. 2, §3.2 · p. 3Results
Test MSE falls from 4.1783 to 4.0130 with λ = 64 versus Adam/BP, and the method also improves PSNR and SSIM.
Source: §4.2 · p. 3Takeaways & Limitations
The experiments indicate that compact shared-kernel networks can make global forward-only learning practical, with structure and aligned credit assignment more important than simply increasing population size.
Source: §5 · p. 4The reported evaluation counts are not a strict equal-compute comparison, and the paper makes no claim of wall-clock superiority.
Source: §4.4 · p. 4
Abstract
from arXiv · showhide
Backpropagation remains the dominant training paradigm, using backward gradients and local updates that can suffer from vanishing or exploding gradients in deep networks. Although the Forward-Forward algorithm avoids conventional backpropagation, it remains mainly local or layer-wise, motivating global whole-network learning using forward evaluations only. For shared-kernel wavelet neural networks, we therefore propose a CMA-ES-based global forward-only learning framework. We further introduce structure-aware parameter decomposition and channel-specific credit assignment, factorizing the search into three subspaces. On BSDS500, our method reduces test MSE from 4.1783 to 4.0130 and improves PSNR from 42.621 to 42.787~dB over Adam/BP without backward propagation. Beyond these gains, training requires only 6 training-set coverages, compared with 1500 epochs for BP. Ablations show that structural decomposition and aligned credit assignment are more important than increasing population size, demonstrating the feasibility of whole-network forward-only learning for compact reconstruction networks. These results suggest a potential route toward forward-only optimization of compact reconstruction models for resource-constrained imaging and edge or near-sensor systems.
1. INTRODUCTION
The paper asks whether extreme parameter sharing in wavelet reconstruction networks can make whole-network learning without backpropagation tractable. It proposes CMA-ES search aligned with network structure, using parameter decomposition and channel-specific credit assignment.
1. INTRODUCTION
- Shared-kernel wavelet networks use cross-scale parameter sharing to keep the trainable parameterization compact, potentially making whole-network search tractable.The paper frames this as using compactness not only for model efficiency but also to enable a different learning regime.Source: §1 · p. 1
- Backpropagation-free methods such as Forward-Forward avoid backward propagation but remain predominantly layer-local, while whole-network population search is generally impractical for large networks.The paper uses “global” to mean population-based search over the complete parameterization, not guaranteed convergence to a global optimum.Source: §1 · p. 1
- The proposed framework evaluates complete network candidates using forward reconstruction losses and adapts a CMA-ES population, without automatic differentiation or backward propagation.Structure-aware parameter decomposition and channel-specific credit assignment align the search with the network’s intrinsic structure.Source: §1 · p. 2
- The paper develops structure-aware optimization through parameter decomposition and channel-specific credit assignment for whole-network learning.It also reports comparisons with backpropagation, structural ablations, and multi-seed evaluations for reconstruction without backpropagation.Source: §1 · p. 2
2. RELATED WORK
Related work covers compact multiscale reconstruction architectures and learning methods that avoid backpropagation. The paper positions shared-kernel compactness as a way to make global whole-network search tractable.
2. RELATED WORK
- The review groups relevant prior work into compact multiscale reconstruction networks and backpropagation-free alternatives to gradient-based training.These strands motivate examining whether compact architectures can make whole-network search practical.Source: §2 · p. 2
2.1. Compact Multiscale Reconstruction Networks
- Shared-kernel wavelet networks reuse a small set of learnable kernels across scales, extending compact multiscale reconstruction designs.The paper investigates whether this extreme parameter sharing can support a different learning regime, beyond architectural efficiency.Source: §2.1 · p. 2
2.2. Backpropagation-Free Neural Learning
- Prior backpropagation-free approaches include layer-wise or error-driven forward methods, zeroth-order optimization, and evolutionary search over complete candidates.CMA-ES adapts the search mean, covariance, and step size, while whole-network search is difficult in high dimensions.Source: §2.2 · p. 2
3. GLOBAL FORWARD-ONLY LEARNING
This section defines a compact shared-kernel wavelet reconstruction network and trains it by evaluating whole-network candidates with CMA-ES. It then decomposes the search by RGB channel and assigns each parameter block its corresponding channel-specific loss.
3. GLOBAL FORWARD-ONLY LEARNING
- The method combines a compact shared-kernel wavelet network with CMA-ES forward-only search, then decomposes the search and aligns credit assignment with RGB channels.The same RGB forward pass evaluates assembled channel candidates; each output channel depends only on its corresponding parameter block.Source: §3 · p. 2, §3.3 · p. 3
3.1. Shared-Kernel Wavelet Network
- Three learnable kernels, H, G, and K, are reused across resolutions, keeping the trainable parameter count independent of the number of scales.The model reconstructs an image from a Laplacian field y with per-channel DC information; the same three-scalar DC input is used for BP and CMA-ES.Source: §3.1 · p. 2
- The kernels are bias-free channel-wise grouped convolutions, with H and K sized 5×5 and G sized 3×3 for each RGB channel.There are 59 learnable parameters per channel and 177 total.Source: §3.1 · p. 2
3.2. Population-Based Forward-Only Search
- CMA-ES samples complete parameter candidates, ranks them by forward reconstruction loss, and adapts its search distribution without automatic differentiation or backward propagation.The candidates are drawn using a search mean, covariance matrix, and global step size.Source: §3.2 · p. 3
3.3. Structure-Aware Search and Credit Assignment
- Instead of jointly searching the 177-dimensional parameter vector, the method factorizes it into three channel-specific distributions over 59 parameters each.At each population index, one candidate from each channel distribution is assembled and evaluated through the RGB forward pass.Source: §3.3 · p. 3
- Each generation’s candidates and three CMA-ES instances share the same subset of training samples, with a refreshed permutation after each full training-set coverage.Each coverage partitions all N samples into non-overlapping subsets of size M.Source: §3.3 · p. 3
- Each channel’s CMA-ES instance is ranked and updated using the MSE of its corresponding output channel, aligning each parameter block with its direct optimization signal.The R, G, and B instances use L_R, L_G, and L_B, respectively.Source: §3.3 · p. 3
4. EXPERIMENTS
The experiments compare forward-only CMA-ES with Adam/BP on BSDS500 reconstruction and test how parameter decomposition, channel-specific fitness, and population size affect search. They also describe the shared evaluation protocol and qualify computational comparisons.
4.1. Experimental Setup
- The BSDS500-derived dataset uses 256 × 256 patches and eight-way augmentation; all methods share a 177-parameter network and data splits.CMA-ES uses 4,800 generations, corresponding to six training-set coverages; Adam/BP uses 1,500 epochs.Source: §4.1 · p. 3
4.2. Comparison with Backpropagation
- Test MSE falls from 4.1783 for Adam/BP to 4.0130 for the forward-only method at λ = 64, with PSNR and SSIM also improving.Table 1 reports means over ten runs; all ten runs of both forward-only configurations had lower test MSE than all BP runs.Source: §4.2 · p. 3, Table 1 · p. 4
4.3. Structure and Credit Assignment
- Test MSE drops from 940.23 for joint 177-D search to 30.81 with 3 × 59-D decomposition, then to 4.04 with channel-specific credit assignment.The comparison uses λ = 32; the authors characterize decomposition as decisive and channel-specific credit assignment as critical for reaching BP-level reconstruction quality under the tested budget.Source: §4.3 · p. 4, Figure 3 · p. 4
- Increasing population size improves both search formulations, but joint 177-D search at λ = 64 remains far from the BP regime while structured search achieves orders-of-magnitude lower error.For structured search, the largest improvement occurs from λ = 19 to 32, after which gains saturate.Source: §4.3 · p. 4, Figure 4 · p. 4
4.4. Computational Characteristics
- Forward-only CMA-ES uses forward evaluations, whereas BP also propagates gradients through training samples; these counts do not establish wall-clock superiority.The comparison is not equal-compute because candidate evaluations increase with λ and the methods use different GPU parallelism.Source: §4.4 · p. 4
5. CONCLUSION
The paper presents global forward-only optimization for compact shared-kernel wavelet networks, combining CMA-ES with structure-aware decomposition and channel-specific credit assignment. Experiments report consistent gains over Adam/BP and identify network structure and aligned credit assignment as more important than simply increasing population size.
5. CONCLUSION
- The framework optimizes complete compact shared-kernel wavelet networks with forward evaluations only, combining CMA-ES, parameter decomposition, and channel-specific credit assignment.The method uses neither automatic differentiation nor backward propagation.Source: §5 · p. 4
- On image reconstruction, the learner consistently outperforms Adam/BP, while structure and aligned credit assignment matter more than simply increasing population size.The authors conclude that extreme parameter sharing can make global forward-only learning practical for compact neural networks.Source: §5 · p. 4
- Future work will investigate extending the framework to larger architectures and more general vision tasks.Source: §5 · p. 4