PaperlayerView exact revision on arXiv

Source-linked AI summary

Nonlinear Direct Feedback Alignment for Scalable Backpropagation-Free Training

Wu Lin, Yan Fan, Chenxiang Ma, Yinglan Feng, Jia Wang, Ka-Chun Wong, Qiuzhen Lin

arXiv:2610.36718v1cs.LG

TL;DR

DFA avoids BP’s layer-by-layer error propagation, but its fixed feedback matrices have limited capacity to capture nonlinear transformations and its performance remains behind BP. NDFA uses learnable nonlinear surrogate subnetworks to construct hidden-layer feedback and updates them during training. The paper reports that NDFA consistently outperforms DFA-based methods, narrows the gap to BP, and maintains parallel training across layers.

  • Problem

    DFA’s fixed feedback matrices have limited ability to capture the nonlinear transformations from hidden representations to network output, and DFA-based methods remain less accurate than BP in deep networks.

    Source: §1 · p. 2

  • Method

    NDFA constructs hidden-layer feedback through lightweight learnable nonlinear surrogate subnetworks and updates the feedback during training.

    Source: §1 · p. 2

  • Results

    NDFA consistently outperforms DFA-based methods, narrows the gap to BP, and maintains parallel training; on ConvNet it surpasses FDFA by more than 4% and 6% accuracy on CIFAR-10 and CIFAR-100, respectively.

    Source: §1 · p. 2, §5.2 · p. 6, §6 · p. 9

  • Takeaways & Limitations

    NDFA’s reported advantages span convolutional, residual, and transformer architectures, with stronger BP alignment and neural data similarity also reported.

    Source: §6 · pp. 9–10

    An observable gap to BP remains on very deep architectures, and practical training efficiency depends on parallel implementations and hardware support for the surrogate computations.

    Source: §6 · p. 10

Abstract

from arXiv · show

Backpropagation (BP) remains the predominant algorithm for training deep neural networks, but it is biologically implausible and suffers from the backward locking problem, constraining parallelization across layers. Direct Feedback Alignment~(DFA), which propagates the global output error directly to each hidden layer via fixed feedback matrices, has emerged as a promising alternative for addressing these limitations. However, a significant performance gap persists between DFA and BP, which we attribute to the limited ability of fixed feedback matrices to capture the nonlinear transformation from hidden representations to the network output. To address this limitation, we propose Nonlinear Direct Feedback Alignment (NDFA), which constructs each hidden-layer feedback using a lightweight nonlinear surrogate subnetwork. To enhance the effectiveness of the feedback throughout training, we further propose a learning strategy to update the surrogate subnetwork, thereby better approximating the underlying nonlinear transformation. Extensive experiments across residual networks and vision transformers on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets demonstrate that NDFA consistently outperforms existing DFA-based methods and substantially narrows the performance gap to BP, while maintaining efficient parallel training. The code is available at https://github.com/YanFan321/NDFA.

1 Introduction

NDFA replaces DFA’s fixed feedback matrices with learnable nonlinear surrogates to address feedback mismatch. The paper reports that it outperforms DFA-based methods, narrows the gap to BP, and preserves parallel training efficiency.

1 Introduction

  • BP’s layer-by-layer gradient propagation creates backward locking, which hinders training parallelization and raises concerns about biological plausibility.DFA instead sends the global output error directly to hidden layers using fixed random feedback matrices.Source: §1 · pp. 1–2
  • DFA remains less accurate than BP in deep networks, and existing feedback-matrix update methods do not match BP accuracy.These approaches include DKP, FDFA, and GrAPE.Source: §1 · p. 2
  • The authors identify a structural mismatch: BP feedback reflects a nonlinear downstream transformation, whereas DFA uses a fixed linear matrix.They argue that simply updating DFA’s feedback matrices cannot resolve this mismatch.Source: §1 · p. 2
  • NDFA uses a lightweight nonlinear surrogate for each hidden layer’s feedback and updates the surrogate during training to better approximate the underlying transformation.The paper evaluates convolutional and transformer-based architectures on CIFAR-10, CIFAR-100, SVHN, and ImageNet.Source: §1 · p. 2
  • NDFA outperforms existing DFA-based methods, narrows the performance gap to BP, and preserves parallel training efficiency across diverse datasets and architectures.The reported experiments also show strong neural data similarity.Source: §1 · p. 2

2 Related work

Related work explores biologically motivated, local, parallel, and forward-only alternatives to BP. These approaches can address particular training constraints, but the cited work reports performance limitations on challenging tasks or deep architectures.

2 Related work

  • Biologically motivated alternatives include random feedback connections, direct label projection to hidden layers, and layer-wise targets, but these methods struggle to match BP on challenging tasks and deep architectures.Feedback Alignment uses random feedback connections rather than symmetric weights.Source: §2 · p. 2
  • Decoupled Neural Interfaces, SoftHebb, and other local-learning approaches promote layer-wise parallel learning, but limited cross-layer coordination and global-objective guidance often constrain performance.These approaches use synthetic gradients, local Hebbian learning, or auxiliary networks with local objectives.Source: §2 · pp. 2–3
  • Forward-only methods include PEPITA, Forward-Forward, and forward-gradient approaches; they can improve memory efficiency but often have high-variance gradient estimates and noticeable gaps to BP.They use extra forward passes, positive and negative samples, or forward-mode directional derivatives.Source: §2 · p. 3

3 Background

BP computes parameter gradients through sequential error propagation, which makes hidden-layer updates wait on downstream layers. DFA sends the global output error directly to hidden layers, removing that inter-layer error-propagation dependency.

3.1 Backpropagation (BP)

  • BP propagates error signals sequentially from the output layer toward hidden layers, then uses those errors to compute weight gradients.The gradient rule uses the hidden-layer error and the preceding layer’s representation.Source: §3.1 · p. 3
  • BP’s hidden-layer gradient computation waits for errors from downstream layers, preventing parallelization across layers.This dependency is the backward locking problem.Source: §3.1 · p. 3

3.2 Direct feedback alignment (DFA)

  • DFA computes each hidden-layer error by projecting the global output error through a fixed random feedback matrix, enabling hidden layers to update in parallel.The layer’s activation derivative is also applied to the projected error.Source: §3.2 · p. 4

4 Method

NDFA replaces DFA’s fixed feedback matrices with learnable nonlinear surrogate subnetworks that approximate each hidden layer’s downstream mapping. The method also trains the surrogates to track that mapping while allowing feedback computation and surrogate updates to overlap with later layers’ forward pass.

4.1 Feedback mismatch between DFA and BP

  • DFA’s fixed, representation-independent linear feedback mismatches BP’s representation-dependent nonlinear downstream mapping, and the mismatch grows with network depth.Effective feedback should depend on hidden representations and induce a nonlinear transformation.Source: §4.1 · p. 4

4.2 Representation-dependent feedback via nonlinear surrogate subnetworks

  • NDFA uses a lightweight nonlinear surrogate for each hidden layer and derives feedback from the surrogate’s Jacobian, making feedback representation-dependent.The surrogate is instantiated as an MLP to balance expressiveness and computational efficiency.Source: §4.2 · pp. 4–5

4.3 Learning nonlinear surrogate subnetworks for effective feedback

  • Because the true downstream subnetwork changes during training, NDFA trains each surrogate to match its input-output behavior on hidden representations encountered in training.The objective uses the main network’s loss and labels; stop-gradient prevents its gradients from updating the main network, and the learning rate is shared.Source: §4.3 · p. 5
  • NDFA can compute feedback and update each surrogate in parallel with subsequent layers’ forward pass, then project the global output error into hidden layers once it is available.Source: §4.3 · p. 5

5 Experiments

Across image-classification benchmarks, NDFA outperforms DFA-based baselines and narrows the gap to BP. Feedback analyses, representation probing, efficiency tests, ablations, and neural-data comparisons examine its behavior and trade-offs.

5.2 Main results

  • On CIFAR-10, CIFAR-100, and SVHN, NDFA achieves the best DFA-based performance across ConvNet and AlexNet, with ConvNet gains over FDFA exceeding 4% and 6% on CIFAR-10 and CIFAR-100.These results substantially reduce the gap to BP.Source: §5.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6
  • On ResNet-32, NDFA leads DFA-based methods across datasets and module counts; on ViT-Tiny, it substantially improves DFA on CIFAR-10 and CIFAR-100.The results include robustness as the number of parallel modules increases.Source: §5.2 · p. 6, Table 3 · p. 6, Table 4 · p. 7
  • On ImageNet, NDFA improves ResNet-32 accuracy by more than 17% and outperforms DFA on both ResNet-32 and ViT-Tiny, where DFA fails to train effectively.The study evaluates both architectures on ImageNet.Source: §5.2 · p. 6, Table 5 · p. 7

5.3.1 Feedback analysis

  • On CIFAR-10 with ConvNet and AlexNet, NDFA improves weight, gradient, and subnetwork alignment and surrogate accuracy; these measures track distinct aspects of feedback quality.Weight alignment improves during training; subnetwork alignment can rise while gradient alignment remains stable.Source: Figure 2 · p. 7, §5.3.1 · pp. 7–8

5.3.2 Representation analysis

  • Linear probing on CIFAR-10 finds that NDFA learns more discriminative representations than DFA-based methods across AlexNet and ResNet-32, especially in middle and deep layers.Its representation trends are closer to BP.Source: Figure 3 · p. 7, §5.3.2 · p. 8

5.4 Parallel training efficiency

  • Despite extra surrogate computation, NDFA retains significant acceleration over BP and FDFA while outperforming DFA-based methods in accuracy.The efficiency comparison conservatively counts surrogate computation in backward time, although it could in principle overlap with subsequent layers’ forward pass.Source: §5.4 · p. 8, Table 6 · p. 9

5.5 Ablation study

  • Across ConvNet and ResNet-32 on CIFAR-10 and CIFAR-100, learnable feedback improves on fixed feedback, nonlinear feedback improves on linear feedback, and combining both performs best.The ablation attributes complementary gains to learning the feedback and modeling nonlinear downstream transformations.Source: §5.5 · p. 9, Table 7 · p. 9

5.6 Neural data similarity

  • On the Mante 2013 dataset, NDFA has lower Procrustes distance than BPTT in the high-accuracy regime, indicating closer resemblance to recorded neural data.Both methods progressively approach the neural data as task performance improves.Source: §5.6 · p. 9, Figure 4 · p. 9

6 Conclusion

NDFA replaces fixed DFA feedback with learnable nonlinear surrogate subnetworks and consistently outperforms DFA-based methods while retaining parallel training. It improves alignment with BP and neural-data similarity, though a gap to BP remains on very deep architectures and surrogate computation adds overhead.

6 Conclusion

  • NDFA constructs hidden-layer feedback through learnable nonlinear surrogate subnetworks to address DFA's limited capture of nonlinear downstream transformations.The paper's analysis identifies this feedback mismatch as a key limitation of fixed DFA matrices.Source: §6 · p. 9
  • NDFA consistently outperforms DFA-based methods while maintaining parallel training, with stronger BP alignment and representations more similar to biological neural recordings.The conclusion also reports more discriminative intermediate representations.Source: §6 · pp. 9–10
  • An observable performance gap to BP remains on very deep architectures, and surrogate subnetworks add computational overhead.Practical efficiency depends on parallel implementations and hardware support; rigorous convergence analysis is left for future work.Source: §6 · p. 10

A Implementation details for image classification experiments

Image-classification experiments cover four datasets and four architectures, with module-wise partitions varied across ResNet-32, ViT-Tiny, and ImageNet settings. Learning rates are tuned separately for each method to support fair comparisons.

A Implementation details for image classification experiments

  • Experiments use CIFAR-10, CIFAR-100, SVHN, and ImageNet, spanning small-scale natural images, fine-grained classes, street-view digits, and large-scale classification.ImageNet has 1,000 classes, approximately 1.2 million training images, and 50,000 validation images.Source: A Implementation details for image… · p. 14
  • Evaluation covers a ConvNet, AlexNet, ResNet-32, and ViT-Tiny; convolutional feature maps are pooled and flattened before entering the corresponding surrogate subnetwork's first linear layer.The described ConvNet uses convolutional layers, max pooling, and a 128-unit fully connected hidden layer.Source: A Implementation details for image… · p. 14
  • Learning rates are tuned separately for each method because optimal rates vary across learning rules.Experiments use four NVIDIA RTX A6000 GPUs; hyperparameters are summarized in Table 8.Source: A Implementation details for image… · pp. 14–15, Table 8 · p. 15
  • Module-wise partitions vary by task: ResNet-32 uses K = 4 or 16 on CIFAR datasets, ViT-Tiny uses K = 5 or 13, and both ImageNet backbones use K = 3.The partitions range from grouped stages or blocks to individual residual or transformer blocks.Source: A Implementation details for image… · p. 15

B Additional experiments

Additional experiments test NDFA on time-series classification and language modeling, and compare it with other backpropagation-free methods on CIFAR-10. NDFA performs comparably to BP on the two additional tasks and has the highest accuracy among the compared CIFAR-10 methods.

B Additional experiments

  • On FordA and WikiText-2, NDFA consistently outperforms DFA and achieves performance comparable to BP.The experiments use ViT-Tiny trained for 100 epochs; results are reported in Table 9.Source: B Additional experiments · p. 15, Table 9 · p. 15
  • On CIFAR-10 with a ConvNet, NDFA achieves the highest accuracy among Forward-Forward, LG-FG-A, DNI, PredSim, and other compared BP-free methods.The comparison uses Table 10; some baselines are reported from their original papers, and LG-FG-A uses a specialized architecture and twice NDFA's training budget.Source: B Additional experiments · p. 16, Table 10 · p. 16

C Additional analyses

Analyses report stronger Jacobian alignment for NDFA than DFA, stable decreases in network and surrogate losses, and improved accuracy with larger surrogate learning rates. Parameter-matched ablations favor nonlinear over linear surrogate subnetworks.

C Additional analyses

  • NDFA's layer-wise and average Jacobian cosine similarities consistently exceed DFA's across layers and training, with alignment increasing nearer the output.First-layer alignment drops after an early peak, while the average declines more moderately; Figure 5 compares layer-wise alignment with BP for NDFA and DFA.Source: C Additional analyses · p. 16, Figure 5 · p. 17
  • Main-network and three surrogate-subnetwork losses gradually decrease with very small standard deviations, indicating stable optimization across runs.Loss decreases for deeper surrogate subnetworks as well.Source: C Additional analyses · p. 16, Table 11 · p. 16
  • NDFA remains stable across surrogate learning rates, and larger surrogate learning rates yield improved accuracy.The authors suggest further learning-rate tuning could produce additional accuracy gains; results are in Table 12.Source: C Additional analyses · p. 17, Table 12 · p. 17
  • Parameter-matched nonlinear surrogate subnetworks outperform their linear counterparts in both learnable and fixed NDFA variants.The comparisons cover CIFAR-10 and CIFAR-100 using ConvNet and ResNet-32; results are reported in Table 13.Source: C Additional analyses · p. 17, Table 13 · p. 17

D Implementation details for neural data similarity

The neural-data experiment evaluates task performance and representational similarity on a context-dependent decision-making task using a CTRNN. NDFA applies learned nonlinear feedback to recurrent hidden states, and similarity is measured against recorded neural activity.

D Implementation details for neural data similarity

  • The Mante 2013 dataset tests context-dependent integration of sensory inputs and supports evaluation of both task performance and biological plausibility.Trials include fixation, target presentation, motion or color context cues, a delay, and a saccadic response.Source: D Implementation details for neural… · p. 18
  • A CTRNN combines input-to-hidden projection, recurrent hidden-state dynamics, and a hidden-to-output readout, with nonlinear activations.NDFA projects global output error to recurrent hidden states through a learned nonlinear surrogate pathway that maps hidden states to output space.Source: D Implementation details for neural… · p. 18
  • Neural similarity is measured by Procrustes distance between CTRNN hidden representations and recorded neural responses; lower distance means closer alignment after optimal orthogonal transformation.The evaluation compares BPTT and NDFA at matched task performance.Source: D Implementation details for neural… · p. 18

E Influence of surrogate subnetwork architecture

The architecture analysis finds that a two-layer, 128-dimensional surrogate is an effective lightweight choice, while increasing depth improves accuracy at a computational cost.

E Influence of surrogate subnetwork architecture

  • Increasing surrogate hidden dimension from 128 to 512 yields only marginal performance changes across ConvNet and ResNet-32 on CIFAR-10 and CIFAR-100.The authors therefore use hidden dimension 128 in the main experiments.Source: E Influence of surrogate subnetwork… · p. 19
  • Increasing surrogate depth consistently improves classification accuracy across both architectures and datasets, but raises backward computation cost and reduces training acceleration.Deeper surrogates gradually diminish NDFA’s parallel-training advantage over BP.Source: E Influence of surrogate subnetwork… · p. 19
  • The experiments use a two-layer MLP with hidden dimension 128 as a balance between feedback quality and computational efficiency.This is the lightweight surrogate setting adopted throughout the main experiments.Source: E Influence of surrogate subnetwork… · p. 19

F Surrogate subnetworks with layer-dependent capacities

This section discusses layer-dependent surrogate capacity and the computational and memory characteristics of NDFA surrogates, while noting that LLMs are not important, original, or non-standard components of the core method development.

F Surrogate subnetworks with layer-dependent capacities

  • Assigning shallower surrogates to earlier layers increased computational cost and lowered accuracy, while efficiency gains shrank as surrogate depth approached the BP path.The highest-capacity surrogate’s depth was already close to the seven downstream layers in the first AlexNet layer; the authors identify principled capacity design as future work.Source: F Surrogate subnetworks with… · pp. 19–20
  • In AlexNet on CIFAR-10, deeper surrogates for earlier layers improved accuracy; the 6→5→4→3→2→1 setting approached BP performance with a 1.10× backward-pass speedup.This setting substantially improved accuracy over the uniform two-layer surrogate configuration.Source: F Surrogate subnetworks with… · p. 19, Table 16 · p. 20

G Computational overhead

  • NDFA introduced negligible FLOPs overhead and used less peak GPU memory than BP on CIFAR-10 across ConvNet, AlexNet, and ResNet-32.The paper attributes lower memory use to not storing the end-to-end computational graph.Source: Table 17 · p. 20, G Computational overhead · p. 20
  • As ConvNet depth increased, NDFA’s additional computational overhead remained negligible relative to BP.The scaling analysis used CIFAR-10 ConvNets with progressively more convolutional layers and two fully connected layers.Source: Table 17 · p. 20, G Computational overhead · p. 20

NeurIPS Paper Checklist

  • The paper marks LLM involvement as N/A because LLMs are not important, original, or non-standard components of its core method development.Source: NeurIPS Paper Checklist · p. 27

Related papers

Loading 2610.36718v1…