Source-linked AI summary
Training of Physical Neural Networks
Ali Momeni, Babak Rahmani, Benjamin Scellier, Logan G. Wright, Peter L. McMahon, Clara C. Wanjura, Yuhang Li, Anas Skalli, Natalia G. Berloff, Tatsuhiro Onodera, Ilker Oguz, Francesco Morichetti, Philipp del Hougne, Manuel Le Gallo, Abu Sebastian, Azalia Mirhoseini, Cheng Zhang, Danijela Marković, Daniel Brunner, Christophe Moser, Sylvain Gigan, Florian Marquardt, Aydogan Ozcan, Julie Grollier, Andrea J. Liu, Demetri Psaltis, Andrea Alù, Romain Fleury
TL;DR
Growing AI models make traditional digital GPUs increasingly untenable because of energy, latency, and memory-processing separation costs. This paper reviews physical neural-network training, spanning backpropagation, approaches minimizing digital-electronic computation, and physical forward-model methods, while highlighting reproducibility, efficiency, and emerging local-learning results.
Problem
As AI models grow, reliance on traditional digital GPUs is becoming untenable because of high energy consumption and latency from separating memory and processing.
Method
The paper reviews PNN training broadly, covering backpropagation, methods that minimize digital-electronic computation, and in-silico or in-situ training using physical forward models.
Results
Local-learning proposals match global-learning accuracy on classification tasks with up to 100 classes, while in-silico training offers reproducibility and transparency compared with some in-situ systems.
Takeaways & Limitations
The reviewed training techniques indicate that PNNs can pursue scalable hardware implementations while retaining practical advantages such as reproducible training and local-learning performance.
Takeaways & Limitations
It remains unclear whether local-learning methods can reproduce backpropagation performance beyond small laboratory demonstrations, and optical hardware can limit analog scaling advantages.
Abstract
from arXiv · showhide
Physical neural networks (PNNs) are a class of neural-like networks that leverage the properties of physical systems to perform computation. While PNNs are so far a niche research area with small-scale laboratory demonstrations, they are arguably one of the most underappreciated important opportunities in modern AI. Could we train AI models 1000x larger than current ones? Could we do this and also have them perform inference locally and privately on edge devices, such as smartphones or sensors? Research over the past few years has shown that the answer to all these questions is likely "yes, with enough research": PNNs could one day radically change what is possible and practical for AI systems. To do this will however require rethinking both how AI models work, and how they are trained - primarily by considering the problems through the constraints of the underlying hardware physics. To train PNNs at large scale, many methods including backpropagation-based and backpropagation-free approaches are now being explored. These methods have various trade-offs, and so far no method has been shown to scale to the same scale and performance as the backpropagation algorithm widely used in deep learning today. However, this is rapidly changing, and a diverse ecosystem of training techniques provides clues for how PNNs may one day be utilized to create both more efficient realizations of current-scale AI models, and to enable unprecedented-scale models.
Introduction
The review frames physical neural networks as alternatives to increasingly untenable digital-GPU scaling and surveys training strategies for exploiting analog hardware. It covers backpropagation, approaches that reduce digital computation, local objectives, and implications for larger models.
- Context: AI systems increasingly support applications including personal assistants, healthcare, weather prediction, and material design.The passage presents these applications as part of AI’s broad influence and continuing expansion.
- Motivation: Traditional digital GPUs face high energy consumption, low throughput, high latency, and widening computation and data-transfer gaps as AI models grow.The memory–processing separation contributes to latency during training and inference.
- Scope: PNNs are reviewed from a domain-agnostic training perspective rather than only through technological advances, applications, or inference capabilities.The review spans optical, photonic, and analog-electronic platforms.
- Training approaches: The article examines backpropagation-based methods, approaches minimizing digital-electronic computation, and local training algorithms using system dynamics or local objectives.These strategies aim to learn analog-system parameters while reducing dependence on digital processing.
- Scaling: The review also discusses how analog computers may handle larger models and how their training can be made more efficient.This connects training-method design with scaling considerations for analog computing systems.
Historical Overview of Analog Computing and PNNs
Analog neural computing developed from early neural models and learning rules into physical implementations such as reservoirs, memristive systems, and neuromorphic platforms. These systems can reduce training complexity, but substrate dynamics and memory-dependent tasks impose practical constraints.
- Learning foundations: Backpropagation, Boltzmann Machines, Hebbian learning, and STDP represent successive learning approaches relevant to analog and neuromorphic computation.STDP uses temporally asymmetric spike correlations, while local event-based learning avoids extra energy for non-local transmission.
- Physical implementations: Physical implementations span optical, electronic, mechanical, exciton-polariton, multiplexed, and quantum reservoir systems.Quantum proposals seek to exploit Hilbert-space scaling for high-dimensional information mapping.
- Reservoir and extreme learning: Reservoir computing and Extreme Learning Machines treat the physical network as a nonlinear transformation whose output layer is trained.Only the final output weights are trained, while input and internal coupling weights remain constant.
- Reservoir and extreme learning: Reservoir computing can use recurrent physical dynamics for memory-dependent tasks or feed-forward systems for static inputs.Its physical substrate may be any sufficiently high-dimensional nonlinear system, with neurons embedded in its degrees of freedom.
- Weight implementation: Software weights provide flexibility and easy integration, whereas hardware weights can improve inference speed, latency, energy use, and in-memory scaling.Hardware weights require training strategies compatible with the physical device.
- Constraints: Memory-dependent tasks require inputs to drive the physical system according to its intrinsic timescales, which can be challenging for ultra-fast systems.Such systems entail fast modulation bandwidths.
Box1: PNNs
PNNs are physical systems with adjustable weights that perform computation using continuous physical variables, either by mimicking digital operations or by exploiting native physical transformations. Their training can also target noise-resilient inference.
- Definition: PNNs use adjustable physical weights θ to learn and perform a desired computing task, with analog encoding in part or all of the system.Their goal is to surpass digital hardware in performance or efficiency.
- PNN categories: Isomorphic PNNs implement strict operation-by-operation transformations such as matrix-vector multiplication, whereas broken-isomorphism PNNs directly train native physical transformations.Broken-isomorphism designs may gain advantages by abandoning mathematical isomorphism, but required universal-computation features can be unknown.
- Noise resilience: PNNs can be trained for deterministic inference that remains resilient to noise even when the signal-to-noise ratio is approximately 1 and quantum noise dominates.This is presented as an alternative training objective for physical systems.
Training Techniques of PNNs
PNN training spans in-silico optimization, hybrid physics-aware backpropagation, and related physical or digital-model approaches, each balancing fidelity, scalability, efficiency, and hardware realism.
- In-Silico Training: In-silico training digitally emulates and optimizes hardware degrees of freedom before deployment, enabling rapid, reproducible, and cost-effective exploration of scalable PNN designs.It also supports controlled study of theoretical architectures beyond current fabrication and material constraints.
- In-Silico Training: In-silico training is limited by inaccurate digital forward models, missing hardware phenomena, costly simulations, and computational overhead that can exceed conventional digital-network training.Discretizing continuous physics can cause computational requirements to grow sharply with physical size and input/output channels.
- Physics-aware BP Training: Physics-aware training uses the physical system for the forward pass and a differentiated digital model for the backward pass, requiring approximate gradient alignment rather than a perfect model.This hybrid approach can serve as a drop-in replacement for in-silico training while retaining benefits of in-situ learning.
- Physics-aware BP Training: Physics-aware training mitigates experimental noise and model mismatch while retaining backpropagation-like time scaling, but slow physical updates and complex digital models constrain large-scale use.Physics-informed machine learning is identified as a potential way to improve modeling of complex PNNs.
Box2: Inference and Training Processes in PNNs
PNNs combine physical inference with training methods that may use digital twins, feedback alignment, local learning, or physical dynamics, trading gradient fidelity against hardware efficiency and scalability.
- Inference and Training Processes in PNNs: PNNs train physical parameters to minimize task loss, commonly using a digital twin for gradient computation before deploying the optimized hardware for analog inference.The training loop executes a forward model, computes loss, backpropagates error, and updates parameters with gradient descent.
- Feedback Alignment: Feedback alignment avoids transporting forward weights backward by using fixed random feedback projections, reducing computational cost but often sacrificing performance.Direct feedback alignment broadcasts error signals directly to all layers, but physical implementations remain scarce and can degrade accuracy on convolutional and recurrent networks.
- Physical Local Learning: Local learning eliminates inter-layer gradient communication by letting each block compute and apply its own training signal, enabling distributed and asynchronous optimization.Local methods have matched global-learning accuracy on classification tasks with up to 100 classes in reported studies.
- Physical Local Learning: Dividing networks into independently trainable blocks can reduce GPU memory usage by 2.8 times, with memory generally decreasing by approximately a factor of k for k blocks.This is a compute-efficiency advantage of local parallelism rather than evidence that local methods match backpropagation at arbitrary scale.
- Physical Local Learning: Local learning remains far from demonstrating backpropagation-level performance beyond small laboratory scales, making guaranteed high-dimensional scaling an essential requirement.Exact matching is not required if efficiency improves substantially, but scaling evidence remains unresolved.
- Gradient-Free and Physical-Dynamics Training: Physical training methods can achieve gradient descent without a digital twin and may provide energy gains of 4 orders of magnitude compared with GPU-based neural networks.Other reported physical implementations include software-equivalent MNIST accuracy for binary equilibrium propagation and physical weight updates through equilibration.
Towards Implementation of Analog Efficient Large Models
Large-model scaling is motivating alternatives to digital hardware, including analog PNNs whose physical scaling may improve efficiency for sufficiently large models. Realizing this opportunity requires architectures and training strategies designed around physical constraints, while current PNNs remain behind digital systems.
- Large-model scaling: Current large-model training requires weeks or months of optimization on thousands of accelerators and trillions of tokens, while inference incurs substantial memory and floating-point costs.Model partitioning across accelerators also causes I/O overhead during inference.
- Efficiency strategies: Efficient digital-model strategies include sublinear attention, quantization, parameter-efficient fine-tuning, and hardware-aware attention implementations.Quantization-aware training has produced ternary-parameter models matching full-precision transformer performance while saving energy.
- Large-model scaling: Transformer attention has quadratic complexity in sequence length, while MLP layers impose overhead through very large weight matrices.These costs affect time, energy, and memory-related implementation demands as models and contexts grow.
- Analog large models: Plausible optoelectronic hardware could be roughly 100x more efficient than 2023 state-of-the-art digital electronics for Transformer models.The estimate accounts for splitting models across optical processing units and loading parameters and inputs from memory.
- Analog large models: Analog PNNs may offer better efficiency than digital systems at sufficiently large model scales despite analog-hardware overheads.Their different energy-scaling behavior is presented as the central potential advantage for future large models.
- Analog large models: Optical dot-product energy scales as 1/N, whereas digital operation cost is normally fixed at approximately 1.This scaling is identified as an optical advantage for matrix-vector multiplication.
- Limitations and design requirements: PNN scaling advantages are not guaranteed because algorithms, data movement, electronic modulation, memory access, and optical hardware can constrain them.Commercial viability may require energy improvements of thousands or millions of times over digital electronics.
- Training and scalability: PNN training may exploit physical phenomena for scalable local learning and provide scaling advantages in energy, size, and speed beyond inference gains.The passage frames training acceleration as promising but distinct from the more immediate inference opportunity.
Emerging PNN Technologies
Emerging PNN technologies span quantum, probabilistic, photonic, light–matter, sensor-integrated, and biological systems. Their promise is accompanied by constraints involving noise, error rates, input/output overhead, programmability, and scalability.
- Quantum, photonic, and probabilistic systems: Quantum, probabilistic, photonic, light–matter, and hybrid systems provide distinct hardware paradigms for PNN computation.Examples include soft quantum neurons, spatial photonic Ising machines, gain-based systems, and probabilistic bits for generative models.
- Quantum, photonic, and probabilistic systems: Optics offers spatial parallelism and dissipationless light propagation that can provide advantages over electronic systems.These properties motivate photonic optimizers and other optical PNN implementations.
- Intelligent sensors: Sensor-integrated PNNs can combine sensing with over-the-air computation, reducing the overhead of encoding inputs into and reading outputs from a separate physical system.Intelligent sensors can select task-relevant information during acquisition, yielding improvements in latency and other metrics.
- Intelligent sensors: Current intelligent-sensor training generally uses end-to-end error backpropagation through analog acquisition and digital post-processing, leaving alternative training paradigms unexplored.The physical layer can nevertheless be made programmable or nonlinear for task-reconfigurable sensing and richer acquisition-time computation.
- Integrated wave-based systems: Wave-based computing is increasingly being integrated with sensing, communications, and data storage across emerging architectures.Examples include dynamic metasurface antennas, privacy-preserving cameras, and computational imagers.
- Challenges and hybrid systems: Practical quantum and advanced-physics PNNs must address limited qubits, computational error rates, noise, learning-algorithm adaptation, and architectural scalability.Hybrid systems combining quantum or photonic units with classical elements are proposed as possible practical pathways.
- Biological PNNs: Biological systems, such as octopus-inspired soft robotic arms, may provide new reservoir-computing substrates but require training compatible with biological vulnerability.This extends PNN exploration beyond engineered physical devices.
Outlook
PNNs may serve applications from large generative models to adaptive edge sensors, but their training requirements differ across settings. The field therefore needs training methods that balance model independence, training efficiency, hardware utilization, and robustness rather than seeking one universal algorithm.
- Outlook: Different PNN applications impose different training constraints, from infrequent, energy-flexible server updates to frequent, power-limited edge retraining.The diversity of use cases makes selecting an appropriate method and understanding trade-offs the central open challenge.
- Outlook: An ideal method would be model-free, rely on few hardware assumptions, and let designers optimize PNN hardware for inference speed and energy.This would avoid requiring a mathematical description of the physical hardware during training.
- Outlook: Training should improve speed and energy costs over conventional neural-network training while using the PNN hardware itself rather than relying heavily on digital electronics.The comparison assumes the same task, accuracy, and training data.
- Outlook: Methods should exploit the full expressivity of PNN hardware and tolerate, or cheaply compensate for, copy-to-copy variation, drift, and noise.These requirements address both hardware capability and practical imperfections.
- Outlook: No known method simultaneously satisfies all desired properties, but recent approaches are expanding the trade-off space and may become more general, efficient, and robust.The authors anticipate further advances enabling practical and widespread PNN use.