Source-linked AI summary
Circuit realization and hardware linearization of monotone operator equilibrium networks
Thomas Chaffey
TL;DR
The paper asks which neural networks can be realized in analog hardware and develops a correspondence between resistor-diode circuits and ReLU equilibrium networks. It also presents hardware linearization for computing gradients directly in circuits, with a device-level simulation demonstrating training robustness to nonidealities.
Problem
It is unclear which classes of neural networks can be learned in analog hardware, despite interest in analog devices for computation and memory.
Source: §I · p. 1Method
The paper maps RTGD circuit behavior to ReLU monotone operator equilibrium networks and uses hardware linearization to compute circuit gradients directly.
Source: §I · p. 1Results
The circuit kernel corresponds to fixed points of a ReLU monotone operator equilibrium network, and device-level simulation indicates hardware linearization is effective and robust to device nonidealities.
Source: §IV · p. 5, §IX · p. 12Takeaways & Limitations
Cascading crossbar arrays with adjustable synaptic gains allows realization of network classes beyond the restricted class available from a single array.
Source: §IX · p. 11Parasitic reactance may limit the frequency at which hardware linearization can be performed.
Source: §VIII · p. 11
Abstract
from arXiv · showhide
It is shown that the port behavior of a resistor-diode network corresponds to the solution of a ReLU monotone operator equilibrium network (a neural network in the limit of infinite depth), giving a parsimonious construction of a neural network in analog hardware. We furthermore show that the gradient of such a circuit can be computed directly in hardware, using a procedure we call hardware linearization. This allows the network to be trained in hardware, which we demonstrate with a device-level circuit simulation. We extend the results to cascades of resistor-diode networks, which can be used to implement feedforward and other asymmetric networks. We finally show that different nonlinear elements give rise to different activation functions, and introduce the novel diode ReLU which is induced by a non-ideal diode model.
I. INTRODUCTION
The paper addresses which neural networks analog hardware can realize, establishing a correspondence between restricted passive circuits and ReLU equilibrium networks. It also introduces hardware linearization for circuit-gradient computation and develops the circuit-modeling framework used in the analysis.
I. INTRODUCTION
- The paper frames the unresolved question as which neural-network classes can be learned in analog hardware, amid interest in reducing transport and computation overhead through analog devices.Memristive crossbars can perform matrix-vector calculations in one operation.Source: §I · p. 1
- RTGD circuits correspond one-to-one with ReLU monotone operator equilibrium networks, whose fixed points represent weight-tied networks in the infinite-depth limit.A circuit forward pass applies voltage or current excitations and reads the responses; inference is theoretically instantaneous because the circuit is non-dynamic.Source: §I · p. 1
- Hardware linearization computes circuit gradients by replacing nonlinear elements with open- or short-circuit approximations, enabling direct device training rather than training a model first.For reciprocal circuits, the gradient computation simplifies further.Source: §I · pp. 1–2
- In reciprocal resistive networks, perturbing each device with a voltage or current and measuring the output gives the gradient; hardware measurements can also compensate for device nonidealities.The paper demonstrates this advantage in a preliminary device-level circuit simulation.Source: §I · p. 2
II. PRELIMINARIES
- The analysis studies non-dynamic RTGD circuits—LTI resistors, ideal transformers, ideal gyrators, and ideal diodes—and their driving-point current-voltage behavior.An m-port interacts with its environment through m terminal pairs.Source: §II · p. 2
- Element extraction models the circuit by replacing resistors with internal ports and deriving a hybrid-matrix description from Kirchhoff’s current and voltage laws.The extracted representation uses dual current-voltage variables for external ports and resistors.Source: §II · p. 3
III. FROM RESISTOR–DIODE NETWORKS TO KERNEL BEHAVIORS
The section derives resistor-diode circuit behavior by separating the diodes from the RTG network and representing each part with an operator relation. Reconnecting them yields a kernel equation for the circuit’s port behavior.
III. FROM RESISTOR–DIODE NETWORKS TO KERNEL BEHAVIORS
- Extracting the diodes separates an RTG network from a network of disconnected diodes, whose behaviors are represented by a hybrid matrix and a diagonal monotone operator, respectively.Reconnection gives the port behavior as the kernel of the sum of a positive semi-definite matrix and a diagonal monotone operator.Source: §III · p. 4
- For an RTGD circuit with n diodes, the theorem provides a partition of driving-point voltages and currents, diode variables, and matrices defining the circuit relation.The matrix H is positive semi-definite, as is D.Source: §III · p. 4
- The resulting kernel equation describes internal circuit currents and voltages and can be solved by building the circuit, applying driving-point inputs, and measuring outputs.The paper introduces an optimization algorithm with steps corresponding to neural-network layers as another way to solve the equations.Source: §III · p. 4
IV. FROM KERNEL BEHAVIORS TO EQUILIBRIUM
The forward/backward iteration turns the circuit kernel equation into a ReLU network update. Its fixed point is a monotone operator equilibrium network, and every RTGD network has a corresponding ReLU monDEQ, with a converse circuit construction.
IV. FROM KERNEL BEHAVIORS TO EQUILIBRIUM
- The forward/backward iteration applies a ReLU update with weight matrix (I − αH), and its fixed point is the infinite-depth limit of a weight-tied network.When H ⪰ 0 and ψ consists of ideal diodes, the result is a ReLU monDEQ.Source: §IV · pp. 4–5
- A monDEQ maps input u to a fixed point when the iteration has a fixed point; contraction is a sufficient condition guaranteeing existence.The paper’s examples satisfy the stated sufficient condition.Source: §IV · p. 5
- Different algorithms can solve the same kernel equation and yield the same fixed points, while having different convergence properties.The paper therefore treats the circuit kernel equations as the fundamental object.Source: §IV · p. 5
- Every RTGD circuit realizes a ReLU monotone operator equilibrium network, and every such network has a corresponding RTGD circuit.The circuit kernel is the set of fixed points of the network.Source: §IV · p. 5
A. Example 1: Crossbar arrays
The crossbar example configures a resistor array as an equilibrium layer through its port assignments. Its circuit structure forces Stieltjes matrices, restricting the maps it can learn.
A. Example 1: Crossbar arrays
- A crossbar array can implement an equilibrium layer under a particular port configuration, but represents only a restricted class of monDEQs.The array is a resistor network with a fully connected bipartite graph between input and output nodes.Source: A. Example 1: Crossbar arrays · p. 5
- The port partition assigns voltage inputs to tree edges and current inputs to cotree edges, then circuit cut-set and loop analysis yields the kernel representation.The illustrated spanning tree contains the input ports, diodes, and one resistive edge.Source: A. Example 1: Crossbar arrays · p. 5, Figure 4 · p. 5
- For the crossbar circuit, D11 is zero while D22 and H are Stieltjes matrices.The characterization assumes no resistive edge is open or short.Source: A. Example 1: Crossbar arrays · p. 6
- The diode subcircuit’s admittance matrix H is a weighted Laplacian, and full rank plus symmetry makes it Stieltjes.Its diagonal entries are positive and its off-diagonal entries are nonpositive.Source: A. Example 1: Crossbar arrays · p. 6
- Because H and D22 are Stieltjes, this crossbar construction restricts the class of maps that can be learned.The paper proposes interconnecting multiple crossbars to alleviate this limitation.Source: A. Example 1: Crossbar arrays · p. 6
V. HARDWARE LINEARIZATION OF EQUILIBRIUM NETWORKS
The paper computes equilibrium-network gradients by hardware-linearizing diodes at an operating point and solving the resulting implicit equation in hardware. For reciprocal resistive circuits, it also computes parameter-matrix derivatives in hardware.
V. HARDWARE LINEARIZATION OF EQUILIBRIUM NETWORKS
- At the ideal-diode switching point, the analogous result concerns a directional derivative rather than the gradient.The paper sets this case aside, noting the switching point is measure zero in the ideal diode operator’s graph.Source: §V · pp. 6–7
- At an operating point where no diode is switching, the hardware-linearized circuit computes the gradient of the network output with respect to its parameters.The gradient is obtained by differentiating the equilibrium condition rather than backpropagating through layers.Source: §V · pp. 6–7
- Hardware linearization replaces each diode with a local linearized element so the implicit gradient equation can be solved in hardware in a single step.The hardware-linearized circuit is built at the specified operating point and with selected offsets.Source: §V · p. 7
- For reciprocal resistive circuits with resistances and conductances as parameters, the required matrix derivatives can also be computed in hardware.The procedure applies unit voltage across a resistor or unit current through a conductor, then reads outputs and diode responses.Source: §V · p. 7
- The computation requires precomputed derivatives of H, B, C, and D except in the stated reciprocal resistive special case.In that special case, the parameter-matrix derivatives can also be obtained in hardware.Source: §V · p. 7
- Reciprocity makes the hybrid matrix signature symmetric, and the gradient method uses this property to compute circuit gradients through element excitations.The method is related to in-situ backpropagation in photonic networks and adjoint sensitivity analysis.Source: §V · pp. 7–8
- Hardware linearization computes the circuit-output gradient precisely, unlike contrastive learning’s contrastive-cost gradient and equilibrium propagation’s small-nudging approximation.The method assumes each diode can be switched; pairing diodes with CMOS circuitry is suggested as a possible implementation.Source: §V · p. 8
VI. CASCADE NETWORKS
Cascading RTGD circuits with amplifier gains as synaptic weights yields a lower-triangular network structure. This construction addresses the symmetry restriction of a single reciprocal crossbar and includes feedforward ReLU networks.
VI. CASCADE NETWORKS
- Cascades of RTGD subcircuits connected by ideal amplifiers can produce a lower-triangular H, addressing the signature-symmetry restriction of circuits without gyrators.The amplifier gains σ are treated as synaptic weights.Source: §VI · p. 8, Figure 5 · p. 8
- Cascades include feedforward ReLU networks and Deep Resistive Networks, which are stated to be universal approximators of continuous functions.Cascades may be built from several circuits or implemented sequentially with one reconfigurable array.Source: §VI · p. 8
- A cascade can pass only selected outputs from one subcircuit to the next, provided the connected inputs and outputs have compatible voltage or current types.The resulting kernel behavior is formed by substituting each preceding subcircuit’s outputs into the next circuit’s kernel equations.Source: §VI · pp. 8–9
A. Hardware linearization of cascade networks
For cascaded RTGD circuits, hardware linearization propagates gradients by linearizing the downstream circuit and feeding it the upstream gradient. The measured output gives the derivative of the downstream output with respect to an upstream parameter.
A. Hardware linearization of cascade networks
- To obtain a downstream output’s derivative with respect to an upstream parameter, first hardware linearize the upstream circuit and then apply its gradient as input to the linearized downstream circuit.The downstream circuit’s output is the desired derivative.Source: A. Hardware linearization of cascade… · p. 9
- Hardware linearization at the input u provides a circuit that can be driven with the input derivative du/dϑ.This is the first step in the theorem’s procedure.Source: A. Hardware linearization of cascade… · p. 9
- The output of the hardware-linearized circuit is dy/dϑ, obtained by differentiating the circuit behavior at its operating point.The proof differentiates the kernel behavior with respect to ϑ.Source: A. Hardware linearization of cascade… · p. 9
B. Example 2: feedforward networks
A crossbar configuration with diagonal H reduces the kernel equation to a ReLU layer, and cascading such layers realizes arbitrary feedforward ReLU networks. The approach also accommodates equilibrium layers.
B. Example 2: feedforward networks
- With diagonal H, the crossbar kernel equation can be solved explicitly as a ReLU, and cascading layers with positive and negative synaptic weights realizes arbitrary feedforward ReLU networks.The configuration uses p ports for voltage inputs/current outputs and q ports for current inputs/voltage outputs.Source: B. Example 2: feedforward networks · pp. 9–10, Figure 6 · p. 10
- Neglecting input and output port currents makes the output voltage equal the diode voltage, yielding a ReLU layer with full entrywise negative weights −B.The matrix H^-1 is diagonal, and each reverse-biased diode entry is unaffected by scalar multiplication.Source: B. Example 2: feedforward networks · p. 10
- Duplicating each layer’s input and negating one copy removes the negative-weight restriction, making the combined weight matrix arbitrary.The construction combines the two input copies through the matrices B1 and B2.Source: B. Example 2: feedforward networks · p. 10
- Adding crossbar arrays with the Section IV-A port configuration also implements ReLU neural networks with equilibrium layers.The feedforward configuration is presented as a special case of the broader theory.Source: B. Example 2: feedforward networks · pp. 9–10
VII. OTHER ACTIVATIONS
The section turns from ideal diodes and ReLU to other activation implementations, examining non-ideal diodes and a saturation activation.
VII. OTHER ACTIVATIONS
- The section shifts from ideal diodes and their ReLU counterpart to non-ideal diode behavior.The non-ideal diode is examined first.Source: §VII · p. 10
- A circuit implementation for a saturation activation is also presented.This follows the discussion of non-ideal diodes.Source: §VII · p. 10
- The section covers both a change in diode model and a separate saturation activation implementation.The passage identifies these as the section’s two topics.Source: §VII · p. 10
A. Non-ideal diodes and the “diode ReLU”
The paper extends its activation implementations beyond ideal-diode ReLU: the Shockley diode yields a smoothed diode ReLU, while opposing Zener diodes realize saturation monDEQs.
A. Non-ideal diodes and the “diode ReLU”
- The Shockley diode model yields a smoothed ReLU activation called the diode ReLU.Its dual current-to-voltage mapping can be implemented using the admittance of a reverse-biased diode.Source: A. Non-ideal diodes and the “diode ReLU” · p. 10
- The diode ReLU is plotted using vT = 25.852 mV, n = 1, and is = 1 pA.These are the figure’s stated parameter values.Source: Figure 7 · p. 10
- Replacing ideal diodes with the Shockley equation gives a realization theorem for monDEQs with diode ReLU activations.The passage says the remaining theory is unchanged.Source: A. Non-ideal diodes and the “diode ReLU” · p. 10
B. Saturation
- The resolvent of the admittance from opposing Zener diodes gives the saturation activation.The activation is illustrated in Figure 8.Source: B. Saturation · pp. 10–11, Figure 8 · p. 11
- Replacing forward-biased diodes with opposing Zeners and reverse-biased diodes with anti-parallel CRDs realizes saturation monDEQs in analog hardware.The electrical dual of the opposing Zener pair is a parallel interconnection of a forward-biased and reverse-biased CRD.Source: B. Saturation · p. 11
VIII. COMPUTATIONAL EXPERIMENT
The computational experiment finds that hardware linearization trains a crossbar cascade comparably under realistic diode models and shows smaller error transients than nominal-model gradients under device and update errors. The conclusions connect passive nonlinear circuits to equilibrium networks and identify further activation realizations and research directions.
VIII. COMPUTATIONAL EXPERIMENT
- With realistic diode models, hardware-linearization and idealized-model training curves are almost identical.The simulation uses a 50 V breakdown voltage, reverse-bias saturation current of 1×10^-13 A, and ideality factor 1.05.Source: §VIII · p. 11
- With resistance and gradient-update errors, equilibrium backpropagation on the nominal model shows much larger error transients than hardware linearization on the true circuit model.Both methods eventually outperform the ideal, noise-free procedure from the preceding experiment.Source: §VIII · p. 11
IX. CONCLUSIONS
- Certain passive nonlinear networks correspond to equilibrium neural networks, and cascaded crossbar arrays with adjustable synaptic gains realize a class with universal approximation properties.A single crossbar array realizes only a restricted class of networks.Source: §IX · p. 11
- The paper gives saturation activation realizations and introduces diode ReLU, induced by non-ideal diodes.Source: §IX · pp. 11–12
- Existing circuit simulators do not scale well enough for learning applications, motivating future demonstrations of hardware linearization in machine learning.Source: §IX · p. 12
- Future directions include realizing dynamic neural networks as nonlinear RLC networks and extending the results beyond Euclidean monotonicity.Source: §IX · p. 12
APPENDIX: AUXILIARY RESULTS
The appendix establishes local differentiability of circuit solutions away from diode switching points and develops a tangent-cone argument for the parameter derivative.
APPENDIX: AUXILIARY RESULTS
- With positive-definite Hϑ and no diode at its switching point, the solution derivative is characterized through a tangent-cone inclusion.The proof reduces the derivative claim to that inclusion after differentiating the local solution relation.Source: APPENDIX: AUXILIARY RESULTS · pp. 12–13
- The proof constructs a local C1 solution branch by using the linear equation for the remaining variables and the assumed invertibility of Hϑ.Source: APPENDIX: AUXILIARY RESULTS · p. 12
- Within an operating region, conducting and non-conducting diode variables are fixed at zero, and the remaining variables satisfy a linear equation.Source: APPENDIX: AUXILIARY RESULTS · p. 12