PaperlayerView exact revision on arXiv

Source-linked AI summary

An all-transistor physical-learning transformer can be trained to perform in-context learning

Michael C. Abbott, Benjamin B. Machta

arXiv:2610.12571v1cs.LGcond-mat.soft

TL;DR

Physical-learning circuits can mimic MLPs, but multiplication for transformer attention interrupts reverse error propagation. The paper adds explicit backward elements in dyadic circuits and builds MOSFET-based transformer-like attention. It presents the circuit as learnable with approximate reverse components and uses an in-context learning task as a test.

  • Problem

    Physical-learning circuits can represent MLPs, but transformer attention’s multiplications can block error signals from propagating backward through ideal voltage multipliers.

    Source: §1 · pp. 1–2

  • Method

    The paper uses dyadic circuits with explicit backward error elements and constructs MOSFET-based attention with floating loss elements and local adaptive optimization.

    Source: §1 · p. 2

  • Results

    The proposed transformer-like circuit uses realistic approximate reverse components that allow learning, and the paper frames a difficult in-context learning task as a test.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work presents transformer-like architecture and reverse-pass training as extensions of physical learning beyond MLPs.

    Source: §5 · pp. 16–17

    The circuit is an idealized sketch rather than a realistic engineering design, and its simulations omit limitations of the built components.

    Source: §3.5 · p. 14

Abstract

from arXiv · show

Most systems trained by equilibrium propagation or coupled learning are in the same class as multi-layer perceptrons: while they may be universal function approximators, they struggle to learn associative tasks which are easy for transformers. Equilibrium propagation relies on a bidirectional flow of information through the physical system between input and output. The reverse flow is broken by more complex nonlinear systems, such as those performing multiplication which is needed to express dot-product attention. Here we explicitly restore this bidirectional flow by introducing extra circuit elements for this purpose into a double or dyadic circuit. We then use these ideas to construct an attention mechanism exploiting the exponential current law of sub-threshold MOSFET devices. We show that in simulations this circuit can be trained to perform a simple in-context learning task, despite using realistic reverse components which transmit only an approximate gradient. For this training to succeed we need cross-entropy loss (not mean squared error). We generate nudges appropriate for this loss using a novel floating circuit element, based on the same MOSFET sub-circuit used for softmax. We train the resistors in this circuit using the Lion optimiser, which can be implemented using one capacitor per parameter to store local memory of past gradient estimates.

1 Introduction

Physical-learning circuits can model MLPs, but transformer attention’s multiplications interrupt the reverse error flow needed for physical training. The paper restores that flow in dyadic circuits and applies them to transformer-like attention, local training rules, and an in-context learning task.

1 Introduction

  • Transformer attention multiplies inputs, but an ideal bilinear voltage-controlled current source blocks error propagation to earlier circuit elements.Reciprocal resistor and diode networks can mimic MLPs, which are universal function approximators, while transformer architectures dominate current neural networks.Source: §1 · pp. 1–2
  • Dyadic circuits carry forward computation in paired node-voltage means and error signals in their differences, with added elements passing error backward through nonlinear components.The steady state is generally a saddle point of a dyadic potential rather than an energy minimum.Source: §1 · p. 2
  • The proposed transformer-like attention exploits MOSFET exponential nonlinearity, using a bump circuit analogous to Bahdanau attention and a shared-tail-current softmax.Realistic components approximate the dyadic double of a MOSFET, and the learnable parameters are adjustable resistors.Source: §1 · p. 2
  • The paper implements cross-entropy as a floating circuit element and describes local adaptive optimization, including Adam and Lion.These optimizers retain memory of past updates for each parameter, preserving physical-learning locality.Source: §1 · p. 2
  • The test task matches the final input character to earlier characters, retrieves nearby input, and returns it, an operation associated with learned transformer induction heads.The task uses character embeddings and is described as difficult for an MLP despite smaller input matrices than MNIST images.Source: §1 · p. 2
  • The introduction conjectures that systems with pairwise reciprocal interactions, whose states minimize energy, cannot efficiently learn in-context learning tasks.Source: §1 · p. 2

2 Circuits adapted for physical learning

This section develops physical-learning circuits that preserve local equilibrium-propagation updates while extending them to paired or dyadic circuits, circuit-implemented losses, and nonlinear multiplication. It describes approximate MOSFET implementations that exploit exponential voltage or current dependence for attention and local optimization.

2 Circuits adapted for physical learning

  • The MLP circuit encodes adjustable parameters as resistors and implements ReLU with oppositely oriented diodes, while inverted inputs accommodate the inability of resistors to have negative resistance.Its output resistors cannot amplify, so the inputs are amplified before being supplied to the circuit.Source: §2 · p. 3, Figure 1 · p. 3

2.1 Double and dyadic circuits

  • The doubled circuit estimates resistor gradients from paired voltage differences; for resistors and diodes the estimate is exact at sufficiently small nudges that do not switch a diode.Dyadic circuits following make the estimate valid for any nudge strength, but require diodes whose paired states switch together.Source: §2.1 · pp. 4–5

2.2 Loss as a circuit component

  • A floating circuit element can generate the nudge within the circuit, allowing input and target voltages to be applied together and gradients to be estimated after settling.For MSE, the element is a voltage-controlled current source; for logit cross-entropy, the nudge uses a MOSFET bank circuit to implement softmax.Source: §2.2 · p. 6

2.3 Perfect voltage multiplier

  • A dyadic multiplier adds reverse-facing multiplier components to restore the output-to-input influence needed for learning through multiplication.The construction is a special property of multiplication: a dyadic circuit can be specified by the rules of, but is not guaranteed to use the same components as the original circuit.Source: §2.3 · pp. 7–8

2.4 Aside on automatic differentiation

  • The reverse multiplier mirrors the pullback for multiplication: the output error signal is multiplied by an input, while a resistor’s dyadic element is simply another resistor because its current law is multiplication.Independent paired diodes are not an exact dyadic double because their effective resistances should depend on the same voltage difference.Source: §2.4 · p. 8

2.5 Ideal sub-threshold MOSFET

  • Sub-threshold MOSFETs are used for their exponential current-voltage relationship, which the paper uses to implement attention.The passage contrasts this regime with using triode-regime MOSFETs as slightly nonlinear adjustable resistors.Source: §2.5 · p. 8
  • Crude approximately dyadic circuit blocks, though much worse than O(β^2), are sufficient for learning despite the complexity of the MOSFET reverse-error component.Current-source circuits use transistor gates to control how current is shared among drains, with exponential device laws providing useful functional forms.Source: §2.5 · p. 9

2.6 Adaptive, local, optimisation rules

  • Lion stores one number per parameter, and its scale-invariant adaptive updates use a learning rate measured in conductance, suiting circuits with exponential voltage dependence.Diagonal-Hessian adaptive rules preserve physical-learning locality; the section points to Lion as a simpler implementation than Adam.Source: §2.6 · p. 9

3 Transistor transformer circuit

The section builds a transistor-based attention circuit using a bump similarity score, a softmax-like bank, and differential-pair value mixing. A doubled circuit supplies approximate reverse-gradient signals, but the design remains an idealised sketch.

3 Transistor transformer circuit

  • The proposed transformer-like circuit replaces dot-product attention with a bump-based similarity score and uses approximate reverse elements to train resistor weights.The key approximation assumes positive diagonal key weights; the learned weights are implemented with resistor arrays.Source: §3 · p. 10

3.1 Scores Sst as currents from “bump”

  • Delbrück’s bump circuit produces a current score that peaks when its two inputs agree, while Gilbert-cell reverse components provide an approximate gradient that still supports movement toward or away from the peak.The approximation differs substantially from the exact gradient away from the bump centre.Source: Figure 5 · p. 10, §3.1 · p. 11

3.2 Attention Ast as currents from “bank”

  • A transistor bank distributes fixed tail current among outputs according to input voltages, implementing softmax-like attention with a reverse block that closely follows ideal dyadic behaviour.The bank’s output currents represent the attention weights.Source: §3.2 · pp. 11–12, Figure 6 · p. 11

3.3 Current to voltage amplifier

  • A resistor-feedback transimpedance amplifier converts bump output currents into voltage inputs for the bank, while also addressing impedance and sign incompatibilities between the stages.The amplifier’s effective gain is set by its resistor and voltage-controlled current source.Source: §3.3 · p. 12, Figure 7 · p. 12

3.4 Value mixing with differential pairs

  • Differential pairs use bank currents to mix token values across time, producing the softmax-weighted combination; their approximate reverse signal reaches the bank current but not the token inputs.The mixer’s tanh nonlinearity is described as harmless when input amplification is controlled.Source: §3.4 · pp. 13–14, Figure 8 · p. 13

3.5 Limitations

  • The circuit is an idealised sketch rather than a realistic engineering design, and its simulations omit limitations of the built components.The authors note that many parameters are tuned and that more realistic, elegant, lower-component designs may exist.Source: §3.5 · p. 14
  • The doubled circuit is larger by an O(1) factor rather than scaling with parameter count, and the model has 444 learned parameters.The 444 parameters exceed the breadboard self-learning circuits of by an order of magnitude but are fewer than the 642 TaOx cells used for an MLP with 1000 parameters in [38].Source: §3.5 · p. 14, Table 1 · p. 14

4 Minimal in-context learning task

The paper tests whether a small transformer-like circuit can learn associative recall from examples by matching a query character and retrieving its paired number. Simulations show successful training with exact and approximate gradients, with cross-entropy and an adaptive optimiser required for training to succeed.

4 Minimal in-context learning task

  • The associative recall task predicts the number paired with the last query letter by finding its earlier matching letter and copying the following number.Many distinct letters are used so in-context learning is preferred over memorizing a fixed letter-number mapping.Source: §4 · p. 15
  • Concatenating each letter and number into one token makes the studied task solvable with a single attention layer, unlike alternating letter and number tokens, which require two layers or one induction head.The illustrated task uses D = 12 and T = 5.Source: §4 · p. 15, Figure 9 · p. 15
  • Training succeeds with exact or approximate gradients, but requires cross-entropy loss and an adaptive optimiser such as Adam or Lion rather than mean squared error and gradient descent.The tested resistors are randomly initialized log-uniformly from 10^3 to 10^8 Ω and constrained to that range during training.Source: §4 · p. 15, Figure 10 · p. 16
  • Evaluation uses new embedding vectors outside the training set, with 256 test examples generated from 200 new random vectors.Figure 10 tracks both final-output accuracy and whether the highest score occurs at the time containing the solution.Source: §4 · p. 15, Figure 10 · p. 16

5 Conclusion

The paper presents a transformer-like architecture for physical learning and uses it to address an in-context learning task that reciprocal physical-learning systems are conjectured to struggle to learn efficiently. Its conclusion connects this design to broader analogue-learning approaches and derives doubled circuit elements that carry forward computation and error signals.

5 Conclusion

  • The paper constructs a transformer-like architecture for physical learning, extending physical-learning translations beyond MLPs and convolutional nets.The authors frame this as engineering a known architecture and translating its reverse pass and training law to another substrate.Source: §5 · p. 16
  • Toy in-context learning tasks may diagnose post-MLP learning abilities because associative recall appears difficult for feed-forward networks such as MLPs.The authors connect this task to their interest in learning mechanisms relevant to biology, while explicitly rejecting a literal brain-as-transformer analogy.Source: §5 · p. 16
  • The authors conjecture that reciprocal pairwise systems that minimize energy cannot efficiently learn ICL tasks, while emphasizing that inductive-bias claims are hard to prove.They distinguish expressivity from learnability: MLPs and circuit relatives can express any function, but that does not establish which tasks they learn.Source: §5 · pp. 16–17
  • Doubled circuits add explicit reverse propagation for non-reciprocal circuits, using realistic components that transmit approximate gradients at an order-1 factor more reverse elements.The text contrasts this with a larger circuit that can preserve forward properties while implementing explicit reverse propagation.Source: §5 · p. 17

5.1 The broader landscape of unconventional learning

  • Analogue hardware may offer energy and time advantages, while hybrid training can compensate for physical defects but is thought to remain largely in the MLP learning class.The paper notes that a voltage multiplier can require fewer transistors than a digital floating-point multiplier; hybrid approaches can also be applied to the transistor circuit described here.Source: §5.1 · p. 17
  • Transformer-like Hopfield models often use energy differently from reciprocal steady-state circuits, and the cited CET model learns memorization but not ICL in the authors’ adaptation.The CET model’s output is described as minimizing over output and hidden degrees of freedom while holding the input fixed.Source: §5.1 · pp. 17–18

A Deriving current rules for bidirectional information flow

  • The dyadic formalism doubles a system’s state so the mean of paired voltages gives the forward solution and their difference carries an error signal for gradient estimation.It describes stationary states using a dyadic potential and extends the construction to circuits decomposed into components acting on local subsets of nodes and parameters.Source: A Deriving current rules for… · pp. 23–24

A.1 Resistor

  • A resistor’s dyadic implementation consists of two resistors sharing the original conductance, corresponding to the difference of two single-copy potentials.The original resistor current is tied to the voltage difference across its terminals.Source: §A.1 · pp. 24–25

A.2 Gilbert cell — figure 3

The appendix develops doubled implementations for nonlinear circuit elements, from Gilbert cells and diodes to MOSFET-based bump and bank blocks. These constructions approximate reverse currents, and the bank variant described here was found to learn ICL but not memorization.

A.3 Diode

  • The diode’s doubled implementation makes both paths respond to the mean voltage difference, whereas independent diodes can introduce a discrepancy comparable to the error signal.The discrepancy occurs when one diode is on and the other is off.Source: §A.3 · p. 26

A.4 MOSFET

  • The dyadic MOSFET component includes current flowing back from the gate, providing a reverse current path through the device.The appendix derives six currents for the dyadic MOSFET.Source: §A.4 · p. 26

A.5 Bump — figure 5

  • The bump circuit uses a physical approximation that linearizes the response around the top of the bump.The bump output current is given by a bump-shaped function of the voltage difference, with fixed positive τ and Itail.Source: §A.5 · pp. 26–27

A.6 Bank — figure 6

  • The bank approximates forward currents using each paired voltage separately and uses a more complicated reverse circuit after a crude variant learned ICL but not memorization.The chosen circuit pulls current from all input pairs; the authors report that the TIA stage handles these currents well.Source: §A.6 · p. 27

A.7 Pair — figure 8

The pair block is a bank variant with the tail current supplied as an input. Its doubled circuit uses paired input nodes and a mean-based forward response, with a simplified reverse construction.

A.7 Pair — figure 8

  • The pair block takes v1, v2 = −v1, and i3 as inputs, with output currents at nodes 4 and 5.It is introduced as a variant of the bank in which tail current is one of the inputs.Source: §A.7 · pp. 27–28
  • In the doubled pair, the output nodes sit between two copies of the inputs, and the forward response depends on the mean of paired voltages.The figure description also notes independently created current paths to reduce interference.Source: Figure S2 · p. 28
  • The reverse construction discards one layer of terms and linearizes tanh around v1 = v2.Source: §A.7 · p. 28

B Cross-entropy loss as a floating circuit element

The appendix describes a floating circuit element that generates cross-entropy nudges using a MOSFET bank, alongside a discussion of adaptive optimization. It frames memory-based optimization as addressing both gradient conditioning and noise smoothing in the single-sample physical-training setting.

B Cross-entropy loss as a floating circuit element

  • The appendix presents loss elements for both MSE and cross-entropy, with the cross-entropy circuit using a bank whose outputs are converted to voltages for comparison with targets.The MSE element reads mean outputs and computes its nudge current by comparing them to the target.Source: B Cross-entropy loss as a floating… · pp. 28–29, Figure S3 · p. 29
  • The floating loss circuit implements the cross-entropy gradient by forming mean outputs, adjusting softmax temperature, and comparing bank-generated softmax currents with target probabilities.The circuit uses a fixed total tail current in the bank, and amplification allows the softmax temperature to be adjusted.Source: B Cross-entropy loss as a floating… · p. 29

C Adaptive optimisation rules with memory

  • Adaptive optimization is motivated as combining landscape preconditioning with smoothing noisy gradients, especially since physical training naturally uses a batch size of one.The text describes the gradient estimate as noisy and explains that moving averages of past estimates can smooth it.Source: C Adaptive optimisation rules with… · p. 30

C.1 Adam [10]

Adam maintains two per-parameter memories of the gradient: estimates of its first and second moments. Its update is essentially the first-moment estimate divided by the square root of the second, with corrections for early steps.

C.1 Adam [10]

  • Adam stores first- and second-moment estimates of each parameter’s gradient, updated using decay rates β1 and β2.The typical values given are β1 = 0.9 and β2 = 0.999.Source: §C.1 · p. 30
  • Adam’s update is essentially −ηm/√v, with additional corrections for starting both memories at zero.The corrections apply only for small t.Source: §C.1 · p. 30
  • Scaling a gradient by 10 leaves Adam’s parameter update unchanged because its first and second moments scale by 10 and 100, respectively.Source: §C.1 · p. 30

C.2 Lion [11]

Lion uses a single memory per parameter to form fixed-size signed updates. The paper uses this rule for physical training and describes a possible capacitor-based circuit implementation, which it does not simulate.

C.2 Lion [11]

  • Lion stores one memory ψ per scalar parameter and updates it using the gradient at each step.Source: §C.2 · p. 31
  • With β2 = 0.99, ψ averages about 100 past gradients; β1 = 0.9 lets a disagreeing gradient about 10 times larger move the parameter immediately.Source: §C.2 · p. 31
  • Lion uses fixed-size parameter steps of ±η, and multiplying the gradient by 10 does not change the update.When β1 = β2, Lion simplifies to using the memory directly for the update.Source: §C.2 · p. 31
  • The physical training uses Lion; a proposed local implementation stores ψ in one capacitor per parameter and updates conductance according to its sign.The suggested RC time is about 100 training examples.Source: §C.2 · p. 31
  • The capacitor-based Lion circuit is not simulated because the simulations are restricted to DC.Source: §C.2 · p. 31

D Transitions between ICL and memorisation

The authors examine when their one-layer circuit learns in-context learning (ICL) rather than memorizing class-label associations. In this task, increasing the number of classes, burstiness, or noise makes memorization harder and increases the likelihood of ICL.

D Transitions between ICL and memorisation

D.1 Single circuits and the VF algorithm

Training without the double circuit can partly fit output values for memorization, but does not train the key and query resistors needed for ICL. A substantially larger resistor MLP also memorizes K=10 but was not observed to learn ICL.

D.1 Single circuits and the VF algorithm

  • The VF algorithm applies the equilibrium-propagation update despite the absence of an energy function.Source: §D.1 · p. 32
  • Without doubling, VF training partly fits output-value resistors for K=10 memorization but learns nothing at K=100 because gradients do not reach key and query resistors.Source: Figure S5 · p. 33, §D.1 · p. 33

D.2 Memorisation in an MLP-like circuit

  • The 4,352-resistor MLP, about an order of magnitude larger than the transformer, memorizes K=10 but was not seen to learn ICL.Source: §D.2 · p. 33

D.3 Convergent Energy Transformer [63]

The adapted Convergent Energy Transformer memorizes the small-class task but remains at chance on the larger in-context learning task. The section also compares artificial neural networks and describes the solver used to validate the trained circuit.

D.3 Convergent Energy Transformer [63]

  • For the CET, the authors adapted the model from and evaluated its forward computation with constrained hidden states.The setup used matrix inputs and vector outputs.Source: §D.3 · pp. 33–34
  • The adapted CET memorized K=10 classes but achieved chance ICL accuracy at K=100, matching the resistor MLP’s failure to learn ICL.The experiment used D=12, Dout=6, T=5, and varied Dmem and γ.Source: §D.3 · p. 34, Figure S6 · p. 34

D.4 Artificial neural networks

  • Ordinary neural networks can be encouraged to perform ICL by increasing K, while the compared RNNs showed intermediate performance.The comparison used the same task, and the models had widely varying parameter counts.Source: §D.4 · p. 34, Figure S7 · p. 35

E Nonlinear circuit solver

  • Linear circuits require one matrix inversion, while piecewise-linear circuits can use the fast algorithm of [75] only for diodes connected to fixed voltages.Source: E Nonlinear circuit solver · pp. 34–35
  • For nonlinear MOSFET circuits, a custom Newton–Raphson solver changes variables to track small nudges despite poorly conditioned exponentials.Ngspice solved the trained model with the same ICL accuracy, although the model still used ideal components, including idealized transistors.Source: E Nonlinear circuit solver · p. 35
Loading 2610.12571v1…