PaperlayerView exact revision on arXiv

Source-linked AI summary

Compute-in-Memory Attention: A Time-Domain Analog Softmax Circuit with RC-Tunable Temperature

Ankur Singh, Ashish Gautam, Shruti R. Kulkarni, Guojing Cong

arXiv:2609.04266v1cs.ARcs.LG

TL;DR

Softmax’s exponentiation and normalization complicate its integration with analog compute-in-memory attention scores. This paper presents a programmable-temperature time-domain analog softmax that processes those scores through a shared ramp and RC sampling. The 128-element circuit achieves 24.46 mV RMSE, and its hardware-aware Transformer validation loss is within about 2.5% of the ideal-softmax baseline.

  • Problem

    Analog CIM produces attention-score voltages, but integrating softmax’s nonlinear exponentiation and vector-wide normalization can require conversion overhead or dedicated peripheral hardware.

    Source: §I · p. 1

  • Method

    A shared ramp converts CIM score voltages into timing events that trigger RC-decay sampling for exponential weights, followed by in-circuit normalization with temperature controlled by ramp slope and RC time constant.

    Source: §I · p. 2

  • Results

    The 128-element circuit achieves an RMSE of 24.46 mV, and hardware-aware Transformer validation loss is 1.7868 versus 1.7435 for ideal softmax.

    Source: §I · p. 2

  • Takeaways & Limitations

    The MemTorch-based hardware-aware Transformer evaluation confirms applicability to Transformer inference, with validation loss about 2.5% above the ideal-softmax baseline.

    Source: §I · p. 2

Abstract

from arXiv · show

Softmax is a key operation in Transformer attention, but its exponentiation and normalization add significant overhead in compute-in-memory (CIM) accelerators, especially when analog attention scores must first be converted to the digital domain. This work presents a tunable-temperature analog softmax circuit in GlobalFoundries 22-nm fully depleted silicon-on-insulator (FDSOI) technology that operates directly on CIM-generated score voltages without intermediate analog-to-digital conversion. Each input score is converted into a time-domain event using a shared falling ramp. The corresponding comparator transition samples an RC-decaying reference to generate an exponential weight, which is then processed by an in-circuit normalization stage. In contrast to analog softmax circuits that rely on transistor weak-inversion behavior for exponentiation, the proposed architecture controls the softmax response through the ramp slope and RC time constant, enabling programmable effective temperature. The 128-element architecture is evaluated using transistor-level and post-layout extracted simulations, including multi-level input vectors, capacitance variation and mismatch, process and temperature variation, monte carlo analysis, and shared-interconnect parasitics. The complete 128-element implementation occupies 9453.42~$\mu\mathrm{m}^{2}$ including the shared global ramp circuitry, while each replicated softmax element occupies 70.2~$\mu\mathrm{m}^{2}$. The circuit achieves a 242.97-ns evaluation latency at 13.44~mW total power, corresponding to 25.5~pJ per output element. The simultaneous 128-element evaluation achieves an RMSE of 24.46~mV relative to the ideal softmax response. The extracted circuit characteristics are further incorporated into a MemTorch-based hardware-aware Transformer model, where the proposed softmax achieves a validation loss within 2.5\% of the ideal-softmax baseline.

I. INTRODUCTION

Softmax adds repeated computation to Transformer attention, while integrating its exponential mapping and vector-wide normalization with analog CIM scores remains difficult. The paper proposes a direct time-domain analog softmax whose effective temperature can be tuned through circuit parameters.

I. INTRODUCTION

  • Softmax recurs across Transformer attention heads and layers and requires exponentiation, maximum search, accumulation, and normalization.These operations make softmax a non-negligible component of attention computation.Source: §I · p. 1
  • Analog CIM naturally generates attention-score voltages, but softmax requires nonlinear exponentiation and vector-wide normalization, often leaving conversion or dedicated peripheral hardware in the path.An analog attention design instead used HardSigmoid because sequence-dimension interconnection for softmax normalization was difficult to implement efficiently.Source: §I · p. 1
  • Weak-inversion transistor softmax is constrained to a narrow input-voltage range, with practical operation also affected by temperature, leakage, tail-current accuracy, and device mismatch.As inputs approach moderate or strong inversion, the current-voltage characteristic departs from the desired exponential dependence.Source: §I · pp. 1–2
  • The proposed 22-nm analog softmax generates exponential weights through a shared ramp, comparator-based event detection, and RC-decay sampling rather than weak-inversion MOS behavior.It operates on CIM-generated score voltages and includes in-circuit normalization.Source: §I · p. 2

A. Temperature-Programmable Softmax

  • Ramp slope and RC time constant set the effective softmax temperature, allowing the score-to-weight mapping to be tuned for varying CIM score conditions.Lower temperature sharpens the distribution, while higher temperature makes it smoother.Source: A. Temperature-Programmable Softmax · p. 2

B. Time-Domain Analog Softmax Architecture

The architecture converts CIM score voltages to time-domain events with a shared falling ramp, samples an RC-decaying reference to form exponential weights, and normalizes those weights in circuit. Its initialization prepares the capacitors and selects the RC operating condition before evaluation.

B. Time-Domain Analog Softmax Architecture

  • Each comparator detects when the shared falling ramp reaches its CIM score voltage, converting the score to a time event that triggers local pulse generation and RC-decay sampling.The sampled exponential weights are stored as VE,n and passed to the normalization block, which generates normalized output voltages VP,n.Source: B. Time-Domain Analog Softmax… · pp. 2–3, Figure 1 · p. 3
  • Operation proceeds through initialization, exponential-weight generation, and approximate in-circuit normalization.The three phases are shown in the architecture timing sequence.Source: B. Time-Domain Analog Softmax… · p. 3, Figure 1 · p. 3

1) Initialization phase:

  • Initialization precharges the ramp and exponential-generator capacitors to approximately VDD, programs the memristive element to a high-resistance state, and resets storage and output capacitors.The resistance is programmed when the desired operating condition is configured, not for every input vector.Source: 1) Initialization phase · p. 3
  • During evaluation, a bias-controlled current discharges the ramp capacitor while the programmed memristive resistance discharges the exponential-generator capacitor.VBIAS controls ramp slope SR; the exponential-generator time constant is τE, and the resistance is assumed approximately constant over the selected operating range.Source: 1) Initialization phase · p. 3

2) Exponential-weight generation phase:

Each analog CIM score is converted into a comparator switching time, which controls sampling of an RC-decaying reference to form an exponential weight. The ramp slope and RC time constant program the effective temperature and softmax response.

2) Exponential-weight generation phase:

  • The comparator switches when the common falling ramp reaches the corresponding analog CIM score voltage.The comparator transition is processed into a narrow complementary sampling pulse that enables the transmission gate.Source: 2) Exponential-weight generation phase · pp. 3–4
  • A larger input score produces an earlier comparator transition, and its sampling pulse samples the decaying reference onto the exponent-storage capacitor.The stored voltage represents the exponential weight for that input.Source: 2) Exponential-weight generation phase · p. 4
  • Because AE is common across branches, the sampled voltage is proportional to the exponential softmax numerator.Source: 2) Exponential-weight generation phase · p. 4
  • The effective temperature can be programmed through the ramp slope or RC time constant: smaller Teff sharpens the response, while larger Teff smooths it.The ramp bias controls score-to-time conversion, and programmable resistance selects the RC operating point.Source: 2) Exponential-weight generation phase · p. 4

3) Approximate in-circuit normalization phase:

The normalization stage applies stored exponential weights to matched NMOS branches sharing a source node, then transfers branch currents to output capacitors through PMOS current mirrors. Its first-order softmax response depends on the ramp and RC conditions, with circuit nonidealities evaluated in transistor-level simulations.

3) Approximate in-circuit normalization phase:

  • Stored exponential weights drive matched NMOS transistors with a shared source node terminated by reference current sink IREF.Source: 3) Approximate in-circuit… · p. 4
  • During normalization, PMOS current mirrors transfer NMOS branch currents to output capacitors, which are initially reset near zero.The VSAMP-controlled switch enables the mirror network after exponential weights have been stored.Source: 3) Approximate in-circuit… · pp. 4–5
  • The square-law normalization stage adds a factor of two to the exponential coefficient, and its effective output temperature depends on the circuit parameters.The first-order closed-form relation neglects the common offset V0 in the dominant input-dependent term.Source: 3) Approximate in-circuit… · pp. 4–5
  • Increasing ramp slope increases effective temperature and reduces γth, while increasing the RC time constant also reduces γth.Finite common-source voltage, threshold voltage, channel-length modulation, and current-mirror nonidealities contribute to differences between theoretical and simulated responses.Source: 3) Approximate in-circuit… · p. 5

Z TSAMP

The output relation approximates vector-wise normalized exponential weights, with its first-order inverse-temperature coefficient set by ramp and RC conditions. The approximation assumes equal output capacitors and a nearly constant current ratio during the short normalization interval.

Z TSAMP

  • The output relation assumes equal output capacitors and an approximately constant current ratio during the short normalization interval.Source: Z TSAMP · p. 5
  • Apart from the common output-scaling factor VFS, the circuit approximates a vector-wise normalized exponential response whose inverse-temperature coefficient is determined by ramp slope and RC time constant.Source: Z TSAMP · p. 5
  • Table I lists comparator-circuit transistor dimensions, while Table II covers nominal circuit parameters and theoretical softmax gain.Source: Table I · p. 5, Table II · p. 5

A. Softmax Accuracy and Programmable Temperature

The nominal circuit reproduces softmax transfer and evaluates 128-element multi-level vectors; capacitor parasitics and mismatch affect accuracy, while ramp adjustment can compensate systematic capacitance shifts.

A. Softmax Accuracy and Programmable Temperature

  • Approximately 4.3% RMSE versus ideal softmax characterized the controlled nominal transfer, whose output branches moved complementarily and intersected near equal inputs.Nominal parameters yielded γth = 11.85 V−1 and a fitted γsim = 11.36 V−1.Source: A. Softmax Accuracy and Programmable… · pp. 5–6
  • Eight interleaved input levels produced distinct ramp-crossing times, so earlier write-enable sampling captured larger VEXP values than later sampling.The same eight-level sequence was distributed across all 128 channels.Source: A. Softmax Accuracy and Programmable… · p. 6, Figure 4 · p. 6
  • 24.46 mV RMSE over 128 elements versus ideal softmax accompanied preserved input ordering and normalized distribution under simultaneous multi-level excitation.The evaluated vector used eight interleaved voltage levels across the 128 channels.Source: A. Softmax Accuracy and Programmable… · p. 6

B. Analog Non-idealities

  • Femtfarad-scale parasitics can alter sampled voltages and increase deviation from theoretical softmax, while transmission-gate turn-off produces a small storage-node perturbation.The shared ramp is also susceptible to interconnect loading and distributed resistance.Source: B. Analog Non-idealities · p. 7
  • Increasing CP to 2.5 fF raised RMSE to approximately 110 mV versus the nominal theoretical response, while ramp-slope recalibration can recover the desired characteristic after systematic capacitance shifts.CC and CP were independently varied over ±25%.Source: B. Analog Non-idealities · p. 7
  • Grouped mismatch across 16-element capacitor groups produced 23.6 mV RMSE for CC and 32.7 mV for CP versus the nominal 128-channel response.The assigned capacitances spanned approximately 1.8–2.37 fF around the nominal 2 fF.Source: B. Analog Non-idealities · p. 7, Figure 8 · p. 7

C. Variability Analysis

The circuit’s softmax response was evaluated across temperature, process corners, and Monte Carlo variation. It maintained expected softmax behavior across the considered variability conditions, with a reported 3.4% output spread in Monte Carlo analysis.

C. Variability Analysis

  • γ varied from 11.36 V−1 at 27°C (TT) to 19.2 V−1 at 80°C across −30°C to 80°C; process-corner responses preserved complementary behavior.Across process corners, the reference response agreed more tightly with the theoretical characteristic and intersected the swept output near equal inputs.Source: Figure 10 · p. 8, C. Variability Analysis · p. 8
  • Monte Carlo output was approximately Gaussian (µ = 0.348 V, σ = 12 mV; 3.4% relative spread), and tested variability conditions preserved expected softmax behavior.The statistical models included transistor and modeled passive-component variations, but excluded the externally modeled memristor.Source: Figure 11 · p. 8, C. Variability Analysis · p. 8

D. Layout and Post-Layout Extraction Results

The layout combines replicated softmax elements with a shared ramp and peripheral circuitry, while post-layout extraction evaluates area, power, parasitics, and programmable response. The extracted 128-channel circuit retains programmable temperature and tracks the theoretical response on a multi-level input vector.

Layout and Area/Power

Shared-Ramp Extraction

  • The shared VRAMP network has approximately 210 fF parasitic capacitance and 120 Ω resistance; its extracted distributed RC network is retained in post-layout simulation.The explicit ramp capacitor was reduced from 200 fF in the schematic to 50 fF in layout, with residual slope shift compensated through VBIAS.Source: D. Layout and Post-Layout Extraction… · pp. 8–9

Post-Layout Programmability

Post-Layout Extraction Results

IV. APPLICATION AND COMPARISON

The MemTorch evaluation inserts the circuit-derived softmax into a hardware-aware Transformer attention path after mapping attention scores to circuit voltages. Its validation loss stays close to ideal softmax and below sigmoid and hard-sigmoid alternatives across the reported temperature conditions.

A. Transformer-Level Evaluation Based on MemTorch

B. Comparison with Prior Softmax Hardware

The design reports 128-way parallel softmax in a single 242.97-ns evaluation, with the smallest per-element area among compared implementations. Its time-domain exponential generation and temperature control distinguish it from prior hardware approaches; scaling beyond 128 elements remains future work.

B. Comparison with Prior Softmax Hardware

  • The 128-element architecture occupies 9453.42 µm2, or 70 µm2 per element—the smallest per-element area among compared implementations—and evaluates in 242.97 ns.The array consumes approximately 105 µW per element; energy is approximately 25.5 pJ per output element.Source: B. Comparison with Prior Softmax… · p. 11
  • The circuit generates exponential response through a shared ramp and RC decay, with temperature programmed by ramp slope and RC time constant over a demonstrated 0.30–1.10 V input range.Compared analog designs typically use MOSFET subthreshold behavior, while digital implementations approximate exponentiation with arithmetic or lookup methods.Source: B. Comparison with Prior Softmax… · pp. 11–12

C. Future Works and Scalability

  • Scaling beyond 128 elements remains future work because added branches are expected to increase shared-ramp parasitic loading and alter ramp slope and γth.The authors propose hierarchical groups with replicated ramp drivers and identify physical crossbar integration and full hardware-level Transformer evaluation as future work.Source: C. Future Works and Scalability · p. 12

V. CONCLUSION

  • The conclusion reports post-layout evaluation of the 128-channel array at 9453.42 µm2, 242.97 ns, and 13.44 mW, with approximately 25.5 pJ per output element.MemTorch-based Transformer evaluation further demonstrates applicability to CIM-based attention processing.Source: §V · p. 12

Related papers

This paper cites

Loading 2609.04266v1…