PaperlayerView exact revision on arXiv

Source-linked AI summary

RAPID: Row-Parallel Arithmetic Processing in DRAM

William C. Tegge, João Paulo Cardoso de Lima, Shouzhi Fang, Jeronimo Castrillon, Alex K. Jones

arXiv:2610.02502v1cs.AR

TL;DR

Existing DRAM-PUM architectures use bit-serial layouts that conflict with conventional row-major data, making PUM–host transitions costly. RAPID adds migration and inversion cells for row-parallel arithmetic, and reports higher end-to-end throughput than SIMDRAM across its DDR3 and DDR4 MLPerf evaluations.

  • Problem

    Existing DRAM-PUM’s bit-serial layouts are incompatible with conventional row-major data, requiring transposition at PUM–host boundaries.

    Source: §I · p. 1, D. Limitations of Existing PUM… · p. 4

  • Method

    RAPID adds migration cells for horizontal movement and inversion cells, and its compiler jointly selects arithmetic algorithms and layouts.

    Source: C. Global Optimization via Dynamic… · p. 8, §VIII · p. 13

  • Results

    RAPID achieves 5.9× higher end-to-end throughput on DDR4 and 94× on DDR3 than SIMDRAM across 19 MLPerf benchmarks.

    Source: §VIII · p. 13

  • Takeaways & Limitations

    RAPID preserves CPU-compatible layouts while providing row-parallel arithmetic with lower end-to-end throughput tradeoffs than bit-serial alternatives in the evaluated workloads.

    Source: §VIII · p. 13

    RAPID repurposes existing DRAM cell structures at subarray extremities; the reported area overhead is less than 1%.

    Source: D. Limitations of Existing PUM… · p. 4

Abstract

from arXiv · show

Processing-using-memory (PUM) architectures perform computation directly within DRAM to reduce costly data movement between memory and processors. Because charge-sharing operations are confined to individual bitlines, existing DRAM-PUM architectures reorganize data into column-oriented, bit-serial representations. This organization is fundamentally incompatible with the row-oriented, word-parallel layouts used by conventional processors and accelerators, requiring expensive data-layout transformations whenever computation transitions between PUM and conventional execution. In this paper, we present RAPID, a Row-parallel Arithmetic Processing-In-DRAM architecture. RAPID augments the DRAM subarray with two lightweight extensions: migration cells that enable localized horizontal data movement between neighboring bitlines and inversion cells that provide efficient in-array logical inversion. These primitives enable RAPID to operate directly on row-parallel, bit-parallel data, preserving CPU-compatible layouts while exploiting the massive parallelism of the DRAM subarray. In particular, RAPID demonstrates that localized horizontal communication is sufficient to realize shallow arithmetic networks and efficient parallel reduction for multiplication, reducing arithmetic latency while preserving throughput and eliminating costly data-layout transformations, all while maintaining the conventional DRAM array organization. We demonstrate the feasibility and overhead of augmenting DRAM subarrays with migration and inversion cells through detailed transistor-level layout and SPICE-validated circuit simulations. Using the RAPID compiler it is possible to evaluate the performance and data reorganization tradeoffs to ensure the best execution across combined CPU and PUM. Evaluating RAPID on 19 MLPerf benchmarks, there is a 5.9x higher end-to-end performance compared to SIMDRAM for DDR4 PUM execution.

I. INTRODUCTION

Existing DRAM-PUM layouts exploit bitline-local operations but conflict with conventional row-major data, making transitions costly. RAPID adds localized horizontal movement and inversion to support row-parallel arithmetic while retaining CPU-compatible layouts.

I. INTRODUCTION

  • Existing DRAM-PUM architectures use column-oriented, bit-serial data that is fundamentally incompatible with conventional row-oriented layouts, complicating interoperability and requiring costly transformations.Because bitlines isolate neighboring bit positions, existing designs lack native horizontal communication; many arithmetic structures need only local communication.Source: §I · p. 1
  • RAPID combines migration cells for horizontal movement and inversion cells to support row-parallel, bit-parallel arithmetic in CPU-compatible layouts.Its arithmetic constructions include logarithmic-depth prefix addition and carry-save reduction for multiplication.Source: §I · p. 2
  • RAPID’s compiler jointly selects arithmetic algorithms and data layouts across the dependence graph to minimize computation and conversion costs.The compiler supports both bit-serial and bit-parallel execution, accounting for conversion before results reach the CPU or later bit-parallel computation.Source: §I · p. 2

A. DRAM Organization

  • Conventional DRAM subarrays connect cells on each bitline to a shared sense amplifier, using wordlines to activate rows and access stored charge.In the open-bitline architecture, each sense amplifier is shared between adjacent subarrays.Source: A. DRAM Organization · p. 3

B. Processing-Using-Memory (PUM)

PUM reuses DRAM charge-sharing behavior for in-memory operations, while existing bit-serial layouts remain incompatible with row-major host data. RAPID targets that gap with low-overhead horizontal communication.

B. Processing-Using-Memory (PUM)

  • PUM uses DRAM charge sharing for computation, and RowClone copies an entire row through back-to-back activations without routing data through the memory controller.Ambit builds Boolean logic from RowClone, majority, and NOT operations; SIMDRAM maps complex operations to majority and NOT sequences.Source: B. Processing-Using-Memory (PUM) · p. 3

C. Bit-Serial and Bit-Parallel Algorithms

  • Bit-serial addition uses sequential carry propagation, whereas bit-parallel prefix addition reduces depth from O(n) to O(log n).Bit-serial execution maps each full-adder step across packed elements, trading per-element latency for aggregate throughput.Source: C. Bit-Serial and Bit-Parallel… · p. 3

D. Limitations of Existing PUM Architectures

  • Existing PUM architectures typically use vertical, bit-serial layouts because charge-sharing computation is confined to bitlines, requiring transposition at PUM–host boundaries.Earlier dedicated shifting logic enables bit-parallel computation on row-oriented data but incurs substantial estimated periphery area overhead.Source: D. Limitations of Existing PUM… · p. 4
  • RAPID repurposes DRAM cell structures at subarray extremities for horizontal communication at less than 1% area overhead.This contrasts with dedicated shifting logic co-located with sense amplifiers, estimated to incur up to 60% area overhead.Source: D. Limitations of Existing PUM… · p. 4

III. LIGHTWEIGHT HARDWARE EXTENSIONS FOR PUM

RAPID adds in-subarray inversion, shifting, and broadcast operations using lightweight hardware extensions, enabling data manipulation without leaving the memory bank. RowBroadcast uses migration cells for in-place expansion of a selected bit.

III. LIGHTWEIGHT HARDWARE EXTENSIONS FOR PUM

  • RAPID performs inversion, shifting, and broadcast within the subarray without sending data outside the memory bank or adding dedicated shifting logic to the sense amplifiers.The architecture uses migration cells for movement and inversion cells for logical inversion.Source: §III · p. 4

A. In-Subarray Inversion (RowInvert)

  • RowInvert uses inversion rows at the subarray’s top and bottom and requires one AAP command to invert a row.A second AAP command is needed to recombine and return inverted data from the inversion cells.Source: A. In-Subarray Inversion (RowInvert) · p. 4

B. In-Subarray Shifting (RowShift)

  • A full in-row RowShift uses two migration-cell rows to split and recombine data with a one-position offset.The operation requires four AAP commands and takes approximately 150 ns; the recombine steps execute serially.Source: B. In-Subarray Shifting (RowShift) · p. 4

C. In-Subarray Broadcast (RowBroadcast)

  • RowBroadcast isolates a selected bit with a positional mask and propagates it across a contiguous range of columns.It expands a single-bit control to match a multi-bit operand, including for multiplication partial-product generation.Source: C. In-Subarray Broadcast (RowBroadcast) · p. 4
  • RowBroadcast’s sequential in-place method avoids the explicit OR merges used by doubling-based broadcast and needs no dedicated shift-array hardware beyond migration cells.It takes n − 1 hops per bit, the same hop count as the described doubling approach.Source: C. In-Subarray Broadcast (RowBroadcast) · p. 5

IV. RAPID BIT-PARALLEL COMPUTATION

RAPID maps parallel arithmetic onto row-oriented data using horizontal shifts and logic. Kogge–Stone addition provides logarithmic depth, while carry-save reduction compresses multiplication’s partial products before a final addition.

IV. RAPID BIT-PARALLEL COMPUTATION

  • The section motivates RAPID’s arithmetic approach as parallel operations combined with structured horizontal communication rather than inherently sequential carry propagation.Multiplication requires partial-product generation, carry propagation, and multi-operand reduction across bits of different significance.Source: §IV · p. 5

A. Element-Wise Kogge–Stone Addition (KSA)

  • Kogge–Stone prefix addition reduces individual additions to O(log n) depth by mapping prefix levels to horizontal shifts and bitwise logic.The construction uses power-of-two shift distances; RowShift results are masked to correct duplicated boundary values.Source: A. Element-Wise Kogge–Stone Addition… · p. 5
  • The paper excludes synthesis of AP/AAP commands from bitwise primitives, noting that prior work addresses this topic.The cited prior works are,, and.Source: A. Element-Wise Kogge–Stone Addition… · p. 5

B. Bit-Parallel Element-Wise Multiplication

  • Multiplication forms shifted partial products by broadcasting each multiplier bit across columns, masking the multiplicand, and aligning the result with RowShift.The multiplier bit is isolated using a positional mask before replication.Source: B. Bit-Parallel Element-Wise… · p. 6
  • RAPID uses carry-save addition to eliminate intermediate carry propagation when reducing multiplication partial products, deferring it to one final addition.Sequential Kogge–Stone reduction would require n − 1 carry-propagating additions at O(log n) depth each, or O(n log n) total depth.Source: B. Bit-Parallel Element-Wise… · p. 6

1) Multi-Operand Reduction:

Carry-save reduction compresses multiplication’s partial products before one final prefix addition, reducing depth versus sequential addition. The section also describes GEMV/GEMM layout tradeoffs and a compiler that jointly selects layouts and algorithms.

1) Multi-Operand Reduction:

  • Each carry-save stage processes three operands into a sum and shifted carry without carry propagation, reducing the operand count by one at unit depth.The carry vector is aligned to the next-higher significance with one RowShift per stage.Source: 1) Multi-Operand Reduction · p. 6
  • O(n) + O(log n) depth reduces multiplication from O(n log n) for sequential Kogge–Stone addition and O(n^2) for sequential ripple-carry addition.Iterative 3:2 carry-save stages reduce n partial products to two operands, followed by one Kogge–Stone addition.Source: 1) Multi-Operand Reduction · p. 6

C. Implications for GEMV and GEMM Computation

  • The evaluation targets single-batch GEMV directly, computing one output row per layer; M input rows are handled as independent GEMVs across banks.GEMV multiplies a K × N weight matrix by a K-element input vector to produce an N-element output.Source: C. Implications for GEMV and GEMM… · p. 6
  • Bit-parallel layout avoids bit-packing, but partial-product masks are built once per weight row and reused across its dot products; horizontal layout enables lower-latency reduction.Mask construction costs O(n log n) per row, while each dot product applies masks and reduces n partial products.Source: C. Implications for GEMV and GEMM… · pp. 6–7

V. THE RAPID COMPILER

  • The RAPID compiler co-optimizes arithmetic algorithms and data layouts across the dependence graph because a layout mismatch can make transposition costs outweigh algorithmic gains.The compiler pass identifies PIM-vectorizable instructions, assigns algorithms and output layouts, and leaves instructions failing its eligibility test to the CPU.Source: §V · p. 7

A. Data Layout and RowShift Semantics

RAPID’s vertical and horizontal layouts give RowShift different meanings, matching bit-serial and bit-parallel arithmetic. The cost model includes host-side layout conversion, which can change which addition algorithm is cheapest.

A. Data Layout and RowShift Semantics

  • Vertical layout stores each bit position across a separate DRAM row, while horizontal layout stores all bits of an element in one row.Vertical layout suits bit-serial ripple-carry addition; horizontal layout suits wide-word arithmetic.Source: A. Data Layout and RowShift Semantics · p. 7
  • RowShift moves neighboring array elements under vertical layout but shifts bits within each word under horizontal layout.The compiler can mark array-neighbor operations as fixed-vertical and propagate that requirement to dependent loads.Source: A. Data Layout and RowShift Semantics · p. 7

B. Per-Operation Cost Model

  • Layout conversion reads an operand to the host, transposes its bit-matrix on the CPU, then writes it back; its cost scales with operand footprint.The conversion cost is direction-independent, and CPU bit-transposition dominates memory transfer by 8× at the evaluated bandwidth.Source: B. Per-Operation Cost Model · p. 7
  • For N = 65,536 and int32 operands, layout conversion takes about 73 µs on DDR3, making the Kogge–Stone path 28× more expensive than ripple-carry after conversion.Although Kogge–Stone is 2.4× cheaper than ripple-carry when inputs are already horizontal, a vertical input incurs the conversion cost.Source: B. Per-Operation Cost Model · pp. 7–8

C. Global Optimization via Dynamic Programming

RAPID uses dynamic programming to jointly choose each operation’s algorithm and output layout across a dependence tree. Its evaluation combines circuit-level modeling with architectural simulations across two DRAM generations and paired ARM platforms.

C. Global Optimization via Dynamic Programming

  • The optimizer computes minimum subtree latency for each candidate output layout bottom-up, then propagates the root’s selected layouts and algorithms top-down.It accounts for transposition costs when a child’s output layout differs from its parent’s required input layout.Source: C. Global Optimization via Dynamic… · p. 8

VI. EXPERIMENTAL SETUP

  • RAPID’s evaluation combines DRAM-circuit layout implementations, PUM simulation, and execution simulation on embedded-class architectures.The transistor-level layout includes sense amplifiers, migration and inversion cells, DRAM cells, and routing.Source: §VI · p. 8, A. Layout Evaluation of Migration and… · p. 8

A. Layout Evaluation of Migration and Inversion Cells

  • The circuit model uses extracted parasitics and calibrated MIM capacitors in place of unavailable trench capacitors; the modeled 512-cell bitline capacitance is about 143 fF.Monte Carlo simulations characterize process-variation effects on latency, energy, and reliability.Source: A. Layout Evaluation of Migration and… · p. 8
  • The architectural evaluation covers DDR3-1333 and DDR4-2400 paired with Cortex-A9 and Cortex-A72, respectively.The ARM platforms measure bit-serial layout-reorganization time; RAPID host code is simulated on an in-order RISC-V core.Source: A. Layout Evaluation of Migration and… · p. 8

B. Compilation and Simulation Methodology

The evaluation compiles PyTorch models through LLVM and measures PUM operations with cycle-accurate simulation. It tests layout selection on fused and single-purpose 32-bit kernels with different layout requirements.

B. Compilation and Simulation Methodology

  • The LLVM-based RAPID pass identifies PIM-vectorizable regions, jointly selects layouts and algorithms, and lowers selected operations to in-DRAM command sequences.Operations outside the vectorizable region, including softmax and normalization, remain host code; the same pass also supports C/C++ kernels through clang.Source: B. Compilation and Simulation… · p. 9
  • Per-operation latency and energy come from cycle-accurate NVMain-PIM simulation, with less than 0.15% error against composed multi-operation traces.The simulated operations include n-bit multiplication and addition for operand widths 4, 8, 16, and 32.Source: B. Compilation and Simulation… · p. 9
  • The GEMV composition evaluates one output row (M = 1); bit-serial rows pack 65,536 elements, while bit-parallel packing depends on element width and guard bits.At 32 bits, the stated bit-parallel packing is 1,040 elements per row.Source: B. Compilation and Simulation… · p. 9

C. Layout-Selection Kernels

  • The layout-selection evaluation uses fifteen 32-bit integer kernels on 8,192-element vectors, including five fused pipelines and ten single-purpose kernels.The workload mix includes stencils, a prefix sum, multiply/add chains, and pipelines whose neighboring stages prefer different layouts.Source: C. Layout-Selection Kernels · p. 9, Table III · p. 9

D. Application Workloads

The evaluation spans MLPerf GEMV workloads, arithmetic variants, circuit validation, and trace-based simulation. RAPID’s arithmetic uses majority logic and row-parallel partial-product amortization.

D. Application Workloads

  • The evaluation extracts GEMV layers from 19 MLPerf models and analyzes over 20,000 layer–width configurations across four operand widths.The models span language, recommendation, vision, graph, and point-cloud workloads; layer dimensions and MAC counts cover broad ranges.Source: D. Application Workloads · p. 9

E. Higher-Order MAJ Fidelity

  • PEX-extracted Monte Carlo SPICE trials show MAJ5 reliably resolves within sense-amplifier voltage margins across tested conditions.The simulations use a 45 nm FreePDK layout and vary transistor width and length across realistic process-variation levels.Source: E. Higher-Order MAJ Fidelity · p. 10

F. NVMain-PIM Trace-Based Simulation

  • Cycle-accurate NVMain-PIM simulation supplies per-operation latency and energy metrics for single-operation comparisons and workload-level GEMV scaling.The per-operation costs are obtained from DRAM command traces.Source: F. NVMain-PIM Trace-Based Simulation · p. 10

G. Circuit-Level Validation and 45nm Layout

  • RAPID’s migration and inversion extensions use a DRC/LVS-clean 45 nm layout, and 400,000 Monte Carlo trials confirm reliable RowInvert operation across tested variation levels.Post-layout parasitics for the bitline sense amplifiers, migration cells, and routing are simulated in LTSpice.Source: G. Circuit-Level Validation and 45nm… · p. 10

VII. EVALUATION

  • The evaluated configurations vary data layout, logic primitives, movement and inversion mechanisms, core adders, and multi-operand reduction strategies.RAPID is evaluated as BP-M35-MR-KS-PR, alongside bit-serial and other bit-parallel configurations.Source: §VII · p. 10

A. Core Arithmetic Evaluation

  • RAPID’s bit-parallel M35 logic provides a 3x speedup over AOI, with majority-logic benefits more pronounced than in bit-serial computation.For bit-serial computation, MIG provides about 2x speedup for addition and 30%–70% gains for multiplication; M35 adds 11%–14% for addition and about 5% for multiplication.Source: A. Core Arithmetic Evaluation · p. 10

B. Partial-Product Amortization in Bit-Parallel GEMV

1) Amortization Mechanism:

RAPID’s row-wide execution shares partial-product generation across packed GEMV positions. This amortization reduces end-to-end latency relative to bit-serial execution, including the cost of transposition.

1) Amortization Mechanism:

  • A single row-wide broadcast produces different partial products at packed positions because each position contains a different weight, while the scalar operand is shared.The broadcast and extraction cost grows from 34.4% of multiplication at 4-bit precision to 69.5% at 32-bit precision.Source: 1) Amortization Mechanism · p. 10

3) Impact on End-to-End Competitiveness:

2) Quantifying the Benefit:

  • At 32-bit precision, the amortization factor ranges from 132x for small-N benchmarks to over 670x when all 1,040 packed positions are used.The factor compares one shared row-wide broadcast against independent partial-product generation at each active position.Source: 2) Quantifying the Benefit · p. 11, Table IV · p. 11

3) Impact on End-to-End Competitiveness:

  • For BERT-Large at 32-bit precision, RAPID achieves 6.3x lower end-to-end latency than SIMDRAM on DDR4 and a 97x gap on DDR3.The DDR4 comparison includes SIMDRAM’s data transposition; the gap widens on DDR3 because transposition is substantially more expensive there.Source: 3) Impact on End-to-End Competitiveness · p. 11

C. End-to-End GEMV Comparison

RAPID executes GEMV in row-major, bit-parallel layouts, whereas bit-serial baselines require transposition before and after PUM execution. Across the evaluated benchmarks, those costs reverse some compute-only comparisons and favor RAPID end-to-end.

C. End-to-End GEMV Comparison

  • The compiler selects bit-parallel, row-major execution for every evaluated GEMV, requiring no bit-serial fallback or intermediate transposition.Bit-serial baselines require a transposition roundtrip for each GEMV operator to return results to CPU-compatible format.Source: C. End-to-End GEMV Comparison · pp. 11–12

1) PUM-Only Compute:

  • For single-tile benchmarks, RAPID’s Kogge–Stone addition yields 2.2–2.3x lower PUM-only latency than SIMDRAM’s ripple-carry addition.For large-N benchmarks, SIMDRAM achieves up to 3.3x lower PUM-only latency by fitting all elements in one row, but transposition reverses that advantage end-to-end.Source: 1) PUM-Only Compute · p. 12

2) End-to-End Including Transposition:

  • On DDR3, RAPID achieves 18.6–96.9x speedups over SIMDRAM across benchmarks; on DDR4, it achieves 4.1–6.6x speedups.For LLaMA-2 70B on DDR4, RAPID’s PUM compute is 3.2x slower, yet it is 6.0x faster end-to-end.Source: 2) End-to-End Including Transposition · p. 12

3) Throughput Analysis:

  • SIMDRAM delivers 2.7–2.8x higher PUM-only throughput, reflecting its 63x per-row parallelism advantage at 32-bit precision.RAPID packs 1,040 elements per row at that precision, compared with SIMDRAM’s 65,536.Source: 3) Throughput Analysis · p. 12
  • RAPID achieves 5.9x higher end-to-end throughput on DDR4 and 94x on DDR3, as transposition reverses the PUM-only throughput comparison.Fig. 6 reports end-to-end GEMV throughput at 32-bit precision, sorted by RAPID’s speedup over SIMDRAM including transposition.Source: Figure 6 · p. 12, 3) Throughput Analysis · p. 12

D. Layout Selection Against Fixed-Layout Policies

The compiler’s dynamic programming (DP) layout selection outperforms fixed layouts on mixed pipelines and matches the better fixed policy on single-purpose kernels. The conclusion reports gains over fixed-layout choices across evaluated kernels.

D. Layout Selection Against Fixed-Layout Policies

  • The evaluation compares DP against fixed vertical-only and horizontal-only policies on fifteen 32-bit integer kernels, including fused pipelines and single-purpose kernels.The DP minimizes modeled plan cost, so it is never worse than a fixed policy by construction.Source: D. Layout Selection Against… · p. 12
  • On fused pipelines, each fixed policy costs 1.75–3.47x more than DP, with geometric-mean penalties of 2.4x for vertical-only and 2.0x for horizontal-only.Fig. 7 presents modeled plan cost relative to DP; horizontal-only is better on four pipelines, while vertical-only is better on blur_mask.Source: D. Layout Selection Against… · pp. 12–13, Figure 7 · p. 13
  • On ten single-purpose kernels, DP matches the better fixed policy, while the wrong fixed choice incurs geometric-mean penalties of 4.8x for vertical-only and 2.7x for horizontal-only.Fig. 8 presents fixed-policy costs relative to DP; horizontal-only is right for seven kernels and vertical-only for three shift kernels.Source: Figure 8 · p. 13, D. Layout Selection Against… · p. 13

VIII. CONCLUSIONS

  • Across mixed kernels, per-operation layout selection reduces latency by 2.0x and 2.4x versus horizontal-only and vertical-only layouts, and by 1.8x versus the better fixed choice per pipeline.No fixed layout is best across all kernels.Source: §VIII · p. 13

Related papers

Loading 2610.02502v1…