Source-linked AI summary
p-DCIM: A 28nm Probabilistic SRAM DCIM based Ising Machine for Near-Memory Context Selection in LLM-RAG Pipeline
Madhav Vadlamani, Rahul Singh, Yuyao Kong, Haomei Liu, Zheng Zhang, Shimeng Yu
TL;DR
LLM-based set selection adds a costly inference pass to RAG, motivating an alternative for evidence selection. The paper formulates selection as a QUBO and uses a dual-mode SRAM DCIM Ising macro to construct and solve it. The evaluated flow reduces total query energy by 67% while maintaining competitive downstream answer quality.
Problem
LLM-based set selection adds another transformer-inference pass, with repeated HBM–GPU weight movement that incurs substantial latency and energy.
Source: §I · p. 1Method
A dual-mode SRAM DCIM macro reuses its array and signed accumulation datapath to construct embedding-derived QUBO coefficients and perform in-memory annealing updates.
Source: §I · pp. 2–3Results
67% lower evaluated total query energy, with competitive downstream answer quality and approximately 0.9 µJ selection energy.
Source: §VI · p. 11Takeaways & Limitations
The demonstrated near-memory flow keeps candidate embeddings near HBF storage during selection and returns only selected passage identifiers to the GPU.
Source: §VI · p. 11
Abstract
from arXiv · showhide
We present a fabricated SRAM DCIM Ising machine that performs set-level context selection for large language model (LLM) with retrieval-augmented generation (RAG). In RAG, the evidence-selection stage is increasingly delegated to repeated language-model reranking, which adds substantial energy to the inference pipeline. We reformulate evidence selection as a quadratic unconstrained binary optimization (QUBO) problem and solve it on a dual-mode digital compute-in-memory (DCIM) macro: the same multiply-and-accumulate datapath computes the embedding-similarity scores that populate the QUBO coefficients and evaluates the per-spin Hamiltonian differences that drive the anneal. The annealing temperature is set by a digital plane-count control along with the SRAM pseudo-read random bit flips, giving graded cooling entirely from on-chip digital configuration. The macro is taped-out in a 28 nm process [0.16 mm2, 0.41 mW] and is proposed to be integrated as part of the logic base-die in hybrid high-bandwidth memory (HBM) and high-bandwidth flash (HBF) stacks in a GPU package, where only the selected passage identifiers return to the host. Replacing the language-model selection pass with the near-memory Ising solver yields a macro normalized selection-energy estimate more than six orders of magnitude lower than the LLM-based baseline and reduces the evaluated total energy per query by 67%, while maintaining comparable token-level F1 on the evaluated examples.
I. INTRODUCTION
The paper targets the energy and latency cost of LLM-based set selection in RAG, proposing a fabricated near-memory Ising accelerator that constructs and solves a QUBO. Its evaluated flow reduces selection-stage energy by more than six orders of magnitude relative to an LLM-based selector and total query energy by 67%, with competitive answer quality.
I. INTRODUCTION
- Set-level QUBO selection models pairwise passage interactions, unlike independent top-K ranking, but LLM-based selection adds another transformer pass and substantial latency and energy.The objective can represent relevance, coverage, redundancy, complementarity, and context-size constraints.Source: §I · p. 1
- The dual-mode macro reuses one SRAM array and signed accumulation datapath to construct embedding-derived QUBO coefficients and evaluate local Hamiltonian differences during annealing.This shares storage and arithmetic resources across construction and optimization.Source: §I · pp. 2–3
- SRAM pseudo-read errors provide annealing perturbations, while a digital magnitude-plane schedule provides graded cooling with the sign bit protected.The array operates at fixed reduced supply; successive magnitude bit-planes are protected as annealing progresses.Source: §I · pp. 2–3
- The proposed HBF integration keeps candidate embeddings and pairwise processing near storage, returning only selected passage identifiers to the GPU.The fabricated macro occupies 0.16 mm2 and consumes approximately 0.41 mW in probabilistic operation.Source: §I · p. 3
- More than six orders of magnitude lower selection-stage energy than an LLM-based selector and 67% lower evaluated total query energy, with competitive HotpotQA answer quality.The reported result uses a measurement-parametrized model for larger workloads.Source: §I · p. 3
II. QUBO FORMULATION
The formulation represents requirement-aware passage selection as a QUBO over binary selection variables. Its objective balances relevance and requirement coverage against context size and redundancy, rewards complementary evidence, and converts constraints into penalties.
II. QUBO FORMULATION
- Each candidate passage receives a binary selection variable, while requirement coverage, relevance, redundancy, and complementarity are represented using passage and pairwise terms.Redundancy captures overlapping evidence; complementarity captures jointly useful support for different requirements.Source: §II · p. 3
- The objective balances query relevance and requirement coverage against context size and redundancy, while rewarding complementary evidence.The parameter λ controls the cost of including passages within the maximum context budget K.Source: §II · pp. 3–4
- Coverage and context-budget constraints are converted into equalities using binary-encoded slack variables, then added as squared penalties to form the unconstrained QUBO.Negating the original maximization objective and collecting its terms produces the standard QUBO form.Source: §II · p. 4
- The resulting QUBO contains passage-selection, coverage, and auxiliary slack variables; optimized passage selections are passed unchanged to the answer generator.The DCIM datapath solves the QUBO.Source: §II · p. 4
III. FABRICATED P-DCIM ARCHITECTURE AND DUAL-MODE OPERATION
p-DCIM uses one SRAM DCIM array and signed accumulation datapath to construct and optimize the context-selection QUBO in two operating modes.
III. FABRICATED P-DCIM ARCHITECTURE AND DUAL-MODE OPERATION
- The same SRAM DCIM array and signed accumulation datapath serve both QUBO construction and stochastic annealing, rather than using separate engines.This reuse is the architecture’s central feature.Source: §III · p. 4
- In deterministic mode, the datapath computes embedding-derived quantities used to populate the QUBO matrix.Source: §III · p. 4
- In probabilistic mode, the array stores quantized coefficients while the datapath evaluates local energy differences for sequential QUBO updates.Source: §III · p. 4
A. Fabricated SRAM-DCIM Macro
The fabricated macro combines a bit-sliced SRAM compute array with a signed adder tree, supporting both QUBO evaluation modes and frequent coefficient writes.
A. Fabricated SRAM-DCIM Macro
- The macro contains a 64×64 logical DCIM array supporting signed 8-bit coefficients, distributes eight coefficient bit planes across 512 physical wordlines for 32 Kb total SRAM capacity, and occupies 0.16 mm2.It operates at 500 MHz and is characterized through an FPGA/AXI-based measurement platform.Source: A. Fabricated SRAM-DCIM Macro · p. 4
- The compute-enabled memory element is a 12T SRAM cell with a 6T storage latch, 4T NOR compute gate, and 2T transmission-gate cell multiplexer.A signed digital adder tree combines cell outputs into weighted accumulations used in both operating modes.Source: A. Fabricated SRAM-DCIM Macro · p. 4
- The macro uses one protected sign plane and multiple magnitude planes; its bit-sliced organization retains coefficients and participates in their arithmetic evaluation.The fabricated macro uses B = 8.Source: A. Fabricated SRAM-DCIM Macro · p. 4
- 0.406 mW in deterministic mode and 0.413 mW in probabilistic mode are the measured active powers of the macro.The SRAM array dominates measured area; the remaining area includes control, read/write circuitry, drivers, multiplexing, and the signed adder tree.Source: A. Fabricated SRAM-DCIM Macro · p. 4
- SRAM suits this flow because each query produces a new coefficient matrix and annealing periodically restores nominal coefficients after noise injection.Its low write latency and effectively unrestricted write endurance support frequent writes, alongside controlled read-disturb stochasticity.Source: A. Fabricated SRAM-DCIM Macro · p. 4
B. Deterministic Mode: QUBO Construction
In deterministic mode, the shared SRAM-DCIM MAC datapath computes embedding dot products used to construct QUBO coefficients. Constraint and penalty terms are combined with embedding-derived coefficients before they are quantized and written back to the array.
B. Deterministic Mode: QUBO Construction
- The signed MAC datapath evaluates query–candidate, requirement–candidate, and candidate–candidate similarities used to form linear and pairwise QUBO coefficients.The same array stores the coefficients for subsequent probabilistic operation.Source: B. Deterministic Mode: QUBO Construction · p. 5, Figure 3 · p. 6
- Constraint- and penalty-derived coefficients are generated digitally and combined with embedding-derived coefficients in the host control path before quantization and write-back.If embeddings exceed the array width, dot products use multiple MAC passes with partial results accumulated.Source: Figure 3 · p. 6, B. Deterministic Mode: QUBO Construction · p. 6
- Deterministic operation uses VDDM = 0.9 V, suppressing pseudo-read disturbances while the array computes and programs QUBO coefficients.The same SRAM array and signed accumulation datapath are reused for annealing after the supply is reduced.Source: B. Deterministic Mode: QUBO Construction · p. 6
C. Probabilistic Mode: In-Memory QUBO Annealing
In probabilistic mode, the same array and adder tree evaluate local energy changes and update binary variables sequentially. SRAM pseudo-read disturbances perturb stored coefficient magnitudes to provide annealing stochasticity, with periodic coefficient refreshes and digitally reduced noise as cooling proceeds.
C. Probabilistic Mode: In-Memory QUBO Annealing
- The same DCIM array and signed adder tree evaluate each variable’s local energy change, and its sign determines whether the variable is flipped.Only the sign is needed for the update, unlike deterministic coefficient construction, which uses the full signed MAC result.Source: C. Probabilistic Mode: In-Memory QUBO… · p. 7
- For general QUBOs with nonzero diagonal coefficients, pinned-one embedding maps each diagonal term to a coupling with an auxiliary variable clamped to one.This lets the same datapath evaluate single-variable energy changes without a separate diagonal-add path.Source: C. Probabilistic Mode: In-Memory QUBO… · p. 7
- Pseudo-read disturbances perturb exposed coefficient magnitude bits to provide stochastic exploration, while periodic buffer refresh restores nominal coefficients and retains the optimization state.Sequentially committed updates use the perturbed effective local fields without dedicated random-number-generation peripherals.Source: C. Probabilistic Mode: In-Memory QUBO… · p. 7
- No separate optimization datapath is needed: coefficients written after deterministic construction are reused by the same array and adder tree for repeated annealing evaluations.The state is updated sequentially as the anneal proceeds.Source: C. Probabilistic Mode: In-Memory QUBO… · p. 7
D. SRAM Pseudo-Read Stochasticity and Digital Cooling
Reduced-voltage SRAM pseudo-read errors supply annealing randomness, while digital control of exposed coefficient planes sets its effective strength. Protecting more planes progressively reduces coefficient perturbation without fine analog voltage regulation.
D. SRAM Pseudo-Read Stochasticity and Digital Cooling
- The digital schedule protects the sign plane and steps exposed magnitude planes through k = 7, 5, 3, 1, with each stage lasting one quarter of the anneal.Fewer exposed planes mean weaker perturbations and increasingly deterministic operation.Source: D. SRAM Pseudo-Read Stochasticity and… · p. 7
- At VDDM = 0.6 V, measured RMS coefficient perturbation decreases monotonically as magnitude planes are protected.Removing higher-significance planes produces a larger reduction because adjacent bit planes differ by a factor of two in numerical weight.Source: D. SRAM Pseudo-Read Stochasticity and… · p. 7
- Device-level pseudo-read errors provide randomness, while digital plane-enable signals control its strength without fine analog VDDM regulation or dedicated noise-generation circuitry.The resulting cooling trajectory remains programmable.Source: D. SRAM Pseudo-Read Stochasticity and… · p. 7
E. End-to-End Dual-Mode Control Flow
The accelerator first constructs and programs QUBO coefficients in deterministic mode, then switches to probabilistic operation for annealing. In-memory evaluation, sequential updates, and periodic refreshes continue under the digital cooling schedule until the prescribed iteration count is reached.
E. End-to-End Dual-Mode Control Flow
- At VDDM = 0.9 V, the accelerator computes, quantizes, and programs QUBO coefficients before switching to VDDM = 0.6 V for probabilistic operation.The probabilistic phase uses pseudo-read disturbance and in-memory local-field evaluation.Source: E. End-to-End Dual-Mode Control Flow · p. 7, Figure 6 · p. 8
- Both modes use the shared SRAM-DCIM array and signed adder tree, with full signed MAC output used for coefficient construction and only the local-energy-difference sign used for spin updates.The same SRAM-DCIM substrate supports embedding scoring, QUBO solution, and annealing stochasticity.Source: Figure 6 · p. 8, E. End-to-End Dual-Mode Control Flow · p. 8
- During probabilistic operation, sequential spin updates and periodic coefficient refreshes repeat under the digital cooling schedule until the prescribed iteration count is reached.The final passage-selection state is returned for downstream answer generation.Source: E. End-to-End Dual-Mode Control Flow · p. 8, Figure 6 · p. 8
IV. NEAR-MEMORY RAG SYSTEM ARCHITECTURE
The architecture moves set-level context selection near the HBF-resident vector database, avoiding GPU-side processing of the full candidate pool. A proposed HBF base-die p-DCIM constructs and solves the QUBO locally, returning selected passage identifiers to the GPU.
A. Data Movement in Conventional RAG Systems
- Conventional RAG transfers retrieved candidates to the GPU, where context selection may require another LLM inference pass using weights streamed from HBM.For candidate pools of size N, pairwise QUBO terms can require O(N^2) similarity evaluations.Source: A. Data Movement in Conventional RAG… · p. 8
B. Proposed HBF-Integrated p-DCIM Architecture
- The proposed HBF-base-die p-DCIM keeps retrieved embeddings in the HBF subsystem during QUBO construction and selection, returning only selected passage identifiers to the GPU.The HBM stack stores LLM weights and is not accessed by p-DCIM during context selection.Source: Figure 7 · p. 9, B. Proposed HBF-Integrated p-DCIM… · p. 9
- The same SRAM-DCIM array and adder-tree datapath compute QUBO coefficients from embedding similarities and then solve the QUBO in probabilistic mode.The HBF-integrated organization is proposed; the fabricated work does not include the integrated HBM/HBF system.Source: B. Proposed HBF-Integrated p-DCIM… · p. 9
V. EXPERIMENTAL RESULTS AND SYSTEM EVALUATION
The evaluation combines measurements from the fabricated macro with a measurement-parametrized model for larger RAG workloads. It examines HotpotQA context selection, downstream answer quality, and system-level energy and latency.
A. Experimental Methodology and End-to-End Demonstration
- The evaluation directly executes QUBOs that fit the fabricated 64 × 64 macro and models larger RAG instances using measured SRAM pseudo-read statistics and finite-precision datapath characteristics.The model incorporates signed 8-bit coefficient quantization, sequential spin updates, periodic coefficient refresh, and digital plane-count cooling.Source: §V · p. 9, A. Experimental Methodology and… · p. 10
V. EXPERIMENTAL RESULTS AND SYSTEM EVALUATION
- The system evaluation compares downstream answer quality and energy per query across RAG pipeline stages, while measuring latency against an LLM-based GPU set-selection baseline.The evaluation figure reports answer-quality metrics and energy by pipeline stage.Source: §V · p. 9, Figure 9 · p. 10
A. Experimental Methodology and End-to-End Demonstration
- The HotpotQA evaluation supplies the query, requirements, and candidate passages before selection, with requirement generation and final answer generation remaining on the GPU.The fabricated chip selects complementary evidence passages and rejects an irrelevant candidate in the representative example.Source: A. Experimental Methodology and… · p. 10, Figure 8 · p. 10
B. QUBO Solution and RAG Answer Quality
On the evaluated workload, replacing the GPU’s LLM-based selection pass with p-DCIM reduces total query energy while maintaining competitive HotpotQA answer quality. The fabricated macro solves small QUBOs on-chip, while larger HotpotQA-scale results use a measurement-parametrized model.
B. QUBO Solution and RAG Answer Quality
- Across all ten measured instances that fit the fabricated macro, p-DCIM reached the brute-force minimum-energy state.Larger HotpotQA-derived QUBOs were evaluated with a measurement-parametrized model.Source: B. QUBO Solution and RAG Answer Quality · p. 10
- On evaluated HotpotQA examples, p-DCIM achieves token-level F1 of 0.8524 versus 0.7857 for both SetR and SetRO.Exact Match is 0.70 for p-DCIM and both LLM-based selectors; Gold-Contains is 0.90 for p-DCIM versus 0.80 for both selectors.Source: B. QUBO Solution and RAG Answer Quality · p. 10
C. Selection Latency and Energy
- More than six orders of magnitude: p-DCIM reduces selection-stage energy versus the SetR GPU selection pass, which consumes approximately 358 J per query.The p-DCIM estimate is approximately 0.9 µJ per selection, based on measured probabilistic-mode power of 0.413 mW and interpreted as macro-normalized, not a full-array projection.Source: C. Selection Latency and Energy · p. 11
- 67%: replacing only the selection stage reduces evaluated total query energy from approximately 536 J to 175 J.Requirement generation and final answer generation remain unchanged on the GPU.Source: C. Selection Latency and Energy · p. 11
VI. CONCLUSION
- The dual-mode architecture reuses the SRAM array and signed accumulation datapath for deterministic similarity-based QUBO construction and probabilistic local-energy evaluation during annealing.SRAM pseudo-read disturbances provide stochasticity, while a digital magnitude-plane schedule controls annealing strength.Source: §VI · p. 11
- HotpotQA-scale selection and the proposed HBF-integrated system are evaluated using a measurement-parametrized model derived from measured chip characteristics.The HBM/HBF-integrated system itself is proposed rather than fabricated.Source: §VI · p. 11