Source-linked AI summary
Processing Data Where It Makes Sense: Enabling In-Memory Computation
Onur Mutlu, Saugata Ghose, Juan Gómez-Luna, Rachata Ausavarungnirun
TL;DR
Modern systems move data to computation even as memory scaling becomes difficult and data movement imposes major latency and energy costs. The paper develops low-cost PIM approaches that compute in or near memory, reports broad application and architectural benefits, and identifies programming and runtime support as adoption challenges.
Problem
Growing working sets, difficult DRAM scaling, and the high latency and energy cost of moving data leave conventional processor-centric systems poorly matched to data-intensive workloads.
Method
The paper explores low-cost PIM through massively parallel bulk operations within DRAM and processing logic in 3D-stacked memory, with cross-layer design and adoption analysis.
Results
PIM improves data-intensive applications and reduces data movement, including 13.8× average system-performance improvement and 87% average energy reduction for Tesseract graph workloads.
Takeaways & Limitations
Computation can be placed in or near memory to reduce unnecessary data movement, while PIM designs can target real platforms with low-cost changes.
Takeaways & Limitations
Widespread PIM adoption still requires easier programming models, compiler and library integration, and runtime support for scheduling, coherence, and virtual memory.
Abstract
from arXiv · showhide
Today's systems are overwhelmingly designed to move data to computation. This design choice goes directly against at least three key trends in systems that cause performance, scalability and energy bottlenecks: (1) data access from memory is already a key bottleneck as applications become more data-intensive and memory bandwidth and energy do not scale well, (2) energy consumption is a key constraint in especially mobile and server systems, (3) data movement is very expensive in terms of bandwidth, energy and latency, much more so than computation. At the same time, conventional memory technology is facing many scaling challenges in terms of reliability, energy, and performance. As a result, memory system architects are open to organizing memory in different ways and making it more intelligent, at the expense of higher cost. The emergence of 3D-stacked memory plus logic as well as the adoption of error correcting codes inside DRAM chips, and the necessity for designing new solutions to serious reliability and security issues, such as the RowHammer phenomenon, are an evidence of this trend. Recent research aims to practically enable computation close to data. We discuss at least two promising directions for processing-in-memory (PIM): (1) performing massively-parallel bulk operations in memory by exploiting the analog operational properties of DRAM, with low-cost changes, (2) exploiting the logic layer in 3D-stacked memory technology to accelerate important data-intensive applications. In both approaches, we describe and tackle relevant cross-layer research, design, and adoption challenges in devices, architecture, systems, and programming models. Our focus is on the development of in-memory processing designs that can be adopted in real computing platforms at low cost.
1. Introduction
Growing application working sets and costly data movement are worsening the main-memory bottleneck. The paper presents processing-in-memory as a data-centric alternative and discusses adoption challenges for two PIM approaches.
- Growing application working sets make main memory a significant bottleneck, while capacity, energy, cost, and performance become increasingly difficult to scale together.
- Moving data from DRAM to the CPU incurs substantial latency and energy costs, especially when cached data is not reused.
- Processor-centric systems treat memory as an unintelligent storage worker, creating inefficiencies as applications become increasingly data-centric.
- Processing-in-memory, also called near-data processing, performs operations in memory or with processing logic inside the DRAM subsystem.
- The paper examines massively parallel DRAM bulk operations and logic-layer acceleration in 3D-stacked memory, alongside device, architecture, systems, and programming challenges.
2. Major Trends Affecting Main Memory
Main memory faces simultaneous scaling challenges in capacity, bandwidth, latency, reliability, and energy. Technology scaling and workloads make these constraints increasingly costly and difficult to manage.
- DRAM scaling is increasingly challenged in capacity, energy efficiency, and reliability as applications demand higher memory capacity and performance.
- Core counts double every two years while DRAM capacity doubles every three years, reducing memory capacity per core by approximately 30% every two years.
- Over twenty years, DRAM chip capacity improved around 128×, bandwidth around 20×, and latency declined only 30%.
- Smaller DRAM cells become less reliable, and aggressive scaling creates security vulnerabilities such as RowHammer-induced bit flips in adjacent rows.
- DRAM consumes energy even when unused because periodic refresh is required, while increasing capacity, complexity, and off-chip placement worsen energy challenges.
3. The Need for Intelligent Memory Controllers to Enhance Memory Scaling
Intelligent memory controllers can address memory-scaling challenges by adapting reliability, refresh, latency, energy, and parallelism mechanisms to device and application characteristics. The paper also positions controller intelligence as a path toward computation near data.
- Programmable memory controllers can manage reliability, latency, power, and application computation using knowledge of DRAM and workload characteristics.
- PARA probabilistically refreshes rows adjacent to activated rows, providing RowHammer protection with low power, performance, and chip-area overhead.
- Per-row retention-time tracking can avoid unnecessary worst-case refreshes, reducing refresh-related energy waste and performance loss.
- At 55 °C rather than 85 °C, read and write DRAM latencies can be reduced by 33% and 55%, respectively, across 115 real modules.
- Profiling manufacturing variation can reduce DRAM access latencies by as much as 57% for cells that reliably support lower timing parameters.
- DRAM modifications and intelligent controllers can exploit subarray-level parallelism, fast and slow rows, and charge-aware timing to improve access behavior.
4. Perils of Processor-Centric Design
Processor-centric systems separate computation from storage, making data movement a major source of latency, bandwidth pressure, and energy use. These costs motivate data-centric architectures that compute where data resides.
- The von Neumann separation of computation and storage requires data to move back and forth across an off-chip bus.
- Accessing memory requires requests, DRAM transfers, cache placement, and CPU execution, creating a costly multi-step path.
- More than 62% of system energy was observed in data movement across four major mobile consumer workloads.
- Narrow off-chip buses limit memory bandwidth, while complex cache and latency-hiding mechanisms consume die area and energy.
- Caches can be inefficient for streaming, random-access, and graph workloads because fetched cache-line data is often not reused or used fully.
- Off-chip interconnects add significant latency and energy to every DRAM data transfer and limit system performance and energy efficiency.
- Memory accesses consume two to three orders of magnitude more energy than complex additions, with one cited estimate at approximately 115×.
- Data-centric architectures aim to perform computation with minimal data movement and where the data resides.
5. Processing-in-Memory (PIM)
Processing-in-memory equips the memory subsystem to perform computation, reducing unnecessary movement between memory and compute units. The paper presents two approaches: minimally modifying DRAM for bulk operations and using logic in 3D-stacked memory for broader application acceleration.
- PIM gives the memory subsystem computation capability to eliminate unnecessary data movement between memory and compute units.
- Approach I: Minimally Changing Memory Chips: The first approach minimally modifies DRAM to exploit its architecture and circuitry for bulk processing operations.
- Approach II: PIM using 3D-Stacked Memory: The second approach places general-purpose processing logic in 3D-stacked memory, using high internal bandwidth and low latency between logic and memory layers.
5.1. Approach I: Minimally Changing Memory Chips
The first PIM approach minimally changes DRAM to perform bulk data movement, bitwise operations, and data reorganization inside memory. These mechanisms reduce movement costs and improve bandwidth utilization across several workloads.
- Minimal-change PIM exploits DRAM interconnects and analog behavior to perform computation without a dedicated logic layer.
- RowClone: Bulk data copy and initialization are important bandwidth-intensive operations because operating systems and data centers spend significant time performing them.
- RowClone: RowClone uses DRAM structure to copy or initialize data inside a DRAM chip through Fast Parallel Mode and Pipelined Serial Mode.
- RowClone: 11.6× lower latency and 74.4× lower energy for a 4kB bulk page copy require only 0.01% DRAM chip area overhead.
- Ambit: Ambit performs bulk bitwise operations by using DRAM charge sharing for AND and OR and sense-amplifier inverters for NOT.
- Ambit: 44× higher throughput than an Intel Skylake processor and 32× higher throughput than an NVIDIA GTX 745 GPU are achieved across seven bitwise operations.
- Gather-Scatter DRAM: GS-DRAM reorganizes data across chips so one command can gather strided values concurrently, improving access to non-unit-stride patterns.
- Gather-Scatter DRAM: 3× database speedup for transactional workloads, 2× for analytics, and 10% faster matrix multiplication demonstrate GS-DRAM’s workload benefits.
5.2. Approach II: PIM using 3D-Stacked Memory
The second PIM approach uses processing logic in 3D-stacked memory to access nearby memory with high bandwidth and low latency. The paper applies this approach to graph processing, consumer workloads, GPUs, and lightweight PIM-enabled instructions.
- PIM engines in 3D-stacked memory can execute application portions or entire applications while accessing stacked memory through high-bandwidth, low-latency connections.
- Tesseract: Graph Processing: Graph processing stresses memory bandwidth through large random accesses across large memory regions and limited cache efficiency.
- Tesseract: Graph Processing: Tesseract moves functions to data and uses programmable PIM acceleration, message passing, and specialized prefetchers for graph workloads.
- Tesseract: Graph Processing: 13.8× average performance improvement and 87% average energy reduction were measured across five graph-processing workloads with large real-world graphs.
- Consumer Workloads: Consumer workloads spend 62.7% of total system energy on average moving data across the memory hierarchy.
- Consumer Workloads: PIM cores and fixed-function accelerators implement simple consumer-workload operations while reducing average energy by 55.4% and execution time by 54.2%.
- GPU Systems: 3D-stacked memory can connect GPU cores directly to DRAM layers, but computation offloading and data mapping across memory cubes must be enabled.
- GPU Systems: TOM combines compiler offloading analysis with runtime resource decisions and page placement that co-locates data with executing PIM cores.
6. Enabling the Adoption of PIM
Widespread PIM adoption requires programming, runtime, coherence, and virtual-memory mechanisms that integrate near-data execution without imposing excessive burden or overhead. The paper surveys proposed solutions while identifying unresolved design questions.
- Programming Model and Code Generation: PIM adoption requires programming models, compiler support, and libraries that hide architectural details from most programmers.The paper identifies programming models and compiler/library assistance as open research questions.
- Programming Model and Code Generation: PIM-enabled instructions suit small computations but can incur overhead for large tasks because information must frequently move between PIM logic and the CPU.The paper proposes integrating PIM instructions with compiler-based methods or library calls to reduce programmer burden.
- PIM Runtime: Scheduling and Data Mapping: Runtime systems must select code, schedule offloading, map data across memory modules, and share PIM engines among threads and cores.These decisions determine when PIM execution is worthwhile and whether it can maximize performance and energy efficiency.
- Memory Coherence: LazyPIM maintains coherence without issuing a coherence request for every memory access, reaching within 5.5% of ideal performance and 4.4% of ideal energy consumption.It speculatively acquires permissions and later checks compressed, batched coherence lookups for violations.
- Virtual Memory Support: IMPICA performs virtual-memory pointer chasing inside 3D-stacked memory and improves linked-list, hash-table, and B-tree performance by 92%, 29%, and 18%.On DBx1000, it improves transaction throughput by 16%, response time by 13%, and reduces overall energy consumption by 6%.
- Virtual Memory Support: Efficient PIM address translation and access protection remain challenging for general applications, especially those using large amounts of virtual memory.The paper presents IMPICA as a tractable solution for pointer chasing but identifies broader virtual-to-physical translation as unresolved.
- PIM Runtime: Scheduling and Data Mapping: PIM-managed concurrent data structures can outperform state-of-the-art concurrent structures executed on multiple cores, while broader data-structure support remains future work.The cited work specifically motivates extending PIM-managed designs to hash tables, search trees, and priority queues.
7. Conclusion and Future Outlook
Modern processor-centric systems incur substantial data-movement costs because computation is separated from memory. The paper advocates data-centric PIM designs and emphasizes that programming and runtime support remain necessary for broad adoption.
- Conclusion and Future Outlook: Data movement is a major performance and energy bottleneck because modern systems perform computation only in processors and accelerators away from data.The processor-centric design therefore moves large amounts of data through the memory hierarchy.
- Conclusion and Future Outlook: PIM enables a data-centric design by placing computation in or near memory to minimize data movement.The paper frames this as a paradigm shift away from the rigid separation between computing and memory units.
- Conclusion and Future Outlook: The paper demonstrates DRAM-based bulk operations and 3D-stacked-memory logic as two approaches for improving performance and reducing energy across applications and platforms.The approaches respectively use minimal memory-chip changes and embedded computation in the logic layer.
- Conclusion and Future Outlook: Widespread PIM adoption still requires easy-to-use programming models, compiler and library abstractions, and runtime support for scheduling, sharing, coherence, and virtual memory.The paper identifies these mechanisms as key remaining challenges and provides benchmarks, identification methodology, and simulation infrastructure to support future research.