PaperlayerView exact revision on arXiv

Source-linked AI summary

CHIPSMORE: Compute-in-Interconnect and -Memory Chiplets for Multi-Mode Multi-Request LLM Inference Acceleration

Yue Jiet Chong, Yimin Wang, Zhen Wu, Zixuan Wang, Wei Zhang, Xuanyao Fong

arXiv:2608.30509v1cs.AR

TL;DR

LLM inference varies across adaptation modes, context lengths, and request concurrency, challenging utilization, memory efficiency, and scalable performance. CHIPSMORE combines heterogeneous CIM with compute-in-interconnect, hierarchical KV memory, shared-weight multi-request execution, and state-aware reconfiguration. Compared with Nvidia H100, it achieves up to 2.38× higher throughput and 27× higher energy efficiency on Mistral-7B inference while eliminating weight replication.

  • Problem

    LLM inference spans adaptation modes, context lengths, and request concurrency, while existing CIM accelerators commonly support only fixed or limited workload settings.

  • Method

    CHIPSMORE combines heterogeneous RRAM-ACIM and SRAM-DCIM processing with programmable IPCN computation, hierarchical KV memory, shared-weight multi-request execution, and state-aware resource management.

  • Results

    Compared with Nvidia H100, CHIPSMORE achieves up to 2.38× higher throughput and 27× higher energy efficiency on Mistral-7B inference while eliminating weight replication.

  • Takeaways & Limitations

    CHIPSMORE supports base-mode and LoRA inference across varying model sizes, context lengths, and batch sizes with replication-free multi-request execution and favorable power scaling.

Abstract

from arXiv · show

Large language model (LLM) inference exhibits substantial variability across adaptation modes, context lengths, and request concurrency, creating challenges for maintaining high utilization, memory efficiency, and scalable performance on compute-in-memory (CIM) accelerators. This paper presents CHIPSMORE, a multi-mode and multi-request LLM inference accelerator that integrates compute-in-interconnect and CIM to support both base-mode and low-rank adaptation (LoRA) inference under diverse workloads. CHIPSMORE employs heterogeneous processing elements consisting of resistive RAM analog compute-in-memory (RRAM-ACIM) and static RAM digital compute-in-memory (SRAM-DCIM) interconnected through a programmable Inter-PE computational network (IPCN). A composable hierarchical key-value (KV) memory scheme dynamically allocates router scratchpad, SRAM-DCIM, and embedded DRAM (eDRAM) resources according to workload requirements, enabling scalable support for long-context and batched inference. Furthermore, a non-replicated multi-request execution pipeline exploits request-level parallelism without duplicating pretrained weights, while a state-aware resource reconfiguration mechanism selectively retains runtime states and power-gates inactive resources to improve energy efficiency. Evaluation using cycle-accurate hardware-software co-simulation demonstrates that CHIPSMORE effectively sustains high throughput across varying model sizes, context lengths, and batch sizes while maintaining favorable power scaling. Compared with Nvidia H100, CHIPSMORE achieves up to $2.38\times$ higher throughput and $27\times$ higher energy efficiency on Mistral-7B inference while eliminating weight replication for multi-request serving.

I. INTRODUCTION

CHIPSMORE targets variable LLM inference workloads spanning adaptation modes, context lengths, and request concurrency. It combines heterogeneous CIM, programmable in-network computation, hierarchical memory, non-replicated serving, and state-aware power management.

  • LLM inference hardware requirements vary with model size, context length, adaptation mode, and request concurrency.
  • Existing CIM accelerators commonly assume fixed contexts, single memory roles, or single-batch execution, while weight replication increases storage, area, and leakage.
  • CHIPSMORE unifies adaptation-aware execution, KV-cache scaling, and multi-request serving through workload-oriented orchestration.
  • Static-weight operations execute in heterogeneous CIM macros, while runtime-generated operations execute through a programmable IPCN for communication and computation co-execution.
  • Its hierarchical KV memory dynamically combines router scratchpad, SRAM-DCIM, and eDRAM according to workload mode and capacity requirements.
  • The non-replicated pipeline shares weight-bearing chiplet clusters across requests, while state-aware clustering and power gating retain needed states and power off stateless resources.

II. BACKGROUND AND MOTIVATION

LoRA and multi-request serving create distinct storage and utilization demands for CIM accelerators. CHIPSMORE addresses these demands by combining heterogeneous memory resources with shared-weight concurrent execution and scalable runtime-state storage.

  • LoRA adds task-specific low-rank matrices while keeping pretrained parameters unchanged, creating distinct storage requirements for static and reconfigurable parameters.
  • Pretrained weights suit non-volatile memory, whereas LoRA parameters benefit from rapidly reconfigurable resources, motivating RRAM-ACIM and SRAM-DCIM together.
  • Multi-request inference improves decode parallelism and utilization because a single request generates only one token per autoregressive step.
  • Concurrent requests can reuse identical pretrained weights, but each maintains an independent KV cache whose capacity grows with retained context length.
  • Supporting concurrency therefore requires higher runtime-state capacity without replicating identical model weights, which otherwise increases storage, area, and power.

C. KV-Cache Scaling

KV-cache demand grows with context length and request concurrency, making scalable memory allocation central to CIM-based LLM acceleration. CHIPSMORE addresses this through a unified architecture combining hierarchical KV storage with persistence-aware compute and communication.

  • Each active request stores attention states in a KV cache, whose capacity grows with retained context length and request concurrency.
  • For B requests and T retained tokens, KV capacity depends on layers, K/V heads, head dimension, and bits per K/V element, with a factor of two for keys and values.
  • As context length and concurrency increase, aggregate KV requirements can exceed local-memory capacity, requiring flexible allocation that preserves locality.
  • Existing CIM accelerators often address adaptation, KV-memory management, and multi-request execution independently, leaving coupled workload dimensions under-orchestrated.
  • CHIPSMORE provides a unified chiplet architecture that jointly orchestrates computation, communication, memory allocation, and runtime-state management.

A. Processing Element (PE)

CHIPSMORE’s processing element combines nonvolatile analog and reconfigurable digital CIM to support static pretrained weights and LoRA parameters. Its surrounding IPCN also performs communication and runtime-data computation.

  • Each PE integrates RRAM-ACIM and SRAM-DCIM macros for heterogeneous storage and computation.RRAM-ACIM supports frozen pretrained weights, while SRAM-DCIM supports reconfigurable LoRA parameters.
  • RRAM-ACIM stores pretrained parameters persistently and performs matrix-vector multiplications directly in analog crossbar arrays.Its density and nonvolatility suit large weight matrices that remain unchanged during inference.
  • SRAM-DCIM stores LoRA matrices and executes their digital MAC operations without permanently modifying pretrained weights.Its fast write capability supports task-specific adaptation parameters.
  • SRAM-DCIM can alternatively serve as general-purpose on-chip storage when adaptation parameters are absent.This gives the PE a unified structure across operating modes.
  • The IPCN combines data communication with in-network computation for runtime-generated activations and intermediate tensors.Its routers execute DMAC operations, while PEs execute SMAC operations on locally stored weights.

1) Network Program Memory (NPM):

The IPCN uses programmable instruction storage and centralized control to reconfigure communication and computation schedules while overlapping instruction loading with network execution. Its routers provide the distributed communication and in-network-computing substrate.

  • The NPM stores IPCN instructions and enables software-defined control of data movement and in-network computation.Network behavior can be reconfigured for different AI workloads.
  • NPM banks separate active instruction execution from concurrent updates by the configuration controller.This double-buffered organization supports continuous operation.
  • The NMC fetches and decodes instructions, then coordinates routing, local-memory access, and DMAC operations across the IPCN.Centralized sequencing supports synchronized, workload-specific dataflow patterns.
  • Each unit router is both a communication node and an in-network computing engine in the 2D-mesh IPCN.A router is coupled with heterogeneous PEs to support both DMAC and SMAC execution.
  • Router data-I/O ports use FIFO buffers to support concurrent transfers across inter-router and router-PE connections.The controller coordinates packet routing, local accesses, internal transfers, and computation.

5) IPCN Instruction Set Architecture (ISA):

CHIPSMORE’s programmable IPCN ISA adapts routing, computation, and memory operations across workloads, while its memory and chiplet organization support capacity scaling beyond a single compute tile.

  • 5) IPCN Instruction Set Architecture (ISA):: The IPCN ISA exposes programmable communication and in-network computation while preserving a uniform hardware structure.This allows execution behavior to adapt across AI workloads.
  • 5) IPCN Instruction Set Architecture (ISA):: Each IPCN instruction is a 30-bit control word encoding routing, computation, and local-memory operations.Supported primitives include unicast, broadcast, and router-executed computation.
  • C. Embedded DRAM (eDRAM): Each compute tile includes multi-bank eDRAM for high-density storage beyond local router scratchpad and SRAM-DCIM capacity.Its 2T0C capacitorless cells support logic-process integration, non-destructive reads, and reduced refresh overhead.
  • D. Inter-CT Communication: A chiplet-based architecture connects multiple compute tiles to scale model size, memory capacity, and distributed execution.Within-tile communication uses the IPCN, while inter-tile communication is required for larger systems.
  • D. Inter-CT Communication: UCIe provides each compute tile with a standardized high-bandwidth, low-latency die-to-die interface.It carries activations, intermediate tensors, and memory data between neighboring chiplets.
  • D. Inter-CT Communication: CHIPSMORE localizes computation and storage within compute tiles whenever possible to limit inter-tile transfers.Adding compute tiles supports larger models, longer contexts, and higher request concurrency.

IV. WORKLOAD-ORIENTED RESOURCE ORCHESTRATION

CHIPSMORE orchestrates spatial mapping, hierarchical KV allocation, and non-replicated request scheduling to support base and LoRA inference across varying contexts and concurrency.

  • The resource-orchestration framework combines unified spatial mapping, hierarchical KV allocation, non-replicated multi-request execution, and state-aware reconfiguration.These mechanisms address variation in adaptation mode, context length, and request batch size.
  • A. Unified Spatial Mapping: Spatial mapping places pretrained and LoRA projection matrices using the same crossbar-oriented geometric decomposition.This preserves locality and enables a uniform execution model across inference modes.
  • A. Unified Spatial Mapping: Intermediate tensors are stored near their corresponding weight partitions, enabling local aggregation and subsequent accesses across operating modes.The spatial organization remains invariant, avoiding remapping when workloads adapt.
  • B. Hierarchical KV Memory Allocation: The hierarchical KV scheme dynamically uses router scratchpad, SRAM-DCIM, and eDRAM according to locality, density, and workload requirements.Router scratchpad is the first-tier KV storage because of its proximity to IPCN and PE resources.
  • B. Hierarchical KV Memory Allocation: In base mode, SRAM-DCIM is repurposed for KV storage; in LoRA mode, it is reserved for adaptation matrices.This mode-dependent policy reuses on-chip memory without dedicated provisioning.
  • B. Hierarchical KV Memory Allocation: KV data is striped across adjacent chiplets when a single compute tile cannot provide sufficient aggregate capacity.Capacity striping extends KV storage beyond one tile.

C. Non-Replicated Multi-Request Pipeline

CHIPSMORE increases multi-request throughput by interleaving requests across a layer-wise pipeline while sharing pretrained weights. State-aware reconfiguration preserves volatile runtime state and reduces power in inactive resources.

  • Non-Replicated Multi-Request Pipeline: The non-replicated pipeline raises throughput without duplicating pretrained weights stored in RRAM-ACIM arrays.Avoiding replication reduces the associated RRAM area, chiplet count, leakage power, and system cost described for replicated designs.
  • Non-Replicated Multi-Request Pipeline: CHIPSMORE injects requests at temporal offsets across statically assigned transformer layers, allowing multiple requests to occupy different pipeline stages simultaneously.This layer-wise organization forms a pipeline across unique weight-bearing chiplet clusters.
  • Non-Replicated Multi-Request Pipeline: Pretrained weights are shared across active requests, while each request retains independent sequence context, KV cache, and generation state.Intermediate activations move through the pipeline while request-specific runtime state remains separate.
  • Non-Replicated Multi-Request Pipeline: State-aware reconfiguration retains resources containing LoRA parameters or active KV-cache data while transitioning completed clusters toward reduced-power states.Power management accounts for heterogeneous retention requirements across nonvolatile weights and volatile runtime state.
  • Non-Replicated Multi-Request Pipeline: Power scales primarily with active computation and retained runtime state rather than total model capacity.As context and concurrency increase, additional power is incurred predominantly by resources required for KV retention.

A. Performance Benchmark

Performance declines with larger models and longer contexts because computation, pipeline depth, KV capacity, and memory traffic increase. Larger batches improve throughput by filling the shared pipeline, but contention and pipeline overhead make the gains sub-linear.

  • Performance Benchmark: Throughput decreases as model size and context length increase because larger workloads require more computation and deeper layer pipelines.Qwen3-14B is particularly affected by its greater transformer depth and dimensions.
  • Performance Benchmark: Longer contexts reduce throughput through quadratic prefill attention, growing decode-time KV access, and increased retained KV capacity.The attention-score matrix scales as O(T^2) during prefill, while decoding accesses previously retained tokens.
  • Performance Benchmark: Extra-long contexts can activate eDRAM when router scratchpad and SRAM-DCIM capacity is insufficient, adding memory-access latency and interconnect traffic.The hierarchical memory scheme avoids off-chip accesses but does not eliminate the throughput cost of greater eDRAM reliance.
  • Performance Benchmark: 3.91×, 2.71×, and 2.77× end-to-end throughput improvements occur for Llama 3.2-1B, Mistral-7B, and Qwen3-14B, respectively, when batch size rises from 1 to 4 on short workloads.The gains result from interleaving requests while sharing one physical pretrained-weight copy.
  • Performance Benchmark: Batch-size throughput gains are sub-linear, especially for larger models, because of pipeline fill/drain latency, exclusive stage occupancy, and shared-resource contention.The contended resources include IPCN communication, memory bandwidth, and KV-cache access paths.
  • Performance Benchmark: Hierarchical allocation repurposes SRAM-DCIM for KV storage when LoRA support is unnecessary, increasing effective on-chip capacity and reducing external-memory requirements.KV requirements grow proportionally with context length across base and LoRA modes.

C. Performance Comparison of LoRA vs Base Modes

LoRA throughput remains close to base-mode throughput across most operating points, with long-context degradation driven mainly by KV-memory contention rather than adaptation computation. Higher batch sizes improve utilization and amortize added memory latency.

  • Performance Comparison of LoRA vs Base Modes: LoRA throughput remains close to base-mode throughput across most operating points, indicating relatively small low-rank adaptation-computation overhead.The dominant degradation source is the altered KV-memory allocation policy.
  • Performance Comparison of LoRA vs Base Modes: ∼71% of corresponding base-mode performance occurs for larger models at long context and batch size one when LoRA reserves SRAM-DCIM for adaptation weights.Half of the KV data is forced into eDRAM, increasing retrieval latency during prefill and decode attention.
  • Performance Comparison of LoRA vs Base Modes: 0.95× performance factor is reached at batch size four for the same long-context larger-model workloads.Request-level parallelism improves pipeline utilization and amortizes the additional memory latency.
  • Performance Comparison of LoRA vs Base Modes: 0.94× performance factor is reached for Llama 3.2-1B at batch size four after extra-long-context memory overflow affects LoRA allocation.The smaller model’s KV footprint avoids this effect at shorter contexts and batch size one.
  • Performance Comparison of LoRA vs Base Modes: ∼5% throughput penalty remains for most long-context and high-concurrency operating points.The reported overhead is primarily attributed to KV-memory resource contention rather than low-rank adaptation computation.
  • Performance Comparison of LoRA vs Base Modes: Pipeline utilization increases with batch size as independent requests fill idle stages without replicating pretrained weights.Deeper models such as Mistral-7B remain harder to fully populate than the 16-layer Llama 3.2-1B pipeline.

E. Time-to-first-token (TTFT) and Inter-token latency (ITL)

TTFT and ITL increase with model size and context length, while batching improves occupancy and amortizes pipeline overhead. Inter-chiplet utilization remains below saturation, but IPCN and router activity dominate power and LoRA consumes more power than base mode.

  • Time-to-first-token (TTFT) and Inter-token latency (ITL): 14× higher TTFT results when prompt length increases from short to extra-long, reflecting quadratic attention workload and increased communication.Increasing batch size from 1 to 4 raises TTFT by only ∼20% because of interleaved pipeline execution.
  • Time-to-first-token (TTFT) and Inter-token latency (ITL): ITL generally increases with model size and context length because decoding requires more KV-cache access, attention processing, and inter-tile communication.Additional requests can improve occupancy and amortize pipeline fill/drain overhead.
  • Time-to-first-token (TTFT) and Inter-token latency (ITL): Below 25% maximum UCIe utilization is observed even at batch size four, indicating that inter-chiplet bandwidth is not a system bottleneck.The majority of data movement remains localized within compute tiles.
  • Time-to-first-token (TTFT) and Inter-token latency (ITL): IPCN and router activity is the dominant contributor to overall system power across evaluated configurations.The subsystem performs inter-router communication, collective operations, and DMAC computations during inference.
  • Time-to-first-token (TTFT) and Inter-token latency (ITL): LoRA mode consistently consumes higher power than corresponding base-mode configurations because SRAM-DCIM remains active for adaptation weights and computations.This also increases eDRAM utilization compared with base mode.
  • Time-to-first-token (TTFT) and Inter-token latency (ITL): Increasing batch size generally improves energy efficiency because pipeline overhead is amortized while throughput increases faster than power consumption.Efficiency decreases with model size and context length, while power scaling remains more favorable than throughput scaling.

H. Comparison with SoTA

CHIPSMORE combines heterogeneous CIM processing, programmable in-network computation, hierarchical KV memory, and state-aware execution to support diverse LLM inference workloads. It delivers strong throughput and energy efficiency while avoiding pretrained-weight replication for multi-request serving.

  • 1112.5 tokens/s and 3003.9 tokens/s are achieved at batch sizes 1 and 4, respectively, with average power of 30.7 W and 46.4 W.
  • CHIPSMORE integrates heterogeneous RRAM-ACIM and SRAM-DCIM processing elements with a programmable IPCN for persistence-aware partitioning and in-network computation.
  • A hierarchical KV-memory organization dynamically coordinates router scratchpad, SRAM-DCIM, and eDRAM resources for workload-dependent KV-cache requirements.
  • A non-replicated multi-request pipeline improves hardware utilization without duplicating pretrained weights.
  • CHIPSMORE supports base-mode and LoRA inference with scalable KV-cache management, replication-free multi-request execution, and favorable power scaling.
  • 2.38× higher throughput and 27× higher energy efficiency than Nvidia H100 are achieved on Mistral-7B inference while eliminating weight replication for multi-request serving.
Loading 2608.30509v1…