Source-linked AI summary
Corona: System Implications of Emerging Nanophotonic Technology
Dana Vantrease, Robert Schreiber, Matteo Monchiero, Moray McLaren, Norman P. Jouppi, Marco Fiorentin, Al Davis, Nathan Binkert, Raymond G. Beausoleil, Jung Ho Ahn
TL;DR
Many-core processors will require orders-of-magnitude more memory and inter-core bandwidth than electrical pins and interconnects may provide. The paper presents Corona, a 3D many-core system using optical memory connections and a photonic crossbar, and reports 2 to 6 times better performance on memory-intensive workloads than an electrical alternative while dissipating less interconnect power.
Problem
Many-core scaling creates bandwidth demands that electrical pins and interconnects may not meet at acceptable performance, power, and area.
Method
The paper designs Corona, using optically connected memories, a DWDM photonic crossbar, and optical arbitration in a 3D many-core NUMA system.
Results
2 to 6 times better performance on memory-intensive workloads was found for optically connected systems, while dissipating much less interconnect power than electrical systems.
Takeaways & Limitations
Nanophotonics can be a compelling solution to memory and network-on-chip bandwidth walls while ameliorating the power wall.
Takeaways & Limitations
Integrating many photonic devices and designing power-efficient analog control electronics remain important unresolved problems.
Abstract
from arXiv · showhide
We expect that many-core microprocessors will push performance per chip from the 10 gigaflop to the 10 teraflop range in the coming decade. To support this increased performance, memory and inter-core bandwidths will also have to scale by orders of magnitude. Pin limitations, the energy cost of electrical signaling, and the non-scalability of chip-length global wires are significant bandwidth impediments. Recent developments in silicon nanophotonic technology have the potential to meet these off- and on- stack bandwidth requirements at acceptable power levels. Corona is a 3D many-core architecture that uses nanophotonic communication for both inter-core communication and off-stack communication to memory or I/O devices. Its peak floating-point performance is 10 teraflops. Dense wavelength division multiplexed optically connected memory modules provide 10 terabyte per second memory bandwidth. A photonic crossbar fully interconnects its 256 low-power multithreaded cores at 20 terabyte per second bandwidth. We have simulated a 1024 thread Corona system running synthetic benchmarks and scaled versions of the SPLASH-2 benchmark suite. We believe that in comparison with an electrically-connected many-core alternative that uses the same on-stack interconnect power, Corona can provide 2 to 6 times more performance on many memory-intensive workloads, while simultaneously reducing power.
1 Introduction
Many-core scaling will drive memory and inter-core bandwidth needs beyond what electrical pins and on-chip networks can practically provide. The paper presents Corona as a 3D nanophotonic architecture intended to address these bandwidth demands at acceptable power levels.
- Bandwidth Challenges: Orders-of-magnitude growth in memory and inter-core bandwidth demand threatens to outpace pin-count and electrical-interconnect scaling.The ITRS roadmap predicts less than 2x pin-count growth over the next decade.
- Nanophotonic Opportunity: Nanophotonics can support global communication with length-independent endpoint energy and higher bandwidth density through DWDM.Recent devices approach the dimensions of electrical buffers and wires.
- Nanophotonic Opportunity: 3D packaging lets optics, logic, DRAM, non-volatile memory, and analog circuitry occupy separate stacked die connected through TSVs.Using the third dimension also eases layout and reduces worst-case wire lengths.
- Corona: Corona combines 256 cores, 64 four-core clusters, and an all-optical DWDM crossbar in a 3D NUMA system.The crossbar targets cache coherence with near-uniform on-stack and memory communication latencies.
- Paper Scope: The paper describes nanophotonic technology, the Corona architecture, and a comparison with a comparable all-electrical many-core alternative.Its stated contribution is to evaluate compatibility with future CMOS and communication performance per area and energy.
2 Photonic Technology
The paper describes silicon nanophotonic components for dense, energy-efficient communication and identifies integration and control electronics as remaining challenges. Ring resonators provide wavelength-selective modulation, injection, and detection within DWDM networks.
- Photonic Advantages: Photonic interconnects can improve energy efficiency at high speeds and long distances while carrying many channels per waveguide.This increases bandwidth density and reduces the number of wires needed.
- Required Components: A complete nanophotonic network requires waveguides, light sources, modulators, photodiodes, and injection switches built in a CMOS-compatible process.The components collectively carry, encode, detect, and route optical signals.
- Required Components: Germanium absorbs light and converts it into an electrical signal for photonic detection.Unstrained germanium is paired with light around 1.3 µm for fiber-compatible off-stack communication.
- Modulation: A narrow-linewidth laser supports dense channel packing, while a continuous-wave laser with a separate modulator provides an alternative encoding approach.Wavelength-selective silicon modulators exceeding 10 Gb/s have been demonstrated.
- Ring Resonators: Ring resonators selectively suppress, transfer, or detect resonant wavelengths through coupling, attenuation, or germanium absorption.The same ring structure can function as a modulator, injector, or wavelength-selective detector.
- Other Components: A broadband splitter divides optical power and data across waveguides while leaving the unsplit signal unchanged apart from reduced strength.It operates across all wavelengths rather than selecting a resonant wavelength.
- Open Challenges: Integrating many DWDM devices on one chip, correcting fabrication variation, and designing efficient analog control electronics remain important problems.The authors still regard DWDM photonic networks as a credible response to rising many-core bandwidth needs.
3 The Corona Architecture
Corona is a tightly coupled, highly parallel NUMA architecture designed to reduce the management burden of scaling programs and data across many cores. Its optical interconnect provides high bandwidth and near-uniform latency for up to 1024 simultaneous threads.
- Architecture Goals: Corona uses homogeneous cores and caches, a near-uniform-latency crossbar, fair arbitration, and one byte per flop of core-to-memory bandwidth.These design choices aim to lessen programmer, compiler, and runtime management burdens in a scalable NUMA system.
- System Scale: 256 multithreaded in-order cores support up to 1024 threads, 10 teraflops of computation, 20 TB/s of on-stack bandwidth, and 10 TB/s of off-stack memory bandwidth.These are stated peak architectural capabilities.
- System Organization: The conceptual system view and sample layout include optical waveguides, optical memory connections, and other interconnect components.The layout covers both on-stack and off-stack optical communication paths.
3.1 Cluster Architecture
Corona’s clusters combine private L1 caches, shared L2 caches, and hubs that route traffic among computation, memory, coherence, and optical components. The architecture uses modest, power-conscious design choices, with resource and power estimates grounded in scaled reference cores.
- Each cluster’s hub routes traffic between the shared L2, directory, memory controller, network interface, optical bus, and optical crossbar.
- The cluster design uses modest subcomponent parameters rather than optimizing branch prediction, execution units, cache sizes, or cache policies.These choices target a 16 nm process in 2017.
- Cores: Dual-issue, in-order, four-way multithreaded cores are selected primarily to reduce power across hundreds of cores.
- Cores: Power estimates use scaled Penryn and Silverthorne reference cores, yielding 82–155 watts for the processor, caches, memory controller, and hub.
- Corona uses a MOESI directory protocol backed by a broadcast bus, but coherence is included only for die-size and power estimation, not system simulation.The simulations therefore provide a first-order indication of potential performance.
3.2 On-Stack Photonic Interconnect
Corona’s on-stack network uses an optical DWDM crossbar, broadcast bus, and token arbitration to provide high-bandwidth communication while managing coherence traffic and contention.
- 39 W is the estimated photonic interconnect power, including the analog circuit layer and photonic-die laser power.
- Optical Crossbar: The crossbar’s many-writer single-reader channels are replicated 64 times to form a fully connected 64 × 64 interconnect.Each channel originates at its destination cluster, passes the other clusters, and terminates at its origin.
- Optical Crossbar: 20 TB/s total crossbar bandwidth connects Corona’s clusters through 256-wavelength channels carried on bundled waveguides.Each wavelength signals at 10 Gb/s using modulation on both clock edges.
- Optical Broadcast Bus: The broadcast bus sends invalidation messages on a first pass and enables clusters to read and snoop them on a second pass.This avoids redundant unicast invalidates that could congest the crossbar.
- Arbitration: The distributed optical token scheme fairly allocates channels in round-robin order while tolerating variability in request, modulation, and propagation times.Under low contention, a cluster may wait up to 8 processor clock cycles for an uncontested token.
- Arbitration: Figure 5 depicts wavelength-specific tokens circulating among clusters, with requesting clusters diverting tokens and transmitting on the corresponding channels.
3.3 Optically Connected Memory
Optically connected memory uses a 3D optical-and-DRAM stack and a fiber loop to provide high external bandwidth with small expansion overhead and similar access latency across modules.
- 10 TB/s target external memory bandwidth is pursued because an electrical interconnect would require over 160 W at the assumed interconnect energy.
- An optical link connects the processor to a string of OCM modules using paired fibers for half-duplex communication and controller-scheduled transfers.The memory controller is the master and target modules divert or modulate optical signals for communication.
- The OCM stack combines one optical die with multiple DRAM dies, organized so a single mat reads or writes an entire cache line.3D stacking minimizes interconnect delay and power between the optical loop and DRAM mats.
- Adding OCMs to the fiber loop requires modulators and detectors but no additional lasers, keeping incremental communication power small.Because light passes through without buffering or retiming, access latency remains similar across modules.
3.4 Chip Stack
Corona separates digital, memory, optical, and analog functions across a 3D die stack, using face-to-face bonding and TSVs for compact interconnects.
- Most signal activity and heat reside in the top die, which contains the clustered cores and L1 caches.
- The processor die bonds face-to-face with the L2 die, directly connecting clusters to L2 caches, hubs, memory controllers, and directories.
- The bottom die contains optical structures and bonds face-to-face with analog electronics controlling detector circuits, ring resonance, and modulation.
- Signal through-silicon vias connect L2 endpoints to the analog die, while only power, ground, and clock vias traverse all three die.Face-to-face bonds carry most die-to-die signals and reduce layout impact.
4 Experimental Setup
The evaluation combines synthetic network workloads with scaled SPLASH-2 applications and trace-driven simulation across optical and electrical network and memory configurations.
- Synthetic workloads stress specific interconnect features, while scaled SPLASH-2 applications provide realistic performance measurements.The SPLASH-2 applications remain essentially unmodified, with larger datasets used when possible.
- 1024-threaded application instances generate L2-miss and synchronization traces for a separate network simulator.The network simulator models requests through on-stack and off-stack transactions.
- Finite buffers, queues, and ports model bandwidth, latency, back pressure, and capacity limits throughout the simulated memory system.
- Network Configurations: The optical crossbar provides 20.48 TB/s bisection bandwidth with at most 8 clocks of signal propagation.
- Memory Configurations: OCM supplies 10.24 TB/s off-stack bandwidth, whereas ECM supplies 0.96 TB/s; both use 20 ns memory latency.
- Configurations: The study compares XBar/OCM, HMesh/OCM, LMesh/OCM, HMesh/ECM, and LMesh/ECM to separate memory and interconnect effects.
5 Performance Evaluation
Corona’s optical memory and crossbar configurations improve performance on bandwidth- and latency-sensitive workloads, while electronic meshes can become power-prohibitive. Benefits vary by application according to memory bandwidth, latency, and traffic burstiness.
- Synthetic benchmarks: 3.28 geometric-mean speedup results from replacing ECM with OCM in HMesh systems on synthetic benchmarks.Adding the photonic crossbar provides a further 2.36 speedup on those benchmarks.
- SPLASH-2 applications: Four SPLASH-2 applications are fully adequate on LMesh/ECM systems because their cache-miss rates and main-memory bandwidth demands are low.The applications are Barnes, Radiosity, Volrend, and Water-Sp.
- SPLASH-2 applications: Fast memory benefits Cholesky, FFT, Ocean, and Radix only when paired with the fast crossbar, while LU and Raytrace gain mainly from OCM latency improvements.LU and Raytrace do not require substantially more bandwidth than ECM provides.
- Bandwidth and latency: 2–5 TB/s requirements characterize three synthetic tests and four applications that benefit most from the XBar/OCM configuration.LU and Raytrace instead appear to benefit mainly from lower latency because their memory traffic is bursty.
- Bandwidth and latency: Average L2-miss latency includes queue waiting and interconnect transit, and bursty traffic can produce high latency even at low overall bandwidth.LU and Raytrace improve dramatically with OCM and further with the optical crossbar.
- Power: 100 W or more can make electronic-mesh network power prohibitive for applications with significant memory demands, despite lower performance.For applications fitting in L2, the photonic crossbar can dissipate more power than electronic meshes when mesh leakage is ignored.
6 Related Work
Prior optical interconnect proposals use hierarchical buses, circuit-switched meshes, or earlier optical crossbars, whereas Corona integrates optical arbitration and DWDM communication into a 3D chip stack. Related electrical and wireless approaches provide alternative communication paths but retain mesh or coupling trade-offs.
- Optical interconnects: Corona’s crossbar differs from several optical CMP proposals by using optical rather than electrical arbitration and control.Earlier proposals include an optical global layer and an optical circuit-switched mesh managed by electrical control packets.
- Optical crossbars: Earlier optical crossbars relied on expensive VCSELs and free-space gratings, unlike Corona’s solution integrated into a modern 3D chip stack.The comparison concerns wavelength demultiplexing and physical integration.
- Alternative interconnects: Radio-frequency, capacitive, and inductive approaches offer alternative chip or package communication mechanisms, while Chang et al. combine a 2D mesh with RF shortcuts and frequency-division channels.The RF proposal uses frequency-division multiple access similarly to Corona’s DWDM approach.
- Electrical CMP interconnects: Electrical CMP examples include crossbars in 8- and 16-core Niagara chips and 2D meshes in 80-core Polaris and 64-core Raw chips.The cited mesh designs are easy to lay out but incur high nonlocal latency and limited bisection bandwidth.
7 Conclusions
Corona addresses projected many-core bandwidth growth with optically connected memory and an optical crossbar in a 3D architecture. Simulations indicate substantially better performance and lower interconnect power than electrical-only alternatives on memory-intensive workloads.
- Motivation: Orders-of-magnitude growth in memory and inter-core bandwidth is expected as per-socket many-core performance increases over the coming decade.Nanophotonics is presented as potentially providing the required bandwidth at acceptable power levels.
- Corona architecture: 10 TB/s of memory bandwidth comes from 64 optically connected memory modules through 128 DWDM fibers.Corona’s photonic crossbar provides near-uniform latency and 20 TB/s of on-stack bandwidth.
- Evaluation: A simulated 1024-thread Corona system runs synthetic benchmarks and scaled versions of the SPLASH-2 suite.The simulations evaluate the architecture’s potential benefits across both benchmark types.
- Results: 2 to 6 times better performance is reported for memory-intensive workloads, with much less interconnect power than systems using only electrical interconnects.The authors conclude that nanophotonics can address memory and network-on-chip bandwidth walls while ameliorating the power wall.