PaperlayerView exact revision on arXiv

Source-linked AI summary

Dissecting the Graphcore IPU Architecture via Microbenchmarking

Zhe Jia, Blake Tillman, Marco Maggioni, Daniele Paolo Scarpazza

arXiv:1912.03413v1cs.DCcs.ARcs.PF

TL;DR

The report asks how the IPU’s architecture determines AI/ML performance and how that performance compares with theoretical limits. It answers through microbenchmarks of memory, interconnects, host transfers, matrix multiplication, and convolution. The measurements characterize the IPU’s performance behavior while noting that some collective results and system-scale conclusions are limited by the tested configuration.

  • Problem

    The report investigates how the IPU’s architecture affects memory, communication, and computation performance for AI/ML workloads.

  • Method

    The authors use microbenchmarks spanning local memory, on- and off-chip interconnects, host connectivity, matrix multiplication, convolution, and collective operations.

  • Results

    The study reports up to 31.1 TFlops/s single-precision and 124.5 TFlops/s mixed-precision theoretical throughput per chip, with measured single-precision GEMM reaching 18.9 TFlops/s per IPU.

  • Takeaways & Limitations

    The benchmarks provide performance models for reasoning about IPU applications from their computation and communication steps.

  • Takeaways & Limitations

    The evaluation uses one eight-board single-host configuration, and all-to-all results could not be scaled beyond half an IPU because of transfer count and local-memory demands.

Abstract

from arXiv · show

This report focuses on the architecture and performance of the Intelligence Processing Unit (IPU), a novel, massively parallel platform recently introduced by Graphcore and aimed at Artificial Intelligence/Machine Learning (AI/ML) workloads. We dissect the IPU's performance behavior using microbenchmarks that we crafted for the purpose. We study the IPU's memory organization and performance. We study the latency and bandwidth that the on-chip and off-chip interconnects offer, both in point-to-point transfers and in a spectrum of collective operations, under diverse loads. We evaluate the IPU's compute power over matrix multiplication, convolution, and AI/ML primitives. We discuss actual performance in comparison with its theoretical limits. Our findings reveal how the IPU's architectural design affects its performance. Moreover, they offer simple mental models to predict an application's performance on the IPU, on the basis of the computation and communication steps it involves. This report is the natural extension to a novel architecture of a continuing effort of ours that focuses on the microbenchmark-based discovery of massively parallel architectures.

Architecture

The IPU is a massively parallel processor built from many MIMD tiles with distributed local memory, specialized interconnects, and fine-grained execution. Its design supports irregular workloads while retaining strong arithmetic throughput on regular AI/ML computations.

  • Fine-grained parallelism: The IPU uses true MIMD parallelism to execute fine-grained, irregular computations with irregular data accesses.Individual processing threads can follow distinct code and execution paths without the large coherent warps required by SIMD/SIMT architectures.
  • Architecture: Each IPU contains 1,216 tiles, each combining one core with 256 KiB of local memory.The device has no memory beyond tile-local memories and the register file.
  • Interconnect: The exchange provides on-chip tile communication, while IPU links connect processors and PCIe links connect CPU-based hosts.These components form the processor’s main communication paths.
  • Arithmetic throughput: 31.1 TFlops/s in single precision and 124.5 TFlops/s in mixed precision are the IPU’s reported per-chip theoretical throughputs.Mixed precision uses FP16 multiplicands with products accumulated into FP32 results.
  • Arithmetic throughput: The IPU’s theoretical advantage over GPUs grows by roughly twofold in per-board and energy-efficiency comparisons.This comparison concerns theoretical advantage and energy efficiency at board scale.
  • Arithmetic throughput: Measured single-precision performance makes the IPU a clear winner against an Nvidia V100, while mixed-precision comparison requires more nuanced analysis.The reported comparison distinguishes theoretical limits from measured workload performance.

1.4 Memory Architecture

The IPU replaces conventional shared memory hierarchies with distributed, high-bandwidth SRAM local to tiles. This organization requires partitioning data across tiles but supports low-latency computation, multi-IPU scaling, and on-chip residency for many models.

  • Memory organization: Each tile provides 256 KiB of local memory, totaling 304 MiB per IPU, with 45 TB/s nominal aggregate bandwidth and 6-cycle latency.Each tile’s contiguous 21-bit address space is shared by six hardware execution contexts.
  • Memory performance: IPU local memory has lower latency than comparable Turing GPU memories and latency comparable to L2 caches in modern Intel CPUs.Its capacity also exceeds equivalent-latency CPU and GPU memory layers by one to two orders of magnitude.
  • Programming model: Software must partition working sets across tile memories and exchange data when operands are non-local.Poplar and its compiler can orchestrate placement and transfers from descriptions of operand flow.
  • Model placement: The 304 MiB per-IPU memory is typically sufficient for many contemporary AI/ML models to remain entirely on-chip.Larger models can be sharded across IPUs and boards.
  • Interconnect performance: On-chip all-to-all transfers reach 7.7 TB/s actual aggregate throughput, with 6.3 GB/s simultaneously available per tile.Measured tile-to-tile latency is 165 nanoseconds or lower and does not degrade above that value under load.
  • Inter-IPU scaling: The tested board configuration achieved 108 GB/s actual bandwidth between its two IPUs, while larger systems remain outside the study’s single-host, eight-board scope.Multi-IPUs expose cumulative memory and compute resources as a virtual device.

1.6. THE BULK SYNCHRONOUS PARALLEL MODEL

The IPU organizes execution through Bulk Synchronous Parallel supersteps that separate local computation, communication, and barrier synchronization. Its exchange and IPU-link network implements these phases, making topology and interconnect performance central to application behavior.

  • BSP phases: Each BSP superstep consists of local computation, communication, and barrier synchronization phases executed sequentially.Processes compute on local memory, exchange data, and wait until all processes reach the barrier.
  • BSP implementation: The IPU is a true BSP machine whose hardware and programming model enforce and optimize all three phases.Programmers express graph vertices that compute on local data while the system handles phase-specific execution.
  • Hardware support: The exchange accelerates all-to-all communication and barrier synchronization, while IPU cores directly access only local memories.The programming model separates computation and communication through hardware-supported phases.
  • Performance implications: Interconnect performance is the primary factor determining BSP communication and synchronization performance and therefore affects overall application performance.The report benchmarks intra- and inter-IPU transfers as BSP communication and barrier phases.
  • Topology effects: Tile and network proximity increase communication latency: neighboring IPUs are fastest, while each additional hop costs 0.16 microseconds on average.The farthest tested pair, IPUs 7 and 10, reaches approximately 1.76 microseconds latency.
  • Bandwidth: Measured bidirectional bandwidth between the two IPUs on each board is 108 GB/s, or 84% of nominal peak, and peak endpoint bandwidth is not proximity-dependent.Monodirectional bandwidth is nearly half the bidirectional value.

2.2. MULTI-IPUS

Multi-IPUs federate neighboring physical IPUs into virtual devices that expose combined memory and compute resources. Their topology and numbering distinguish PCI Device IDs from proximity-based DNC IDs, while benchmark timing accounts for single- versus multi-IPU overheads and SDK version.

  • Multi-IPUs are built from neighboring IPUs in power-of-two sizes: 2, 4, 8, or 16.
  • Device IDs follow PCI enumeration and do not encode proximity, whereas consecutive DNC IDs identify physically neighboring IPUs.
  • Within a Multi-IPU, tiles are ordered by DNC ID, with each physical IPU contributing a contiguous tile range.
  • SDK version 1.0.49 is part of the benchmark specification because subsequent releases may change performance through incremental optimizations.
  • Single-IPU timings use cycle-accurate on-device timestamps, while multi-IPU timings use repeated host measurements to amortize overheads.

Local Memory

Each IPU tile has private local memory with fixed latency and independent performance. Aggregate bandwidth depends strongly on access width, instruction density, block size, and whether code uses compiler-supported vectorization or hand-written assembly.

  • Each tile contains 256 KiB of directly accessible local memory, while remote-tile memory requires the exchange.
  • 6 clock cycles is the fixed local-memory read latency, independent of access pattern, stride, working-set size, thread count, or concurrent tiles.
  • 31.1 TB/s is the theoretical aggregate read-bandwidth limit, derived from 16 bytes per cycle, 1.6 GHz, and 1,216 tiles.
  • 7.59 TB/s is achieved by naive Poplar/C++ load loops, roughly one quarter of the theoretical limit, with lower instruction density potentially reducing performance further.
  • 30.7 TB/s, or 98.6% of the theoretical limit, is achieved by hand-written assembly using zero-overhead loops with 128-bit loads.
  • The float2 approach doubles access width for contiguous 32-bit data, but float4 does not make the compiler emit 128-bit loads.
  • 8 KiB blocks suffice to reach 95% of peak local-memory bandwidth, while distributed local memories avoid congestion as more tiles participate.
  • Theoretical write bandwidth is 15.5 TB/s, and aligned vector types provide the same performance benefits for writes as for reads.

Interconnect

The interconnect evaluation uses microbenchmarks of point-to-point and collective communication to measure how latency and bandwidth change with operation scale across on-chip and off-chip links. The selected primitives are intended to support early performance estimates for parallel applications.

  • The benchmarks measure latency and bandwidth degradation as point-to-point and collective operations scale across on-chip and off-chip interconnects.
  • The primitive set covers communication patterns commonly found in parallel applications and is intended to support early performance estimates.
  • The chapter treats point-to-point transfers first, followed by separate sections for each collective operation.

4.1 Point-to-point Transfers

The study characterizes point-to-point latency and bandwidth across on-chip and off-chip IPU interconnects under congestion-free and loaded conditions. Results expose topology-dependent latency penalties, scalable on-chip throughput, and bandwidth differences across links and system scales.

  • Method: The benchmarks cover point-to-point transfers across on-chip and off-chip topologies under congestion conditions ranging from global silence to full load.The experiments exercise same-IPU, same-board, inter-board, and multi-hop communication patterns.
  • Latency: 0.13 microseconds is the average on-chip latency, while crossing an IPU link adds approximately 0.5 microseconds.Reaching the second IPU on the same board can be marginally more expensive than reaching a directly connected IPU on another board.
  • Latency under Load: +24% on-chip latency versus 4.0...7.7× off-chip slowdown under load.Congestion affects IPU-link communication substantially more than on-chip communication.
  • Latency under Load: 1.93 ns/message on 16 IPUs versus 1.76 ns/message on a single-board 2-IPU system.Randomized concurrent transfers therefore show only a minimal increase in average per-message latency at this scale.
  • Bandwidth: A single thread reaches full bandwidth, and additional threads cause no material performance change on or across chips.Multi-threading is not required for peak bandwidth and does not degrade it when used.

4.2 Broadcast

Broadcast benchmarks vary destination scale, topology, message size, and on-chip versus off-chip placement. Broadcast latency scales roughly logarithmically, while bandwidth saturates quickly on chip and degrades when communication extends off chip.

  • Method: The experiments vary broadcast scale from one tile to the full 16-IPU test system and compare same-board, direct inter-board, and indirect inter-board topologies.Scale denotes destination tile count, or IPU count when whole IPUs participate.
  • Latency: Less than 2 microseconds is required for a minimum-size broadcast across 16 IPUs.A one-word, 32-bit message is used to measure minimum latency.
  • Latency: 0.2 microseconds is sufficient for one tile to broadcast one word across its IPU, while an off-chip two-IPU broadcast adds approximately 0.5 microseconds.A broadcast across two IPUs takes approximately 0.7 microseconds.
  • Latency: Broadcast latency grows roughly logarithmically with tile count within a chip and with IPU count across chips.The off-chip cost is dominated by traversing an IPU Link; local destinations can be reached in parallel with remote delivery.
  • Bandwidth: 12 GB/s is the approximate on-chip per-tile broadcast bandwidth saturation level, reached with a destination set of four tiles.Extending the broadcast off chip lowers average per-transfer bandwidth from 12.4 to 5.6 GB/s, with negligible difference between on-board and inter-board communication.
  • Message Size and Scaling: 1 KiB messages achieve only 15-30% of peak bandwidth, and larger system scales require larger messages to approach peak performance.Aggregate broadcast bandwidth grows monotonically with system scale, while per-tile bandwidth decreases roughly logarithmically with system size.
  • Message Size and Scaling: Bandwidth generally increases monotonically with block size without degradation, except for slight declines above 64 KiB in whole-IPU broadcasts across boards.The exception occurs in topologies (c) and (d).

4.3 Gather

The gather section defines and benchmarks the reverse communication pattern of broadcast across matching scales and topologies. Its design is constrained by the destination tile’s need to store one input buffer per source.

  • Definition: A gather sends one distinct message from multiple source tiles to one destination tile.The destination buffer must hold the product of message size and source count, while each source stores one message-sized buffer.
  • Method: The benchmarks use the same scales as broadcast, reverse the data-flow direction, and study collective operations without unrelated system load.The destination tile’s local memory limits the message size and therefore the operation’s aggregate bandwidth.
  • Latency: 0.8 microseconds is the latency of a gather involving an entire IPU.The benchmark uses a minimum-size 32-bit message.

4.3. GATHER

Gather performance is evaluated across increasing system scales and topologies using minimum-latency and maximum-message-size bandwidth experiments. Latency rises with IPU-link hops and system diameter, while local memory capacity constrains bandwidth experiments.

  • Latency: 25 microseconds is the latency of a whole-system gather across 16 IPUs, with increasing Multi-IPU sizes growing sub-linearly.The results are reported for varying scales and topologies.
  • Latency: Two-hop gather experiments take roughly twice the latency of one-hop experiments.This relationship appears for both the two-IPU and whole-IPU topology comparisons.
  • Topology: Gather latency is slightly longer across directly connected inter-board IPUs than between IPUs on the same board.The same pattern is reported for corresponding broadcast and scatter comparisons.
  • Latency: Gather latency approximately doubles when system size doubles because network diameter also roughly doubles.This scaling result connects system expansion to the total communication path length.
  • Bandwidth: Peak gather bandwidth uses the largest message size allowed by local memory capacity, with the message size reported separately for each scale and topology.Different scales have different maximum usable message sizes.

4.4. SCATTER

Scatter operations remain highly scalable across IPU systems, with performance shaped primarily by source-tile memory limits and message size. The section measures scatter latency and bandwidth across scales and topologies.

  • Operation: Scatter operations send distinct messages from one source tile to multiple destinations and reverse the direction of gather operations.
  • Latency: 0.9 microseconds: a whole-IPU scatter completes only marginally slower than an equal-scale gather.The experiment uses 32-bit messages and scales from one tile to the whole system.

4.4. SCATTER

The collective-operation results show strong scaling for scatter and reduction, while memory footprint limits all-to-all experiments to half an IPU and single-word messages. Reduction bandwidth scales well, but strong scaling exposes parallelization costs.

  • Scatter latency: Scatter latency differences across 2-IPU topologies are smaller than in gather experiments, although direct connectivity remains marginally better.
  • Scatter latency: Doubling system size roughly doubles network diameter and total latency, while scatter completes almost twice as quickly as equal-scale gather.
  • Scatter bandwidth: 16-IPU scatter bandwidth is only 6.6% below 2-IPU bandwidth, while off-chip bandwidth decreases from 6.3 to 5.6...5.2 GB/s.
  • All-to-all: Whole-IPU and multi-IPU all-to-all operations could not be benchmarked; results stop at half IPU, where latency is 0.55 microseconds.Each tile needs n input and n output buffers, so only single-word messages support the largest tested scale.
  • Reduction latency: 11...16% more expensive: fully distributed reduction latency is comparable to a sequential operation with equal input size on one tile.The study spans tiles, IPUs on one board, multiple boards, and a 16-IPU system.
  • Reduction latency: 2.1 microseconds versus 14.51 microseconds: reducing 19,456 values costs more on a 16-IPU system than on one IPU under strong scaling.The fixed problem size exposes the cost of parallelization.
  • Reduction bandwidth: 6.76 TB/s and 92.2% parallel efficiency: weak-scaling on-chip reduction bandwidth grows strongly with system size.
  • Reduction bandwidth: 64.7 TB/s: a 16-IPU system reaches this peak aggregate off-chip reduction throughput, with 59% parallel efficiency.Throughput grows 9.4× from 1 to 16 IPUs, while per-tile bandwidth degrades approximately 41%.

4.7. HOST CONNECTIVITY

Host-to-IPU transfers are evaluated as potential bottlenecks in hybrid CPU/IPU applications, with latency and bandwidth measured through PCI Express. Latency is stable across topology and destination-IPU count, while bandwidth rises with PCIe lane count.

  • Motivation: Bandwidth is relevant for streaming workloads, while latency matters for inference applications that alternate output production between CPU and IPU.
  • Scope: Results are representative only of SDK 1.0.49, and later SDK and driver releases may optimize host-to-IPU connectivity.
  • Method: The measurements use graph::DataStream transfers to distributed IPU tensors, with host-side latency averaged over multiple synchronous transfers.
  • Latency: At least 8.81 µs: host-to-IPU transfer latency remains stable across PCI Express topologies and does not increase when communicating with multiple IPUs.A slight increase with destination-tile count results from transfers requiring multiple maximum-size PCI Express transactions.
  • Bandwidth: Host-to-IPU bandwidth increases with the number of involved PCIe lanes, reaching approximately 6 GB/s for one IPU connected through 8 PCIe Gen3 lanes.

4.7. HOST CONNECTIVITY

The report benchmarks IPU and GPU connectivity and computation, finding strong IPU single-precision and energy-efficiency results alongside important operand-size, software-version, and workload constraints.

  • Host connectivity: 13.78 GB/s is available when four IPUs share 16 PCIe lanes, while transfers to all 16 IPUs reach 55.04 GB/s.The measurements characterize concurrent host-to-IPU transfers across different transfer topologies.
  • Matrix multiplication: 31.1 TFlops/s single-precision and 124.5 TFlops/s mixed-precision theoretical throughput are available per IPU chip.The IPU’s 1,216 tiles and AMP units provide these theoretical limits at 1.6 GHz.
  • Measurement scope: Actual performance depends on vendor library implementations, with these IPU results limited to SDK version 1.0.49.The GEMM experiments use Graphcore’s poplin and NVIDIA’s cuBLAS libraries, so measured results reflect software optimization as well as hardware.
  • Matrix multiplication: 18.9 TFlops/s single-precision GEMM performance reaches 60% of the theoretical limit and outperforms the V100 per chip.Two IPUs on a C2 board can deliver 37.8 TFlops/s on independent matrix operands.
  • Memory constraints: The GPU fits substantially larger matrix operands: roughly ∼50,000× ∼50,000 versus 2,944×2,944 on one IPU.This memory-capacity difference persists for mixed precision, where the largest IPU operands are 2,688×2,688 versus roughly ∼72,000× ∼72,000 on a 32-GB GPU.
  • Matrix multiplication: 58.9 TFlops/s versus 90.0 TFlops/s gives the V100 higher mixed-precision GEMM performance than the IPU.The IPU reaches 47.3% of its theoretical limit, compared with 72.0% for the GPU.
  • Energy efficiency: A single IPU delivers higher efficiency than two IPUs and the V100 in both single and mixed precision.The report evaluates efficiency as throughput divided by nominal board power.
  • Convolution: The IPU supports smaller CNN batch sizes, typically peaking at 8...32 versus 512 or 1,024 for the GPU.For ResNeXt with G=32, the IPU shows >100x advantage for certain convolutions, while reported speedups can exceed 700x for some GPU-unoptimized CNNs.

5.3. PSEUDO-RANDOM NUMBER GENERATION

The report compares bulk pseudo-random number generation on IPUs and GPUs as output blocks grow. Dedicated IPU PRNG hardware reaches much higher throughput, but the comparison excludes randomness quality.

  • Throughput comparison: 944 Gsamples/s versus 192 Gsamples/s gives the IPU 4.9× the V100’s aggregate PRNG throughput.The comparison is per chip and uses uniform-distribution pseudorandom samples.
  • Limitations: Randomness quality is outside the paper’s scope, despite the IPU’s higher throughput.The authors note that the IPU algorithm provides lower randomness quality than the fastest algorithm used in the comparison.
  • Benchmark scope: The benchmark focuses only on pseudorandom numbers from a uniform distribution.The study does not cover other requested distributions.
  • IPU implementation: Each IPU tile includes PRNG acceleration circuitry implementing a variant of xoroshiro128+.The benchmark invokes poprand::uniform, reaching 944 Gsamples/s for sufficiently large blocks.
  • Block-size dependence: Larger output blocks increase throughput on both IPUs and GPUs until performance saturates similarly as a function of output size.Figure 5.5 presents bulk generation throughput against requested block size in a per-chip comparison.
Loading 1912.03413v1…