PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer Accelerator (TFA): A Macro-Op INT8 Hardware Chip for Transformer Inference and Machine Translation

Shashank

arXiv:2608.23582v1cs.ARcs.CLcs.LG

TL;DR

Transformer inference at the edge is constrained by memory traffic and difficult per-tensor INT8 activation quantization. TFA uses a compact, compiler-programmed, bit-exact macro-op engine to execute pretrained transformer pipelines. It achieves end-to-end zero-mismatch verification and translation execution, but performance and energy results remain partly simulation-based or projected.

  • Problem

    Single-batch transformer decode is memory-bound, while activation outliers make direct per-tensor INT8 quantization unreliable.

  • Method

    TFA uses a host-programmed memory-to-memory engine with eight macro-ops, bit-exact integer semantics, and compiler-side Hadamard reparameterization and quantization.

  • Results

    Zero golden-model mismatches were achieved with 100% functional coverage and 94.96% DUT-scoped code coverage, including the compiled T5 translation pipeline.

  • Takeaways & Limitations

    TFA demonstrates end-to-end, bit-exact execution of pretrained encoder-decoder transformers using compact hardware and compiler-managed quantization.

  • Takeaways & Limitations

    Performance, throughput, and energy figures primarily come from RTL simulation or pre-synthesis projections rather than representative silicon measurements.

Abstract

from arXiv · show

We present the Transformer Accelerator (TFA), a synthesizable, parameterizable INT8 memory-to-memory engine for transformer inference. One time-multiplexed datapath handles prompt processing and autoregressive generation. TFA implements matrix multiplication, softmax, RMSNorm, elementwise, and copy/gather operations through eight 512-bit macro-op descriptors. Offline-compiled programs are fetched, validated, and dispatched through AXI interfaces, supporting encoder, decoder, and encoder-decoder models. The RTL combines an output-stationary multiply-accumulate array with ping-pong buffers that overlap DMA and compute, bit-exact reciprocal-square-root and divide units, key-value-cache and embedding addressing, and an abort-safe zero-padding write engine. A UVM environment byte-compares outputs against a bit-exact golden model. Across 25 tests and 34 constrained-random runs, TFA achieved zero mismatches, 100% functional coverage, and 94.96% code coverage. We compiled the t5-small encoder-decoder pipeline for English-to-French, German, and Romanian translation. On ten multilingual proverbs, TFA executed 70,320 descriptors and matched 37.9 MB of golden-model output with zero mismatches. INT8 output matched the floating-point reference token-for-token on five sentences; the rest produced valid alternative translations. Randomized-Hadamard reparameterization recovered about 11 dB of per-tensor INT8 signal-to-noise ratio across layers. The verification configuration achieved about 20x end-to-end speedup over a 22-thread CPU, while larger designs are projected to reduce energy per token by about 1000x. After RAM inference recoding, logic area fell to 2.73 mm2, and the design completed design-rule-clean synthesis and place-and-route on SkyWater sky130. TFA demonstrates end-to-end, bit-exact execution of pretrained transformers using compact hardware and compiler-managed quantization.

I. INTRODUCTION

TFA is a compact, host-programmable INT8 engine aimed at the memory-bound regime of transformer inference. Its minimal macro-op ISA, compiler-managed quantization, and bit-exact verification enable end-to-end execution of pretrained encoder-decoder translation models.

  • System concept: TFA executes transformer workloads as streams of eight fixed-width memory-to-memory macro-operations rather than through a general-purpose processor.The host compiler resolves descriptors, addressing, and tiling offline.
  • Contributions: The accelerator combines a compact descriptor ISA, a time-multiplexed streaming datapath, bit-exact integer numerics, and byte-comparing golden-model verification.The contributions include ping-pong buffering, integer-reproducible operation specifications, and UVM validation.
  • End-to-end demonstration: The pretrained T5 translation pipeline ran across three target languages and 70,320 operations with zero golden-model mismatches.The demonstration covered the encoder, cross-attention, and autoregressive decoder; randomized-Hadamard reparameterization was folded into the weights.
  • Workload motivation: Decode is memory-bound at batch size one because each generated token rereads the resident weights and growing KV cache.Prefill is described as compute-bound, whereas autoregressive decode has low arithmetic intensity.
  • Quantization motivation: INT8 W8A8 reduces operand traffic relative to wider formats, but transformer activation outliers make direct per-tensor quantization unreliable.The paper motivates compiler-side handling of activation range difficulty rather than adding floating-point scale machinery to the datapath.
  • Positioning: TFA differs from larger systolic and fixed-function accelerators by combining host-driven macro-ops, a time-multiplexed INT8 backend, and a fully specified numeric contract.The contract gives each operation a unique integer result that software can replay and byte-compare.

III. SYSTEM ARCHITECTURE AND PROGRAMMING MODEL

TFA defines a host-managed boundary in which external memory and IRAM supply data and macro-op programs to a shared tensor engine. The same descriptor vocabulary expresses encoder, decoder, cross-attention, prefill, and decode layer programs.

  • System boundary: TFA is a memory-to-memory engine whose host supplies weights, activations, KV cache, and command programs, while the hardware executes tensor primitives and signals completion.The boundary omits tokenizer, DRAM/PCIe control, cache coherence, and general-purpose processor functionality.
  • Interfaces: One AXI4 master handles bulk data movement, while an AXI4-Lite slave exposes control, lookup-table, and IRAM windows.The host loads or patches descriptors, sets START_PC, and pulses START.
  • Macro-op ISA: The eight-opcode ISA covers GEMM, SOFTMAX, RMSNORM, ELTWISE, COPY, SYNC, NOP, and HALT through coarse-grained tensor operations.GEMM supports requantized INT8 or raw 32-bit output, while COPY supports linear movement or indexed gather.
  • Descriptor format: Each macro-op is a 512-bit descriptor containing opcode and flags, tensor dimensions, addresses, pitches, ring parameters, and requantization fields.The descriptor is sixteen little-endian 32-bit words; reserved fields are ignored for forward compatibility.
  • Layer programs: The host expresses encoder, decoder, and cross-attention layers through short descriptor sequences, while the sequencer autonomously fetches and executes them.Attention options such as GQA, causal masking, and sliding windows change addressing or flags without changing the overall program structure.
  • Prefill and decode: The same program structure serves prefill and decode by changing M and per-token parameters, with KV-ring addressing supporting the single-row decode case.Prefill uses M=S and S×S attention tiles; decode uses M=1 and patches only a small set of per-token fields.

D. Decode Validation Contract

TFA validates every descriptor before issuing memory transactions, enforcing size, ring, and footprint constraints with overflow-safe arithmetic. Its fixed integer semantics then support deterministic replay and byte-exact verification.

  • Validation flow: Before issuing a macro-op, the sequencer checks its descriptor against an exhaustive per-opcode contract and aborts violations with BAD_SIZE.Rejected descriptors never reach an execution engine.
  • Size checks: Size checks require positive active dimensions and bound row-engine dimensions by ROW_MAX, including the softmax accumulator constraint.The stated bound is 1 ≤ N ≤ ROW_MAX for SOFTMAX, RMSNORM, and ELTWISE.
  • Control protocol: The sequencer follows FETCH→VALIDATE→ISSUE→WAIT and reports completion or errors through STATUS and the interrupt.The host initiates execution through START_PC and START, then observes the result through the control interface.
  • Memory safety: Footprint checks verify that each operand’s highest reachable byte remains within the address space.The calculation uses descriptor- and opcode-derived rows, row_bytes, and pitch values.
  • Overflow handling: 41-bit unsigned arithmetic is required because a 40-bit address-width configuration can produce a legal sum above 2^40, whose carry-out must reject the descriptor.A modular 40-bit adder could wrap and falsely accept an out-of-range operand.
  • Deterministic numerics: Bit-exact opcode semantics let the golden model replay programs and byte-compare outputs without tolerance checks.The same deterministic definitions are used by the compiler, RTL, and UVM scoreboard.

B. RMSNorm

TFA computes RMSNorm with integer accumulation, reciprocal-square-root processing, and compiler-supplied requantization. The design explicitly handles zero rows and bounds intermediates for deterministic execution.

  • Pipeline: RMSNorm accepts an INT8 row and gain, computes an integer sum of squares, derives a reciprocal square root, and applies a reciprocal before producing outputs.The reciprocal-square-root operation uses the serial tfa_isqrt unit.
  • Edge case: An all-zero input row produces y_i = 0 exactly without division-by-zero or undefined behavior.The specified path is s_s = 0, q = 0, followed by zero outputs.
  • Requantization: Per-op integer pipelines use compiler-chosen M_A and SHIFT as the only scale-bearing parameters, leaving the hardware scale-agnostic.The figure caption describes fixed INT8 or RAW32 input-output functions.
  • Related row semantics: Softmax separately defines masked visibility and an all-zero-row guard, illustrating the broader deterministic treatment of row operations.The max(·, 1) guard makes the zero-row case well-defined for any monotone host-provided exponential table.

D. Eltwise, Copy, and Gather

TFA’s elementwise and copy operations implement transformer residual, gating, embedding, and data-movement primitives through compiler-specified integer arithmetic and shared DMA infrastructure.

  • Elementwise and Copy: ELTWISE ADD supports residual arithmetic with two operand scales, while MUL optionally applies SiLU to realize the SwiGLU gate in one operation.COPY is byte-exact, and gather uses 32-bit indices to fetch embedding rows.
  • Elementwise and Copy: The datapath manipulates integers without interpreting tensor scales, leaving quantization policy to the compiler.This separation supports compiler-side transformations such as randomized-Hadamard reparameterization without changing the ISA or silicon.
  • Shared Infrastructure: TFA dispatches one validated macro-op at a time, with the active engine owning shared read and write DMA clients through ready/valid interfaces.The sequencer validates descriptors before issuing transactions and advances only after writes retire.
  • Shared Infrastructure: The sequencer performs exhaustive footprint checks in hardware using widened arithmetic and routes validation failures or aborts to safe completion paths.The wider sum includes carry-out, preventing modular-wrap false acceptance of out-of-bounds descriptors.
  • Shared Infrastructure: The GEMM core uses an output-stationary T × T MAC array with signed 32-bit accumulators and zero-masked edge lanes for uniform full-tile execution.Explicit re-signing is required for packed signed SystemVerilog array elements.

D. Ping-Pong Operand Buffering

TFA sustains GEMM execution with banked operand buffers and concurrent fill, compute, and writeback control, allowing DMA activity to overlap computation.

  • D. Ping-Pong Operand Buffering: Four double-buffered operand SRAMs hold A and B tiles across banks, supporting row-major and lane-scattered fills with one-cycle-interval column reads.The banked organization is designed to keep the read channel continuously busy.
  • D. Ping-Pong Operand Buffering: Three concurrent GEMM FSMs separate tile filling, returned-data writing, and MAC computation while a free-running bank toggle passes tiles between ping-pong banks.The toggle remains continuous across k0, m0, and n0 boundaries.
  • D. Ping-Pong Operand Buffering: The compute FSM accumulates across K-tiles, writes only on the final K-tile, and overlaps drain with the next tile’s fill on disjoint DMA channels.RUN and DRAIN do not overlap because they share the accumulators.
  • D. Ping-Pong Operand Buffering: For decode with M = 1, A loading reduces to one row while B streaming dominates, making ping-pong buffering particularly relevant to sustained reads.The reported arrangement delivers measured read-bus utilization.

E. Writeback and Ring/RAW32 Addressing

TFA’s writeback supports INT8 requantization, RAW32 outputs, and linear or ring-buffer addressing, while vector and softmax engines provide bit-exact row-wise normalization and attention probabilities.

  • E. Writeback and Ring/RAW32 Addressing: GEMM writeback requantizes accumulators to INT8 with optional ReLU, while RAW32 emits signed 32-bit values for outputs such as language-model logits.The row element size is 1 byte for INT8 or 4 bytes for RAW32.
  • E. Writeback and Ring/RAW32 Addressing: Ring-buffer destinations use wrapped row counters, with validation guaranteeing that one GEMM cannot lap its own ring.The same modulo-addressing scheme supports ring-buffer key/value reads.
  • E. Writeback and Ring/RAW32 Addressing: Verification found and fixed a packed-signed sign-extension defect and a one-cycle ping-pong pointer race in writeback control.The race fix latches the completing bank index before pointer toggling.
  • E. Writeback and Ring/RAW32 Addressing: RMSNorm loads γ once per descriptor, computes row-wise sums of squares, then uses serial square-root and reciprocal units before requantized output.The square root takes 24 iterations and the restoring divide takes 32 cycles per row.
  • E. Writeback and Ring/RAW32 Addressing: The softmax engine processes masked rows through maximum finding, LUT exponentiation with sum accumulation, serial division, and probability writeback.Fully masked rows use a zero-row short circuit.

C. Copy and Gather

TFA’s copy engine supports linear transfers and indexed gathers, while DMA movers handle alignment, burst legality, and abort-safe completion across shared AXI channels.

  • C. Copy and Gather: Linear copy streams each source row directly to its destination, while gather prefetches up to 64 indices and uses them to address embedding or KV rows.Gather indices are read from ADDR_A and source rows from ADDR_B + idx · PITCH_B.
  • C. Copy and Gather: Shared read-stream command discipline separates index and row data at command boundaries, with FIFO writes enabled only during parsing states.This prevents data from different command types from being misclassified.
  • C. Copy and Gather: A prior FIFO qualification defect duplicated the last parsed index during refills beyond 64 entries and was caught by constrained-random testing.The corrected push condition requires both a held beat and a parse-state FSM condition.
  • C. Copy and Gather: The read DMA splits commands at 4 KB and burst-size boundaries, realigns arbitrary byte offsets, and drains in-flight data cleanly after abort or read errors.A discard mode sinks remaining beats before asserting idle.
  • C. Copy and Gather: The write DMA drains every already-issued burst after abort or starvation, padding missing beats with zero data and cleared write strobes while preserving WLAST.This provides abort-safe completion for AXI writes.

E. CSR, IRAM, and LUTs

TFA uses a parameterized, memory-to-memory architecture in which compiled descriptors control tensor operations, while ping-pong buffering overlaps data movement with computation. Its GEMM datapath uses output-stationary accumulation and continuous operand-bank scheduling to sustain AXI traffic across tiles.

  • CSR, IRAM, and LUTs: TFA exposes AXI4-Lite control, IRAM descriptor storage, and exponential/SiLU LUT windows through a synchronous CSR interface.The 512-bit descriptor path is implemented with 16 parallel 32-bit simple-dual-port RAMs so the sequencer reads one descriptor per cycle.
  • Parameterization: A single parameter set scales the hierarchy across SIM, EDGE, and PERF geometries by configuring MAC dimensions, AXI width, address width, buffering, and descriptor depth.The reference points range from an 8×8 array with 64-bit AXI data to a 128×128 array with 512-bit AXI data.
  • GEMM dataflow: TFA’s GEMM keeps a TR × TC accumulator tile in the MAC array while streaming A and B sub-tiles through KTILE contraction chunks.The output-stationary schedule drains requantized or RAW32 results after completing the K reduction.
  • GEMM dataflow: Ping-pong operand banks fill contraction chunk n+1 while the array runs chunk n, with a free-running toggle preserving overlap across k0, m0, and n0 boundaries.The compute pointer consumes banks in fill order, avoiding an explicit producer/consumer handshake counter.
  • Bus utilization: At EDGE and PERF, the B fill is the bottleneck and the structural utilization ceiling approaches ≈95% and ≈93%, respectively.SIM reaches ≈80% because TC equals LANES and fill and run take similar time; the ≥85% requirement applies only to product geometries.

C. Attention as Address Computation

TFA maps attention variants onto shared GEMM and softmax engines using descriptor-controlled addressing and masking parameters. Its verification environment reconstructs device state from observed control traffic and byte-compares replayed outputs against a bit-exact golden model.

  • Attention mapping: Self-attention and cross-attention share the same GEMM and softmax datapaths; descriptor address arithmetic selects the KV source and softmax parameters select masking behavior.Supported options include bidirectional, causal, sliding-window, and grouped-query attention.
  • Roofline regime: Batch-1 autoregressive generation is memory-bound because each token streams the resident weight set once, making effective read bandwidth—not peak compute—the throughput limiter.With INT8 weights, the arithmetic intensity is approximately one MAC per byte read, below the ridge for SIM, EDGE, and PERF.
  • Prefill limitation: The v1 streaming schedule makes prefill GEMM bus-bound because A and B sub-tiles are fetched per output tile without inter-tile weight reuse.A documented v1.1 B-stationary loop order with a C-strip accumulator is intended to amortize weight fetches and lift arithmetic intensity above the ridge.
  • Verification: The UVM environment uses an active AXI-Lite master, a randomized-latency reactive AXI4 memory model, reset injection, and protocol-error injection.Stimulus includes directed, constrained-random, corner, error, abort/race, sweep, and performance scenarios.
  • Verification: The scoreboard reconstructs device state solely from the AXI-Lite monitor stream, replays each started program through tfa_gold, and byte-compares every output region.It also predicts completion or error outcomes using the architectural integer arithmetic, rounding, saturation, and validation sequence.

D. Coverage Closure

Coverage closure combined broad functional stimulus with reviewed code-coverage waivers and adversarial verification, while constrained-random testing exposed functional, protocol, and environment defects.

  • Coverage closure: 100% functional coverage and 94.96% DUT-scoped code coverage were achieved across 34 merged runs with zero UVM_ERROR and zero UVM_FATAL.The raw code-coverage score was 87.94%; reviewed waivers raised it to 94.96%, with FSM and assertion coverage reaching 100%.
  • Coverage closure: Reset and abort sweeps covered transient FSM behavior, while waivers were limited to reviewed reachability judgments for asynchronous-reset arcs, one-cycle abort windows, and static attributes.Functional reset and abort behavior was separately proven by dedicated sweep tests.
  • Representative bugs: Adversarial review found six DMA hangs or protocol violations, and bit-exact simulation exposed eight additional signed-arithmetic, buffering, addressing, and reset-related defects.The defects included an AXI VALID violation, abort deadlocks, packed-signed sign-extension errors, ping-pong races, and stale ring bases.
  • Representative bugs: A gather sequence exceeding 64 indices exposed an index-FIFO duplicate flood masked by smaller gathers, demonstrating the value of coverage-driven stimulus.The fix qualified FIFO writes by both a held beat and a parse-state condition.
  • Coverage closure: The verified methodology established bit-exact RTL agreement before the end-to-end transformer experiment tested ISA expressiveness and INT8 adequacy.The subsequent experiment addressed whether the eight macro-ops could run a pretrained encoder-decoder transformer and whether per-tensor INT8 was numerically sufficient.

A. Compiling a Pretrained Transformer onto the ISA

The compiler maps a complete pretrained T5 encoder-decoder into TFA programs while moving quantization difficulty into an exact, compiler-side basis transformation.

  • Compilation flow: The offline flow targets t5-small and exercises bidirectional encoder attention, causal decoder attention, cross-attention, FFN operations, and RMSNorm pre-normalization.The complete pipeline is chosen because it covers every transformer block type required by the demonstration.
  • Compilation flow: Calibration records maximum absolute values at 194 activation sites to derive symmetric max-abs INT8 scales, while the compiler emits tiled descriptor programs and a memory image.The LM head uses a RAW32 GEMM over 32,128 logits so argmax can be computed directly from chip-written logits.
  • Compilation flow: 2,095 descriptor programs, 70,320 descriptors, and a 77.6 MB memory image encode the ten-sentence multilingual workload.Weights and sentence inputs share one serialized arena, keeping the image weight-sized across the workload.
  • Quantization challenge: Naive per-tensor W8A8 produces incoherent decoder tokens because residual outliers reach approximately 9531 versus a median near 72, a 133× disparity.The outlier structure annihilates ordinary channels under a shared per-tensor scale.
  • Quantization solution: Randomized-Hadamard reparameterization rotates residuals and folds the transformation into weights, preserving the floating-point computation because RMSNorm is rotation invariant.The chip remains unaware of the rotation; it is implemented entirely during compilation.
  • Quantization solution: Approximately 11 dB of residual INT8 SNR is recovered at every encoder layer, increasing SNR from approximately 24 dB to 36 dB.The gain is described as roughly two effective bits of precision and restores end-to-end translation from garbage to the exact reference stream.

D. Running on the RTL

The compiled T5 workload runs on cycle-accurate RTL with independent per-token and byte-level checks, producing bit-exact agreement with the compiler and golden model across ten multilingual translations.

  • RTL execution: The RTL test loads the compiled image, programs LUTs and IRAM, starts each program, and checks chip-written RAW32 logits against the expected token at every decode step.The verification sequence uses a first-max-wins argmax rule matching numpy.argmax.
  • RTL execution: 37,905,664 bytes of chip-written output were byte-compared across ten simulations with zero mismatches and zero UVM_ERROR or UVM_FATAL.The resulting chain is RTL = golden = compiler for all 2,095 programs, 70,320 descriptors, and 155 token checks.
  • Translation results: Five of ten INT8 translations matched the floating-point greedy reference token-for-token; the other five remained valid alternatives differing in punctuation, connective choice, or one word.The exact matches included all three German cases and two Romanian cases.
  • Translation results: The eight-opcode ISA executed the full encoder-decoder pipeline, including cross-attention, autoregressive KV-cache decode, and a full-vocabulary RAW32 LM head, without model-specific hardware.The same operator set composes into encoder, decoder, and encoder-decoder transformers, while task prefixes select three language pairs.
  • RTL performance context: The performance experiment reuses the verified datapath on cycle-accurate RTL and compares randomized-memory and ideal-memory profiles against a 22-thread CPU baseline.This setup distinguishes measured SIM results from projected EDGE and PERF configurations.
  • RTL performance context: 101 tok/s under hostile verification memory and 115 tok/s with ideal memory correspond to 20× and 22× the 5.13 tok/s CPU baseline.These are end-to-end measurements for the 64-MAC, 8-byte-bus SIM configuration.

C. Utilization and Traffic

Measured counters confirm that TFA approaches its modeled memory ceiling, while traffic remains dominated by weight-heavy operations and memory inference recoding substantially reduces implementation area.

  • Utilization: 79.3% AXI read-bus utilization approaches the predicted 80% structural ceiling, while the MAC array reaches 65.8% and the chip is busy for 99.3% of wall-clock time.The measurements validate the utilization model across the full workload.
  • Traffic: 711 MB of measured reads versus 548 MB modeled reads shows a 30% SIM excess caused by A/B tile re-reads in the v1 output-stationary dataflow.Larger EDGE/PERF tiles reduce this re-read factor in the projections.
  • Energy: 12.7 J/token for the CPU baseline compares with 4.3 mJ/token at EDGE and 7.8 mJ/token at PERF, improving energy per token by 2,974× and 1,627×.The CPU power is assumed and TFA power is pre-synthesis estimated; DRAM energy is excluded from both sides.
  • Caveats: SIM throughput is measured on RTL, whereas EDGE and PERF results are roofline projections, and the CPU baseline is idiomatic float64 numpy rather than an optimized inference stack.The comparison uses a 1 GHz reference design point and explicitly states these scope conditions.
  • Caveats: The documented v1 inefficiency is tile re-reading, which larger product tiles reduce and a proposed v1.1 B-stationary option removes for prefill.This is the nearest architectural boundary on the reported traffic results.
  • Implementation: The design completed open-source synthesis and place-and-route to a design-rule-clean GDS-II layout on SkyWater sky130.This provides a manufacturability result beyond cycle-accurate RTL simulation.
  • Traffic: The LM head, FFN, and attention projections dominate read traffic, whereas control-class operations account for most descriptors but under 1% of bytes.This separates descriptor-count overhead from the byte movement that bounds decode.
  • Implementation: Recoding memories for RAM inference reduced whole-chip logic area from 5.44 to 2.73 mm2, removing comparator-decode and flip-flop-memory artifacts.The changes preserved behavior through full regression and enabled compact RAM mapping.

C. Register-transfer to GDS-II

TFA progressed from verified RTL to a design-rule-clean SkyWater sky130 layout, while physical results exposed implementation-dependent timing, area, power, and dataflow boundaries. The paper frames these results as a manufacturability demonstration and identifies concrete paths toward higher-performance implementations.

  • Physical implementation: Design-rule-clean routing completed the full chip hardening flow on SkyWater sky130, with the routed die occupying a 5.48×5.48 mm area.The flow used SiliconCompiler asicflow with Yosys, OpenROAD, and KLayout; the layout streamed out cleanly with zero design-rulecheck violations.
  • Timing and throughput: 5.76 MHz was the post-route maximum frequency: hold timing passed with +0.029 ns worst slack, but setup timing retained a large negative slack.The result reflects generic, non-retimed mapping, flip-flop-mapped memories, and a fully combinational requantization datapath; production would pipeline the source.
  • Timing and throughput: 0.74 GOPS at 5.76 MHz and 128 GOPS at the 1 GHz design point follow from the 8×8 output-stationary INT8 MAC array.The array contains 64 MAC units and performs 128 INT8 operations per cycle; a 32×32 tile at 1 GHz is projected at about 2 TOPS.
  • Scope and limitations: The reported physical, throughput, and energy results are scope-limited: SIM values come from RTL simulation, while EDGE and PERF values are pre-synthesis projections.The post-route frequency and per-token energy estimates should be read as order-of-magnitude open-flow figures at 130 nm rather than characterized product results.
  • Scope and limitations: The v1 streaming dataflow re-reads tiles and caps prefill arithmetic intensity, motivating stationary-B buffering and descriptor indirection as future architectural changes.The paper also identifies compiler integration for RoPE, billion-parameter INT8 validation, and decoder-only end-to-end execution as remaining work.
Loading 2608.23582v1…