PaperlayerView exact revision on arXiv

Source-linked AI summary

Implementation and Evaluation of BitNet Inference on a CGLA by Signed-Int4 Instructions

Takuto Ando, Yasuhiko Nakashima

arXiv:2609.27453v1cs.AR

TL;DR

LLM decoding repeatedly transfers model weights and activations, making memory traffic and its energy cost part of the decode path; BitNet’s ternary–int8 arithmetic also differs from conventional GEMM. This paper maps that operation to a programmable CGLA with reusable signed-int4 instructions, reporting a 0.390 ns projected product time and a 2.52-token/s decode with one offloaded call.

  • Problem

    BitNet’s ternary-weight, integer-activation dot products differ from conventional int8 or floating-point GEMM, while existing BitNet accelerators use specialized datapaths.

    Source: §I · p. 1

  • Method

    The paper maps BitNet ternary–int8 dot products to reusable signed-int4 OP_SMA4 streams on a programmable CGLA, splitting each int8 activation into two fragments for reconstruction.

    Source: §I · p. 1

  • Results

    2.52 tokens/s was measured for the CGLA-offloaded BitNet C++ decode, while the 145 MHz FPGA measurement projects to 0.390 ns per product at 840 MHz.

    Source: §I · p. 1, §V · p. 7

  • Takeaways & Limitations

    OP_SMA4 provides a reusable signed-int4 instruction that supports arbitrary signed-int4 operands rather than a BitNet-only datapath.

    Source: §I · p. 1

    The integrated end-to-end run offloads one selected call, and the 840 MHz timing and 1.82 W CGLA values rely on a 28 nm model and frequency-only scaling rather than silicon measurements.

    Source: E. Interpretation and Limitations · p. 6

Abstract

from arXiv · show

Large language model (LLM) inference transfers model weights and activations for every generated token, making memory traffic and its energy cost part of the decode path. BitNet b1.58 represents its low-bit weights by ternary values and uses integer activations. However, this arithmetic does not match conventional int8 or floating-point general matrix multiplication, and existing BitNet accelerators implement it in specialized datapaths. We instead map this operation to a CPU-Grounded Linear Array (CGLA), a programmable ASIC with explicit direct memory access, local memories, and reusable compiler-visible integer lanes. The mapping adds OP_SMA4 as a reusable signed-int4 multiply-accumulate instruction rather than a BitNet-only datapath. Each ternary weight occupies one signed 4-bit lane. Each int8 activation is split into two signed-int4 fragments and reconstructed by shift-and-add. Frequency scaling of the 145 MHz FPGA measurement to an 840 MHz 28 nm CGLA achieved 0.390 ns per signed-int4 product. We showed that CGLA-offloaded BitNet C++ execution measures 2.52 tokens/s.

I. INTRODUCTION

LLM decoding repeatedly moves weights and activations, making memory bandwidth and access energy part of decode cost; this paper maps BitNet’s ternary–int8 dot product to a programmable CGLA using reusable signed-int4 instructions. It evaluates the instruction, tiled projections, and an integrated runtime call, reporting a 2.52-token/s decode with one offloaded call.

I. INTRODUCTION

  • Repeated weight and activation transfers make memory bandwidth and memory-access energy relevant to decode latency and energy alongside peak arithmetic throughput.The motivation passage attributes this to each generated token requiring another pass through weights and activations.Source: §I · p. 1
  • BitNet’s ternary-weight and integer-activation dot products do not match conventional int8 or floating-point GEMM, while dedicated accelerators use specialized datapaths.The paper instead targets a programmable accelerator without adding a BitNet-only datapath.Source: §I · p. 1
  • OP_SMA4 reuses signed-int4 multiply–accumulate lanes, while two streams cover each int8 activation and the host reconstructs the result by shift-and-add.The instruction accepts arbitrary signed-int4 operands, not only ternary weights.Source: §I · p. 1
  • OP_SMA4 is evaluated as a reusable signed-int4 dot-product instruction, with its 2.26 ns FPGA measurement projected to 0.390 ns per product at 840 MHz.Source: §I · p. 1
  • The mapping validates two activation-fragment streams over 4096 outputs and uses sparse correction for positive values from 120 to 127.Each ternary weight occupies one signed-int4 lane, while an int8 activation is split into two signed-int4 fragments and reconstructed by shift-and-add.Source: §I · p. 1
  • 2.52 tokens/s was measured for a 19-token BitNet C++ decode with one offloaded call, including a 16.3 ms offload interval.The CGLA interval is only one selected call; remaining host execution is part of the integrated run.Source: §I · p. 1
  • Each CGLA processing element uses pipelined ALUs, double-buffered registers, and local memories, while the Processing System controls DMA transfers.The programmable logic contains the CGLA array.Source: Figure 1 · p. 2

II. RELATED WORK

The related-work discussion distinguishes BitNet model and runtime foundations from the low-bit execution tools used around the arithmetic kernel.

A. BitNet Models and Software Runtimes

  • BitNet and BitNet b1.58 define the low-bit model family, while the technical report supplies the model reference and BitNet C++ supplies the runtime tensor layout.The evaluated BitLinear call combines packed I2 S ternary-weight blocks with q8 activation blocks, then applies activation and weight block scales to integer partial sums.Source: A. BitNet Models and Software Runtimes · p. 2
  • T-MAC and llama.cpp describe lookup, packing, and CPU-side low-bit execution around the arithmetic kernel.CGLA offload retains the runtime tensor order, block arithmetic, and scaling sequence; model loading and token-generation control remain on the host.Source: A. BitNet Models and Software Runtimes · p. 2

B. Dedicated BitNet Hardware

Prior BitNet hardware commonly specializes ternary arithmetic, whereas this work uses a fixed signed-int4 instruction on a programmable CGLA and preserves its existing transfer and scheduling path.

B. Dedicated BitNet Hardware

  • Dedicated BitNet accelerators specialize datapaths for ternary arithmetic, whereas this implementation uses a reusable signed-int4 instruction and software decomposition of int8 activations.The comparison includes FPGA, ASIC, CiROM, and LUT-oriented designs; Bit Fusion dynamically composes bit-level processing elements for different operand widths.Source: B. Dedicated BitNet Hardware · p. 2

C. Coarse-Grained Reconfigurable Arrays (CGRAs) and CGLA for LLMs

  • Prior CGRA and CGLA studies cover LLM operations or kernels but do not map BitNet-style ternary BitLinear kernels onto these arrays.This work maps the ternary BitLinear kernel to a signed-int4 instruction and evaluates instruction cost, tiled projection, and the integrated-call boundary.Source: C. Coarse-Grained Reconfigurable… · p. 2

D. CGLA Architecture and Compilation Model

  • CGLA exposes an ordered linear array and fixed instruction interface, with compiler scheduling instead of two-dimensional PE placement-and-routing.The FPGA prototype places the CGLA core in programmable logic; DMA transfers operands from DDR to local memory modules, and scheduled rows execute and drain results.Source: D. CGLA Architecture and Compilation… · p. 2
  • Fig. 2 maps a ternary weight to a signed-int4 lane and an int8 activation to two fragments, whose products are reconstructed by shift-and-add.The mapping retains the DMA, local-memory, synchronization, and row-scheduling path while adding the reusable instruction.Source: Figure 2 · p. 3, D. CGLA Architecture and Compilation… · p. 3

III. PROPOSED METHOD

The method represents BitNet’s ternary–int8 dot product with two signed-int4 fragment products and runs the selected operation through the host–CGLA transfer, execution, and reconstruction path.

A. BitNet Ternary–int8 Dot Product

  • Each BitNet ternary–int8 product maps to two signed-int4 fragment products, with eight packed products per 32-bit lane and reconstruction after the partial sums.The ternary weight is stored directly; “int4 product” denotes one fragment product, while “original MAC” denotes one ternary–int8 product.Source: A. BitNet Ternary–int8 Dot Product · p. 3

B. Host–CGLA Execution Sequence

  • The host prepares ternary weights and activation fragments in DMA-visible buffers, then DMA stages an operand tile into local memory for scheduled PE-row execution.The selected dot product enters the CGLA stack at the packed-buffer boundary.Source: B. Host–CGLA Execution Sequence · p. 3
  • After 32-bit partial sums drain to a host-visible buffer, host code reconstructs and scales them; oversized projections repeat staging, execution, and drain across tiles.Layer execution accumulates returned tile partials; reported CGLA time includes staging, row execution, and drain, while model loading, tokenization, sampling, and layer scheduling remain outside the interval.Source: B. Host–CGLA Execution Sequence · p. 3

C. OP SMA4 Instruction and Mapping

OP_SMA4 maps BitNet’s ternary–int8 dot product onto reusable signed-int4 lanes by splitting each activation into two fragments. Shift-and-add reconstructs the integer dot product, with sparse correction for the positive activation values that exceed the signed-int4 fragment range.

C. OP SMA4 Instruction and Mapping

  • Each ternary weight is stored directly in a signed 4-bit lane, while each int8 activation is represented as two signed int4 fragments: a sign-extended low-nibble residue and an adjusted high fragment.The decomposition is a = 16ahi + alo, with ahi = (a − alo)/16.Source: C. OP SMA4 Instruction and Mapping · p. 3
  • For activations from 120 to 127, the encoded high fragment is −8, so the host adds 256w per affected element to correct the partial dot product.The correction is zero when the activation block contains no value from 120 to 127.Source: C. OP SMA4 Instruction and Mapping · p. 4
  • The host reconstructs the integer dot product by shifting the high-fragment sum left four bits and adding the low-fragment sum.The reconstructed integer partial is then multiplied by the activation scale; the accumulated output is multiplied by the I2 S weight scale.Source: C. OP SMA4 Instruction and Mapping · p. 4
  • In the fused-output schedule, CGLA combines the two sums before output transfer, while sparse correction and block scaling remain on the host.The archived schedule spans 64 rows, with rows 2–3 issuing signed-int4 operations, rows 5–7 reducing and combining sums, and row 63 storing the result.Source: C. OP SMA4 Instruction and Mapping · p. 4

D. Activation-Only Int4 Comparison Baselines

The comparison evaluates four activation-only single-stream int4 approximations against the exact two-stream mapping. The alternatives replace the two signed-int4 activation fragments with one quantized block, producing approximate dot products.

D. Activation-Only Int4 Comparison Baselines

  • The exact path uses two signed-int4 activation streams, whereas each of four baselines uses one quantized stream and produces an approximate dot product.All paths use the same ternary weights, signed-int4 instruction, and CGLA call boundary; activation representation and host-side reconstruction differ.Source: Figure 3 · p. 4, D. Activation-Only Int4 Comparison… · p. 4
  • The four activation-only baselines are max-abs scaling, power-of-two scaling, asymmetric affine scaling, and activation-MSE scale search.The MSE search tests ten scales from 0.5 to 2.0 times the max-abs scale and selects the one with the lowest activation-reconstruction mean squared error.Source: D. Activation-Only Int4 Comparison… · p. 4
  • Each baseline fixes its quantization parameters from the activation block alone before the CGLA call.Weights, CGLA outputs, and the exact ternary–int8 dot product do not select those parameters.Source: D. Activation-Only Int4 Comparison… · p. 4

IV. EXPERIMENTS AND DISCUSSION

The evaluation separates instruction timing, layer projection, and an integrated BitNet C++ call, with measurements spanning the CGLA software stack and model execution. Correctness checks report matching reference outputs and zero normalized integer-dot error for the corrected two-stream path.

A. Evaluation Setup

  • The evaluation separates instruction timing, layer projection, and one integrated BitNet C++ call to distinguish arithmetic cost from runtime overhead.The integrated experiment runs BitNet b1.58 2B-4T with two host threads and records ten 19-token runs as five pairs after warm-up and prompt evaluation.Source: A. Evaluation Setup · pp. 4–5
  • Kernel times are repeated-invocation wall-clock medians that include input staging, array execution, and output drain for each loaded-word count.Measurements use the CGLA software stack and the cacheable DDR path exposed by /dev/mem; scalar host code checks the output.Source: A. Evaluation Setup · p. 5
  • The device comparison assigns 0.6485 W to host execution and 2.07 W to active CGLA offload; CPU/GPU energy uses stated power or TDP-context estimates.Tiled BitNet layer projections use the measured product costs.Source: A. Evaluation Setup · p. 5
  • The corrected two-stream output has zero normalized integer-dot error, and the tested split path matches over 4096 outputs for inputs from −128 to 119.In the selected projection comparison, values from 120 to 127 make up 2.07% of processed activations and use sparse correction; all CGLA partial sums match the packed-int4 host reference.Source: A. Evaluation Setup · p. 5

B. Signed-Int4 Instruction Results

The OP_SMA4 benchmark includes staging, array execution, and output drain; its reported product cost is measured on the FPGA prototype and separately projected by clock scaling to the 28 nm CGLA model.

B. Signed-Int4 Instruction Results

  • 0.390 ns per signed-int4 product is the clock-scaled projection at 840 MHz; the measured FPGA cost is 2.26 ns per product at W = 32768.Scaling changes only the clock, from 145 to 840 MHz.Source: B. Signed-Int4 Instruction Results · p. 5
  • At W = 64, 1024, 4096, 16384, and 32768 loaded words, the measured kernel times are 15.0, 32.8, 87.5, 304, and 592 µs.Each invocation includes input staging, array execution, and output drain.Source: B. Signed-Int4 Instruction Results · p. 5
  • A two-parameter fit across the five word counts gives a marginal cost of 0.0176 µs per loaded word, with R2 > 0.999.At W = 32768, each operand buffer occupies 128 KiB in DDR and the invocation uses a 30.1 KiB resident LMM window.Source: B. Signed-Int4 Instruction Results · p. 5

C. BitNet Projection and Integrated Measurement

The integrated BitNet C++ runtime measures 2.52 tokens/s at the decode boundary, while the projected CGLA path is estimated at 0.258 J/token. The measurement offloads one selected call, with most decode time remaining on the host.

C. BitNet Projection and Integrated Measurement

  • Approximately 0.08 s at 145 MHz and 14 ms at 840 MHz are projected for the exact two-stream mapping of a 6912 × 2560 layer.The layer contains 1.77×10^7 ternary–int8 MAC operations, does not fit one LMM tile, and these are layer-time estimates rather than a directly timed invocation.Source: C. BitNet Projection and Integrated… · p. 5
  • The integrated runtime records a 7.53 s median for 19 generated tokens, or 2.52 tokens/s, with a 16.3 ms offload interval and approximately 7.51 s on the host.This decode-boundary measurement excludes model loading and prompt evaluation and offloads one selected call.Source: C. BitNet Projection and Integrated… · p. 5
  • Fused-output execution reduced median E2E time from 8.91 to 7.53 s across five run pairs, while fused in-array combination reduced drain time from 2.34 to 1.64 ms.The latter cut CPYOUT volume from 1.56 to 0.78 MB; active interval fell from 17.0 to 16.4 ms.Source: C. BitNet Projection and Integrated… · p. 5
  • 2.52 tokens/s and 0.258 J/token are reported for the fused CGLA path; its modeled energy is the lowest among four implementations, while W2A8 reaches 406.5 tokens/s.The comparison uses the 28 nm CGLA projection; the preferred operating point depends on the required decode rate and energy budget.Source: C. BitNet Projection and Integrated… · p. 5, Figure 4 · p. 6
  • The comparison does not establish accelerator-induced full-model speedup or energy saving because almost all decode time remains on the host.Source: C. BitNet Projection and Integrated… · p. 6

D. Activation-Only Baseline Comparison

The comparison tests exact two-stream reconstruction against four single-stream activation-only int4 baselines on one projection. Single-stream paths shorten some execution times but introduce normalized partial-dot error.

D. Activation-Only Baseline Comparison

  • 0.0821 normalized error at 27% lower path time is the asymmetric affine baseline’s result relative to the exact path.The comparison uses one 2560 × 2560 projection, one decoded token, and two host threads; the exact path takes 0.595 s with zero sparse-corrected integer error.Source: D. Activation-Only Baseline Comparison · p. 6
  • The max-abs, power-of-two, and asymmetric-affine baselines report normalized errors of 0.102, 0.0979, and 0.0821, respectively.Their path times are 0.359 s, 0.387 s, and 0.432 s, respectively.Source: D. Activation-Only Baseline Comparison · p. 6
  • Activation-MSE takes 1.97 s and gives 0.0942 normalized error because of host-side candidate evaluation, while its CGLA phases take 0.142 s.All four baselines execute their quantized activation stream through the same OP_SMA4 kernel; no perplexity or downstream-task evaluation establishes whether their errors are acceptable for deployment.Source: D. Activation-Only Baseline Comparison · p. 6

E. Interpretation and Limitations

The paper maps BitNet ternary–int8 dot products to a reusable signed-int4 instruction on a programmable CGLA. It reports a 0.390 ns per-product 28 nm projection and 2.52 tokens/s at a decode boundary with one offloaded call.

E. Interpretation and Limitations

  • The evaluation boundary limits the claims: the integrated run offloads one selected call, layer times are projections, and 840 MHz timing and 1.82 W use a 28 nm model rather than silicon measurements.Frequency-only scaling assumes the measured interval scales with core clock, including transfer overheads that may scale differently in silicon.Source: E. Interpretation and Limitations · p. 6
  • The software-visible instruction behavior is specified, but incremental decoder, register-path, and arithmetic-unit area and power are not isolated.The conclusion identifies host and DMA staging/drain optimization and full-model execution as future work.Source: E. Interpretation and Limitations · p. 6, §V · p. 7

V. CONCLUSION

  • 0.390 ns per product at 840 MHz is the frequency-only 28 nm projection from the 2.26 ns-per-product FPGA measurement at 145 MHz.The projected CGLA path is estimated at 0.258 J/token under the stated phase-accounted model, below the three CPU/GPU rows in the device comparison.Source: §V · p. 7
  • The conclusion reports 2.52 tokens/s at the decode boundary with one offloaded call and separately reports input staging, array execution, and output drain.Source: §V · p. 7
Loading 2609.27453v1…