PaperlayerView exact revision on arXiv

Source-linked AI summary

TOMOS: Eliminating Data Duplication in Mixed-Precision Matrix Computations

Vasileios Georgiou, Petros Drineas

arXiv:2610.19047v1cs.MS

TL;DR

Mixed-precision computation often needs multiple representations or repeated conversions of the same matrix, increasing storage and data-movement costs. TOMOS stores each FP32 entry as primary and complementary words, exposing low-precision access and exact FP32 reconstruction from one representation. Its kernels achieve throughput comparable to vendor implementations, and converging LSQR implementations report speedups of 1.42× to 1.63× over the FP32 baseline.

  • Problem

    Mixed-precision methods often need duplicate matrix representations or repeated conversions, while data representation can remain a source of redundant storage, conversion overhead, or numerical error.

    Source: §I · p. 1

  • Method

    TOMOS stores primary and complementary words in one matrix representation, with specialized kernels accessing low-precision values or reconstructing FP32 values locally during computation.

    Source: A. Contributions · pp. 1–2

  • Results

    Converging LSQR implementations achieve speedups ranging from 1.42× to 1.63× over the baseline FP32 implementation.

    Source: C. End-to-end benefits:… · p. 10

  • Takeaways & Limitations

    TOMOS translates kernel-level accuracy and data-movement properties into end-to-end LSQR speedups and memory savings without redundant matrix storage.

    Source: D. Discussion · p. 10

    Benchmarks used synthetic data to control data statistics, and the study is inconclusive about GEMV portability.

    Source: 2) Limitations · p. 10

Abstract

from arXiv · show

Mixed-precision algorithms exploit reduced-precision hardware, however computing with the same matrix at distinct precisions typically requires duplicate storage, repeated conversions, or operation-emulation. We introduce TOMOS, a composite-precision matrix format that provides implicit 32-bit and 16-bit access from a single representation. Each FP32 entry is split into a 16-bit primary word for low-precision computation and a complementary word that enables exact FP32 reconstruction. The encoding scheme improves numerical quality of the low-precision view through subnormal representations, and allows exact recovery of high-precision inputs. TOMOS uses FP32-sized storage, with data movement scaling with compute precision. Optimized GEMM and GEMV kernels achieve throughput comparable to vendor libraries on NVIDIA A100 GPUs, while a mixed-precision least-squares solver achieves up to 1.6x speedup over FP32 without redundant matrix allocations.

I. INTRODUCTION

Mixed-precision workloads need multiple precision views, but storing copies or repeatedly converting data can increase memory footprint and traffic. TOMOS addresses this representation gap with a single layout for multiple compute precisions, exact FP32 recovery, and low-precision kernels that use less data movement.

I. INTRODUCTION

  • Mixed-precision execution can require duplicate representations or repeated conversions, increasing memory footprint, traffic, instructions, and synchronization overhead.Scientific methods may need high-precision stability while selectively using lower precision, yet duplicated formats or dynamic-range sensitivity remain concerns.Source: §I · p. 1

A. Contributions

  • TOMOS introduces FP32/FP16 and FP32/BF16 layouts that expose multiple precision views without explicit copies.The representation uses FP32-sized storage: low-precision operations access primary words, while FP32 operations combine primary and complementary words for exact recovery.Source: A. Contributions · pp. 1–2
  • TOMOS encodes each FP32 entry in a primary low-precision word and a complementary word that enables exact FP32 reconstruction.Rounding information is retained and reversible; special encodings handle values outside the normal FP16 range.Source: A. Contributions · pp. 1–2
  • Specialized GEMM and GEMV kernels move data according to compute precision, accessing only primary words for low precision and reconstructing FP32 values locally.On NVIDIA A100 GPUs, the kernels achieve throughput comparable to vendor implementations.Source: A. Contributions · p. 2
  • Nine mantissa bits plus rounding information produce lower FP16 GEMM and GEMV error than ten mantissa bits obtained by direct truncation.The primary FP16 encoding sacrifices one mantissa bit to store control information.Source: A. Contributions · p. 2

A. Notation

  • Floating-point values use sign, fractional, and exponent fields; subnormals represent values below the smallest normalized exponent by shifting the fraction right.The notation also defines bias correction as the source–target exponent-bias difference added during conversion; FP32-to-FP16 uses 112.Source: A. Notation · p. 2

B. Related Work

Prior approaches address precision reduction through segmentation, compression, arithmetic emulation, or shared exponents, whereas TOMOS focuses on exposing multiple native precision views from one stored matrix. Its FP32/FP16 representation divides each value into primary and complementary words for low-precision access and exact FP32 reconstruction.

B. Related Work

  • CPMS partitions FP64 values into segments, while TOMOS instead exposes native FP16 and FP32 views from an FP32-sized representation.CPMS varies the number of mantissa segments and retains 11 exponent bits.Source: B. Related Work · p. 2
  • Compression methods target storage reduction, while TOMOS preserves the highest represented precision's storage footprint and reorganizes it for access at multiple compute precisions,,.TOMOS is therefore a representation for multiple views, rather than a compression scheme.Source: B. Related Work · pp. 2–3
  • APE and MixPert emulate 32-bit numerics with reduced-precision instructions, whereas TOMOS provides multiple precision views of the same stored data,.Ozaki-style mantissa splitting instead evaluates partial products and accumulates them to emulate higher precision [20], [21].Source: B. Related Work · p. 3
  • Unlike block floating-point formats, TOMOS does not share exponents across value groups and preserves exact reconstruction of each FP32 entry,,.TOMOS' components form a storage representation from which low- or high-precision values can be accessed directly.Source: B. Related Work · p. 3

III. TOMOS: COMPOSITE PRECISION REPRESENTATIONS

  • The FP32/FP16 representation splits each FP32 value into primary p and complementary c words; c combines with p to reconstruct the original value exactly.The primary word approximates FP16 with five exponent and nine mantissa bits; the words also encode control information for bounds and rounding.Source: §III · p. 3
  • FP32-decode reconstructs f from the primary and complementary encodings, while FP16-decode converts special-value words into valid FP16 words.FP32/FP16-encode generates the two encodings.Source: §III · p. 3

A. Rounding reversal in FP32 reconstruction

TOMOS preserves exact FP32 reconstruction by recording how the primary encoding was rounded, and uses special encodings for values outside the normal FP16 exponent range. Decode optimizations trade execution-path divergence against instruction and resource costs.

A. Rounding reversal in FP32 reconstruction

  • The complementary word’s rounding-direction flag reverses the primary encoding’s rounding, enabling exact FP32 reconstruction.The encoder rounds the nine most significant mantissa bits; the complementary word stores the reverse-rounding direction.Source: A. Rounding reversal in FP32… · pp. 3–4

B. Special Encodings of the FP32 / FP16 Representation

  • For FP16 underflows, TOMOS stores a shift count in exponent bits to recover subnormal mantissa information; for overflows, it stores an offset to the maximum FP16 exponent.Special encodings can be decoded as subnormals or mapped to zero; extended supernormal values can use an 8-exponent-bit format.Source: B. Special Encodings of the FP32 /… · p. 4

C. Decode Device-Function Optimizations

  • Decode optimization targets unpacking information attached to the primary exponent, while encode cost is amortized because encoding occurs once at initialization.Conditional decoding risks warp divergence; branchless decoding avoids it but uses more resources and assembly instructions, and fast-decode maps subnormals to zero.Source: C. Decode Device-Function Optimizations · p. 4

D. FP32 / BF16 Representation

The FP32/BF16 format provides access to FP32 and BF16 computations using primary encodings with an 8-bit exponent. Its encoding can optionally retain rounding information.

D. FP32 / BF16 Representation

E. Multiprecision Access to Contiguous Matrix Entries

TOMOS exposes FP16 and FP32 matrix views from primary and complementary arrays, reconstructing FP32 values locally during matrix operations. Its GEMM implementations use tiling and pipelining, with workspace decoding used to address register pressure.

E. Multiprecision Access to Contiguous Matrix Entries

  • TOMOS exposes FP16 values through the primary array and reconstructs FP32 matrix actions from both arrays in registers and in parts.Reconstructing the entire matrix would undermine the goal of minimizing data duplication.Source: E. Multiprecision Access to Contiguous… · p. 5

IV. MULTIPRECISION BLAS ON THE NVIDIA A100

  • Composite-precision BLAS kernels support FP32, FP16, or BF16 computation over native and composite storage layouts.The implementations include GEMM and GEMV variants, including transposed GEMV.Source: §IV · p. 5

A. GEMM Kernels

  • The FP32/Fp16 GEMM kernel uses a two-layer asynchronous pipeline to interleave FMA work with tile transfers, but not decode steps.A three-stage asynchronous-load pipeline consistently slowed benchmarks because of increased shared-memory requirements.Source: A. GEMM Kernels · p. 5
  • When workspace holds only a small subset of matrix rows, the memory-reduction benefits remain unaffected.Batching decoded A tiles into global-memory workspace reduces shared-memory pressure and allows reuse of reconstructed FP32 values, at the cost of workspace allocation.Source: A. GEMM Kernels · p. 5

B. GEMV Kernels

For large-scale GEMV, TOMOS decodes composite-layout entries in groups and updates dot products with FMA instructions. The evaluation also examines standalone kernels, numerical accuracy, and a mixed-precision least-squares solver.

B. GEMV Kernels

  • The best-performing FP32/FP16 GEMV accesses primary and complementary entries in groups, decodes them to FP32, and updates dot products with FMA instructions.The FP32 reference maps each thread to an output row and uses vectorized loads.Source: B. GEMV Kernels · p. 6
  • Branchless decoding is crucial for high-throughput GEMV, while workspace techniques do not help these memory-bound kernels.Source: B. GEMV Kernels · p. 6

V. EVALUATION METHODOLOGY

  • The evaluation measures standalone-kernel performance and accuracy, then uses the kernels in a mixed-precision least-squares solver to assess end-to-end performance and memory usage.It also examines rounding and subnormal representations in low-precision computations.Source: §V · p. 6

A. Standalone Kernel Evaluation - Performance

The evaluation compares TOMOS kernels with native and vendor baselines across compute-bound GEMM and memory-bound GEMV, while separately testing numerical accuracy as dimensions and input magnitudes vary.

A. Standalone Kernel Evaluation - Performance

B. Standalone Kernel Evaluation - Numerical Accuracy

  • Relative Frobenius error compares promoted kernel outputs with high-precision matrix products after inputs are demoted to FP32 or FP16.A is stored in composite form, while B uses native floating-point storage.Source: 1) Notation · p. 6

2) Test cases:

  • The numerical tests vary square-matrix dimension, input magnitude, and the rounding scheme used to convert FP32 inputs for TOMOS FP16 GEMM.The rounding comparison uses the same setting as the magnitude-scaling test.Source: 2) Test cases · p. 6

C. Mixed-Precision Linear Regression Solvers

The solver evaluation pairs LSQR with a CountSketch-based randomized preconditioner and compares mixed-precision operation with FP32. The supplied kernel results report workspace-based GEMM throughput and memory savings, while the reproducibility passage specifies the hardware and software environment.

C. Mixed-Precision Linear Regression Solvers

1) GEMM: Can composite storage match native throughput:

2) GEMV: What is the cost of decoding in a memory-bound kernel?:

GEMV exposes decoding costs more than GEMM because it is memory-bound, and decode strategy strongly affects its throughput. The numerical comparisons find competitive FP32/FP16 accuracy, with distinct stability outcomes across GEMV implementations.

2) GEMV: What is the cost of decoding in a memory-bound kernel?:

  • Branchless and fast decoding recover much of GEMV’s lost throughput, while workspace-based decoding provides little benefit for the memory-bound operation.FP32/BF16 GEMV matches FP32 performance; fast decode reduces latency for FP32/FP16 GEMV.Source: 2) GEMV: What is the cost of decoding… · p. 7

1) Error analysis for scaling matrix-dimensions:

  • FP32 TOMOS implementations have accuracy comparable to native implementations, while composite FP32/FP16 remains competitive with native FP16 despite one fewer primary mantissa bit.Across GEMV implementations, the FP32/FP16 method is more accurate because it accumulates in FP64; BF16 GEMV is the least numerically stable.Source: 1) Error analysis for scaling… · pp. 7–8

2) GEMV: What is the cost of decoding in a memory-bound kernel?:

2) Error analysis for magnitude-scaling inputs:

The section evaluates TOMOS under magnitude scaling and tests whether rounding offsets the primary word’s lost mantissa bit. It then examines LSQR convergence, speed, and memory use with implicit mixed-precision storage.

2) Error analysis for magnitude-scaling inputs:

  • Magnitude-scaling results are consistent with dimension-scaling results, with no variability across scaling factors.Figure 8 shows the magnitude-scaling error plots for square matrices.Source: 2) Error analysis for… · p. 8, Figure 8 · p. 9

3) Can rounding compensate for the lost mantissa bit:

C. End-to-end benefits: Mixed-Precision LSQR

  • On matrix-1, explicit mixed precision converges in 70 iterations, branchless implicit storage takes 3 more, while fast-decode stagnates.Fast-decode maps FP16 subnormals to zero, causing information loss.Source: C. End-to-end benefits:… · pp. 9–10
  • 1.42×–1.63× speedups over FP32 and ~33% lower memory footprint versus explicit-copy LSQR are reported for converging implementations.Figure 11 reports results for two matrices.Source: C. End-to-end benefits:… · p. 10
  • ~95% workspace reduction is achieved by TOMOS-32/16 kernels when low-precision copies are counted as workspace.The comparison treats those copies as workspace necessary to reduce runtime.Source: C. End-to-end benefits:… · p. 10

D. Discussion

The discussion summarizes TOMOS’s kernel and LSQR benefits, while noting tradeoffs between decode throughput and numerical robustness. It distinguishes TOMOS from arithmetic-emulation approaches and identifies portability and benchmark-scope limitations.

D. Discussion

  • Decode strategies that maximize throughput may sacrifice subnormal information, while branchless decoding is more robust at additional computational cost.Initialization-time matrix statistics can inform kernel selection based on application requirements.Source: D. Discussion · p. 10

1) Comparisons to related work.:

  • TOMOS exposes multiple precision views from one physical layout while preserving exact recovery of the original FP32 data.Unlike emulation-based methods, it does not improve FP32 performance through algebraic manipulation of inputs.Source: 1) Comparisons to related work · p. 10

2) Limitations:

  • The benchmarks use synthetic data, and the study is inconclusive about GEMV portability.Extending the format across precision combinations and GPU architectures requires considering supported formats and instructions.Source: 2) Limitations · p. 10

VII. FUTURE WORK

  • Future work will investigate TOMOS in broader deep-learning and scientific-computing workloads, automated optimization, adaptive precision, and software/hardware co-design.Source: §VII · p. 11

Related papers

This paper cites

Loading 2610.19047v1…