PaperlayerView exact revision on arXiv

Source-linked AI summary

DyRA: Dynamic Residual Approximation for Efficient Matrix Multiplication in DNNs

Daewon Chae, Hyunwon Chung, Changwoo Lee, Hun-Seok Kim

arXiv:2610.02882v1cs.LGcs.AI

TL;DR

Structured weight approximations reduce computation, but they do not account for input-dependent output errors. DyRA dynamically approximates and corrects those errors in output space, improving the accuracy–efficiency trade-off across tested models and achieving a 1.52× GPU speedup on DINOv3 with a smaller accuracy drop than the weight-only baseline.

  • Problem

    Weight-centric approximations do not account for input activations, so the realized residual product can produce large output errors even when the weight residual is small.

    Source: §2.1 · p. 2

  • Method

    DyRA augments a static structured weight approximation with an input-adaptive correction, directly optimizing low-rank factors of the residual output using alternating least squares.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    DyRA consistently improves the accuracy–efficiency trade-off across vision, language, and speech models; on DINOv3 it achieves 1.52× GPU speedup with 1/3 of the weight-only baseline’s accuracy drop at the same speedup.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Output-domain approximation improves the accuracy–efficiency trade-off across low-rank, Monarch, and BLAST approximations and preserves dense prediction performance under aggressive FLOPs reduction.

    Source: §5 · p. 9

    DyRA is most effective for large matrix–matrix multiplications and is not primarily targeted at token-by-token vector–matrix workloads such as autoregressive decoding.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large-scale foundation models achieve strong performance across diverse tasks, but their size makes inference costly, largely due to dense matrix multiplications. Prior work reduces this cost by replacing dense weight matrices with efficient structured forms such as low-rank factorizations. However, these methods approximate weights rather than the output activations that determine inference accuracy. Consequently, small weight-space errors can be amplified by input activations, producing large output errors. In this work, we propose DyRA, an input-adaptive method that improves structured matrix multiplication approximation by correcting residual output errors during inference. We show that matrix multiplication can be approximated more effectively by directly optimizing low-rank factors of the output. DyRA builds on this insight by dynamically approximating and correcting the output error introduced by structured weight approximations. This combines efficient structured computation with input-dependent correction, yielding a more faithful approximation of full matrix multiplication under the same computational budget. Across vision, speech, and language models, DyRA consistently improves the accuracy-efficiency trade-off over structured weight approximations alone. Notably, DyRA achieves a 1.5$\times$ end-to-end GPU speedup for DINOv3 while reducing accuracy degradation by more than 3$\times$ relative to weight-only baselines.

1 Introduction

Weight-only structured approximations can produce input-dependent output errors and degraded features. DyRA corrects those errors dynamically, improving the accuracy–efficiency trade-off across models and preserving DINOv3 feature structure.

1 Introduction

  • Structured weight methods reduce inference computation, but they optimize weights without accounting for input activations, which can amplify residuals into large output errors.Prior approaches include low-rank and block-structured variants such as Monarch and BLAST.Source: §1 · p. 1
  • At 60% FLOPs reduction, DINOv3 PCA visualizations show noisy, degraded features for weight-only approximations, while DyRA preserves structure close to the original model.The first three principal components are mapped to RGB.Source: Figure 1 · p. 2
  • DyRA adds an input-adaptive residual correction, optimizing low-rank factors in output space with alternating least squares during inference.It avoids explicitly forming the dense residual product while correcting errors from the structured weight approximation.Source: §1 · p. 2
  • 1.52× GPU speedup on DINOv3 incurs 1/3 of the weight-only baseline’s accuracy drop at the same speedup; at 60% FLOPs reduction, mIoU drops 1.7 versus 54.1 points.Source: §1 · p. 2

2 Proposed Method

DyRA augments a structured weight approximation with an input-adaptive low-rank correction to the residual output. Its one-step ALS correction uses low-dimensional computations and, under a matched FLOPs budget, yields lower output error than a static low-rank baseline.

2.1 Overview

  • Because output error depends on both the residual weights and input activations, DyRA adds an input-dependent correction to the structured weight approximation.The method approximates the residual product at inference time to produce a more faithful estimate of the full product within a fixed computation budget.Source: §2.1 · pp. 2–3

2.2 Dynamic Residual Approximation

  • DyRA approximates the realized residual product XWres with low-rank factors optimized by ALS, avoiding explicit formation of the full residual output.The factor updates use low-rank intermediate products; ridge regularization is used in practice for numerical stability.Source: §2.2 · p. 3

2.3 Analysis of DyRA with One-Step ALS Residual Approximation

  • DyRA shares the additional left singular directions of a static rank-(k + r) approximation but optimizes its right factor for the current input.When X^T X = I, the one-step correction reduces to the static rank-(k + r) approximation; otherwise, input feature directions are reweighted.Source: §2.3 · pp. 4–5
  • Under a matched FLOPs budget, DyRA achieves lower normalized output error than a static low-rank baseline across ViT layers.The static baseline increases its rank to account for the dynamic correction’s computational overhead.Source: Figure 3 · p. 5, §2.3 · p. 5
  • The dynamic correction adds an O(c1c2r) cost over increasing static rank, while remaining cheaper than the dense product when r is small relative to c1 and c2.The additional term accounts for constructing the input-adaptive subspace.Source: §2.3 · p. 5

2.4 Implementation of DyRA

  • Custom Triton kernels fuse residual reconstruction with output merging, while the ALS linear-system solver uses Cholesky for small r and Newton–Schulz for larger r.The Figure 4 caption reports improved accuracy–efficiency trade-offs across low-rank, Monarch, and BLAST baselines for ViT-L ImageNet classification.Source: §2.4 · p. 5, Figure 4 · p. 6

3 Experimental Results

DyRA improves the accuracy–efficiency trade-off of structured weight approximations across vision, speech, and language tasks, including when evaluated at test time, with GPU latency, or after additional adaptation. The evaluations also examine input-aware decomposition, fine-tuning, and choices of residual rank and ALS iterations.

3 Experimental Results

  • DyRA consistently improves downstream performance over structured weight approximations under matched computational budgets across vision, language, and speech tasks.The evaluation covers pre-trained models without fine-tuning, end-to-end GPU latency, and settings with calibration or fine-tuning.Source: §3 · p. 6

3.1 Test-Time Model Approximation and Acceleration Across Domains

  • DyRA achieves lower WER than static weight approximation baselines on LibriSpeech Clean under matched computational budgets.Table 1 reports WER at different FLOPs reduction ratios.Source: §3.1 · p. 6, Table 1 · p. 6
  • At 60% FLOPs reduction on DINOv3 semantic segmentation, DyRA limits the mIoU drop to 1.7 points, versus 54.1 points for static weight approximation.The comparison also covers depth estimation, where DyRA achieves a better accuracy–efficiency trade-off.Source: §3.1 · p. 6, Figure 5 · p. 7
  • On LLaDA-8B, DyRA improves HumanEval and GSM8K accuracy by more than 20 percentage points over the FLOPs-matched low-rank baseline.The baseline scores below 4% on both tasks; DyRA brings results substantially closer to the original model.Source: §3.1 · p. 7, Figure 7 · p. 8

3.2 Runtime Analysis

  • On DINOv3 depth estimation, DyRA reaches 1.3× end-to-end speedup with about 0.6% accuracy degradation; the low-rank baseline needs a 3.7× larger accuracy drop for 1.52× speedup.The latency evaluation uses an NVIDIA A40 GPU.Source: Figure 6 · p. 7, §3.2 · p. 7

3.3 Further Evaluations with Augmented Methods

  • On ImageNet at around 70% FLOPs reduction, adding input-aware decomposition raises accuracy from 65.4% to 68.8% over DyRA with vanilla low-rank decomposition.DyRA also outperforms input-aware decomposition alone, and the combination further improves LLaDA results on HumanEval and GSM8K.Source: §3.3 · p. 8, Figure 7 · p. 8
  • After 15 epochs of ImageNet fine-tuning, DyRA outperforms the static low-rank baseline across FLOPs budgets and slightly surpasses the dense model at around 60% FLOPs reduction.The evaluation replaces all ViT-L linear layers with DyRA-augmented low-rank approximations.Source: Figure 8 · p. 8, §3.3 · p. 8
  • Across residual ranks r = 4, 8, 16, and 24, DyRA beats the corresponding static baseline; all tested ALS iteration counts improve over that baseline.One ALS iteration already provides strong performance, supporting the default t = 1 as a cost–accuracy trade-off.Source: §3.3 · pp. 8–9, Figure 9 · p. 9

4 Related Works

Prior work uses low-rank and block-structured matrices to reduce dense-layer costs, and some methods incorporate calibration activations into static weight decompositions. DyRA instead targets the sample-dependent output error during inference; its use of ALS for matrix products differs from ALS methods aimed at matrix reconstruction.

4 Related Works

  • Low-rank approximations and more expressive block-structured forms such as Monarch and BLAST are used to reduce dense linear-layer costs.Other approaches learn efficient weight structures directly, including GBLR and Einsum Linear.Source: §4 · p. 9
  • Input-aware decompositions use calibration activations but remain static at inference, unlike DyRA’s on-the-fly approximation of sample-dependent output error.DyRA can be combined with structured or input-aware decompositions under a fixed computational budget.Source: §4 · p. 9
  • Earlier ALS applications include low-rank matrix reconstruction and structured recovery, whereas DyRA uses ALS to approximate matrix products directly.Its target is efficient matrix multiplication rather than reconstruction of a single matrix.Source: §4 · p. 9

5 Conclusion

DyRA approximates realized outputs and adds an input-adaptive residual correction to structured weight approximations, improving their accuracy–efficiency trade-off. The conclusion also identifies workload boundaries and describes structured matrix forms used in the paper.

5 Conclusion

  • DyRA approximates the realized output product and uses an input-adaptive residual correction to improve the accuracy–efficiency trade-off across low-rank, Monarch, and BLAST approximations.The correction is optimized during inference using ALS; the conclusion also reports preservation of dense prediction performance under aggressive FLOPs reduction and practical GPU speedups.Source: §5 · p. 9
  • DyRA targets large matrix–matrix multiplications; token-by-token autoregressive decoding is not its primary application.The method remains applicable to autoregressive-model prefilling, where the full prompt induces large matrix multiplications.Source: §5 · pp. 9–10

A DyRA Algorithm

  • The algorithm summary covers offline preparation and inference-time dynamic residual correction.The supplied algorithm passage identifies these stages but does not specify their steps.Source: A DyRA Algorithm · p. 13

B Background on Structured Matrices

  • Low-rank approximation uses a rank-r factorization, while Monarch uses block-diagonal factors and a permutation to create structured block-wise patterns.For low rank, truncated SVD is a standard choice and keeps the top-r singular components as the best rank-r approximation under the Frobenius norm.Source: B Background on Structured Matrices · p. 13
  • BLAST shares row and column bases across matrix blocks, while input-aware decomposition uses calibration inputs to reduce output-space discrepancy.Input-aware methods may use activation-based diagonal scaling or a Cholesky factor of the input covariance matrix.Source: B Background on Structured Matrices · pp. 13–14

C Proof of Proposition 1

The proof shows that, with isotropic inputs, a one-step DyRA residual correction recovers the static rank-(k+r) truncated SVD result. It derives the one-step factors using orthogonality of singular vectors.

C Proof of Proposition 1

  • The proof begins with the one-step approximation XW ≈ X c W + L(1)R(1), then derives its low-rank factors from the residual.Orthogonality of the singular vectors is used in the factor updates.Source: C Proof of Proposition 1 · p. 14
  • In the isotropic case, one-step DyRA reduces to the static rank-(k + r) truncated SVD.The derivation substitutes the residual’s singular-vector factors into the ALS updates and uses orthogonality.Source: C Proof of Proposition 1 · p. 14

D Experimental Details

The experimental details span image classification, dense prediction, speech recognition, and language modeling, with structured approximations evaluated at specified ranks or FLOPs-reduction targets. Additional setups cover fine-tuning and comparisons with calibration-based input-aware decomposition.

D Experimental Details

  • Image classification evaluates ViT-L/32 on ImageNet-1K with residual rank 8, varying low-rank, Monarch, and BLAST configurations against FLOPs-matched weight-only baselines.DyRA is applied to all linear layers, including MLP layers and attention projections.Source: D Experimental Details · p. 15
  • Dense prediction evaluates ViT-7B-based DINOv3 on ADE20K segmentation and NYUv2 depth estimation, using residual rank 48 and FLOPs-matched structured baselines.Static low-rank, BLAST, and Monarch configurations are varied for each task.Source: D Experimental Details · p. 15
  • Speech recognition measures WER on LibriSpeech test data with Wav2Vec2-Large, searching configurations at 25% and 50% FLOPs reduction.Reported configurations specify MLP, attention, and residual ranks for BLAST and low-rank variants.Source: D Experimental Details · p. 15
  • Language modeling evaluates LLaDA-8B on HumanEval and GSM8K using low-rank DyRA, with more aggressive FLOPs reduction for query and key projections than MLP layers.The setup uses two total-FLOPs targets, with static and residual ranks specified separately for query/key and MLP layers.Source: D Experimental Details · pp. 15–16
  • A separate ViT-L/32 ImageNet experiment fine-tunes DyRA and FLOPs-matched static low-rank baselines for 15 epochs; input-aware comparisons use calibration data from each evaluation task’s training split.HumanEval uses GSM8K training data for calibration because it has no dedicated training split.Source: D Experimental Details · p. 16

E.1 Autoregressive LLM Prefilling

The section tests DyRA during autoregressive LLM prefilling, where extended prompts are processed in forward passes. At a 20% matrix-multiplication FLOPs reduction, it improves over matched low-rank and BLAST approximations.

E.1 Autoregressive LLM Prefilling

  • On ViT-B/16, combining DyRA with low-rank, Monarch, or BLAST improves the accuracy–efficiency trade-off over static weight-approximation baselines.Figure 10 plots top-1 accuracy against FLOPs reduction.Source: Figure 10 · p. 17
  • The prefilling evaluation processes each prompt and candidate continuation in a forward pass, using perplexity and multiple-choice reasoning benchmarks.Few-shot prompts provide extended contexts for the multiple-choice tasks.Source: §E.1 · pp. 16–17
  • At 20% matrix-multiplication FLOPs reduction, LLaMA-7B DyRA consistently improves over matched low-rank approximations and outperforms BLAST.The evaluation uses perplexity and multiple-choice reasoning benchmarks; results support application to autoregressive-model prefilling.Source: §E.1 · p. 17

E.2 Image Classification with ViT-B

On ViT-B/16 image classification, DyRA improves the accuracy–efficiency trade-off over corresponding FLOPs-matched static weight approximations, extending gains beyond ViT-L.

E.2 Image Classification with ViT-B

  • DyRA consistently improves the accuracy–efficiency trade-off over FLOPs-matched static weight approximations on ViT-B/16 image classification.The results extend the reported gains beyond the larger ViT-L backbone.Source: §E.2 · p. 17

E.3 Convergence of ALS Residual Approximation

The experiment evaluates ALS for approximating residual products across ViT-L linear layers, comparing its rank-16 approximation error with the optimal rank-16 SVD error. ALS typically approaches that optimum within two iterations.

E.3 Convergence of ALS Residual Approximation

  • The experiment uses activations from 4,096 ImageNet validation images and constructs residual weights after a rank-196 truncated SVD of each ViT-L layer's weights.ALS approximates each residual product XWres at target rank 16.Source: §E.3 · p. 17
  • At each ALS iteration t, they compute the relative Frobenius error between residual product XWres and low-rank approximation L(t)R(t), then compare it with the optimal rank-r error from direct SVD of XWres.Source: §E.3 · p. 17
  • ALS typically approaches the optimal rank-16 error within two iterations across ViT-L linear layers.Figure 12 reports results averaged over sampled images, with rapid convergence in nearly all layers.Source: §E.3 · p. 17

E.4 FLOPs-Matched Output Approximation Error

In the additional ViT-L sublayers, DyRA has lower output approximation error than a FLOPs-matched static low-rank baseline. The paper also notes that more efficient inference may broaden deployment of existing AI systems and inherit their societal risks.

E.4 FLOPs-Matched Output Approximation Error

  • DyRA consistently yields lower normalized Frobenius output error than the FLOPs-matched static low-rank baseline across additional ViT-L sublayers.These include query, key, and value attention projections and the second MLP linear layer; Figure 11 reports the additional results.Source: Figure 11 · p. 18, §E.4 · p. 18
  • The comparison uses a rank-196 weight approximation with a rank-16 DyRA residual correction computed in one ALS iteration.The static low-rank baseline's rank is increased until its exact FLOPs match DyRA's total FLOPs, including correction overhead.Source: §E.4 · p. 18

F Broader Impacts

  • DyRA may facilitate broader deployment of existing AI systems, inheriting societal risks associated with those systems and their applications.The paper identifies potential positive impacts through resource efficiency, deployment under limited computational budgets, and reduced inference energy costs.Source: F Broader Impacts · p. 18

Related papers

Loading 2610.02882v1…