Source-linked AI summary
SpecReuse: Spectral Graph Reuse for Efficient Vision GNN Inference on FPGAs
Isabella Bernhardt Eiliya, Anvitha Ramachandran, Dhruv Parikh, Viktor Prasanna
TL;DR
Repeated graph reconstruction at every Vision GNN layer remains a performance bottleneck despite FPGA optimizations to graph construction. SpecReuse uses compact spectral descriptors and an offline-calibrated drift threshold to select graph reuse, achieving substantial inference speedups and energy reductions with small resource overhead.
Problem
FPGA Vision GNN accelerators still execute Dynamic Image Graph Construction at every layer, leaving repeated reconstruction a source of latency and energy consumption.
Source: §I · p. 1Method
SpecReuse uses compact spectral descriptors and spectral similarity to decide whether to reuse a previously constructed graph or run DIGC.
Source: §I · p. 1, C. Spectral Descriptor-Based Graph Reuse · p. 4Results
Up to 2.69× end-to-end inference speedup and approximately 58–65% lower energy per inference were achieved, with 1.07% LUT, 0.56% FF, 0.12% DSP, and 0.88% BRAM overhead.
Source: §I · p. 1Takeaways & Limitations
Graph reuse reduces DIGC activity and lowers latency and energy per inference while preserving classification accuracy and avoiding changes to the underlying ViG architecture.
Source: E. Evaluation Summary · p. 6
Abstract
from arXiv · showhide
Dynamic Image Graph Construction (DIGC) is the primary performance bottleneck in FPGA acceleration of Vision Graph Neural Networks (ViGs), reconstructing graph connectivity at every layer through irregular, memory-intensive computation. Existing FPGA accelerators optimize DIGC but still execute it unconditionally, making repeated graph reconstruction a persistent source of latency and energy consumption. We propose the SpecReuse algorithm, which computes compact spectral descriptors of intermediate features and reuses previously constructed graphs when descriptor drift remains below a calibrated threshold. We further present the SpecReuse accelerator, an FPGA architecture that realizes graph reuse through lightweight hardware for spectral descriptor extraction and reuse control while remaining compatible with existing graph-construction accelerators. Experimental results demonstrate up to a $2.69\times$ speedup in end-to-end inference and approximately 58--65\% lower energy per inference with negligible FPGA resource overhead and minimal loss in classification accuracy.
I. INTRODUCTION
Repeated Dynamic Image Graph Construction (DIGC) remains a source of latency and energy cost in FPGA Vision GNN inference because existing accelerators still rebuild graphs at every layer. SpecReuse instead uses spectral similarity to decide when to reuse a graph, complementing accelerators that reduce the cost of each construction pass.
I. INTRODUCTION
- DIGC repeatedly reconstructs connectivity from intermediate features through similarity computation, neighbor selection, and graph materialization, causing irregular computation and substantial memory traffic.Existing FPGA designs optimize DIGC but still invoke it at every graph layer.Source: §I · p. 1
- Because neighboring ViG layers frequently preserve similar connectivity, graph reuse can reduce how often DIGC executes rather than merely accelerating each pass.This targets repeated work that remains in existing accelerators.Source: §I · p. 1
- SpecReuse adds a lightweight prediction front-end that extracts compact spectral descriptors and uses spectral similarity to decide whether a previously constructed graph can be reused.Reuse bypasses distance computation, sorting, and top-k selection while remaining compatible with existing graph-construction accelerators.Source: §I · p. 1
- Up to 2.69× end-to-end inference speedup and approximately 58–65% lower energy per inference were achieved with small FPGA resource overhead.Offline threshold calibration enabled up to 30.1% graph reuse while limiting Top-1 accuracy degradation to at most 0.54 percentage points.Source: §I · p. 1
II. RELATED WORK
- Prior ViG and FPGA approaches improve graph quality or reduce graph-processing and construction costs, but continue reconstructing connectivity throughout inference or assume a graph is already available.Dedicated graph-construction pipelines reduce the latency of a single pass; Vision GNN accelerators still execute DIGC at every graph layer.Source: §II · pp. 1–2
- SpecReuse targets the complementary problem of reducing DIGC frequency and is orthogonal to existing FPGA graph-construction accelerators because it leaves their DIGC implementation unchanged.It makes graph reuse a separate architectural optimization dimension.Source: §II · p. 2
III. PRELIMINARIES
Vision Graph Neural Networks represent images as graphs and repeatedly update connectivity as feature representations evolve. Message passing uses the dynamically constructed neighborhood to aggregate information and update node features.
III. PRELIMINARIES
- ViGs represent an image as a graph whose vertices are image patches and whose edges encode feature-space relationships between patches.Unlike conventional convolutional networks with fixed receptive fields, ViGs dynamically update connectivity during inference.Source: §III · p. 2
- As node features become more discriminative across layers, graph topology is repeatedly reconstructed so message passing can adapt to the current feature space.The evolving topology enables neighborhood structure to reflect changing semantic representations.Source: §III · p. 2
- At each graph layer, message passing applies learnable message and update functions, with neighborhood aggregation over the dynamically constructed neighbors of each node.The quality of the graph built before message passing directly influences information propagation.Source: §III · p. 2
A. Dynamic Image Graph Construction
DIGC rebuilds graph connectivity before each graph convolution, incurring repeated distance, ranking, and materialization work. SpecReuse estimates structural similarity from compact spectral descriptors and either reuses the prior graph or invokes the baseline DIGC pipeline, using offline-calibrated parameters.
A. Dynamic Image Graph Construction
- DIGC regenerates graph connectivity from current node features before each graph convolution by computing feature-space distances, selecting top-k neighbors, and materializing adjacency.Pyramid ViG reduces this computation in early stages by constructing graphs from a reduced set of conode features.Source: A. Dynamic Image Graph Construction · p. 2
- Repeated pairwise distance computation, neighbor ranking, and graph materialization make DIGC a recurring computational cost, motivating reduced construction frequency rather than a modified DIGC procedure.The method relies on the observation that connectivity often changes gradually between adjacent layers.Source: A. Dynamic Image Graph Construction · p. 2
IV. SPECREUSE ARCHITECTURE
- SpecReuse estimates structural similarity using compact spectral descriptors from a lightweight sparse affinity operator, reusing the previous graph below a drift threshold and otherwise running baseline DIGC.The operator avoids pairwise distance ranking, sorting, and neighbor selection; the policy can integrate with existing FPGA graph-construction accelerators without changing their implementation or graph-convolution pipeline.Source: §IV · pp. 2–3
- The reuse threshold and descriptor dimension are selected offline per backbone and remain fixed during inference.Calibration selects the largest threshold that preserves target accuracy while maximizing reuse, and the smallest descriptor dimension beyond which reuse-decision accuracy does not measurably improve.Source: §IV · p. 3
A. Accelerator Design Motivation
A software or naive hardware implementation of spectral descriptor extraction could erode the savings from skipped DIGC passes. SpecReuse maps descriptor extraction and drift comparison to fixed-latency hardware, with a front-end that decides whether to reuse cached adjacency or run DIGC before the unchanged graph-convolution interface.
A. Accelerator Design Motivation
- Although the affinity operator is lightweight relative to DIGC, Lanczos-based descriptor extraction on general-purpose cores or naive RTL can incur repeated sparse matrix-vector products, irregular memory accesses, and reductions.These costs can erode the algorithmic savings, motivating specialized fixed-latency hardware for extraction and drift comparison.Source: A. Accelerator Design Motivation · p. 3
B. Architecture Overview
- The prediction front-end precedes DIGC: the Spectral Descriptor Extractor computes a descriptor from incoming features, and the Spectral Reuse Estimator compares it with the stored descriptor.The Graph Reuse Controller compares measured drift with the offline-calibrated threshold.Source: B. Architecture Overview · p. 3
- When reuse is selected, the controller bypasses DIGC and sends cached adjacency from the Adjacency Register File to graph convolution; otherwise, baseline DIGC reconstructs the graph and updates the graph and descriptor buffers.The updated adjacency and descriptor become the reference for subsequent decisions.Source: B. Architecture Overview · p. 3
- Graph convolution receives valid adjacency through the same input interface whether it comes from the graph buffer or a new DIGC execution.The downstream graph-convolution pipeline requires no modification.Source: B. Architecture Overview · p. 3
- The five front-end modules are the Spectral Descriptor Extractor, Spectral Descriptor Buffer, Spectral Reuse Estimator, Graph Reuse Controller, and Adjacency Register File.They implement descriptor extraction, drift computation, reuse control, and the persistent descriptor and adjacency state while preserving baseline graph-construction and graph-convolution datapaths.Source: B. Architecture Overview · p. 3
C. Spectral Descriptor-Based Graph Reuse
SpecReuse uses compact spectral descriptors to decide whether a previous graph can be reused, avoiding full graph reconstruction when the descriptor drift stays below a threshold.
C. Spectral Descriptor-Based Graph Reuse
- Graph connectivity often changes gradually between consecutive layers, so reconstructing a graph at every layer can repeat computation while producing highly similar graphs.Source: C. Spectral Descriptor-Based Graph Reuse · p. 3
- SpecReuse compares compact spectral descriptors from a lightweight sparse affinity graph, rather than constructing successive DIGC graphs for direct comparison.Source: C. Spectral Descriptor-Based Graph Reuse · p. 4
- The descriptor retains only the K dominant eigenvalues of the affinity graph, with K ≪ N, as a compact signature of its dominant connectivity characteristics.Source: C. Spectral Descriptor-Based Graph Reuse · p. 4
- Squared Euclidean drift between the current and previous descriptors controls whether to reuse the prior graph or reconstruct one when drift exceeds the threshold.The squared distance preserves Euclidean-distance ordering while avoiding the square-root operation.Source: C. Spectral Descriptor-Based Graph Reuse · p. 4
- When spectral stability permits reuse, SpecReuse bypasses the full DIGC pipeline, including pairwise distance computation, top-k selection, and adjacency materialization.Source: C. Spectral Descriptor-Based Graph Reuse · p. 5
D. FPGA Integration
The SpecReuse front-end precedes DIGC and chooses whether graph reconstruction runs, while preserving the baseline graph-convolution interface.
D. FPGA Integration
- The prediction front-end precedes the entire DIGC pipeline and supplies graph convolution with adjacency from either the graph buffer or a new DIGC execution.Source: D. FPGA Integration · p. 5
- SpecReuse requires no modification to the graph-convolution datapath, which receives valid adjacency through the same interface on both paths.Source: D. FPGA Integration · p. 5
- The front-end operates on a lightweight affinity graph and fixed-length descriptor, and its overhead is amortized whenever a DIGC invocation is eliminated.Its estimator uses elementwise subtraction, squaring, and accumulation, followed by a threshold comparison.Source: D. FPGA Integration · p. 5
V. EVALUATION
The evaluation examines FPGA overhead, graph reconstruction reduction, end-to-end latency, and the threshold trade-off between prediction accuracy and reconstruction reduction.
V. EVALUATION
- The evaluation asks how much end-to-end latency reduction graph reuse provides.Source: §V · p. 5
- The evaluation measures the FPGA overhead introduced by SpecReuse.Source: §V · p. 5
- The evaluation measures how much graph reconstruction SpecReuse eliminates.Source: §V · p. 5
- The evaluation studies how the reuse threshold affects prediction accuracy and graph reconstruction reduction.Source: §V · p. 5
A. Experimental Methodology
The methodology compares baseline DIGC acceleration with the same accelerator augmented by SpecReuse, using ImageNet-1K validation and offline threshold calibration.
A. Experimental Methodology
- Experiments compare the baseline DIGC accelerator with the same accelerator augmented by SpecReuse, alongside CPU and GPU software reference platforms.Source: A. Experimental Methodology · p. 5
- Vision GNN models use ImageNet-1K validation with standard single-crop evaluation; accuracy uses batch size 128 and latency uses batch size one.Source: A. Experimental Methodology · p. 5
- For each backbone, the reuse threshold is calibrated offline to maximize graph reuse while maintaining desired prediction accuracy, then fixed throughout inference.Source: A. Experimental Methodology · p. 5
B. FPGA Implementation Overhead
SpecReuse adds modest FPGA resources while preserving the baseline accelerator’s 300 MHz operating frequency.
B. FPGA Implementation Overhead
- LUT, flip-flop, DSP, and BRAM utilization increase by only 1.07%, 0.56%, 0.12%, and 0.88%, respectively, relative to baseline.The added logic covers spectral descriptor generation, similarity evaluation, reuse control, and graph selection.Source: B. FPGA Implementation Overhead · p. 5
- Table III compares end-to-end inference latency across CPU, GPU, baseline FPGA, and SpecReuse-enhanced FPGA implementations.Its caption reports up to 2.69× speedup over the baseline FPGA implementation.Source: Table III · p. 6
- Table IV reports average energy per inference for GPU and FPGA implementations, with FPGA energy calculated as E = P × t.The calculation uses post-place-and-route Vivado power estimates and measured inference latency.Source: Table IV · p. 6
C. End-to-End Performance
SpecReuse improves end-to-end FPGA performance and energy use; offline threshold calibration balances graph reuse against classification accuracy across backbones.
C. End-to-End Performance
- 2.25×–2.69× speedup over the baseline FPGA accelerator accompanies approximately 58%–65% lower FPGA energy per inference.The speedup primarily results from eliminating redundant DIGC, while remaining network latency is largely unchanged.Source: C. End-to-End Performance · p. 6
D. Reuse Threshold Sensitivity
- Offline-calibrated thresholds enable 18.1%–30.1% graph reuse across evaluated ViG backbones while limiting Top-1 accuracy degradation to at most 0.54 percentage points.The selected thresholds are used for all reported performance and energy evaluations.Source: D. Reuse Threshold Sensitivity · p. 6
- Larger Pyramid ViG variants generally tolerate higher reuse rates than standard ViG models, indicating greater structural similarity between successive graph layers.Source: D. Reuse Threshold Sensitivity · p. 6
E. Evaluation Summary
Across ViG backbones, graph reuse reduces DIGC activity and improves latency and energy outcomes; calibration limits accuracy degradation.
E. Evaluation Summary
- Table V summarizes per-backbone offline threshold calibration, selecting operating points that maximize graph reuse while limiting accuracy degradation.Source: Table V · p. 6
- Reducing graph construction translates into lower end-to-end latency and energy per inference while preserving classification accuracy.The evaluation summary reports reduced DIGC activity across all evaluated ViG backbones.Source: E. Evaluation Summary · p. 6
VI. CONCLUSION
- Across seven ViG and Pyramid ViG backbones, SpecReuse achieves up to 2.69× end-to-end inference speedup and approximately 58–65% lower energy per inference.It adds 1.07% LUT, 0.56% FF, 0.12% DSP, and 0.88% BRAM overhead; calibrated reuse is 18.1–30.1% with at most 0.54 percentage points Top-1 accuracy degradation.Source: §VI · p. 6, Figure 2 · p. 6