PaperlayerView exact revision on arXiv

Source-linked AI summary

VQ-LIC: Shared Vector-Quantized Learned Image Compression on a Resource-Constrained FPGA

Muhammad Fahd Ibrahim Bhatti, Abdullah Bin Faisal, Ahsan Usman, Naveed Anwar Bhatti, Muhammad Ali Siddiqi

arXiv:2609.29727v1cs.AReess.IV

TL;DR

Resource-constrained FPGAs make learned image compression difficult because memory traffic, operation imbalance, and batching affect latency beyond arithmetic count. VQ-LIC shares an edge pointwise engine between analysis and vector quantization, with reconstruction in the cloud. On silicon, the codec reaches 47.98 frames/s at 42.84 mJ/frame, while its mid-rate preset outperforms a similarly sized neural encoder.

  • Problem

    Learned image compression is difficult to deploy on low-cost embedded devices because arithmetic, storage, activation traffic, and entropy-model costs can exceed their resources.

    Source: §I · p. 1

  • Method

    VQ-LIC uses an asymmetric edge–cloud codec whose compact analysis transform and multi-codebook VQ share the edge pointwise engine, with table-driven rANS coding.

    Source: §I · p. 1

  • Results

    47.98 frames/s at 42.84 mJ/frame on silicon, with the mid-rate preset outperforming a similarly sized neural encoder.

    Source: §VII · pp. 12–13

  • Takeaways & Limitations

    A shared pointwise datapath lets the latency model apply to VQ as well as analysis, and post-training codebook refinement reduces arithmetic and storage by 4×.

    Source: C. Compute–Communication Latency Model · p. 8, §VII · p. 12

Abstract

from arXiv · show

Learned image compression (LIC) is hard to deploy on severely resource-constrained FPGAs, since how fast it actually runs depends not just on arithmetic count, but also on memory traffic, imbalance between different operations, and how the hardware batches its work. We present VQ-LIC, an asymmetric edge-cloud codec in which a compact INT8 depthwise (DW)-pointwise (PW) analysis transform and multi-codebook vector quantization (VQ) run at the edge on a reusable DW/PW engine pair, while reconstruction is handled by a larger cloud decoder. Since VQ codeword matching is expressible as a dot product, it is mapped directly onto the same PW engine, removing the need for a separate VQ compute array, to our knowledge a first for FPGA LIC. A novel latency model, derived from deterministic RTL cycle counts of an FPGA's read, DW, PW, and write costs, predicts an analysis transform's per-block latency; since VQ shares the same PW datapath, the model applies to VQ as well. Validated directly against silicon, the model predicts deployed analysis and VQ latency within 0.26\% and 0.05\%, and guides the selection of a three-block $16$-$48$-$64$ transform. Post-training codebook reduction then cuts VQ arithmetic and codebook storage by $4\times$ and shrinks the fixed-width latent representation. On a 220-DSP Zynq-7020, VQ-LIC's mid-rate preset reaches 0.1398 bits per pixel at 28.69 dB PSNR and 13.06 dB MS-SSIM on CLIC~2017, outperforming a similarly sized neural encoder and reaching a rate-distortion range comparable to a codec three orders of magnitude larger. The complete 0.1945-kMAC/pixel analysis-VQ pipeline runs at 47.98 frames per second and 42.84 mJ per frame on silicon, using an order of magnitude fewer DSPs than comparable FPGA LIC accelerators while achieving lower bitrate, higher throughput, and lower energy per frame at a modest PSNR tradeoff.

I. INTRODUCTION

Learned image compression can exceed the arithmetic, storage, and traffic budgets of low-cost embedded devices, while FPGA latency depends on more than MAC count. VQ-LIC responds with a discrete-latent edge–cloud codec that reuses scarce hardware for encoding and quantization.

I. INTRODUCTION

  • FPGA latency depends on arithmetic throughput, operator-specific streaming, memory traffic, and output batching, so MAC count alone does not reliably predict encoding latency.The challenge is fitting the complete edge encoding pipeline within a tightly limited hardware budget.Source: §I · p. 1
  • VQ-LIC reuses the encoder’s multiplier hardware for codeword matching, while table-driven rANS coding avoids learned hyperprior or autoregressive inference on the edge.The asymmetric codec places compression at the edge and reconstruction in a larger cloud model.Source: §I · p. 1

A. Discrete-Latent and Efficient Learned Image Compression

  • Neural LIC typically uses scalar-quantized latents and learned probability models, [2], while later work explores codebooks, multi-codebook quantization, and hierarchical representations [12]–[17].Efficiency work also explores reduced precision,, slimmable models [20], distillation [21], and asymmetric encoders such as MCUCoder [22].Source: A. Discrete-Latent and Efficient… · p. 2

B. FPGA Acceleration of LIC

  • FPGA LIC systems adapt dataflow, accelerators, or network structure –, but prior vector-quantizer designs use a separate compute array [24],.Depthwise-separable convolution reduces arithmetic, but MAC share alone does not determine how limited multipliers should be divided between depthwise and pointwise work [26]–[28].Source: B. FPGA Acceleration of LIC · p. 2

C. Positioning of VQ-LIC

VQ-LIC distinguishes itself from prior FPGA LIC co-designs by deriving a deterministic cycle-count model from realized accelerator behavior and reusing the pointwise engine for VQ. Its edge codec combines an INT8 depthwise–pointwise analysis transform with multi-codebook VQ and a rate-aware codeword assignment.

C. Positioning of VQ-LIC

  • VQ-LIC derives its hardware model by deterministic cycle counting and carries it from encoder architecture selection to VQ latency estimation through the shared pointwise engine.Prior FPGA LIC co-designs adapt network structure and dataflow to the target FPGA; VQ-LIC instead reuses the pointwise MAC array for codeword scoring, avoiding a separate quantizer-specific model.Source: C. Positioning of VQ-LIC · p. 3

III. VQ-LIC CODEC AND TRAINING METHODOLOGY

  • The edge codec combines an INT8 depthwise–pointwise analysis transform and multi-codebook VQ, table-driven rANS coding, and a larger cloud-side decoder.Fig. 1 depicts the end-to-end codec; DWSConv denotes depthwise-separable convolution.Source: §III · p. 3, Figure 1 · p. 3

A. Analysis Transform

  • For 720p input, the selected three-block analysis transform uses channels 16–48–64 and produces a 90 × 160 × 64 latent with 4,491 convolutional weights.Each block uses 3 × 3 depthwise then 1 × 1 pointwise convolution; intermediate activations are requantized to INT8, and the final latent is unsigned INT8.Source: A. Analysis Transform · p. 3

B. Hardware-Aware Product-Codebook Vector Quantization

  • Multi-codebook VQ splits each 64-dimensional latent vector into four 16-dimensional subvectors and scores each against codewords in its own sub-codebook.The assignment uses entropy-constrained VQ, biasing choices toward codewords cheaper to entropy-code; the rate pressure is controlled by βrate.Source: B. Hardware-Aware Product-Codebook… · p. 3, Figure 2 · p. 3
  • The reference configuration uses four 256-entry codebooks, setting a 0.5 bpp uncoded ceiling; fewer sub-codebooks sharply increase search cost, while more approach scalar quantization.With M = 2, K = 65,536 requires 256× more multiplies per latent position; the stated target range is 0.1–0.5 bpp.Source: B. Hardware-Aware Product-Codebook… · p. 3
  • Post-topology refinement retains the K highest-prior codewords from each original 256-entry pool, selecting four 64-entry codebooks without using evaluation-set assignments.The reduced codec is evaluated for reconstruction quality, entropy-coded rate, codebook storage, and modeled shared-engine VQ latency; its uncoded representation is four 6-bit indices per position, or 0.375 bpp before entropy coding.Source: B. Hardware-Aware Product-Codebook… · pp. 3–4

C. Cloud Decoder and Training Objective

The cloud decoder reconstructs images from quantized latents outside the Zynq-7020 hardware boundary, while joint floating-point training combines reconstruction and latent-commitment terms. Three independently trained rate presets use different rate weights.

C. Cloud Decoder and Training Objective

  • The approximately 6.51-million-parameter cloud decoder lies outside the Zynq-7020 boundary and reconstructs images from quantized latents using six Transformer layers and a convolutional synthesis head.Attention uses shifted 14 × 14-token windows, but the complete latent grid is decoded in one forward pass and the image is not tiled.Source: C. Cloud Decoder and Training Objective · p. 4
  • Joint floating-point training optimizes the encoder, VQ codebooks, and decoder with SSIM, ℓ1 reconstruction error, and latent commitment; codebook vectors are updated through EMA.The reconstruction weight is α = 0.84, the commitment weight is β = 0.25, and stop-gradient is denoted sg[·].Source: C. Cloud Decoder and Training Objective · p. 4
  • The independently trained low-, mid-, and high-rate presets use βrate values 1.0, 0.4, and 0.3, respectively.Encoder exploration instead uses the common (4, 256, 16) reference quantizer with βrate = 0; codebook refinement follows topology selection.Source: C. Cloud Decoder and Training Objective · p. 4

D. Floating-Point and Quantization-Aware Training

VQ-LIC uses staged algorithm–architecture co-design to configure its resource-constrained FPGA encoder, while training the encoder, codebooks, and cloud decoder jointly before encoder quantization-aware fine-tuning.

D. Floating-Point and Quantization-Aware Training

  • The encoder, VQ codebooks, and cloud decoder are jointly trained in FP32 for two epochs, then the encoder is fine-tuned with QAT for two more.The codebook is frozen during QAT, while the decoder remains in FP32.Source: D. Floating-Point and… · p. 4

A. Design Constraints and Workload

  • On this platform, minimizing MAC count alone may not minimize latency, which also depends on tensor geometry, operator type, memory traffic, parallelism, and batching.The target is a Zynq-7020 with 220 DSP48E1 slices and a 64-bit DDR-to-PL path.Source: §IV · p. 5, A. Design Constraints and Workload · p. 5

IV. VQ-LIC COMPUTE–COMMUNICATION CO-DESIGN AND FPGA ARCHITECTURE

  • The staged co-design first models and validates accelerator latency, then uses the model to explore encoder topology and refine codebook cardinality.The same model is reused because VQ codeword scoring shares the pointwise datapath.Source: §IV · p. 5, Figure 3 · p. 5

A. Design Constraints and Workload

  • Candidate encoders vary in block count and channel widths; three stride-2 blocks are required, and the reference topology has six blocks.Explored intermediate widths are 16, 32, 48, or 64 channels.Source: A. Design Constraints and Workload · p. 5
  • For a fixed accelerator, each encoder's block geometry determines its read, depthwise, pointwise, and write latency demands.The deployed interface fixes spatial stream width L=8; synthesis, in addition to the latency model, establishes full resource feasibility.Source: A. Design Constraints and Workload · p. 5

B. Accelerator Organization

The accelerator reuses one depthwise–pointwise engine pair across encoder blocks and for VQ. Spatially parallel depthwise filtering feeds a pointwise engine that batches output channels.

B. Accelerator Organization

  • A common depthwise–pointwise engine pair is reused sequentially across encoder blocks, and its pointwise engine also serves VQ.The pointwise parallelism Q governs both convolution and VQ codeword-scoring latency.Source: B. Accelerator Organization · p. 5

1) Depthwise Engine:

  • The depthwise engine filters L adjacent pixels within one channel per cycle, using line buffers to assemble 3×3 neighborhoods.It generates zero-point padding on chip, and its post-processing packs outputs into the 64-bit format expected by the pointwise engine.Source: 1) Depthwise Engine · pp. 5–6, Figure 4 · p. 6

2) Pointwise Engine:

  • The pointwise engine holds output accumulators stationary while input activations stream through Q output-channel banks.Ping-pong buffering overlaps input staging with computation; two packed exact signed products are recovered per DSP48E1, so L=8 and Q=32 evaluate 256 products per cycle using 128 DSP slices.Source: 2) Pointwise Engine · p. 6

3) Shared Pointwise VQ Mode:

VQ-LIC maps codeword matching onto the existing pointwise engine and models its latency with the same scheduler used for convolution. It also fuses depthwise-to-pointwise dataflow within blocks, while retaining memory transfers between blocks and for final-latent VQ.

3) Shared Pointwise VQ Mode:

  • The selected (M, K, Dm)=(4, 64, 16) quantizer uses the pointwise engine for codeword scoring, with each 64-entry sub-codebook evaluated in two passes at Q=32.Across four sub-codebooks this gives eight Q-wide batches, with no zero-padding slots wasted.Source: 3) Shared Pointwise VQ Mode · p. 6
  • The existing MAC array computes each codeword's dot-product term, while precomputed codeword-dependent terms are added to form assignment scores.Only bias storage, score/argmin logic, and index-packing control are added around the reused datapath.Source: 3) Shared Pointwise VQ Mode · p. 6

4) Integer Post-Processing:

  • The pointwise post-processing unit takes Q cycles to retire a batch, so its delay is included in the pointwise scheduling model.The depthwise post-processing unit keeps pace with its engine and adds no steady-state delay.Source: 4) Integer Post-Processing · p. 6

5) Depthwise–Pointwise Fusion and External Memory:

  • Depthwise outputs stream directly into pointwise processing within each block, avoiding an external-memory write/read round trip; outputs between blocks are still written to memory.After the final block, the latent is written and read back for VQ, so convolution and VQ compete for the same input resource and cannot run simultaneously.Source: 5) Depthwise–Pointwise Fusion and… · pp. 6–7
  • The latency oracle models block time as the maximum of read, depthwise, pointwise, and write demands, and reuses the pointwise group scheduler for VQ.Its cycle-count constants are deterministic RTL scheduler counts rather than fitted parameters.Source: 5) Depthwise–Pointwise Fusion and… · p. 7

C. Compute–Communication Latency Model

The model derives block latency from deterministic RTL cycle counts, treating read, depthwise, pointwise, and write costs as competing bottlenecks. It reuses the pointwise schedule for VQ and distinguishes serial frame occupancy from a potentially lower initiation interval.

C. Compute–Communication Latency Model

  • Pointwise group latency reflects overlapping accumulation and retirement for multi-batch outputs, with the slower process setting the total rather than their sum.The RTL uses six-cycle accumulation, two-cycle PPU readout for a full batch, and two output channels per cycle for shadow transfer.Source: C. Compute–Communication Latency Model · p. 7
  • A block’s latency is the maximum of its read, depthwise, pointwise, and write costs, so added arithmetic parallelism helps only while its cost remains the bottleneck.The costs are derived from the accelerator’s cycle-counted RTL behavior, not fitted to measured timing.Source: C. Compute–Communication Latency Model · p. 7
  • VQ timing reuses the pointwise schedule by mapping codeword dimensions to input channels and total codewords to output channels.When K is a multiple of Q, sub-codebook boundaries align with batches, avoiding wasted padded slots; latent reads, index writes, bias formation, and best-score selection add no separate time.Source: C. Compute–Communication Latency Model · p. 8
  • Reducing K lowers VQ arithmetic, codebook storage, and fixed-width index bits, but helps rate–distortion only while retained codewords represent the learned latent adequately.At fixed D = 64 and (M, Dm) = (4, 16), exhaustive scoring uses DK scalar multiplies per latent position and stores DK scalar codebook values; indices use M⌈log2 K⌉ bits.Source: C. Compute–Communication Latency Model · p. 8
  • Serial analysis-plus-VQ time measures single-frame occupancy, not sustained frame initiation interval; finer-grained interleaving could use pointwise-engine idle time during depthwise-bound blocks.The deployed scheduler instead runs coarse-grained analysis and VQ phases; interleaving would require a finer-grained scheduler.Source: C. Compute–Communication Latency Model · p. 8

D. Reference-Workload Hardware Provisioning

The reusable depthwise–pointwise accelerator is provisioned against a representative reference workload, balancing depthwise, pointwise, and communication costs. The design fixes L = 8 and selects Q = 32 within the Zynq-7020 resource limits.

D. Reference-Workload Hardware Provisioning

  • The reference workload sizes the engine to balance depthwise, pointwise, and communication costs without specializing it to one network.The same engine supports different channel dimensions and tensor shapes at runtime.Source: D. Reference-Workload Hardware… · p. 8
  • L is held at 8 because doubling it would require a 128-bit datapath redesign and, at Q = 32, 256 DSP slices—above the device’s 220-slice budget.That count excludes depthwise and post-processing resources.Source: D. Reference-Workload Hardware… · p. 8
  • With L fixed, the design searches Q ∈ {8, 16, 32} and selects Q⋆ = 32 subject to device resource limits.The packed multiplier-array formula is not extended to unsynthesized configurations because complete DSP usage depends on synthesis mapping.Source: D. Reference-Workload Hardware… · p. 8

E. Hardware Cost Oracles and Quantizer Selection

The section uses a platform-specific latency oracle to evaluate encoder and VQ candidates, then validates the model and shared-pointwise VQ implementation against RTL and silicon measurements.

E. Hardware Cost Oracles and Quantizer Selection

  • The fixed accelerator’s tensor geometry yields a platform-specific cost oracle for each encoder, requiring no candidate-specific training or hardware implementation.For VQ, the model predicts speed across K choices, while quality and entropy-coded rate are measured separately to select cardinality.Source: E. Hardware Cost Oracles and Quantizer… · pp. 8–9

V. EXPERIMENTAL METHODOLOGY

  • The timed edge path includes input packing, analysis, shared-pointwise VQ, and PS rANS coding, but excludes image acquisition and cloud decoding.Board timings use the ARM Global Timer after warm-up on a 100 MHz Zynq-7020 implementation.Source: A. Latency Model Validation · p. 9

A. Latency Model Validation

  • Pointwise and depthwise schedules are checked across 100 RTL configurations each and compared with Zynq-7020 measurements; VQ is checked against software on edge cases and full frames.Hardware ablations compare serialized versus split-shadow retirement and fused versus DDR-materialized depthwise–pointwise execution.Source: A. Latency Model Validation · p. 9

B. Encoder Exploration

The selected 16–48–64 encoder reduces weights, modeled latency, and activation traffic relative to the six-block reference. Results also show why tensor-aware latency modeling, rather than useful-MAC counts alone, is used to guide selection.

B. Encoder Exploration

  • Encoder candidates are ranked by modeled latency before training; topology and channel placement are compared at matched modeled-latency levels, then assessed for quality and latency.Candidates share the reference VQ setting, and rate is not a selection criterion because all share the same rate preset.Source: B. Encoder Exploration · p. 9

C. VQ Codebook Refinement

  • For each rate preset, K is refined over {256, 128, 96, 64, 32} with M = 4 and Dm = 16 fixed, retaining the most-used training codes and evaluating rate, quality, storage, and modeled VQ latency.The supplied passages describe the refinement procedure but not its selected cardinalities or results.Source: C. VQ Codebook Refinement · p. 9

D. Evaluation Metrics and Protocol

  • The finalized codec is evaluated on CLIC 2017 and Kodak using PSNR, MS-SSIM, LPIPS, BRISQUE, and coded rate in bpp.PSNR is reported for comparison with prior FPGA LIC work, though the passage says it correlates poorly with perceived visual quality.Source: D. Evaluation Metrics and Protocol · p. 9

VI. EXPERIMENTAL RESULTS AND CO-DESIGN ANALYSIS

  • The evaluation proceeds from model validation through encoder selection and codebook refinement to shared-engine system behavior and rate–distortion performance.The staged protocol separately examines whether predicted-latency ordering differs from useful-MAC ordering.Source: §VI · p. 9

A. Hardware-Model Validation and Encoder Selection

  • The selected 16–48–64 transform’s aggregate latency prediction differs from silicon by 0.264%, and the model identifies every block’s bottleneck correctly.The depthwise schedule was also checked over 100 RTL configurations spanning channel counts, widths, ragged widths, and both supported strides.Source: A. Hardware-Model Validation and… · p. 10
  • Increasing Q from 16 to 32 and adding split-shadow retirement reduce measured PL/DMA latency by 35.6%, while fusion cuts activation traffic from 26.266 to 18.432 MB/frame.The hardware provisioning selects L = 8, Q = 32 before encoder exploration.Source: A. Hardware-Model Validation and… · p. 10, Figure 6 · p. 10
  • A six-block candidate with 21.1% fewer useful MACs than 16–48–64 nevertheless has higher modeled latency, so MAC ordering does not preserve latency ordering.The comparison illustrates that latency depends on changing bottlenecks across tensor geometries.Source: A. Hardware-Model Validation and… · p. 10
  • 16–48–64 achieves the highest proxy PSNR and lowest modeled latency among evaluated candidates, reducing weights by 56.7%, modeled latency by 24.0%, and activation traffic by 10.0% versus the six-block reference.Weights fall from 10,363 to 4,491, and modeled traffic from 18.432 to 16.589 MB/frame.Source: A. Hardware-Model Validation and… · p. 10

B. Codebook Refinement and Shared-Engine Selection

Codebook refinement selects K=64 and retains Q=32 under the device’s pointwise DSP budget. The deployed pipeline overlaps host work across frames, while VQ-LIC’s rate–distortion results compare favorably with similarly sized and larger codecs.

B. Codebook Refinement and Shared-Engine Selection

  • K=64 cuts search arithmetic, codebook storage, and modeled VQ latency by roughly 4× while retaining most reconstruction quality; Q=32 remains selected because Q=64 exceeds the DSP budget.Reducing K further to 32 causes a substantially larger quality loss; Q=64 improves combined modeled analysis–VQ latency by under 1%.Source: B. Codebook Refinement and… · p. 10

C. Final FPGA Implementation and System Performance

  • The implementation uses all 220 DSP48E1 slices, leaving 36.1% of LUTs, 69.6% of registers, and 41.1% of BRAM available.The ideal compute-only initiation-interval floor is 16.416 ms versus 18.342 ms for coarse-grained analysis–VQ execution, leaving roughly 10.5% scheduling headroom.Source: C. Final FPGA Implementation and… · pp. 10–11
  • The deployed codebook reload adds 0.9599 ms, giving a measured serial analysis–reload–VQ latency of 19.3400 ms; Table I also confirms model accuracy for VQ and combined analysis–VQ.These timings distinguish serial accelerator work from the overlapped multi-frame schedule.Source: C. Final FPGA Implementation and… · p. 10
  • 20.8405 ms per frame yields 47.98 frames/s at 720p, while measured energy is 42.84 mJ/frame.Adjacent-frame overlap hides 99.9% of PS entropy-coding time, leaving under 0.004 ms exposed.Source: C. Final FPGA Implementation and… · p. 11, Figure 7 · p. 11

D. Rate–Distortion Performance

  • VQ-LIC outperforms similarly sized MCUCoder across all four metrics and narrows, but does not close, the much larger Ballé codec’s perceptual-metric gap.Against conventional codecs, it also outperforms JPEG, JPEG 2000, and WebP on MS-SSIM, LPIPS, and BRISQUE.Source: D. Rate–Distortion Performance · p. 11

E. Comparison Against a Commodity Edge Platform

VQ-LIC’s edge implementation compares favorably with a commercial platform and prior FPGA LIC accelerators, though the prior-work comparison is not apples-to-apples. The conclusion reports a compact pipeline with measured throughput and energy alongside its rate–distortion results.

E. Comparison Against a Commodity Edge Platform

  • The Zynq-7020 analysis stage is nearly 2× faster and sustains roughly 1.3× higher throughput than the Coral Edge TPU deployment, with under a third of its power and under a fifth of its energy per frame.VQ scoring uses the programmable pointwise datapath, avoiding the separate host-side quantization stage required by the commodity platform.Source: E. Comparison Against a Commodity Edge… · p. 11

F. Comparison with Prior FPGA Accelerators

  • VQ-LIC sustains 47.98 frames/s with 220 DSP48E1 slices, compared with 2992 DSPs for StreamLIC and 4146 for F-LIC; the counts are not cross-device efficiency ratios.At a modest PSNR tradeoff, it also reaches lower bitrate, higher throughput, and lower energy per frame than its closest competitors in reconstruction quality.Source: F. Comparison with Prior FPGA… · pp. 11–12, Table VI · p. 12

VII. CONCLUSION

  • The selected 16–48–64 transform reduces modeled latency by 24.0% versus the six-block reference, while codebook refinement reduces search arithmetic and storage by 4×.The shared pointwise datapath performs VQ codeword scoring without a dedicated VQ engine.Source: §VII · p. 12
  • The complete 0.1945-kMAC/pixel workload runs at 47.98 frames/s and 42.84 mJ/frame, using an order of magnitude fewer DSPs than comparable FPGA LIC accelerators.The conclusion reports lower bitrate, higher throughput, and lower energy per frame at a modest PSNR tradeoff.Source: §VII · p. 13
  • Measured analysis and VQ latencies match model predictions within 0.26% and 0.05%, respectively.The conclusion states that the full workload fits within the Zynq-7020’s 220 DSP48E1 slices.Source: §VII · p. 13
Loading 2609.29727v1…