Source-linked AI summary
Hardware-Aware Learned Representation Compression for Distributed In-Sensor Vision
Chengwei Zhou, Abu Masum, Xuming Chen, Mehran Moghadam, Sreetama Sarkar, Arnab Sanyal, Md Abdullah-Al Kaiser, M. Hassan Najafi, Sercan Aygun, Gourav Datta
TL;DR
High-resolution image transmission can impose substantial communication, throughput, and energy costs, while sensor-integrated logic constrains conventional deep-network partitioning. OASIS learns compact representations and exports them through structured quantization-and-Huffman coding or an HDC classification path. Across visual wake-word classification, hand tracking, and eye tracking, it reports approximately 2×–4.5× lower system energy while maintaining competitive task performance.
Problem
Moving high-resolution pixel streams before extracting task-relevant information can impose substantial communication, throughput, and energy costs, while prior in-/near-sensor systems often demonstrate only simple workloads or a few early neural-network operations.
Source: §I · p. 1Method
OASIS jointly learns compact sensor-side representations and exports them through structured quantization-and-Huffman coding or Sobol-based HDC hypervectors for associative-memory classification.
Source: §I · p. 2, C. HDC as an Alternative… · p. 5Results
Approximately 2×–4.5× lower system energy is reported across visual wake-word classification, hand tracking, and eye tracking while maintaining competitive downstream-task performance.
Source: §I · p. 2Takeaways & Limitations
OASIS provides a practical hardware-algorithm co-design path for communication-efficient in-sensor vision using FPGA characterization, circuit-simulated CIS models, and a 7-nm ASIC projection.
Source: §VII · p. 12A heterogeneously integrated CIS and custom near-sensor logic die were not fabricated; evaluation instead combines circuit simulation, FPGA implementation and board-level measurements, Vivado analysis, and a 7-nm ASIC projection.
Source: §VII · p. 12
Abstract
from arXiv · showhide
In-sensor computing reduces the cost of transmitting high-resolution image data by performing early-stage processing near the sensor. However, the logic chip integrated with a CMOS image sensor (CIS) is tightly constrained in compute and memory, limiting conventional deep neural network partitioning. We present OASIS, a distributed in-sensor vision framework that uses a lightweight encoder to generate compact, task-relevant representations before off-chip transmission. The encoder is trained end-to-end using task, entropy, and reconstruction objectives, while the decoder is used only during training. OASIS supports two complementary deployment paths. The first applies 4-bit quantization and Huffman coding while preserving the spatial structure required by classification and dense-prediction tasks. The second uses Sobol-based hyperdimensional computing (HDC) to transform the encoder latent into a fixed-dimensional binary hypervector for associative-memory classification. For the SwinViT-based VWW model, mapping a $3\times3\times8$ latent to a 64-dimensional hypervector provides an additional $1.77\times$ communication reduction with less than one percentage point of accuracy loss relative to the 128-dimensional configuration, yielding an overall $18{,}816\times$ reduction compared with raw 8-bit image transmission. We implement the digital near-sensor pipeline on an AMD Xilinx Zynq UltraScale+ FPGA and characterize it using direct board-level power measurements and Vivado post-implementation analysis, together with circuit-simulated CIS models and a 7-nm ASIC projection. Across visual wake-word classification, hand tracking, and eye tracking, OASIS reduces total system energy by approximately $2\times$-$4.5\times$ while maintaining competitive accuracy, demonstrating a practical hardware-algorithm co-design path for communication-efficient in-sensor vision.
I. INTRODUCTION
OASIS treats sensor-side processing as learned representation compression to reduce communication while respecting integrated-logic constraints. It offers structured quantization-and-Huffman and HDC export paths, with reported communication and system-energy reductions across vision tasks.
A. In- & Near-Sensor Computing
- Prior in-/near-sensor demonstrations often target simple or low-resolution datasets or a few early operations, leaving deeper multi-channel, multi-bit workloads difficult under sensor area, memory, and power limits.Conventional systems can still transmit large activation tensors and leave substantial end-to-end computation off-sensor.Source: §I · p. 1, A. In- & Near-Sensor Computing · p. 2
I. INTRODUCTION
- OASIS learns a compact, task-relevant latent on the sensor side rather than simply partitioning an existing deep network across chips.The encoder is trained with task, entropy, and reconstruction objectives; the reconstruction decoder is removed for inference.Source: §I · p. 2
- The structured export path retains spatial organization with low-bit quantization and Huffman coding, while the HDC path sends a fixed-dimensional hypervector for classification.The structured path supports classification and dense prediction; HDC replaces the conventional task-specific classifier with associative-memory inference.Source: §I · p. 2
- Approximately 2×–4.5× lower modeled system energy accompanies competitive performance across visual wake-word classification, hand tracking, and eye tracking.The structured path reduces transmitted bits by up to 11,985×; the SwinViT VWW HDC configuration achieves an 18,816× reduction versus raw 8-bit image transmission.Source: §I · p. 2
A. In- & Near-Sensor Computing
- Many analog or mixed-signal designs perform fixed early operations, and analytical per-MAC estimates can omit control, memory, pipeline, and interconnect costs.Some distributed sensor-logic approaches retain conventional backbones without aggressively compressing spatial and channel dimensions.Source: A. In- & Near-Sensor Computing · pp. 2–3
B. Hyperdimensional Computing Engine
OASIS combines compact on-chip encoders with training-only reconstruction supervision and alternative off-chip inference paths. Its HDC background motivates quasi-random hypervectors and lightweight vector operations for classification.
B. Hyperdimensional Computing Engine
- Sobol sequences are used to generate level hypervectors because they improve orthogonality and the reliability of encodings compared with correlated LFSR-generated vectors.HDC combines hypervectors with permutation, binding, and bundling operations.Source: B. Hyperdimensional Computing Engine · p. 3
A. Dual-Branch Autoencoder-based Network
- Compact ResNet- and SwinViT-based encoders aggressively reduce spatial dimensions and control channel growth to limit intermediate memory and transmitted latent size.The encoder and working feature buffers are designed to fit locally without external DRAM accesses during sensor-side processing.Source: A. Dual-Branch Autoencoder-based Network · p. 3
- The reconstruction decoder supplies training supervision but is removed from deployed inference; its expand–contract–expand structure reconstructs the input from the compressed latent.The reconstruction loss provides an additional self-supervised signal during training.Source: A. Dual-Branch Autoencoder-based Network · p. 3
- The default path transmits quantized, Huffman-coded activations for off-chip task inference, whereas HDC maps the latent to a fixed-dimensional hypervector for associative-memory classification.The HDC route eliminates a separate classifier network; the reconstruction branch is active only during training.Source: Figure 1 · p. 4, A. Dual-Branch Autoencoder-based Network · p. 4
B. Training Methodology
OASIS jointly trains reconstruction, entropy, and task objectives so compression preserves information useful to both the input representation and downstream application. Quantization-aware training exposes the model to deployment precision during optimization.
B. Training Methodology
- The joint objective combines reconstruction, entropy regularization, and downstream task losses, with β and γ weighting the entropy and task terms.The reconstruction term retains information about the sensor input, while the task loss preserves features needed by the application.Source: B. Training Methodology · p. 4
- The entropy term acts on quantized latents and encourages redistribution toward fewer frequently occurring symbols when expected coding entropy exceeds Href.The target is a representation better suited to entropy coding, without forcing every latent element to the same value.Source: B. Training Methodology · p. 4
- Quantization-aware training lets task and reconstruction branches observe finite-precision activations during optimization, using a straight-through estimator for the non-differentiable rounding operation.The method uses symmetric uniform quantization and updates the range with a momentum-based procedure.Source: B. Training Methodology · p. 4
- The de-quantized representation uses a scale derived from the maintained encoder-output range and selected bit precision, exposing the encoder to deployment constraints while retaining a differentiable optimization path.The stated scale is qscale = 2 max / (2^n − 1).Source: B. Training Methodology · p. 4
C. HDC as an Alternative Classification Path
The HDC alternative converts an encoder latent into a fixed-dimensional hypervector for classification rather than transmitting a spatial feature map. Sobol-based encoding and associative-memory similarity search define this classification path.
C. HDC as an Alternative Classification Path
- HDC replaces element-wise quantization and Huffman coding with a fixed-dimensional hypervector, so transmitted volume depends on D rather than latent dimensions.This route is limited to classification because hand and eye tracking require spatially structured outputs.Source: C. HDC as an Alternative… · p. 5
- Sobol-sequence level hypervectors improve orthogonality and encoding reliability; binding and bundling combine per-feature vectors into a sample hypervector.Binding uses bipolar multiplication or binary XOR, while bundling accumulates vectors and applies a sign operation.Source: C. HDC as an Alternative… · p. 5
- Training accumulates sample hypervectors into ground-truth class prototypes, and inference predicts the class whose stored prototype scores highest against the query vector.The approach uses single-pass training without gradient backpropagation.Source: C. HDC as an Alternative… · p. 5
- HDC classification uses permute, bind, bundle, and compare operations over D-dimensional vectors instead of a conventional classifier head's multi-bit MAC arithmetic.The distributed representation is described as tolerant to bit-flip errors and hardware noise.Source: C. HDC as an Alternative… · p. 5
D. Huffman Coding
OASIS uses Huffman coding to shorten the representation of frequent quantized symbols while preserving the quantized sequence exactly. Its sensor-side pipeline moves digitized pixels to a logic layer for encoding, with separate implementation strategies for the encoder families and export paths.
D. Huffman Coding
- Huffman coding assigns shorter codewords to frequent quantized values and longer ones to rare values, losslessly recovering the quantized sequence without additional accuracy degradation.The VWW and STB output distributions concentrate around a small subset of 4-bit values.Source: D. Huffman Coding · p. 5, Figure 3 · p. 6
- The HDC path skips Huffman coding because binary hypervectors offer no coding benefit and Sobol-based values are designed to have near-uniform bit distributions.The 64 hypervector bits are transmitted directly.Source: D. Huffman Coding · p. 6
IV. SYSTEM ARCHITECTURE
- Digitized pixel values move through the local die-to-die interface to the logic layer, where the encoder and selected post-encoder path run before compressed data reaches the external processor.Column-parallel readout digitizes pixel voltages before the sensor-side compute layer.Source: §IV · p. 6, A. Active Pixel Sensor · p. 6
B. Efficient In-Sensor Processor
- The compact encoders keep their parameters in on-chip BRAM, while SwinViT processes one local attention window at a time and ResNet reuses scratch buffers across residual blocks.These strategies limit intermediate storage requirements.Source: B. Efficient In-Sensor Processor · p. 6
- The encoder and post-encoder stages use pipelining and parallel memory access, including II = 1 stages and a fully partitioned Huffman lookup table.The Huffman stage packs variable-length codes into 32-bit output words; HDC updates all hypervector dimensions concurrently.Source: B. Efficient In-Sensor Processor · p. 6
V. HARDWARE-VALIDATED SYSTEM ENERGY MODELING
The system energy evaluation combines circuit simulation of the analog front end with FPGA measurements of digital computation. It accounts for the major per-frame energy components and notes that the baseline's sensor-interface energy dominates its total.
V. HARDWARE-VALIDATED SYSTEM ENERGY MODELING
- System-level evaluation combines Monte-Carlo circuit simulation of the analog front end with FPGA hardware measurements of the digital compute pipeline.The simulations sweep mismatch and process, voltage, and temperature variation across randomized trials.Source: §V · p. 7
- The energy model includes sensor readout, data conversion, die-to-die communication, external transmission, encoder execution, and downstream task processing.This scope goes beyond arithmetic energy alone.Source: §V · p. 7
A. Energy Model Formulation
- Total frame energy is expressed as the sum of APS, TSV, sensor-interface, encoder, and task-specific back-end energy contributions.The baseline without in-sensor processing omits TSV energy but incurs higher sensor-interface energy, which dominates its total.Source: A. Energy Model Formulation · p. 7
B. APS Energy Consumption
The energy accounting covers APS operation, communication interfaces, and FPGA-based encoder characterization. It distinguishes low-energy TSV transfer from external-interface transfer and uses measured FPGA energy alongside an ASIC projection.
B. APS Energy Consumption
- For 8-bit pixels, readout and ADC conversion consume approximately 63.6 pJ per pixel in the in-house 22 nm GlobalFoundries FDSOI circuit simulations.The APS energy is estimated across its different operating modes.Source: B. APS Energy Consumption · p. 7
C. Communication Energy
- TSV transfer is estimated at 6.25 pJ per byte, while the external MIPI interface is estimated at 100 pJ per byte.The system reduces external transmitted data, whose interface energy is identified as dominating total energy.Source: C. Communication Energy · p. 7
D. FPGA-Based Encoder Energy Characterization
- FPGA encoder energy is measured from active-minus-idle programmable-logic power multiplied by frame latency, and the 7 nm ASIC projection uses an approximately 20× FPGA-to-ASIC energy ratio.The FPGA pipeline measurement includes control, memory access, pipeline stalls, and interconnect overhead; the projection accounts for routing, node scaling, and memory differences.Source: D. FPGA-Based Encoder Energy… · p. 7
E. Huffman Coding/Decoding Energy
For hand and eye tracking, the reported results quantify task performance under compressed outputs. Huffman coding energy is evaluated separately from neural-network computation and compared with equivalent data transfer and off-chip memory access.
E. Huffman Coding/Decoding Energy
- Huffman encoding and decoding cost 0.96 and 1.15 pJ/byte, respectively.Encoding is primarily symbol-to-codeword lookup and bitstream packing; decoding also recovers symbols from the variable-length representation.Source: E. Huffman Coding/Decoding Energy · pp. 7–8
- Combined entropy-coding energy remains much smaller than transferring equivalent data through the external interface or repeatedly accessing off-chip memory.Source: E. Huffman Coding/Decoding Energy · p. 8
A. Task Performance
- For VWW, OASIS outperformed encoder-only baselines across compression settings for both ResNet and SwinViT while maintaining aggressive compression.Source: A. Task Performance · p. 8
- Hand tracking reached 2D AUC 0.880 versus baseline 0.746 at d=16,s=12, and 3D AUC 0.969 at d=16,s=3 versus baseline 0.958.Source: A. Task Performance · p. 9
- Eye tracking's 4-bit output delivered 192× reduction versus 8-bit image inputs and 24× versus baseline EyeNet output, with 1.4% mIoU degradation.Source: A. Task Performance · p. 9
- ROI prediction raised overall reduction to 364×, but limited annotated frames constrained evaluation and led to mIoU degradation.Source: A. Task Performance · p. 9
B. Efficiency
OASIS sharply reduces off-chip communication and system energy, while its FPGA post-encoder stages add little latency and energy relative to the encoders. The HDC path offers a more compressed, lower-energy backend, though the P2M comparison is not like-for-like.
B. Efficiency
- Approximately 2× lower vision-pipeline energy across most tasks and 4.5× for SwinViT-based VWW are reported versus baseline non-in-sensor systems.The authors attribute the VWW result to a tiny SwinViT encoder that lowers encoder energy by 13.4×; their ResNet encoder lowers it by 1.15×.Source: B. Efficiency · p. 9
- HDC post-encoder energy is 0.729 µJ versus 1.805 µJ for Quant+Huff with ResNet, and 0.520 µJ versus 1.775 µJ with SwinViT.Both post-encoder backends consume less than 0.01% of corresponding encoder energy in the evaluated configurations; the comparison to P2M is explicitly not like-for-like.Source: B. Efficiency · p. 10
- HDC uses 3,258 LUTs and 4,103 FFs versus 5,031 LUTs and 4,960 FFs for Quant+Huff, but its measured dynamic power is 29 mW versus 25 mW.Both use two BRAM18K blocks and no DSP resources; HDC’s shorter execution time results in lower total energy per inference.Source: B. Efficiency · p. 10
- OASIS transfers raw sensor output over a TSV interface with approximately 200× MIPI’s bandwidth, then sends only the compact learned representation over MIPI.The authors report that the P2M compression comparison is not controlled because the systems differ in computation type, placement, and technology context.Source: B. Efficiency · p. 10
- Up to 11,985× and 18,816× less transmitted bit volume are reported for Quant+Huff and HDC, respectively, relative to raw 8-bit image transmission.For the stated configuration, HDC provides an additional 1.57× communication reduction over Quant+Huff.Source: B. Efficiency · pp. 9–10
- Post-encoder latency is negligible relative to encoder execution; HDC takes 0.03242 ms for ResNet and 0.03156 ms for SwinViT, versus 0.06944 ms and 0.06827 ms for Quant+Huff.The reported HDC latency reductions are approximately 53.3% for ResNet and 53.8% for SwinViT.Source: B. Efficiency · p. 10
C. HDC Accuracy–Communication Trade-off
HDC’s accuracy and communication trade-off depends on the encoder representation: retrained ResNet HDC improves on its Quant+Huff accuracy while reducing transmitted size, whereas SwinViT HDC has a modest accuracy decrease. HDC also uses lightweight binary operations and offers stated tolerance to bit-flip errors and hardware noise.
C. HDC Accuracy–Communication Trade-off
- With prototype retraining, ResNet HDC reaches 90.81% accuracy at D = 64, exceeding the corresponding 87.6% Quant+Huff accuracy while reducing transmitted representation size.Without retraining, SwinViT reaches 86.06% at D = 128 and ResNet reaches 89.78% at D = 64; retraining raises these to 86.89% and 90.81%, respectively.Source: C. HDC Accuracy–Communication Trade-off · p. 11
- HDC’s accuracy effect is encoder-dependent: it modestly decreases SwinViT accuracy relative to Quant+Huff but can improve ResNet accuracy while reducing communication cost.The authors attribute the ResNet result to stronger class separability after normalization and projection into hypervector space.Source: C. HDC Accuracy–Communication Trade-off · p. 11
- HDC prototype construction is gradient-free, and binary permute/bind/bundle/compare operations replace multi-bit MAC arithmetic for encoding and associative-memory inference.The distributed hypervector representation is described as intrinsically tolerant to bit-flip errors and hardware noise; Sobol-based level generation improves orthogonality relative to conventional pseudo-random generation.Source: C. HDC Accuracy–Communication Trade-off · p. 11
D. Ablation Study
The ablations show how entropy regularization, quantization, Huffman coding, and the HDC alternative affect communication and accuracy. The reported final performance reflects the synergy of the components rather than any single design choice.
D. Ablation Study
- Huffman coding increases bandwidth reduction from 4,181× to 10,653× with no additional accuracy cost.Source: D. Ablation Study · p. 11
- Reducing output precision from 8 bits to 4 bits changes accuracy from 87.9% to 87.6% and post-Huffman effective bit-width from 3.77 to 1.57 bits.Source: D. Ablation Study · p. 11
- At 2-bit precision, accuracy falls to 70.3% while effective bit-width decreases only to 1.46 bits.Source: D. Ablation Study · p. 11
- Replacing quantization and Huffman coding with HDC raises bandwidth reduction from 10,653× to 18,816× as accuracy changes from 87.6% to 86.0%.Source: D. Ablation Study · p. 11
- Entropy regularization changes classification accuracy from 88.0% to 87.9% while preserving the same communication bandwidth.Source: D. Ablation Study · p. 11
- The final performance is attributed to the synergy of all components rather than any single design choice.Source: D. Ablation Study · p. 12
VII. CONCLUSIONS & DISCUSSIONS
OASIS compresses representations before transmission through quantization with Huffman coding or an HDC path. Across three vision tasks, it reports lower modeled system energy while maintaining competitive accuracy, with evaluation combining measurements, simulation, and an ASIC projection.
VII. CONCLUSIONS & DISCUSSIONS
- Across visual wake-word classification, hand tracking, and eye tracking, OASIS reduces modeled total system energy by approximately 2×–4.5× while maintaining competitive accuracy.Source: §VII · p. 12
- For SwinViT-based VWW, a 64-dimensional hypervector gives an additional 1.77× communication reduction and 18,816× overall reduction versus raw 8-bit image transmission; the structured-feature path achieves up to 11,985× reduction.Source: §VII · p. 12
- The study combines circuit simulation of the CIS front end, FPGA board-level power measurements, Vivado analysis, and a 7-nm ASIC projection because fabricating the integrated CIS and logic die was beyond its practical scope.Future work will explore adaptive compression, emerging-memory integration, higher-resolution applications, and multimodal fusion.Source: §VII · p. 12
VIII. BIOGRAPHY SECTION
The biography section records the authors’ education, appointments, honors, and research interests across computing, engineering, and machine learning.
VIII. BIOGRAPHY SECTION
- Zhou, Datta, Mohammad Masum, and Chen are affiliated with Case Western Reserve University or the University of Louisiana at Lafayette and work across edge AI, in-sensor computing, and emerging computing.Source: §VIII · pp. 13, 15
- Moghadam, Sanyal, and Kaiser have backgrounds in computing and engineering, with research interests spanning hyperdimensional computing, edge AI, and in-sensor or mixed-signal systems.Source: §VIII · p. 14
- Sarkar, Najafi, and Aygun are described through academic appointments, research interests, and honors in areas including multimodal models, approximate computing, and hyperdimensional computing.Source: §VIII · p. 14