PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconfiguring the Imaging Pipeline for Computer Vision

Mark Buckler, Suren Jayasuriya, Adrian Sampson

arXiv:1705.04352v3cs.CV

TL;DR

Photography-oriented imaging pipelines spend energy producing image quality that computer vision does not require. The paper evaluates configurable pipelines and proposes sensor changes for a low-power vision mode; across the tested applications, this mode can substantially reduce imaging energy with little effect on task performance.

  • Problem

    Photography-oriented pipelines produce high-quality images for human consumption, although computer vision algorithms do not require the same image quality, motivating study of their imaging-pipeline costs.

    Source: §1 · p. 1

  • Method

    The authors study algorithm sensitivity to sensor parameters and ISP stages with a reversible pipeline simulation tool, then propose sensor modifications for a configurable vision mode.

    Source: §1 · p. 1

  • Results

    Only demosaicing and gamma compression are important for all but one tested application, and vision mode uses roughly a quarter of photography mode’s imaging-pipeline energy without significantly affecting most studied algorithms’ performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Sensor-circuit design choices can obviate the need for an ISP when supplying a computer vision algorithm.

    Source: §8 · p. 8

    The empirical conclusions are limited to the specific algorithms and datasets studied.

    Source: §8 · p. 8

Abstract

from arXiv · show

Advancements in deep learning have ignited an explosion of research on efficient hardware for embedded computer vision. Hardware vision acceleration, however, does not address the cost of capturing and processing the image data that feeds these algorithms. We examine the role of the image signal processing (ISP) pipeline in computer vision to identify opportunities to reduce computation and save energy. The key insight is that imaging pipelines should be designed to be configurable: to switch between a traditional photography mode and a low-power vision mode that produces lower-quality image data suitable only for computer vision. We use eight computer vision algorithms and a reversible pipeline simulation tool to study the imaging system's impact on vision performance. For both CNN-based and classical vision algorithms, we observe that only two ISP stages, demosaicing and gamma compression, are critical for task performance. We propose a new image sensor design that can compensate for skipping these stages. The sensor design features an adjustable resolution and tunable analog-to-digital converters (ADCs). Our proposed imaging system's vision mode disables the ISP entirely and configures the sensor to produce subsampled, lower-precision image data. This vision mode can save ~75% of the average energy of a baseline photography mode while having only a small impact on vision task accuracy.

1. Introduction

Photography-oriented imaging pipelines spend energy producing image quality that computer vision does not require. The paper studies configurable vision mode and a sensor design intended to reduce capture and processing costs while preserving task performance.

1. Introduction

  • The paper addresses imaging costs that remain beyond inference hardware: capture and processing can consume a larger share of system power as inference gets cheaper.Existing hardware work cited here targets inference, while embedded vision spans the pipeline from photons to task result.Source: §1 · p. 1
  • The proposed sensor uses variable-accuracy ADCs and power-gated selective readout, with subsampling and quantization approximating demosaicing and gamma compression.The design aims to disable the ISP in vision mode.Source: §1 · p. 1
  • The authors evaluate eight applications and use a reversible pipeline simulation tool to study how imaging choices affect vision performance.The applications include classical stereo, optical flow, and structure-from-motion algorithms, as well as CNNs for recognition and detection.Source: §1 · pp. 1–2
  • Vision mode uses roughly a quarter of traditional photography mode’s imaging-pipeline energy without significantly affecting most studied algorithms’ performance.The study also finds that demosaicing and gamma compression are the only important ISP stages for all but one tested application.Source: §1 · p. 2

2. Related Work

Related work targets efficient inference, flexible ISP processing, and vision-oriented sensors, but the paper focuses on the energy cost of the imaging pipeline and its effects across vision algorithms.

2. Related Work

  • Dedicated deep-learning ASICs reduce inference cost, while this work targets the rest of the system by pairing low-power vision implementations with low-power sensing circuitry.The passage notes that image capture and processing can become a larger share of system power as inference becomes more efficient.Source: §2 · p. 2
  • Prior work has explored programmable or selective ISPs and specialized vision sensors; this paper measures sensitivity to traditional ISP choices and proposes sensor changes that subsume critical stages.The authors describe their sensitivity measurement as the first for vision applications across design decisions in a traditional ISP pipeline.Source: §2 · p. 2
  • Unlike compressive sensing approaches that reconstruct images, the proposed pipeline lets vision algorithms work directly on sensor data without additional image reconstruction.The paper also distinguishes its scope from CNN-focused sensor-noise and blurring studies by evaluating both CNN-based and classical algorithms.Source: §2 · p. 2

3. Background & Experimental Setup

The setup models a sensor, an ISP, and a vision application, then uses CRIP to simulate configurable pipelines and evaluates CNN and classical vision benchmarks.

3.1. The Imaging Pipeline

  • The modeled ISP includes denoising, demosaicing, color transformations, gamut mapping, gamma compression, and image compression, while omitting sensor-control and specialized stages such as HDR.The authors describe this as an idealized pipeline because modern ISPs can contain hundreds of proprietary stages.Source: §3.1 · pp. 2–3

3.2. Pipeline Simulation Tool

  • CRIP converts standard images to approximate RAW signals, simulates a specified sensor/ISP configuration, and produces a final RGB image; its reconstructed RAW images had 1.064% average pixel error and 28.81 dB PSNR.CRIP uses reversible ISP stages from Kim et al., whose model linearization error is less than 1%; results are reported only for quantization levels of 8 bits or fewer because ordinary 8-bit images lose sensor detail.Source: §3.2 · p. 3

3.3. Benchmarks

  • The benchmark suite contains five CNN-based and three classical implementations spanning classification, detection, face identification, optical flow, and structure from motion.For CNNs, the authors convert training data and retrain from pretrained weights using the original papers’ learning rates and hyperparameters.Source: §3.3 · pp. 3–4, Table 1 · p. 4

4. Sensitivity to ISP Stages

Across the tested applications, demosaicing and gamma compression are the key ISP stages; denoising also matters for the noise-sensitive Stereo SGBM benchmark.

4. Sensitivity to ISP Stages

  • Without gamut mapping, ResNet44 classification error rises from 6.3% to 6.6%; without color transformation, OpenMVG RMSE rises from 0.408 to 0.445.The authors characterize gamut mapping and color transformations as having minimal effects across benchmarks.Source: §4 · p. 4
  • Disabling gamma compression makes OpenMVG fail to find sufficient features, while removing demosaicing worsens Farneback optical-flow error from 0.227 to 0.448.The authors attribute sensitivity partly to hand-tuned feature extractors that do not account for the Bayer pattern, and to the effect of removing color channels from low-resolution CIFAR-10 data.Source: §4 · p. 4
  • Adding sensor noise increases Stereo SGBM mean error from 0.245 to 0.425, while OpenFace’s apparent improvements when skipping gamut mapping or gamma compression are attributed to training randomness.Across 10 runs, OpenFace’s baseline error varied from 8.22% to 10.35%, with a standard deviation of 0.57%.Source: §4 · p. 4
  • With only demosaicing and gamma compression, ResNet44 top-1 error rises from 6.3% to 7.2%; for SGBM, denoising lowers mean error from 0.33 to its 0.25 baseline.The authors report that CNNs can retrain to adapt to capture-pipeline changes, whereas classical benchmarks are less flexible and can depend on specific ISP stages.Source: §4 · p. 4
  • Demosaicing and gamma compression are the only important stages for all applications except one, which also benefits from denoising; selective ISP use may remain worthwhile for outliers like SGBM.The minimal pipeline retaining demosaicing and gamma compression had similar accuracy to original data for most applications.Source: §4 · p. 4

5. Image Sensor Design

The section proposes adapting the sensor for a low-power vision mode rather than relying on a conventional fixed imaging pipeline. Selective readout, subsampling, and nonlinear ADC quantization substitute for ISP functions.

5. Image Sensor Design

  • Sensor-side subsampling and nonlinear quantization are proposed to replace demosaicing and gamma compression, allowing vision mode to replace the ISP's role.The design uses established sensor techniques in an optional camera mode for embedded vision applications.Source: §5 · pp. 5–6
  • Column-level power gating selectively reads a region of interest or pixel subset to adjust resolution and save energy.The controller can power-gate each column's amplifier and ADC.Source: §5 · p. 5
  • Each 2×2 Bayer block is subsampled into one RGB pixel by dropping one green sample and combining the remaining values.Source: §5 · p. 6
  • The sensor's ADC can use logarithmic quantization instead of linear quantization, serving the data-compression role of gamma compression with fewer bits.The proposed design switches between quantization schemes using a separate capacitor bank for each.Source: §5 · p. 6
  • Vision mode sends uncompressed data directly to the application and assumes real-time processing, while photography mode uses high-precision linear ADCs, ISP processing, and JPEG storage.Source: §5 · p. 6

6. Sensitivity to Sensor Parameters

The sensor-parameter experiments find that logarithmic quantization performs better than linear quantization at matched bitwidth and tolerates reduced bitwidth. Downsampling and replacing ISP demosaicing with sensor subsampling produce small error increases in the tested settings.

6. Sensitivity to Sensor Parameters

  • Replacing ISP demosaicing with sensor subsampling causes a small error increase; Farneback optical flow error rises from 0.332 to 0.375.The comparison excludes CIFAR-10 benchmarks because their 32×32 resolution makes further subsampling unmeaningful.Source: §6 · pp. 6–7
  • Logarithmic ADC quantization yields higher accuracy than linear quantization at the same bitwidth across all benchmarks.For Farneback optical flow, mean error is 0.54 and 0.65 with linear ADC at 8 and 2 bits, versus 0.33 and 0.38 with logarithmic ADC.Source: §6 · p. 7
  • All applications show minimal error increases down to 5-bit logarithmic sampling, motivating a proposed 5-bit vision-mode ADC.OpenMVG average RMSE rises from 0.454 to 0.474, and ResNet20 top-1 error rises from 8.2% to 8.42%.Source: §6 · p. 7
  • At 1/4 resolution, errors were 40.9% for LeNet, 27.71% for ResNet20, and 26.5% for ResNet44, with approximately 1% accuracy loss from full resolution.Source: §6 · p. 7

7. Quantifying Power Savings

The analysis estimates energy savings from lower-precision ADCs, reduced sensor resolution, and disabling the ISP in vision mode. It also profiles ISP stages and notes that its energy estimates omit implementation overheads.

7. Quantifying Power Savings

  • The ADC readout estimate weights each digital level’s energy cost by its probability, using image-value distributions and a SAR ADC capacitor simulation.The model uses CIFAR-10 raw image data for level probabilities and ignores fixed overheads such as control logic.Source: §7 · pp. 7–8
  • The simulated 5-bit logarithmic ADC uses 99.95% less energy than the baseline 12-bit linear ADC.Because ADCs account for 50% of sensor energy, the paper estimates this saves approximately half the sensor’s energy cost.Source: §7 · p. 8
  • Reducing sensor resolution cuts the bulk of sensor power linearly with pixel count because readout, I/O, and the pixel array make up roughly 95% of sensor power.Selective readout powers off parts of the sensor array.Source: §7 · p. 8
  • Software profiling found denoising required more than two orders of magnitude more dynamic instructions, while JPEG compression was also costly; vision mode can bypass the ISP.The profiling used software implementations, which do not directly reflect real ISP hardware costs; bypassing the ISP also avoids compression that is unnecessary when images need not be stored.Source: §7 · p. 8
  • Around 75% of a traditional imaging pipeline’s energy is saved in vision mode, with higher savings possible if resolution is reduced.The estimate combines a sensor using half as much energy as a traditional sensor with disabling the ISP.Source: §7 · p. 8
  • The first-order energy analysis excludes power-gating, muxing, and off-chip communication overheads.The authors plan to measure complete implementations in future work.Source: §7 · p. 8

8. Discussion

The paper argues that mobile imaging pipelines should add a low-power vision mode, and that sensor circuitry can make a separate ISP unnecessary for vision tasks. The background describes the sensor and the typical ISP stages that produce images for human viewing.

8. Discussion

  • A vision mode for mobile imaging pipelines can use sensor circuitry to obviate the ISP when supplying computer vision algorithms.The discussion also notes that the paper's empirical conclusions are limited to specific algorithms and datasets; its first-order energy evaluation does not quantify overheads.Source: §8 · p. 8

A. Background: The Imaging Pipeline

  • A camera system combines a light-responsive sensor that produces RAW data, an ISP that transforms it into an image, and a vision application.During capture, pixel voltages are amplified and quantized by an ADC before the RAW signal is sent to the ISP.Source: A. Background: The Imaging Pipeline · p. 11, §A.1 · p. 11

A.1. Camera Sensor

  • CMOS photodiodes capture monochrome intensity; a color-filter mosaic assigns each pixel a red, green, or blue band, with Bayer using a 2×2 pattern.The Bayer pattern has two green pixels, one red pixel, and one blue pixel.Source: §A.1 · p. 11

A.2. Image Signal Processor

  • A typical ISP denoises and demosaics RAW data, applies per-pixel color and tone operations, and compresses the resulting image.Demosaicing fills in missing color channels; tone mapping adjusts dynamic range, while compression reduces storage and transmission requirements.Source: §A.2 · pp. 11–12

B. Proposed Pipelines

The proposed simplified pipelines retain selected ISP stages rather than the full pipeline. A pipeline with demosaicing and gamma compression performs close to baseline on most benchmarks, with SGBM an exception where denoising matters.

B. Proposed Pipelines

  • The pipeline with demosaicing and gamma compression performs close to baseline on most benchmarks; SGBM is an outlier where denoising has a significant effect.Figure 8 presents absolute error for each benchmark; Figure 10 compares normalized task error across four demosaicing configurations.Source: B. Proposed Pipelines · p. 12, Figure 10 · p. 12
  • The simplified pipelines considered include demosaicing, gamma compression, and denoising, while the two-stage demosaicing-and-gamma pipeline is close to baseline for most benchmarks.SGBM is the stated exception, for which denoising has a significant effect.Source: B. Proposed Pipelines · p. 12
  • Figure 10 compares normalized task error for four demosaicing strategies in configurations with only gamma compression enabled.The demosaic cluster retains original demosaiced data; other clusters simulate subsampling, nearest-neighbor demosaicing, or bilinear interpolation.Source: Figure 10 · p. 12

C. Approximate Demosaicing

Subsampling, bilinear interpolation, and nearest-neighbor demosaicing produce similar vision error, so the authors choose subsampling as the cheapest option. Sensor pixel binning offers a resolution-versus-task-error tradeoff.

C. Approximate Demosaicing

  • The demosaicing stage is useful for the examined vision applications, and subsampling is proposed as a circuit-level replacement for true ISP demosaicing.The compared lower-quality alternatives are bilinear interpolation and nearest-neighbor demosaicing.Source: C. Approximate Demosaicing · p. 12
  • Subsampling, bilinear interpolation, and nearest-neighbor demosaicing lead to similar vision error, so the authors choose subsampling because it eliminates signal processing.Subsampling fills missing channel values from corresponding channels in the Bayer pattern; bilinear interpolation averages local pixels, while nearest-neighbor copies from a nearby pixel.Source: C. Approximate Demosaicing · pp. 12–13

D. Resolution

  • Pixel binning reduces sensor resolution by combining photodiode charge, and the measured increase in error is not remarkably large, though higher capture resolution is generally better when possible.This creates a tradeoff between energy used to capture the image and vision-task error.Source: D. Resolution · p. 13

E. Quantization

The section explores quantization tuned to natural-image statistics as an alternative to logarithmic quantization. CDF-based quantization tends to lower error and often supports fewer bits, but its ADC hardware complexity is unclear.

E. Quantization

  • CDF quantization targets a uniform encoded-value distribution, while logarithmic quantization is presented as a simpler way to replace digital gamma compression.The section notes that information theory associates uniform encoded values with minimum quantization error and maximum entropy.Source: E. Quantization · p. 13
  • The CDF method fits a log-normal curve to raw CIFAR-10 intensity data, inverts its cumulative distribution, and samples uniformly across that CDF to set quantization levels.Figure 12 shows the input and quantized distributions.Source: E. Quantization · pp. 13–14, Figure 12 · p. 14
  • With CDF-based quantization, 4 bits generally suffice for most benchmarks, compared with 5 bits under logarithmic quantization.The CDF technique is less sensitive to bit-width reduction, and error across applications tends to be lower than with logarithmic quantization.Source: E. Quantization · p. 14, Figure 13 · p. 14
  • The main proposal uses logarithmic quantization because logarithmic ADCs are known and can use piecewise-linear approximation, whereas CDF quantization requires arbitrary levels with unclear hardware complexity.The Lloyd-Max algorithm may derive a more precise quantization-level distribution.Source: E. Quantization · p. 14

F. ISP Profiling

Software profiling identifies denoising as substantially more complex than other ISP stages; the measurements are a simpler proxy for stage costs pending hardware-level analysis.

F. ISP Profiling

  • The stage-cost measurements use software profiling because per-stage ISP energy breakdowns are unavailable; hardware-level simulation remains future work.The authors identify denoising implementation as a particular focus for further cost analysis.Source: F. ISP Profiling · pp. 14–15
  • Table 2 reports dynamic instructions, CPU cycles, L1 cache references, and floating-point operations for each ISP stage.Cycles fewer than instructions indicate extracted instruction-level parallelism; L1 cache references upper-bound off-chip memory accesses.Source: F. ISP Profiling · pp. 14–15
  • Denoising is significantly more complex than other ISP stages; most other stages have similar instruction, cycle, cache-reference, and floating-point-operation counts.JPEG compression is an exception for floating-point frequency because its codec is optimized for fixed-point implementation.Source: F. ISP Profiling · p. 15
Loading 1705.04352v3…