CBinfer: Change-Based Inference for Convolutional Neural Networks on Video Data

1 INTRODUCTION

• CNN inference requires billions of operations even for low-resolution images, while on-site processing faces restricted computation and power budgets.Centralized inference also brings infrastructure, energy, privacy, bandwidth, and reliability concerns.Source: §1 · p. 1
• Existing software, hardware, and network-approximation approaches may offer insufficient gains, reach platform limits, lack flexibility or availability, or incur considerable accuracy loss..Source: §1 · p. 1
• The proposed static-camera video method achieves an average 8.6× speed-up over cuDNN with negligible accuracy loss on an urban surveillance dataset.The authors state this brings real-time CNN inference on high-resolution frames within current embedded platforms’ computation and power budgets.Source: §1 · p. 1

2 RELATED WORK

• The related-work survey groups prior approaches around datasets and networks, optimized implementations, accuracy-throughput approximations, and video-based computation reduction.The paper’s evaluation platform, Tegra X1, is an embedded system-on-chip board with an affordable power budget below 15 W for a stationary camera.Source: §2 · p. 1

2.1 Suitable Datasets and Neural Networks

• The urban surveillance dataset provides semantic-segmentation labels and surrounding unlabeled video frames, making it suitable for studying static-camera inference.The related-work discussion notes that many available static-camera datasets target tracking or re-identification rather than multi-class detection or segmentation.Source: §2.1 · p. 2

2.2 Optimized Embedded System Implementations

• GPU CNN implementations commonly use matrix multiplication, with cuDNN and Neon adding assembly-level optimizations and methods such as Winograd or FFT-based convolution.An embedded nonbatched implementation reported that convolutions consumed more than 90% of runtime.Source: §2.2 · p. 2

2.3 Approximations Trading Accuracy for Throughput

• Throughput-oriented approximations include reduced precision, binary weights or activations, pruning, and smaller or simplified networks; CBinfer can combine with these methods.Some methods trade accuracy for throughput, while others require specialized hardware for energy-efficiency gains.Source: §2.3 · p. 2

2.4 Video-based Computation Reduction

• Prior work on video computation includes clockwork CNNs, which update entire frames; CBinfer instead reevaluates only small changed regions.The cited clockwork approach reduces full-frame convolutions by about 40% before accuracy starts to drop on Youtube-Objects, but is limited to certain CNN architectures.Source: §2.4 · pp. 2–3

3 METHODOLOGY

• CBinfer replaces spatial convolution layers with change-based layers that detect changes before each convolution, without modifying CNN training or requiring retraining.The approach is intended for video from a static camera with limited frame-to-frame changes.Source: §3 · p. 3

3.1 Processing Steps

• At each layer, thresholded channel differences flag changed pixels; their filter-support neighborhoods are marked for update, and blockwise extraction produces the affected-pixel indexes and count.The change map is cleared before each pixel checks whether it changes.Source: §3.1 · p. 4
• The algorithm assembles only image-matrix columns for affected outputs, then multiplies by the unchanged trained-filter matrix and updates output feature maps with stored and newly computed values.The matrix multiplication is at worst as time-consuming as full-frame convolution; ReLU is included in the output update.Source: §3.1 · p. 4

3.2 Memory Requirements

• Memory optimization reduces baseline storage to 103M values; CBconv adds ~60M, totaling 163M values (~650 MB, +58%).The baseline reduction comes from sharing the X matrix across convolution layers and reusing output buffers between two layers.Source: §3.2 · p. 4

3.3 Threshold Selection

• Thresholds are swept per layer on sample videos until clear performance degradation appears on validation; evaluations find no per-sequence recalibration is needed and accuracy and speed-up are not overly sensitive.Thresholds are fixed offline after training; zero thresholds were used for functional verification against the non-change-based implementation.Source: §3.3 · pp. 4–5

4 RESULTS & DISCUSSION

• The section plans to report baseline timing, throughput, accuracy-throughput trade-offs, and compute-time breakdowns, including how changes propagate through the network.The supplied passages describe the evaluation plan rather than those results.Source: §4 · p. 5

4.1 Evaluation Environment

• The urban-surveillance evaluation uses 51 training images and 6 validation images at 776 × 1040 resolution, with eight scene-labeling classes and five preceding frames available for each labeled validation image.The trained network’s parameters are reused unaltered.Source: §4.1 · p. 5
• The CUDA/Torch implementation is evaluated on a Jetson TX1 board with JetPack 2.3.1, using cuDNN v5.1.5 as the baseline for the CNN and pixel-wise classification.The baseline library includes Winograd and FFT-based convolution optimizations.Source: §4.1 · p. 5

4.2 Baseline Throughput and Computation Breakdown

• 91.9% (492 ms) of baseline computation time is spent in Layers 1–3, which take 535 ms overall at 1.87 frame/s.These feature-extraction layers use 7 × 7 convolutions and are the focus of the CBconv replacement.Source: §4.2 · p. 5

4.3 Threshold Selection

• Thresholds are selected one layer at a time, choosing each layer’s value after accounting for the thresholds already selected for earlier layers.The selected values are τ1 = 0.04, τ2 = 0.3, and τ3 = 1.0.Source: §4.3 · p. 5
• Accuracy stays on a plateau under jointly scaled thresholds before reaching a clear point of steep error increase.The selected thresholds are τ1 = 0.04, τ2 = 0.3, and τ3 = 1.0.Source: §4.3 · p. 5

4.4 Throughput Evaluations

• Setting all thresholds to zero updates the entire frame and causes a performance loss from change-detection overhead and less cache-friendly matrix generation.Figure 7 evaluates threshold scaling against classification error, changed-pixel count, and throughput.Source: §4.4 · p. 6, Figure 7 · p. 6
• Near 17 frame/s is achieved at threshold factor 1, an 8.6× improvement over the cuDNN baseline of 1.96 frame/s.At this factor, the authors report almost no accuracy loss; most sequences trigger fewer than 3% of possible changes, while a busy-road sequence triggers around 9%.Source: §4.4 · p. 6, Figure 7 · p. 6
• Figure 8 evaluates the throughput-accuracy trade-off across all six video sequences.The evaluation includes sequences with differing throughput behavior, including a busy-road case.Source: Figure 8 · p. 6, §4.4 · p. 6
• On a GTX Titan X, the throughput-threshold trade-off and compute-time breakdown are almost identical to the TX1 results, with scaling up to 11.9×.The authors attribute this scaling to a largely parallelizable workload and the devices’ similar architecture.Source: §4.4 · p. 6

4.5 Accuracy-Throughput Trade-Off

• The paper frames accuracy-versus-throughput tuning as an application-dependent choice, since selecting a CNN already entails choosing its accuracy and computational cost.Source: §4.5 · p. 6
• Updating the entire frame each time gives the lowest throughput while retaining the same accuracy as full-frame inference.Figure 8 evaluates the throughput-accuracy trade-off across all six sequences.Source: §4.5 · p. 6, Figure 9 · p. 7
• Increasing the threshold factor in steps of 0.25 yields significant throughput gains, and for most sequences the trade-off begins only near saturation above 16 frame/s.A busy-sequence outlier behaves differently; the authors suggest adaptive threshold selection using changed-pixel feedback as future work.Source: §4.5 · p. 7

4.6 Compute Time Breakdown

• Within the change-based layers, matrix multiplication takes the largest share of time, change detection exceeds 23.4%, and output updates use a negligible share.Generating the X matrix takes less time than the matrix multiplication; change-index extraction scales linearly with h · w and is not well parallelizable.Source: §4.6 · p. 7
• Convolution and activation time for Layers 1–3 shrinks by more than 12.9×, from 512.8 ms to about 39.7 ms; polling and pixel-wise classification then account for 36% of compute time.The remaining steps total 6.6 ms for polling and 16.0 ms for Layers 4–5 pixel-wise classification.Source: §4.6 · p. 7
• Figure 10 contrasts Layer 2 changed pixels under worst-case propagation without detection against the pixels remaining after applying change detection.The figure also shows single-pixel Layer 1 changes attributed to noise.Source: Figure 10 · p. 7

• Layerwise change detection reduced changes 6.8× in Layer 2 (7.57% to 1.11%) and 1.33× in Layer 3 (2.58% to 1.94%).Source: §4.7 · p. 7
• Although detection is only 22% of Layer 3 compute time, scaling X-matrix generation, multiplication, and output updates exceeds its overhead.Source: §4.7 · p. 7
• Change extraction cannot be dropped; detection must instead be replaced with a quick change-propagation kernel.Source: §4.7 · p. 7

• CBinfer reached 327.6 GOp/s/W and 3577 GOp/s per-frame-equivalent throughput, versus cuDNN's 32.0 GOp/s/W and 413 GOp/s.The reported energy-efficiency boost was about 10×.Source: §4.8 · p. 7
• The board consumed 10.92 W running CBinfer, compared with 12.9 W (680 mA) running the cuDNN baseline.Source: §4.8 · p. 7
• Measurements used a CNN with 210 GOp/frame complexity; idle consumption was 1.7 W normally and 2.5 W at maximum clock frequency.Source: §4.8 · p. 7

• 8.6× average speed-up over a cuDNN baseline was achieved with no significant increase in misclassified pixels using conservative change-detection parameters.Source: §5 · p. 8
• 10× average energy-efficiency boost over per-frame evaluation reached 328 GOp/s/W on the Tegra X1 platform.Source: §5 · p. 8
• The throughput-accuracy analysis reports that throughput can be increased further at the expense of accuracy.Source: §5 · p. 8
• Compute-time analysis found the GPU fully loaded with multiply-accumulate updates of changed pixels using optimized cuBLAS matrix multiplication.Source: §5 · p. 8