PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepCache: Principled Cache for Mobile Deep Vision

Mengwei Xu, Mengze Zhu, Yunxin Liu, Felix Xiaozhu Lin, Xuanzhe Liu

arXiv:1712.01670v5cs.CV

TL;DR

Continuous mobile vision needs efficient CNN inference despite redundant but changing video frames and limited on-device resources. DeepCache matches reusable image regions using video-compression heuristics, propagates them through CNN layers, and reports 18% average inference-time savings, up to 47%, with about 20% lower energy consumption.

  • Problem

    Continuous mobile vision must cache CNN results despite scene variation while balancing cacheability, overhead, and model accuracy.

  • Method

    DeepCache matches reusable input-image regions with video-compression heuristics and propagates them through CNN layers using CNN structure.

  • Results

    18% average and up to 47% inference-time savings were achieved, while system energy consumption fell by around 20%.

  • Takeaways & Limitations

    DeepCache provides an immediately deployable cache for unmodified CNN models with zero developer effort on off-the-shelf mobile devices.

  • Takeaways & Limitations

    The prototype was implemented for CPU inference, although the authors expect it can be ported to hardware accelerators.

Abstract

from arXiv · show

We present DeepCache, a principled cache design for deep learning inference in continuous mobile vision. DeepCache benefits model execution efficiency by exploiting temporal locality in input video streams. It addresses a key challenge raised by mobile vision: the cache must operate under video scene variation, while trading off among cacheability, overhead, and loss in model accuracy. At the input of a model, DeepCache discovers video temporal locality by exploiting the video's internal structure, for which it borrows proven heuristics from video compression; into the model, DeepCache propagates regions of reusable results by exploiting the model's internal structure. Notably, DeepCache eschews applying video heuristics to model internals which are not pixels but high-dimensional, difficult-to-interpret data. Our implementation of DeepCache works with unmodified deep learning models, requires zero developer's manual effort, and is therefore immediately deployable on off-the-shelf mobile devices. Our experiments show that DeepCache saves inference execution time by 18% on average and up to 47%. DeepCache reduces system energy consumption by 20% on average.

1 INTRODUCTION

DeepCache targets costly continuous mobile vision by reusing computation across temporally similar video frames despite scene variation. It combines video-informed input matching with CNN-aware propagation of reusable regions, while supporting unmodified models and reducing inference cost.

  • Motivation: Continuous mobile vision runs CNN inference on resource-constrained devices, motivating computation reuse across redundant consecutive video frames.Temporal locality provides similar or overlapping regions across frames.
  • Challenges: CNN caching must tolerate user, camera, object, and illumination changes while balancing cacheability, lookup overhead, and model accuracy.Existing commodity engines process frames independently, while ad-hoc caches leave benefits untapped.
  • Approach: DeepCache discovers reusable image regions using video-compression heuristics, then propagates them through CNN layers using the model’s internal structure.It avoids applying video heuristics to high-dimensional feature maps whose similarity is difficult to evaluate.
  • Results: 18% average and up to 47% inference-time savings were measured, with about 20% lower system energy, no more than 3% accuracy loss, and under 2% DRAM use.The evaluation used five CNN models and two real-world video datasets on a Nexus 6.
  • Deployment: The prototype runs unmodified CNN models, requires zero developer effort, and is immediately deployable on off-the-shelf Android devices.DeepCache was implemented in ncnn atop Android 6.0.

2 BACKGROUND AND CHALLENGES

This section motivates caching CNN computation in continuous mobile vision and identifies two main obstacles: finding reusable content under scene variation and preserving reuse across layers as regions erode.

  • CNN Background: CNN convolutional layers dominate processing time, contributing at least 60% and up to 90% in the evaluated models.This motivates focusing caching on convolutional layers.
  • CNN Background: Mobile video streams exhibit temporal locality because consecutive frames often contain similar but non-identical regions.This redundancy can support cached CNN results.
  • Challenges: Scene variation makes exact cache-key equivalence unsuitable, requiring systematic similarity evaluation while trading off cacheability, overhead, and accuracy.Relevant variation includes motion, object appearance, and illumination changes.
  • Challenges: Cache erosion causes reusable regions to shrink at deeper layers because output pixels may depend on both reusable and non-reusable inputs.Convolution, pooling, and LRN can erode reuse; fully connected layers may destroy it.
  • Challenges: Early layers offer favorable reuse because they contribute most computation and suffer less erosion, whereas fully connected layers contribute only 11.5% of AlexNet latency.The remaining 88.5% comes from earlier layers that can benefit from caching.

3 SYSTEM OVERVIEW

DeepCache extends a commodity inference engine with a cache spanning video inputs and CNN feature maps. It performs one input-level region lookup, then reuses and transforms cached regions across subsequent layers.

  • Architecture: DeepCache is a lightweight extension that leaves model loading, video ingestion, preprocessing, CNN execution, and output emission otherwise unchanged.The architecture augments commodity inference rather than replacing its components.
  • Architecture: The cache stores recent input frames as keys and recent feature maps for individual CNN layers as values.Keys are equal-sized, fine-grained regions on cached input frames.
  • Value Mapping: DeepCache performs key lookup once at the input, then propagates reusable-region mappings through CNN layers without searching feature maps again.Layer operators transform region boundaries for downstream feature maps.
  • Key Lookup: For each new frame, DeepCache partitions the image and searches recent frames for similar regions using a video-compression-inspired matcher.The default regions are 10x10 pixels, and matching produces rectangle-to-rectangle mappings.

4 IMAGE BLOCK MATCHING

DeepCache matches reusable image blocks between consecutive frames using video-inspired heuristics, then merges suitable matches to preserve reuse despite cache erosion. The matcher balances matching overhead, cacheability, and the need to identify regions that remain useful across CNN layers.

  • Matching strategy: Block-wise matching avoids letting mutated pixels invalidate otherwise similar 10x10-pixel regions.DeepCache uses blocks rather than individual pixels because small pixel mutations can reduce reuse through cache erosion.
  • Matching algorithm: The matcher divides each frame into an N×N grid and searches corresponding blocks in the previous frame using diamond search and PSNR.It estimates average block movement from matches exceeding threshold T, then evaluates candidate blocks at the shared offset.
  • Cache erosion: DeepCache merges adjacent properly matched blocks so reusable regions remain large enough to survive deeper CNN layers.The design rejects individually highest-scoring matches when they would produce small blocks that quickly disappear through cache erosion.
  • Motion handling: Camera motion is captured through average block movement, while independently moving objects are detected and marked non-reusable.The algorithm combines block movements in Step 3 and filters mismatched regions in Step 4.
  • Efficiency: A 2-skip can theoretically save 75% of Step 2’s computation by matching only every other row and column of blocks.The optimization targets the most time-consuming parts of the matching algorithm, especially Steps 2 and 4.

5 CACHE MECHANISMS INSIDE MODEL EXECUTION

DeepCache propagates reusable regions through CNN layers and reuses cached convolution results without repeatedly matching internal feature maps. Its design avoids the high cost and difficult similarity evaluation associated with matching high-dimensional feature maps directly.

  • Propagation: DeepCache dynamically transforms reusable-region boundaries as regions pass through different CNN layer types.The transformation function D_t accounts for how each layer changes the region, including reductions caused by convolution and pooling.
  • Reuse: Reusable convolution regions skip computation and are copied from cached feature maps, while non-reusable pixels undergo convolution.A bit map identifies cached pixels, and the customized convolution traverses the input while skipping reusable ones.
  • Scope of reuse: DeepCache caches only convolutional-layer results because convolution dominates inference time while caching other layer outputs adds memory overhead.This is presented as a trade-off between latency improvement and memory overhead.
  • Accuracy control: DeepCache periodically clears its cache and recomputes a complete frame every N frames, defaulting to 10, to limit accumulated accuracy loss.Repeated reuse can compound disparities because matched blocks are similar but not numerically identical.
  • Design comparison: Matching internal layers can incur up to 35× higher lookup latency than DeepCache, even when covering only 50% of convolutional layers.The comparison varies MIL’s coverage of convolutional layers and reports much higher latency than propagating regions from the input.
  • Limitation of internal matching: Feature-map similarity thresholds may be specific to models, layers, or inputs, making MIL’s reuse decisions difficult to configure without developer effort.DeepCache instead performs key lookup on image regions, where established image-similarity heuristics are available.

6 IMPLEMENTATION

DeepCache is implemented as a portable mobile inference-engine feature compatible with standard CNN models and existing ncnn-based applications. The implementation uses Android GPU acceleration for image matching while exposing optional cache controls to developers.

  • Image matching: The image matcher runs in Android RenderScript, allowing GPU offload and portability across Android devices.RenderScript is described as Android’s counterpart to CUDA and a generic Android API.
  • Compatibility: DeepCache is directly compatible with standard CNN models through its ncnn-based prototype without requiring model changes.The prototype is built atop ncnn, an open-source mobile-optimized engine for Android and iOS.
  • Evaluation: Across five CNN models and test scenarios, Figure 10 reports average processing time for each model over its scenarios.The figure’s supplied description identifies the aggregation but does not state the numerical values or comparison outcome.
  • Implementation scope: The implementation contains 4,030 lines of code and adds custom layers such as atan needed by benchmark models.These custom layers address operators unsupported by the current ncnn implementation.
  • Configuration: DeepCache exposes optional controls for threshold T, block size N, and cache expiration time through APIs compatible with ncnn.Existing ncnn vision applications can use DeepCache without changes, while developers may fine-tune cache behavior.

7 EVALUATION

DeepCache is evaluated on mobile hardware across CNN models and real-world video datasets, showing substantial latency and energy savings with small accuracy losses. Its benefits vary by model, scene dynamics, matching parameters, and the sequencing of matching with inference.

  • Overall results: 18.2% average and up to 47.1% latency savings were achieved across five CNN models and two real-world datasets, outperforming DeepMon on all models and datasets.DeepCache reduced latency by twice as much as DeepMon in the reported comparison.
  • Latency improvement: Latency savings varied by model and scene: REC_1 reached 28.1% on average, while Billiards reached 47.1% and BandMarching only 11.0%.Slow camera motion, mostly static objects, and stable indoor lighting supported higher reuse in Billiards; broken motion reduced it in BandMarching.
  • Layer-level analysis: Savings mainly came from early convolutional layers; DeepCache saved 90.2ms from the third layer, which required 165ms and represented about 18.4% of total model time.Early layers contribute substantially to latency and experience less cache erosion.
  • Accuracy loss: Accuracy losses remained small: at most 3% for top-1 and 1.5% for top-3 activity-recognition accuracy, while DET and DRV median MSEs were 0.00166 and 2.617.For DRV, the reported MSE corresponded to a 2.6-degree offset from the human driver's decision, and periodic full-image execution prevents accumulation.
  • Energy saving: DeepCache reduced energy consumption by 19.7% on average and up to 28.6% for REC_1, compared with 8.0% average savings for DeepMon.Applying DeepCache to REC_3 classification of 10 images saved 66.8J, equivalent to about 40 seconds of video playback on Nexus 6.
  • Parameter choices: The default matching threshold T = 20 achieved 18.3% latency improvement, from 917ms to 748ms, with 2.1% accuracy loss; larger blocks increased both latency savings and accuracy loss.A block size around 10 was empirically suggested for 227x227 images.

8 DISCUSSION

The discussion describes DeepCache’s scope across CNN models, hardware accelerators, and video types. Results are reported for five typical CNN models, with CPU prototyping and mobile-video optimization defining current boundaries.

  • Applicability to other CNN models: Results are reported only for five typical CNN models, although the authors expect applicability to SqueezeNet, MobileNet, and DenseNet.The expectation is based on preserved temporal locality and similar early-convolutional-layer cost dominance.
  • Implementation on accelerators: DeepCache’s inference stage is prototyped on CPU, while accelerator ports are expected to provide further benefits.The authors specifically anticipate avoiding redundant GPU kernels and implementing caching logic on FPGAs.
  • Applicability to other video types: DeepCache’s high-level design can extend to non-mobile videos when adjacent frames contain redundancy.The current optimization targets mobile vision because mobile videos have richer temporal locality and mobile devices are more latency- and energy-sensitive.

9 RELATED WORK

The related work positions DeepCache as complementary to model, hardware, and compression techniques for mobile deep learning. It differs from earlier CNN caches by systematically handling scene variation and exploiting reuse across inference execution.

  • Convolutional Layer Caching: DeepMon and CBinfer use same-position image-block matching or feature-map lookup across all layers, limiting cacheability or increasing overhead.DeepCache instead uses video techniques to search for nearby similar image blocks and avoids indiscriminate all-layer matching.
  • Continuous Mobile Vision: Continuous mobile vision research spans commercial systems and prototypes addressing computation sharing, sensor energy, and image analysis.The cited work includes shared computation and memory objects, analog-domain CNN layers, and other mobile-vision optimizations.
  • Optimizing Deep Learning Execution for Mobile: DeepCache complements smaller models, specialized hardware, and model compression by exploiting temporal locality in input data.Its optimization targets mobile deep vision execution rather than replacing these other approaches.

10 CONCLUSIONS

DeepCache accelerates continuous-vision CNN execution by leveraging temporal locality at both the video input and model levels. Its design combines video-derived locality discovery with model-structure-aware propagation of reusable regions.

  • 10 CONCLUSIONS: DeepCache is a principled cache design that accelerates CNN models for continuous vision by leveraging video temporal locality.It discovers locality from video structure at model input and propagates reusable result regions through model structure.
  • 10 CONCLUSIONS: DeepCache borrows proven video-compression heuristics for input locality but does not apply them to high-dimensional model internals.The internal propagation mechanism instead exploits model structure.
Loading 1712.01670v5…