PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepReDuce: ReLU Reduction for Fast Private Inference

Nandan Kumar Jha, Zahra Ghodsi, Siddharth Garg, Brandon Reagen

arXiv:2103.01396v2cs.LGcs.CR

TL;DR

Private inference runs neural networks directly on encrypted data, but nonlinear operators such as ReLU impose substantial latency costs. DeepReDuce removes ReLU layers based on their differing contributions to accuracy and combines this with knowledge distillation. On CIFAR-100, it improves accuracy by up to 3.5% at iso-ReLU count and reduces ReLUs by 3.5× at iso-accuracy versus the state of the art.

  • Problem

    Private inference computes on ciphertexts, but ReLUs are costly nonlinear operations that make low-latency inference difficult.

    Source: §1 · p. 1

  • Method

    DeepReDuce removes entire ReLU layers according to their criticality and combines ReLU dropping with knowledge distillation from the original Full-ReLU network.

    Source: §1 · p. 2

  • Results

    Up to 3.5% accuracy improvement at iso-ReLU count and 3.5× ReLU reduction at iso-accuracy over the state of the art on CIFAR-100.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DeepReDuce provides a range of networks trading off ReLU count and accuracy for private inference.

    Source: §7 · p. 9

    DeepReDuce assumes the same honest-but-curious threat model and security guarantees as DELPHI, MiniONN, and CryptoNAS, and makes no protocol changes.

    Source: §4 · p. 6

Abstract

from arXiv · show

The recent rise of privacy concerns has led researchers to devise methods for private neural inference -- where inferences are made directly on encrypted data, never seeing inputs. The primary challenge facing private inference is that computing on encrypted data levies an impractically-high latency penalty, stemming mostly from non-linear operators like ReLU. Enabling practical and private inference requires new optimization methods that minimize network ReLU counts while preserving accuracy. This paper proposes DeepReDuce: a set of optimizations for the judicious removal of ReLUs to reduce private inference latency. The key insight is that not all ReLUs contribute equally to accuracy. We leverage this insight to drop, or remove, ReLUs from classic networks to significantly reduce inference latency and maintain high accuracy. Given a target network, DeepReDuce outputs a Pareto frontier of networks that tradeoff the number of ReLUs and accuracy. Compared to the state-of-the-art for private inference DeepReDuce improves accuracy and reduces ReLU count by up to 3.5% (iso-ReLU count) and 3.5$\times$ (iso-accuracy), respectively.

1. Introduction

DeepReDuce removes ReLU layers to reduce ReLU counts while limiting accuracy loss, combining ReLU dropping with knowledge distillation. It targets private inference, where nonlinear operations make encrypted computation costly, and offers networks across a ReLU-count–accuracy frontier.

1. Introduction

  • Up to 3.5% higher accuracy at iso-ReLU count and 3.5× fewer ReLUs at iso-accuracy than the state of the art on CIFAR-100.Figure 1 presents the ReLU-count–accuracy Pareto frontier.Source: Figure 1 · p. 1, §1 · p. 2
  • Private inference processes inputs directly on ciphertexts, while ReLUs are costly enough to account for 93% of ResNet32 online inference time with DELPHI.The supplied introduction describes the high cost of garbled circuits for private ReLU computation.Source: §1 · p. 1
  • ReLU dropping removes entire ReLU layers, exploiting their uneven distribution and differing accuracy contributions for large count reductions with minimal accuracy impact.The paper contrasts wholesale layer removal with the modest ReLU savings from weight and channel pruning.Source: §1 · p. 2
  • DeepReDuce combines ReLU dropping with knowledge distillation from the original Full-ReLU network to improve optimized-network accuracy.The paper identifies these two optimizations as its overall methodology.Source: §1 · p. 2
  • Removing ReLUs can merge adjacent linear transformations, reducing model depth and FLOPs; one ResNet9 comparison reports 24,600 ReLUs and 68.1% accuracy versus 30,720 and 66.2%.The comparison is against the shallower ResNet9 baseline.Source: §1 · p. 2
  • Compared with channel pruning, DeepReDuce provides 2× greater ReLU reduction at similar accuracy.The introduction says existing neural-inference efficiency techniques are insufficient for ReLU reduction and private inference.Source: §1 · p. 2

2. Motivating ReLU Dropping

Conventional CNNs distribute ReLUs unevenly, and experiments show that stages differ in their accuracy contribution. These patterns motivate removing less-critical ReLUs, especially in early layers, while preserving more consequential ones.

2.1. Notation

  • ResNet18 has four stages of repeated residual blocks; its layer-wise FLOPs are even, while ReLUs decrease and parameters increase with depth.The same distribution pattern is reported for other common CNNs.Source: §2.1 · p. 2, §2.2 · p. 3, Figure 3 · p. 3

2.2. ReLU Dropping

  • Early stages contain many ReLUs but tend to be less critical, making them promising targets for reduction.The architecture’s channel increases and spatial downsampling across stages produce a 2× drop in ReLU count across a down-sampling layer.Source: §2.2 · p. 3
  • 24.8%: ResNet18 gains this much accuracy by retaining ReLUs in S3 rather than Conv1, despite equal ReLU counts.Networks with ReLUs in S2 and S3 also achieve high accuracy while using fewer ReLUs than S1.Source: §2.2 · p. 3, Table 1 · p. 3
  • Knowledge distillation gains are position-dependent, with deeper-stage ReLU networks benefiting more than networks retaining ReLUs in S1 or S2.The authors relate this pattern to the claim that teacher knowledge is primarily disbursed in deeper layers.Source: §2.2 · p. 3
  • 1%–3%: Across the tested architectures, alternate-layer ReLU dropping improves iso-ReLU accuracy over channel down-scaling, even with KD.Both strategies reduce ReLUs by 2× in the Table 2 comparison.Source: Table 2 · p. 3, §2.2 · p. 4

3. DeepReDuce

DeepReDuce applies ReLU Culling, Thinning, and Reshaping sequentially to produce Pareto-optimal networks trading accuracy against ReLU count. Its stage-criticality profiling guides culling, and knowledge distillation is used to improve optimized networks’ accuracy.

3. DeepReDuce

  • DeepReDuce sequentially applies Culling, Thinning, and Reshaping to produce Pareto-optimal networks trading accuracy against ReLU count.The optimizations operate at different granularities and are applied to identify high-performing networks quickly.Source: §3 · p. 4

3.1. ReLU Culling

  • Culling removes all ReLUs from a stage, iterating from least to most critical and producing D −1 reduced-ReLU networks for a D-stage network.Criticality is estimated using accuracy when ReLUs are retained in one stage, with KD and stage ReLU cost included in the metric.Source: §3.1 · p. 4
  • Stage criticality order varies by dataset: ResNet18’s least-to-most critical order is S1 < S4 < S2 < S3 on CIFAR-100 and S1 < S2 < S4 < S3 on TinyImageNet.TinyImageNet results also report S1 and S3 as the least- and most-critical stages for ResNet18/34.Source: §3.1 · p. 4, Table 3 · p. 4

3.2. ReLU Thinning

  • Thinning halves the ReLU layers in non-culled stages by dropping alternate layers; it consistently outperforms first-half or last-half dropping and 2× channel scaling.The authors choose alternate-layer dropping to balance search runtime and effectiveness.Source: §3.2 · p. 4

3.3. ReLU Reshaping

  • Reshaping uses channel, feature-map, and compound scaling to reduce ReLU count by 2×, 4×, and 8×, respectively.It is less effective than Culling and Thinning because it tends to introduce a higher accuracy drop, so it is reserved for highly constrained ReLU budgets; scaling factors can also affect accuracy at equal ReLU counts.Source: §3.3 · pp. 4–5

3.4. Improving accuracy using KD

  • Knowledge distillation retrains optimized networks using the Full-ReLU baseline as teacher and typically recovers several percentage points of accuracy.KD is also incorporated when estimating stage criticality, because its accuracy gain depends on stage position and changes the culling order.Source: §3.4 · p. 5

3.5. Putting it All Together

  • DeepReDuce compounds Culling across iterations from least to most critical stages, avoiding exhaustive design-space exploration.It explores 5×(D −1) architectures, while NAS-based methods such as CryptoNAS explore a large design space and train significantly more models.Source: §3.5 · p. 5

4. Methodology

DeepReDuce is evaluated on standard residual and non-residual networks, using established datasets and a private-inference protocol. The experiments retain DELPHI’s threat model and security guarantees.

4. Methodology

  • DeepReDuce is applied to ResNet18/34, VGGNet, and MobileNets.The ResNet architectures follow He et al. (2016).Source: §4 · pp. 5–6
  • Experiments use CIFAR-100 and TinyImageNet, while CIFAR-10 training runs for 150 epochs with cosine learning.CIFAR-100 has 100 classes; TinyImageNet has 200 classes.Source: §4 · p. 6
  • Private inference follows DELPHI: secret sharing handles linear layers and garbled circuits handle ReLUs.Model-weight shares are prepared offline, with linear operations performed online over secret-shared data.Source: §4 · p. 6
  • The threat model assumes an honest-but-curious adversary and makes no protocol changes to DELPHI’s security guarantees.The same setup is used by MiniONN and CryptoNAS.Source: §4 · p. 6

5. Results

Across the evaluated settings, DeepReDuce offers favorable accuracy–ReLU tradeoffs against prior methods and smaller or channel-pruned networks. Its results extend to MobileNetV1, although private-inference latency remains above real-time requirements.

5.1. DeepReDuce Pareto Analysis

  • Each optimization—Culling, Thinning, and Reshaping—appears on the ResNet18 Pareto frontier; Reshaping helps most at low ReLU budgets, while Culling and Thinning dominate higher budgets.Accuracy per kilo-ReLU rises as ReLU count falls; the lowest-accuracy TinyImageNet network is 22.7% less accurate but uses 74.7× fewer ReLUs, yielding 48.8× more accuracy per kilo-ReLU.Source: §5.1 · pp. 6–7

5.2. DeepReDuce Outperforms Prior Work

  • At a 50K ReLU budget, DeepReDuce is 3.5% more accurate than DELPHI; against the merged DELPHI/CryptoNAS Pareto set, it offers up to 3.5× ReLU savings at iso-accuracy.For VGG16 on CIFAR-10, it also yields a 3.4× ReLU reduction at iso-accuracy and 3.6% more accuracy at iso-ReLU count versus SAFENet.Source: Table 6 · p. 7, §5.2 · p. 7, Figure 5 · p. 7

5.3. DeepReDuce Outperforms Shallow ResNets

  • With the same number of layers, DeepReDuce uses 2,000 fewer ReLUs and is 2.3% more accurate than ResNet9; at 24.6K ReLUs it remains 1.9% more accurate than the smaller ResNet9.The comparison also covers ResNet10 models using feature-map and channel scaling.Source: §5.3 · p. 8, Table 7 · p. 8

5.4. DeepReDuce Outperforms Channel Pruning

  • DeepReDuce is 0.73% and 2.83% more accurate on CIFAR-10 and CIFAR-100, respectively, while using 1.4× fewer ReLUs than channel pruning.On CIFAR-100, it is 1% more accurate (71.68% vs. 70.83%) with 2× fewer ReLUs; it uses 1.4× more FLOPs in the comparisons.Source: §5.4 · p. 8, Table 8 · p. 8

5.5. DeepReDuce Network Inference Latencies

  • On CIFAR-100, DeepReDuce achieves a 3.7× latency reduction versus CryptoNAS at iso-accuracy; at iso-latency it offers 3.5% higher accuracy than DELPHI.The fastest model takes 455 ms per inference at 65% accuracy, and the reported inference times remain above 30–60 FPS real-time requirements.Source: §5.5 · p. 8

5.6. Generality Case Study: MobileNets

  • On CIFAR-100 MobileNetV1, DeepReDuce improves accuracy by 10.9% at iso-ReLU and reduces ReLUs 3.8× at iso-accuracy.The results indicate that residual connections are not necessary for DeepReDuce to perform well on non-residual networks.Source: §5.6 · p. 9

6. Related Work

Prior private-inference work combines cryptographic protocols for linear and nonlinear operations, while later methods optimize ReLU use or network structure. The cited approaches include homomorphic encryption, secret sharing, garbled circuits, polynomial approximations, and neural architecture search.

6. Related Work

  • CryptoNets first demonstrated homomorphic encryption for protecting client data during inference, while SecureML focused on privacy-preserving training and also supported inference with high overheads from generic MPC protocols.SecureML uses a two-server protocol.Source: §6 · p. 9
  • MiniONN combines multiplication triplets for linear-layer multiplications with garbled circuits for ReLUs, while Gazelle uses optimized homomorphic encryption for linear layers and garbled circuits for nonlinear layers.DELPHI moves heavy cryptographic operations offline and uses secret sharing for online linear layers.Source: §6 · p. 9
  • DELPHI replaces selected ReLU layers with quadratic functions using neural architecture search; CryptoNAS searches for networks under a ReLU budget, and SAFENet substitutes channel-wise ReLUs with mixed-precision polynomials.The cited methods target ReLU replacement or budgeted network selection through different search and approximation strategies.Source: §6 · p. 9

7. Conclusion and Future Work

The conclusion presents ReLU dropping as a way to tailor networks for private inference and provide accuracy–ReLU tradeoffs. The reported results advance prior performance, but real-time private inference remains an unmet goal.

7. Conclusion and Future Work

  • DeepReDuce prioritizes and removes ReLUs to provide networks spanning different ReLU counts and accuracies.The method is presented as an approach to tailoring networks for private inference.Source: §7 · p. 9
  • On CIFAR-100, DeepReDuce reports 3.5% improvement at iso-ReLU and 3.5× ReLU reduction at iso-accuracy.The paper also evaluates on TinyImageNet, which it describes as uncommon in private-inference work given long runtimes.Source: §7 · p. 9
  • Despite advancing the state of the art, DeepReDuce falls short of the goal of real-time private inference.The authors expect it to inspire further optimizations and new approaches to network design.Source: §7 · p. 9

A. Channel Scaling Vs. Feature map Scaling

Feature-map resolution scaling gives a more accurate option than channel scaling at equal ReLU budgets. DeepReDuce combines the two approaches when targeting very low ReLU counts.

A. Channel Scaling Vs. Feature map Scaling

  • A 2× reduction in each feature-map spatial dimension (ρ=0.5) cuts ReLUs 4×, while channel scaling with α=0.5 cuts them 2×.Source: A. Channel Scaling Vs. Feature map… · p. 12
  • For a 4× ReLU reduction, resolution scaling (ρ=0.5) is preferred over channel scaling (α=0.25), which also reduces parameters and may reduce expressive power.The paper attributes the greater accuracy of resolution scaling to avoiding that parameter reduction.Source: A. Channel Scaling Vs. Feature map… · p. 12, Table 9 · p. 12
  • At iso-ReLU, fmap-resolution-scaled models have higher accuracy with KD than channel-scaled models.The baseline uses α=1 and ρ=1.Source: Table 9 · p. 12

B. VGGNet DeepReDuce Pareto Points

VGG16 stage-wise criticality varies: S5 is least critical, while S2 and S3 are most critical. The described optimization variants use culled networks, and FC-layer ReLUs are retained.

B. VGGNet DeepReDuce Pareto Points

C. ReLUs’ Criticality and Pareto Points for MobileNets

MobileNet ReLU criticality differs across stages, with S1 least critical and S4 most critical. DeepReDuce culls S1 and thins alternate-layer ReLUs, favoring depthwise-layer removal for iso-ReLU accuracy.

C. ReLUs’ Criticality and Pareto Points for MobileNets

E. Layer-wise Distribution of ReLUs

Across the standard networks discussed, ReLUs cluster in initial layers even though FLOPs are evenly distributed and parameters increase in deeper layers. This mismatch means FLOPs- or parameter-based pruning alone does not ensure ReLU optimization.

E. Layer-wise Distribution of ReLUs

Related papers

Loading 2103.01396v2…