Extracting CNNs in the Unknown-Architecture and Feedback-Agnostic Setting

1 Introduction

• Existing CNN extraction attacks assume the architecture is known, although architecture is an undisclosed and valuable model asset.CNN architecture includes layer type, kernel size, stride, padding mode, channel numbers, and pooling structure, making it richer than the list of layer dimensions used for FCNs.Source: §1 · p. 2
• The open question is whether CNN convolutional layers can be extracted without prior architectural knowledge, unlike FCNs whose unknown layer dimensions already have recovery methods.The paper frames this as removing the architectural prior while reusing existing parameter-recovery methods.Source: §1 · pp. 2–3
• The attack formalizes how the spatial structure of recovered vectors reveals convolutional and pooling hyperparameters, independently of the feedback setting.It builds on point search and equation solving, steps shared by existing parameter-recovery methods.Source: §1 · p. 3
• The recovery is modular: it uses vectors already produced by parameter-recovery attacks in raw-output and hard-label settings, whose recovered vectors have identical geometry.Combined with parameter recovery, it recovers each layer's architecture before its parameters.Source: §1 · p. 3
• Every tested model's architecture was recovered exactly in the architecture-unknown setting across both feedback settings and upstream modes, with few additional queries and slightly more running time.The authors also present task-by-task visual evidence of recovering a deliberately architecturally diverse CNN.Source: §1 · pp. 3–4

2.1 Basic Definitions and Notation

• A CNN is described as convolutional layers followed by fully connected layers, with convolution, activation, and pooling operations in its convolutional portion.The final fully connected output layer is an affine transformation.Source: §2.1 · p. 4
• Model architecture is an ordered sequence of layer types, dimensions, and structural hyperparameters, including convolution and pooling choices.The listed choices include layer type, kernel size, and pooling type, window size, and stride.Source: §2.1 · p. 4

2.2 Parameter Recovery in CNNs

• Parameter extraction proceeds in two sequential phases: neuron-signature recovery, which includes point collection and equation solving, followed by neuron-sign recovery.The framework also includes clustering, sorting, and merging during neuron-signature recovery.Source: §2.2 · p. 5
• A convolutional kernel placement is treated as a neuron whose weight vector is nonzero only on its local receptive field.Stacking these weight vectors gives the convolution matrix.Source: §2.2 · p. 5
• Max pooling adds nonlinearity because it selects the maximum within a pooling receptive field, so prior methods account jointly for ReLU activations and pooling selections.A pooling switching point occurs when two responses in one pooling receptive field tie for the maximum.Source: §2.2 · pp. 6–7
• For critical or dual points, small input perturbations form a linear system whose solution gives a recovered vector.The vector is recovered from the network's responses to positive and negative perturbations around the point.Source: §2.2 · p. 7

3.1 Structural Properties of Recovered Vectors

• A recovered vector is a convolution-matrix row or row difference, with support confined to the corresponding receptive fields; RPCPs, PSPs, and FCPs therefore have distinct sparsity patterns.RPCPs associated with the same receptive field, and PSPs associated with the same pair, share a sparsity pattern.Source: §3.1 · p. 8
• The effective position set of a recovered vector consists of its nonzero input indices, while a convolution-matrix row’s set identifies the input positions of its receptive field.The row’s effective-position set is independent of output channel because output channels share the same receptive fields.Source: §3.1 · p. 8
• For target-layer RPCPs and PSPs, input-channel slices have identical effective-position sets, so multiple input channels do not alter spatial sparsity.Recovered vectors associated with the same receptive field or pair of receptive fields are proportional on effective positions.Source: §3.1 · pp. 8–9

3.2 Auxiliary Definitions

• PSPs are classified as horizontal, vertical, or off-axis according to whether their two receptive-field starts differ in neither, one, or both spatial directions.Horizontal starts share a row, vertical starts share a column, and off-axis starts differ in both directions.Source: §3.2 · p. 9
• A spatial box encloses a vector’s effective positions, its shape records the box dimensions, and its effective mask marks covered positions within the box.Figure 2 illustrates these boxes and masks for an RPCP and three PSP types; its symbols distinguish effective, overlapping, and zero-valued positions.Source: §3.2 · pp. 9–10, Figure 2 · p. 10

4.1 Adversarial Goals and Assumptions

• The adversary is assumed to lack hidden-layer types and hyperparameters, while existing CNN extraction attacks recover parameters only when architecture is known.The stated assumptions include arbitrary full-domain inputs, 64-bit computation, and ReLU activations.Source: §4.1 · p. 10, Assumptions · p. 11
• The proposed framework combines architecture recovery and parameter recovery, decomposing architecture recovery into tasks for layer type, convolution, padding, channels, and pooling.The task decomposition is summarized in Table 1.Source: Goals · pp. 10–11, Table 1 · p. 11

4.2 Complete Cryptanalytic Extraction Framework

• Within a layer, the attack first shares the parameter-recovery pipeline’s initial steps, identifies whether the layer is convolutional or fully connected, then recovers the corresponding architecture information.For fully connected layers it recovers width using existing methods; for convolutional layers it handles Tasks 2–6 before connecting to parameter recovery.Source: New Framework · pp. 11–12, Goals · p. 11
• Recovered vectors are convolution-matrix rows or row differences supported on receptive fields, so their geometry supplies signatures for identifying layer types and convolution hyperparameters.RPCP box shape indicates kernel size, offsets between box starts encode stride, and truncated boundary boxes distinguish same from valid padding.Source: Core Idea of the Architecture Recovery · p. 12
• Pooling structure is inferred from recovered-vector patterns: max-pooling switches produce PSPs, while average pooling spreads coefficients evenly across a window and no pooling preserves sparse convolution structure.For average pooling, the spread extent indicates the window size.Source: Core Idea of the Architecture Recovery · p. 13
• Architecture recovery needs only a few PSPs or RPCPs for its qualitative signatures, making its additional query cost constant and independent of kernel parameter count.Parameter recovery instead requires enough PSPs per output channel to exceed that channel’s kernel parameter count.Source: Complexity of the Architecture Recovery · p. 13

5 Identifying the Target Layer Type

• Under the assumed ordering with convolutional layers before fully connected layers, an RPCP or PSP in the target or deeper layers identifies the target as convolutional.The criterion excludes the fully connected case because it and all deeper hidden layers would then be fully connected.Source: §5 · p. 13

5.1 Identifying Point Type

• Recovered-vector supports arising from convolutional layers are unions of target-layer receptive fields, including for deeper-layer RPCPs and PSPs.The union structure follows by projecting receptive fields recursively through intervening layers; pooling affects which positions are covered.Source: Identifying RPCPs · p. 14, Identifying PSPs · p. 15

5.2 Method for Layer-Type Identification

• A fully connected point has support over the full input space and a full-input box, whereas a point with a non-full box is an RPCP or PSP and indicates convolution.The decision rule declares the target convolutional when any retained point has a box different from the full input box.Source: Identifying FCPs · p. 15, §5.2 · p. 15

6 Recovering the Convolution Kernel Size and Stride

• Interior points exclude possible Same-padding effects, and the kernel-size rule uses receptive-field shape and the shortest contiguous nonzero boundary segment.The stride rule follows because convolution windows advance by the stride in both spatial directions.Source: §6 · p. 16

6.2 Method for Convolution Kernel Size and Stride Recovery

• Square receptive-field shapes and boundary-mask patterns identify kernel size, while stride is the minimum nonzero difference between retained window starts.When no RPCPs are available, PSP votes determine kernel size.Source: §6 · p. 16, §6.2 · p. 18

6.1 Identifying the Target Points

• RPCPs have smaller support than PSPs; filters use shape, boundary, and kernel-ratio consistency to separate target-layer points from deeper-layer points.Ratios of non-overlapping PSP entries cancel point-specific scaling and remain determined by the kernel.Source: Identifying Target-layer RPCP · pp. 17–18

7.1 Identifying the Truncated Point

• Boundary candidates are checked against a numerical template using boundary-entry equalities; matched candidates on any boundary establish the presence of truncation.The method repeats the check symmetrically for all four boundaries.Source: §7.1 · pp. 19–20

7.2 Method for Recovering the Padding Mode and the Output Channel Number

• A matched truncated point implies Same padding; if no truncated points are found, the method identifies Valid padding.Under Valid padding, no receptive field intersects a zero-padding region.Source: §7 · p. 19, §7.2 · p. 20
• The method clusters RPCPs, PSPs, and truncated points separately, then merges clusters by kernel values exposed in non-overlapping regions; the merged-class count gives the output-channel number.Source: §7.2 · p. 20

8.1 Recovering the Max-Pooling Window Size and Stride

• PSPs identify MaxPool; their window-start offsets and anchor positions reveal the max-pooling window size and stride.Anchor differences are interpreted in units of the recovered convolution stride.Source: §8 · p. 20, §8.1 · p. 21

8 Recovering the Pooling Window Size and Stride

• The method first checks for PSPs to identify MaxPool, then distinguishes AvgPool from NoPool using restored-vector structure.For average pooling, the pooled coordinates average positions in a pooling receptive field; the method skips that pooling operation and analyzes a subsequent layer's recovered vector.Source: §8 · p. 20, §8.2 · p. 22

8.1 Recovering the Max-Pooling Window Size and Stride

• Max-pooling anchors lie on a grid determined by the pooling stride, and the smallest nonzero anchor difference recovers that stride when adjacent anchors are retained.Source: §8.1 · p. 21

8.2 Recovering the Average-Pooling Window Size and Stride

• Average pooling is distinguished from no pooling by repeated constant blocks in restored vectors; segment lengths, gaps, and overlaps then determine its window size and stride.No pooling retains the sparse convolution structure rather than producing the repeated-block pattern.Source: §8.2 · p. 23

9.1 Summary of Experimental Results

• The evaluation combines in the raw-output setting and in the hard-label setting, testing layer-by-layer and end-to-end pipelines on two CNNs.The tested models use two convolutional layers with 5 × 5 kernels, stride 1, Valid padding, and 2 × 2 max pooling; they differ in output-channel numbers.Source: §9.1 · p. 24
• Every tested model and setting yielded exact layer architectures, and subsequent parameter recovery matched the original attacks' accuracy.Source: §9.1 · p. 24
• Architecture recovery analyzes vectors already produced by point search and equation solving, so its queries are a subset of parameter-recovery queries.For the raw-output (2 + 1) CNN, architecture recovery accounts for 217.71 queries and 24.08 seconds, compared with 220.48 queries and 27.31 seconds for complete parameter recovery.Source: §9.1 · p. 24

9.2 Visualizing the Architecture Recovery Process

• Recovered kernel sizes were 3, 5, and 3, strides 1, 1, and 2, and padding modes Same, Same, and Valid across the three layers, matching ground truth.The figures show kernel-size votes, window-start distributions, and truncated boundary points, respectively.Source: §9.2 · p. 25, Figure 5 · p. 26, Figure 6 · p. 26
• The three layers' output-channel counts matched ground truth, and their pooling structures were recovered as NoPool, AvgPool (2, 2), and MaxPool (3, 2).The figures visualize cluster formation and restored-vector or max-pooling evidence for these tasks.Source: §9.2 · pp. 26–27, Figure 8 · p. 27, Figure 9 · p. 27

• The attack recovers convolutional-layer architecture before parameters, and completes CNN cryptanalytic extraction when combined with methods for fully connected layers.Source: §10 · pp. 27–28
• Recovered vectors’ spatial geometry reveals architecture through sparsity, numerical, and pooling-structure consistency.These signals recover layer type, kernel size and stride, padding mode, output-channel number, and pooling structure.Source: §10 · p. 28
• Experiments across diverse CNNs, including purpose-built models and models used by existing attacks, verified the attack’s effectiveness.Source: §10 · p. 28
• The convolutional-layer approach complements methods that recover fully connected layer widths, together removing the architectural prior from CNN extraction.Source: §10 · p. 28