PaperlayerView exact revision on arXiv

Source-linked AI summary

Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm

Meena Jagadeesan, Ilya Razenshteyn, Suriya Gunasekar

arXiv:2102.12238v4cs.LGstat.ML

TL;DR

The paper asks how controlling weight ℓ2 norm shapes the functions learned by convolutional networks. It defines a function-space regularizer through minimum-norm representations and analyzes its dependence on kernel size and channel counts. The results establish width invariance for single-channel inputs, characterize multi-channel behavior, and identify closed forms for special kernel sizes.

  • Problem

    The paper asks what functions are learned by controlling the ℓ2 norm of network parameters.

    Source: §1 · p. 2

  • Method

    The authors define an induced regularizer as the minimum weight norm needed to realize a function and characterize it for two-layer linear convolutional networks.

    Source: §1 · p. 2, §2 · p. 5, §3.1 · p. 7

  • Results

    For single-channel inputs, the induced regularizer is independent of output width for any kernel size; multi-channel inputs can require width for representation, and the special cases K = 1 and K = D have Fourier-coefficient norm characterizations.

    Source: §4.4 · p. 12

  • Takeaways & Limitations

    Kernel size and channel count shape the function-space inductive bias, while the paper’s analysis motivates a width-invariance hypothesis for gradient-descent-trained networks.

    Source: §6 · p. 14

    For multi-channel inputs, the SDP relaxation is not always tight, and realizing all linear functions can require sufficiently many output channels.

    Source: §4.3 · p. 12

Abstract

from arXiv · show

We provide a function space characterization of the inductive bias resulting from minimizing the $\ell_2$ norm of the weights in multi-channel convolutional neural networks with linear activations and empirically test our resulting hypothesis on ReLU networks trained using gradient descent. We define an induced regularizer in the function space as the minimum $\ell_2$ norm of weights of a network required to realize a function. For two layer linear convolutional networks with $C$ output channels and kernel size $K$, we show the following: (a) If the inputs to the network are single channeled, the induced regularizer for any $K$ is independent of the number of output channels $C$. Furthermore, we derive the regularizer is a norm given by a semidefinite program (SDP). (b) In contrast, for multi-channel inputs, multiple output channels can be necessary to merely realize all matrix-valued linear functions and thus the inductive bias does depend on $C$. However, for sufficiently large $C$, the induced regularizer is again given by an SDP that is independent of $C$. In particular, the induced regularizer for $K=1$ and $K=D$ (input dimension) is given in closed form as the nuclear norm and the $\ell_{2,1}$ group-sparse norm, respectively, of the Fourier coefficients of the linear predictor. We investigate the broader applicability of our theoretical results to implicit regularization from gradient descent on linear and ReLU networks through experiments on MNIST and CIFAR-10 datasets.

1 Introduction

The paper studies the function-space effects of controlling weight norms in convolutional networks, defining the induced regularizer as the minimum weight norm needed to realize a function. It analyzes how kernel size and channel counts shape that regularizer and tests a gradient-descent hypothesis empirically.

1 Introduction

  • The study asks what functions result from controlling parameter ℓ2 norm and treats the minimum norm required to realize a function as its induced regularizer.Weight-norm regularization of parameters is equivalent to regularization by the corresponding induced function-space measure.Source: §1 · p. 2

1.2 Related Work

  • Prior results show that fully connected and full-kernel convolutional linear networks induce different function-space penalties despite realizing the same linear functions.The cited comparison is between the ℓ2 norm of the linear map for fully connected networks and the ℓ1 norm of its Fourier coefficients for full-dimensional convolutional kernels.Source: §1 · p. 2, §1.2 · p. 3

1.1 Our contribution

  • For single-channel inputs, the induced regularizer is independent of output width for every kernel size; for multi-channel inputs, width can affect it, though invariance holds when C ≥ R·K.The authors conjecture invariance for multi-channel inputs when C ≥ R.Source: §1.1 · pp. 2–3

Experiments for gradient descent.

  • The experiments test whether gradient-descent solutions exhibit width-invariant ℓ2 complexity on MNIST and CIFAR-10, including linear and ReLU networks.The reported settings include linear convolutional networks with circular and zero padding, and ReLU networks on MNIST.Source: Experiments for gradient descent · p. 3

1.3 Notation

  • The paper uses zero-based indexing and defines complex-vector notation, the Euclidean norm, and circular convolution for its analysis.Circular convolution is defined for a length-K vector and a length-D vector, with K ≤ D.Source: §1.3 · p. 4

1.4 Multi-channel linear convolutional network

  • The network applies K-sized circular convolutions to a D-dimensional input across C channels, then combines the channels with a single linear output layer.The paper also represents convolution in the Fourier domain using the convolution theorem.Source: §1.4 · pp. 4–5, §1.3 · p. 4

2 Induced regularizer in the function space

The induced regularizer assigns a function the minimum weight norm of any network representation that realizes it. The paper characterizes this cost, including closed forms at endpoint kernel sizes and an SDP formulation for general kernels.

2 Induced regularizer in the function space

  • For any kernel size and output width, the induced regularizer is a norm and equals the value of a tight semidefinite-program relaxation.The paper also states that this regularizer is weakly decreasing as kernel size or output width increases.Source: §2 · p. 5
  • At K = D, the single-channel induced regularizer is twice the ℓ1 norm of the predictor’s Fourier coefficients, promoting Fourier-basis sparsity.At K = 1, the regularizer instead has an ℓ2 form and is basis-agnostic.Source: §2 · p. 5
  • The endpoint-kernel proofs do not extend directly to intermediate kernel sizes because the Fourier representation of the kernel weights is constrained.For K < D, the transformed weights lie in the K-dimensional span of the columns of F_K.Source: §2 · pp. 5–6
  • For K = 2, the Fourier characterization involves maximizing a high-degree rational function, without a clean closed-form solution.The resulting weights suggest a band-pass structure in which nearby Fourier components receive nearby weights.Source: §2 · p. 6

3 Main technical tool: SDP formulation of induced regularizer

The section develops an SDP representation of the induced regularizer and proves the relaxation tight for single-channel inputs. It then derives consequences for channel width, norm structure, patterned vectors, and comparisons with prior work.

3 Main technical tool: SDP formulation of induced regularizer

  • Dropping the rank constraint yields an SDP lower bound on the induced regularizer, and tightness makes that relaxation exact.The formulation lifts the parameters into a positive semidefinite matrix whose objective and constraints are linear in the lifted variable.Source: §3 · pp. 6–7

3.1 Tightness of the SDP Relaxation

  • For single-channel inputs, the SDP relaxation is tight for every K and C, so the induced regularizer depends on K but not C.The proof requires a rank-1 feasible solution matching the SDP optimum; a convolution lemma establishes that sums of two self-convolutions can be represented as one.Source: §3.1 · p. 7

3.2.1 RK,C is independent of number of output channels C

  • The authors hypothesize that single-channel predictors learned by gradient descent are invariant to output-channel count; their experiments also suggest this for some ReLU networks with bias.The preview uses overparameterized linear networks on MNIST digit classification with K = (3, 3) and C ∈ {1, 2, 4, 8}.Source: §3.2.1 · p. 8, Figure 1 · p. 9, Figure 2 · p. 9

3.2.2 RK,C is a norm

  • For single-channel inputs, the induced regularizer is a norm for any kernel size K and output-channel count C.The paper's proof of SDP tightness uses a polynomial representation of convolutions to establish the needed rank-1 solution.Source: §3.2.2 · p. 8, §3.1 · p. 8
  • At K = 1 and K = D, the norm has explicit forms; for intermediate K, it interpolates between Fourier-coefficient ℓ2 and ℓ1 norms.The endpoint formulas are R1,C(w) = 2√D∥ŵ∥2 and RD,C(w) = 2∥ŵ∥1; when K is small relative to D the norm is close to the former, and when K is comparable to D it is close to the latter.Source: §3.2.2 · p. 9

3.3 RK,C for patterned vectors

  • Repeated patterns in w(p) encode periodic-translation invariance, and their induced regularizer is closely related to the cost of realizing the pattern itself.The paper states separate relations for K ≤ P and for K ≥ P when K is an integer multiple of P.Source: §3.3 · p. 9

3.3.1 Connection to previous work

  • Prior work's width threshold for SDP equality can depend on dataset size, whereas this result gives channel-count independence for every C ≥ 1, regardless of dataset or loss.The comparison concerns single-channel-input linear convolutional networks.Source: §3.3.1 · p. 10

4 Networks with multi-channel inputs

For multi-channel inputs, the induced regularizer can depend on output-channel count because some matrix-valued linear maps require multiple output channels. The paper derives an SDP relaxation, proves its tightness at sufficient width, and gives closed-form norms for two kernel sizes.

4 Networks with multi-channel inputs

  • The multi-channel analysis represents inputs with R channels as matrices and defines the induced complexity over matrix-valued linear predictors.The first layer uses R sets of kernel weights, and the second layer maps to a single output.Source: §4 · pp. 10–11

4.1 Role of output channel size C

  • For multi-channel inputs, realizing every linear map requires K·C ≥ min{R, D}, so the model class and induced regularizer can depend on C.Each output channel contributes a matrix of rank at most K.Source: §4.1 · p. 11

4.3 Tightness of SDP Relaxation

  • For R > 1, the SDP relaxation is not always tight; C ≥ R·K is sufficient, while tightness for C ≥ R is conjectured.The authors note that the sufficient bound is not sharp.Source: §4.3 · p. 12

4.2 SDP relaxation for multi-channel input networks

  • The SDP relaxation is a lower bound on the induced regularizer and is equivalent to it when C ≥ R·K.The relaxation represents the objective and constraints using a rank-C positive semidefinite matrix.Source: §4.2 · pp. 11–12, §4.3 · p. 12

4.4 Induced regularizer when K = 1 and K = D

  • For K = 1, the induced regularizer is a scaled nuclear norm when C ≥ min{R, D}.For K = D, it is a Fourier-coefficient ℓ2,1 group-sparse norm when C ≥ 1, encouraging similar weighting across channels and sparsity across frequencies.Source: §4.4 · p. 12

5 Experiments

The experiments test whether the induced regularizer of gradient-descent predictors is invariant to output-channel count, as suggested by the paper’s theory. Across the reported linear-network settings, the observed pattern depends on input channels; MNIST ReLU experiments also show invariance.

Experiments

  • The tested hypothesis predicts invariance of the induced regularizer for normalized predictors when C ≥ R on separable binary classification tasks.The experiments evaluate this hypothesis on small balanced MNIST and CIFAR-10 subsets, mainly using linear convolutional networks trained with stochastic gradient descent.Source: §5 · p. 13
  • The experiments approximate the induced regularizer using trained weight norms, which theoretically provide only an upper bound.The authors report that this approximation was quite accurate in checked instances where the regularizer has a closed form.Source: §5 · p. 13
  • For single-channel MNIST linear networks, estimated regularizers and predictors were nearly invariant to C; for three-channel CIFAR-10, regularizers differed below C = 3 and were invariant once C ≥ 3.For MNIST, the comparisons used K ∈ {1, 3, 8, 16, 28} and C ∈ {1, 2, 4, 8}; for CIFAR-10, K ∈ {1, 3, 8, 20} and C ∈ {1, 2, 3, 4, 8}.Source: §5 · p. 14
  • On MNIST, the estimated induced regularizer was invariant to C for tested ReLU networks, with and without bias parameters.The authors present this as support beyond their theoretical study of linear networks.Source: §5 · pp. 13–14

6 Discussion

The discussion highlights that width and kernel size affect the induced regularizer even in two-layer linear convolutional networks. The results also motivate a hypothesis about gradient-descent training, which the experiments support.

6 Discussion

  • Width and kernel size have interesting effects on the induced regularizer even in two-layer linear networks.The authors also report experimental support for a broader hypothesis about output-channel count in networks learned with gradient descent.Source: §6 · p. 14
  • The authors identify SDP tightness for multiple input channels, gradient-descent limits, pooling, multiple layers, and nonlinearity as future-work directions.Source: §6 · p. 14

A Proof of Theorem 5: SDP tightness

This section proves Theorem 5’s SDP tightness for every kernel size, channel count, and predictor by reducing it to a convolution identity and establishing the needed factorization with polynomial and root constructions. It also summarizes the proof strategy and states the single-channel regularizer in Fourier coordinates.

A Proof of Theorem 5: SDP tightness

A.1 KKT conditions for (SDP)

  • The KKT argument reduces tightness to finding u in R^K satisfying a convolution reduction, which follows by repeatedly applying Lemma 6.Lemma 6 states that for any 1 ≤ K ≤ D and a, b in R^K, some c in R^K satisfies a ⋆ a + b ⋆ b = c ⋆ c, with convolutions in dimension D.Source: §A.1 · pp. 18–19

A.2 Proof of Lemma 6: the convolutional property

  • For K = D, Lemma 6 has a direct Fourier-space construction, but for K < D the vector c lacks an apparent closed form, motivating an implicit construction.The proof technique for the general case may be of independent interest.Source: §A.2 · p. 19

A.2.1 Reducing Lemma 6 to D = 2K −1 Case

  • Lemma 6 for general K and D follows from its special case D = 2K − 1.The same vector c from the special case satisfies the convolution identity for general D.Source: §A.2.1 · p. 19

A.2.2 Introducing the polynomial representation

  • For D = 2K − 1, convolution is represented by polynomial multiplication, with a ⋆ a corresponding to x^(K−1) p_a(x) p_a(1/x), up to coefficient permutation.The target is to construct p in P_(K−1) so that its corresponding polynomial represents a ⋆ a + b ⋆ b; its coefficients yield c.Source: §A.2.2 · pp. 20–21

Step 1. Properties of the complex roots of Q(x)

  • The polynomial Q(x) is real and palindromic, and every unit-modulus root has even multiplicity because Q has the special sum-of-convolutions structure.Palindromicity alone is insufficient: x^2 + 1 is a counterexample, while the even-multiplicity property follows from Lemma 16.Source: §A.2.3 · p. 21, Step 1. Properties of the complex… · p. 22

Step 3. Proof that Sp is the correct multi-set of roots

  • The next proof step is to show that the selected root multiset yields a polynomial p of degree at most K − 1 whose reciprocal product matches Q, then choose a real leading coefficient.The supplied passage outlines this step but does not provide its proof.Source: §A.2.3 · p. 21

Step 2: Constructing the roots of Sp

  • The construction selects half the roots of Q for p by pairing real roots as (α, 1/α) and complex roots in quadruples (α, α, 1/α, 1/α), then includes all zero roots.The root properties used include conjugate pairing, reciprocal pairing for nonzero roots, and even multiplicity for roots on the unit circle.Source: Step 2: Constructing the roots of Sp · pp. 22–23

Step 3. Proof that Sp is the correct multi-set of roots

  • The constructed root multiset has K −1 elements, conjugate-paired complex roots, and yields the same polynomial roots with multiplicities as Q(x).Palindromic root counting handles zero roots, while paired real and complex roots make the nonzero roots match.Source: Step 3. Proof that Sp is the correct… · pp. 23–24

Step 4. Constructing the leading coefficient of p

  • Choosing cp = √γ gives the desired factorization, and coefficient norms show γ > 0, so p has real coefficients and degree K −1.The proof compares the x^(K−1) coefficients, which are the squared norms of the original vectors and the monic polynomial’s coefficient vector.Source: Step 4. Constructing the leading… · pp. 24–25

A.3 Discussion of the proof technique

  • The proof reduces via KKT conditions to CONV-REDUCTION and Lemma 6; its core technical step is Lemma 15 for D = 2K −1.Because c lacks a clean closed-form construction, the proof builds it implicitly from polynomial roots and uses Lemma 16’s additional structure to ensure real coefficients.Source: §A.3 · p. 25

B Proofs for Section 2: Induced regularizer in special cases

  • For C = 1, the unitary Fourier transform preserves the ℓ2 norms, and RK,1(w) minimizes ∥bU∥² + ∥bV∥² subject to bU ⊙ bV = bw.The proof uses the signal-domain and Fourier-domain norms interchangeably for U, V, and w.Source: B Proofs for Section 2: Induced… · p. 25

B.1 Proof of Lemma 3

The appendix develops proofs and bounds for the induced regularizer, including its norm property and the dual SDP formulation. It also derives an upper-bound construction by decomposing vectors into blocks.

B.1 Proof of Lemma 3

  • For K = 1, the proof treats the first-layer weight as a scalar and uses AM-GM in its argument.Source: §B.1 · p. 25

B.2 Induced Regularizer for K = 2

  • For K = 2, equal-norm rescaling gives an equivalent objective 2∥U∥·∥bV∥, and the proof minimizes over a scalar parameter α.Source: §B.2 · pp. 26–27

C.1 Proof of Corollary 7

  • The induced regularizer RK,C(w) is a norm for every K and every output-channel count C.Source: §C.1 · p. 27

C.2 The dual formulation of SDP

  • The dual SDP satisfies strong duality, and the dual formulation supports general bounds on the induced regularizer.Source: §C.2 · p. 28

C.3 Proof of Lemma 8

  • The upper-bound proof splits a vector into disjoint K-sized blocks and bounds the regularizer using the contributions of those blocks.Source: §C.3 · pp. 29–30

C.4 Proof of Lemma 9

Lemma 9 treats periodic vectors and gives results for different relationships between kernel size K and period P. The appendix also describes block structure for matrices in the multi-input SDP.

C.4 Proof of Lemma 9

  • For periodic vectors, Lemma 9 gives all-C results for K ≤ P and for K ≥ P when K is an integer multiple of P.Source: §C.4 · p. 30
  • The proof constructs an upper bound using a feasible single-channel factorization and a lower bound using a dual feasible vector.Source: §C.4 · pp. 30–31

D Appendix for Section 4: Networks with multi-channel inputs

D.1 Realizability of linear functions

For multi-channel inputs, output-channel count can constrain which linear maps the network can realize. The section gives necessary and sufficient channel-count bounds, while noting that their tight characterization remains open.

D.1 Realizability of linear functions

  • K·C ≥ min{R, D} is necessary for the network to realize all linear maps in the multi-channel input space.Each output-channel subnetwork realizes a matrix of rank at most K.Source: §D.1 · p. 33
  • C ≥ min{R, D} is sufficient for realizing every W ∈ R^{D×R}, but is not necessary.The tightest necessary and sufficient conditions remain an open question.Source: §D.1 · p. 34
  • For K = 1 and K = D, the SDP is tight once C is large enough to realize all linear functions, and the induced regularizer has a closed form.The supplied passages establish these special cases, not a closed form for general K.Source: §D.1 · p. 34

D.2 Proof of Lemma 11

When C ≥ RK, the induced regularizer equals the SDP value. The proof establishes this by showing that an optimal SDP solution has rank at most RK.

D.2 Proof of Lemma 11

  • For any W, C ≥ RK suffices for the induced regularizer to equal the SDP value.The rank-constrained formulation is equivalent to the SDP when an optimal solution has rank at most RK.Source: §D.2 · p. 34
  • The proof shows that every optimal SDP solution has rank at most RK, which yields the equality for C ≥ RK.It constructs a spanning set for the relevant column space with at most RK elements.Source: §D.2 · pp. 34–35
  • KKT conditions provide the key ingredient for constructing the bounded spanning set used in the rank argument.The proof uses the KKT conditions to build that set from vectors associated with each input channel.Source: §D.2 · pp. 34–35

D.3 Proofs of Theorems 13-14: induced regularizer for K = 1 and K = D

For the special kernel sizes K = 1 and K = D, the paper derives closed-form induced regularizers. The K = 1 result is a scaled nuclear norm under a channel-count condition, while the K = D result is a scaled ℓ2,1 norm for any channel count.

D.3 Proofs of Theorems 13-14: induced regularizer for K = 1 and K = D

  • For K = 1 and C ≥ min{R, D}, the induced regularizer is a scaled nuclear norm.The proof connects the network formulation to the variational characterization of the nuclear norm.Source: §D.3 · p. 35
  • When K = 1, C ≥ min{R, D} is sufficient to realize any linear predictor W ∈ R^{D×R}.The Fourier-domain factors are otherwise unconstrained, allowing them to realize any rank-min{R,D} matrix.Source: §D.3 · p. 35
  • For K = D, the proof matches an upper bound with a lower bound from the SDP dual to obtain the stated regularizer.The upper bound uses a single output channel; the lower bound constructs feasible dual variables.Source: §D.3 · pp. 36, 38
  • For K = D, the induced regularizer is 2∥cW∥_{2,1} for any C.A single output channel suffices to realize every linear predictor in this full-kernel case.Source: §D.3 · pp. 36, 38

E Experiments for gradient descent

Experiments compare channel counts, kernel sizes, and padding choices in linear convolutional networks, then test whether channel invariance extends to ReLU networks. Across the reported settings, estimated regularizers are largely invariant to output-channel count once the input-channel condition is met, while kernel size affects frequency-domain sparsity.

E Experiments for gradient descent

  • Experiments use binary MNIST and CIFAR-10 classification, circular padding, and predictors normalized to unit training margin before comparison.MNIST uses digits 0 and 1; CIFAR-10 uses automobile and dog.Source: E Experiments for gradient descent · p. 38

E.1 Impact of the number of channels on MNIST

  • On MNIST, estimated regularizers differ minimally across output-channel counts for each kernel size, supporting channel invariance for single-channel inputs.Table 1 reports means and standard deviations over 10 trials; the reported differences are usually smaller than the fixed-channel standard deviation.Source: §E.1 · p. 39, Table 1 · p. 39
  • On MNIST, learned linear predictors appear invariant across channel counts for K < D, while K = D shows differences attributed to multiple minimizing predictors.Figure 3 displays signal- and Fourier-domain predictors across channel counts and kernel sizes.Source: Figure 3 · p. 39, §E.1 · p. 40

E.1.1 Non-linear networks with ReLU activation

  • For ReLU MNIST networks, estimated representation costs are consistent across channel counts, suggesting learned predictors may also be channel-invariant beyond the linear theory.This pattern is reported for networks both with and without biases; the estimate is an upper bound, and costs exclude bias magnitudes.Source: §E.1.1 · p. 40

E.2 Impact of the number of channels on CIFAR-10

  • For CIFAR-10, estimated regularizers vary minimally across channel counts when C ≥3, but differ nontrivially at C = 1, matching the predicted dependence below the input-channel count.The reported comparison uses linear networks on three-channel inputs.Source: §E.2 · p. 41, Table 4 · p. 42

E.2.1 Convolutions with zero padding

  • With zero padding on CIFAR-10, the authors report that Hypothesis 1 still holds, although this padding setting is outside the theoretical results.Table 5 compares estimated regularizers across output-channel counts and kernel sizes.Source: §E.2.1 · p. 42, Table 5 · p. 42

E.3 Varying kernel sizes

  • The kernel size can have a significant impact on predictors, and this observation aligns with the authors’ theoretical findings.Source: §E.3 · p. 42

E.3.1 Effect of kernel size on MNIST

On MNIST, larger kernels are associated with increasingly sparse Fourier-domain predictors; at large kernels, the predictor also develops repeated patterns in the signal domain. The explicit K=1 and K=D solutions show the same contrast between no frequency sparsity and significant sparsity.

E.3.1 Effect of kernel size on MNIST

  • The explicit K=(1,1) solution has no frequency sparsity, whereas the K=(D,D) solution has significant sparsity.The section describes these as minimum ℓ2 and minimum ℓ1 solutions, respectively.Source: §E.3.1 · pp. 42, 46
  • The reported R_K,C values are 2.10 for K=1 and 9.32 for K=D, but the gradient-descent values have not converged to these values.The text attributes the mismatch to slow convergence and notes that the minimum ℓ1 solution may be non-unique.Source: §E.3.1 · p. 46
  • K=(28,28) gradient-descent predictors are highly sparse in the frequency domain, while K=(1,1) predictors are not and K=(3,3) begins to show sparsity.This follows the predicted trend that larger kernels promote frequency sparsity.Source: §E.3.1 · pp. 42, 46
  • With zero-padding to 112 × 112, increasing K expands the predictor's nonzero region; large-kernel predictors show repeated patterns suggestive of restricted periodic translation invariance.The pattern shift size aligns with the size of the repetitions.Source: §E.3.1 · p. 46, Figure 8 · p. 46

E.3.2 Effect of kernel size on CIFAR-10

On CIFAR-10, gradient-descent predictors show greater frequency sparsity at larger kernels, with nontrivial sparsity already visible at K=(3,3). They also show similarities across input channels, which appear to diminish as K grows.

E.3.2 Effect of kernel size on CIFAR-10

  • The theoretical comparison is between a nuclear-norm regularizer at K=1 and an ℓ2,1 regularizer at K=D, both favoring similarities across input channels.The text describes the similarity effect as intuitively stronger for K=1 because the nuclear norm is closely related to rank.Source: §E.3.2 · p. 46
  • For K=D, the explicit minimum ℓ2,1 solution favors similarity across input channels at the expense of greater frequency-domain sparsity.The comparison also includes the optimal ℓ1,1 solution.Source: §E.3.2 · p. 47, Figure 9 · p. 47
  • The minimum ℓ2,1 solution has R_D,C=82.85, versus 114.85 for the gradient-descent predictor's induced regularizer; convergence is slow.The text says the qualitative findings apply to gradient descent despite the limiting value not being reached.Source: §E.3.2 · p. 47
  • Higher kernels favor frequency-domain sparsity in gradient-descent predictors, with nontrivial sparse structures visible even at K=(3,3).Figure 10 compares K values from (1,1) through (32,32) for C=3.Source: §E.3.2 · p. 47, Figure 10 · p. 48
  • The learned K=1 predictor has greater similarity across input channels than the explicit minimum ℓ2 (Frobenius) predictor; channel differences appear to grow with larger K.Some degree of cross-channel similarity remains visible at other kernel sizes.Source: Figure 10 · p. 48, §E.3.2 · p. 49

Related papers

Loading 2102.12238v4…