Resolution Allocation with Learnable Resizing and Spectral Filter Parameterization

1. INTRODUCTION

• Prior work studies learnable resolution mainly through predictive performance, not its interaction with filter parameterization.Spectral filters offer control over spatial extent, and DNArch conjectured that generated kernels may be easier to construct at lower resolutions; that interaction had not been tested.Source: §1 · p. 1
• Spectral filters gain more than spatial filters in all four cropped-resizing settings, with the test-set difference spanning +0.013 to +0.028.The paper reports this pattern across three datasets and two model sizes; Fig. 1 shows held-out test-accuracy gain relative to the fixed schedule at p⋆= 0.12.Source: §1 · p. 1, Figure 1 · p. 1
• The spectral-filter advantage remains positive against a fixed schedule with matched retained area, though it is smaller.The paper also reports that the difference decreases with model size and varies by resizing implementation; full-resolution soft masking yields a larger but more computationally expensive benefit than cropping.Source: §1 · p. 1

2. RELATED WORK

• These methods optimize resolution or scale but do not study its interaction with filter parameterization.Examples include DiffStride, Resolution Layers, Shape Adaptor, and DynOPool.Source: §2 · p. 2
• DNArch [3] jointly learns frequency-domain downsampling and MLP-generated kernels but does not test whether filter parameterization changes the benefit of learned resolution.The paper also situates its resizing implementations alongside Resolution Layers [2] and DiffStride [1], and notes aliasing and the broader DARTS framework.Source: §2 · p. 2

3.1. Complexity Budget

• The complexity measure sums squared effective DCT cutoffs across resize layers and normalizes by the full-resolution value.Effective resolution at a layer is the minimum cutoff encountered so far, because discarded resolution cannot be recovered; C/Cfull is a proxy for retained spatial area, not total end-to-end FLOPs.Source: §3.1 · p. 2
• Training combines categorical cross-entropy with a quadratic penalty that pulls realized complexity toward a target fraction p⋆.The penalty weight is λ = 100; main comparisons use p⋆= 0.12, with seven budgets from 0.08 to 0.50 also swept.Source: §3.1 · p. 2

3.2. Learnable Resizing

• Each of four resize layers learns a scalar cutoff, and a sigmoid frequency mask makes the effective resolution differentiable.Each layer applies a 2D DCT, multiplies coefficients by a mask, and applies an inverse DCT; the fixed, soft, and crop implementations differ in their handling of the spectrum and feature-map size.Source: §3.2 · p. 2
• Crop keeps coefficients whose mask values are not below 0.01, shrinking feature maps while leaving cutoff gradients essentially unaffected.Its retained area averages about 0.24 for both kernel types, roughly twice the fixed-schedule budget value; the soft implementation uses about 6.5× the forward FLOPs and 2.5–3.6× the training time per step of fixed.Source: §3.2 · p. 2

3.3. Spectral Kernels

• The comparison uses standard 3 × 3 spatial convolutions and SpecMLP spectral kernels generated from frequency coordinates.SpecMLP outputs real and imaginary kernel values that multiply the input's real FFT before an inverse transform; the studied kernel replaces the first-stage convolution, with model widths and parameter counts differing by kernel type.Source: §3.3 · p. 2

4.1. Main Result: Learnable Downsampling Gains

• ∆gain was +0.013 to +0.028 on held-out test sets, with spectral kernels gaining more than spatial kernels across all four crop settings.Table 1 reports gains at p⋆=0.12.Source: Table 1 · p. 3, §4.1 · p. 3
• +0.007 to +0.018 validation ∆gain came from increasing fixed retained area; at matched area, learned allocation still gave spectral kernels ∆gain = +0.006 to +0.015 in every setting, significant in two of four.Table 2 includes the matched-area comparison.Source: Table 2 · p. 3, §4.1 · p. 3

4.2. Effect of the Resizing Implementation

• Soft resizing produced the largest ∆gain (+0.020 to +0.049) at about 6.5× fixed-schedule forward FLOPs.The tighter ramp crop yielded smaller ∆gain, significant only for larger-model EuroSAT; the CIFAR-10 threshold sweep ranged from +0.012 to +0.019.Source: §4.2 · p. 3

4.3. Other Factors

• A 100-epoch CIFAR-10 run retained significant test ∆gain of +0.014, versus +0.018 at 30 epochs.A SIREN kernel’s +0.014 gain fell between 3 × 3 (+0.007) and SpecMLP (+0.027), with the ordering significant on validation but not test; a fixed high-ratio schedule also reproduced a similar EuroSAT ∆gain without learning.Source: §4.3 · pp. 3–4

4.4. Budget and Model Size

• Under soft resizing, EuroSAT ∆gain peaked at about +0.026 for p⋆≤0.15, then fell to +0.011 and +0.004 at p⋆=0.50; all 14 points were significant.At 1.8×10^5 parameters, test-set differences were +0.020, +0.022, and +0.008 across the reported settings; under crop, the budget trend was not significant at p⋆=0.25 or 0.50. Absolute accuracy is limited by small backbones and short training.Source: §4 · p. 3, Figure 2 · p. 4, §4.4 · p. 4

4.5. Learned Allocation and Kernel Size

• On EuroSAT under soft resizing, spectral filters learned s1/s2 = 2.09 ± 0.54 versus 1.34 ± 0.11 for spatial filters, and the preference shifted with the spectral kernel’s stage.The allocation pattern was only weakly present on other datasets, so the authors do not treat it as a mechanism; median SpecMLP filters placed 83% of energy within 3 × 3, with no effective-area difference between fixed and soft.Source: Figure 3 · p. 4, §4.5 · p. 4
• Across three initializations, larger EuroSAT models retained positive test-set ∆gain (+0.021, +0.016, and +0.015) and a larger learned spectral first-to-second-stage cutoff ratio.Source: §4.5 · p. 4

• Spectral filters show larger gains than 3 × 3 spatial filters under learned resolution allocation across three datasets and two model sizes.Source: §5 · p. 4
• The effect is modest, largest for small models under tight budgets, and weakens with model size while depending on resizing implementation.Source: §5 · p. 4
• Crop retains about twice the area of the fixed schedule, explaining part—but not all—of the observed gain.Source: §5 · p. 4
• The study tests relatively small models, one backbone, and one spectral parameterization in a single layer.Source: §5 · p. 4
• The remaining gain difference is not fully explained; measuring how spectral filters use frequencies above the cutoff is a proposed next step.Source: §5 · p. 4