Source-linked AI summary
Training-Adaptive Convolutional Sparse Coding via Information Bottleneck for Robust Visual Representation
Meng'en Qin, Yinchen Liu, Mingxuan Cui, Youlu Xing
TL;DR
The paper addresses the lack of explicit control over information trade-offs in intermediate representations, which can hinder robustness under perturbations. It makes the CSC sparsity coefficient trainable and adds label-free post-training adaptation; experiments report competitive clean recognition and improved robustness across corruption types.
Problem
Deep networks typically optimize final task loss without explicitly controlling intermediate representations’ information trade-off, and imbalance between compactness and sufficiency can hinder robustness under input perturbations.
Source: §1 · p. 1Method
TA-CSC unfolds FISTA and learns the sparsity coefficient λ jointly with network parameters, then supports label-free post-training λ adaptation for corrupted inputs.
Source: §1 · p. 2, §2.2 · p. 2Results
TA-CSC reports competitive clean recognition on CIFAR-10, CIFAR-100, and ImageNet, while label-free λ adaptation further improves robustness across all corruption types.
Source: §3.1 · p. 3, §3.2 · p. 3Takeaways & Limitations
As corruption severity increases, TA-CSC-18 imposes stronger sparsity constraints, consistent with λ controlling suppression of task-irrelevant components.
Source: §3.2 · p. 4Evaluation is mainly on ResNet classification, and the relationship between learned λ and information compression is supported primarily by empirical evidence.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Visual signals require compact yet sufficient representations for robust downstream prediction. Convolutional sparse coding (CSC) provides an explicit mechanism for suppressing redundant components while preserving signal content, but its sparsity coefficient is typically fixed and manually selected. We propose a training-adaptive convolutional sparse coding framework for robust visual signal representation. Specifically, we unfold the CSC optimization with the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) and treat the sparsity coefficient as a differentiable variable jointly learned with the network parameters. From the information bottleneck perspective, this coefficient controls the trade-off between information retention and compression: the sparsity term promotes compact representations, while the reconstruction term together with task loss preserves task-relevant signal content. We further introduce a label-free post-training strategy that adjusts the compression strength for corrupted inputs with the main network parameters fixed. Experiments on CIFAR and ImageNet demonstrate competitive clean-data recognition and greatly improved robustness under different input perturbations.
1. INTRODUCTION
Robust visual representations should retain task-relevant information while discarding irrelevant input, but deep networks do not explicitly control this trade-off in intermediate layers. The paper presents training-adaptive convolutional sparse coding as a way to learn compression strength jointly with the representation and downstream task.
1. INTRODUCTION
- The information bottleneck objective seeks representations that retain information relevant to downstream variable Y while discarding input information irrelevant to Y.I(T; X) measures retained input information, I(T; Y) measures task-relevant information, and β controls their trade-off.Source: §1 · p. 1
- Imbalance between compactness and sufficiency can hinder robustness under input perturbations.Deep networks typically optimize the final task loss without explicitly controlling the information trade-off of intermediate representations.Source: §1 · p. 1
- Convolutional sparse coding balances reconstruction fidelity with representation complexity, and its sparsity coefficient λ controls compression strength.The reconstruction term preserves the input signal while the ℓ1 term suppresses unnecessary parameters; λ and the IB coefficient β play analogous roles in balancing compression and preservation.Source: §1 · p. 1
- The proposed TA-CSC framework makes the sparsity coefficient λ differentiable within unfolded FISTA iterations and learns it jointly with the convolutional dictionary and network parameters.It also enables label-free post-training adaptation of λ under corruption while keeping other parameters fixed.Source: §1 · p. 2
2. METHOD
The method unfolds CSC inference with FISTA and learns each layer’s sparsity coefficient through differentiation of the iterations. It also adapts compression after training using unlabeled corrupted inputs while keeping the main network parameters fixed.
2.1. Sparse Inference via FISTA
- Unfolded FISTA outputs each CSC layer’s sparse representation after a fixed number of iterations.The layer output is T_n = F_{λ_n}(T_{n−1}; D_n).Source: §2.1 · p. 2
2.2. Training-Adaptive Convolutional Sparse Coding
- Differentiating unfolded FISTA with respect to λ_n makes the sparsity coefficient a learnable compression variable, jointly optimized with network parameters by backpropagation.The training objective balances task loss, which preserves task-relevant information, against a sparsity term that suppresses redundant components.Source: §2.2 · p. 2
2.3. Label-free Post-training Adaptation for Corrupted Data
- With network parameters fixed, post-training adaptation updates λ using unlabeled corrupted or shifted samples and a loss based on the observed signal and its sparse reconstruction.The loss encourages stronger sparsity while penalizing excessive compression that distorts the signal.Source: §2.3 · p. 3
3. EXPERIMENTS
The experiments evaluate TA-CSC on clean recognition, corrupted inputs, post-training adaptation, and learned λ dynamics. Results cover CIFAR-10, CIFAR-100, and ImageNet, with ablations examining FISTA iterations and adaptation-sample counts.
3. EXPERIMENTS
- The experiments use ResNet-18 and two TA-CSC variants, with two unfolded FISTA iterations and 100 unlabeled corrupted samples as default adaptation settings.TA-CSC-18 replaces the first convolutional layer; TA-CSC-18all replaces all convolutional layers.Source: §3 · p. 3
3.1. Classification Performance on Clean Data
- TA-CSC-18all reaches 97.65% on CIFAR-10, 80.76% on CIFAR-100, and 72.53% on ImageNet, outperforming other methods.These are clean-data results when all convolutional layers are replaced with TA-CSC layers.Source: Table 1 · p. 3, §3.1 · p. 3
3.4. λ Dynamic Behavior Analysis in Training
- During training, λ remains small early, then rises rapidly near accuracy saturation and converges; earlier layers generally have smaller λ than later layers.The learned profile also has four pronounced compression peaks, which the authors associate with ResNet feature-width expansions.Source: Figure 3 · p. 4, §3.4 · p. 4
3.2. Robustness Analysis on Corrupted Data
- Label-free post-training λ adaptation further improves robustness across corruption types, and adapted TA-CSC-18 surpasses SDNet-18 with per-sample λ tuning.Without adaptation, TA-CSC-18 also consistently outperforms ResNet-18 and other CSC baselines.Source: §3.2 · p. 3, Table 2 · p. 4
3.3. Ablation Analysis
- Increasing FISTA iterations gradually improves clean Top-1 accuracy, while increasing adaptation samples from 50 to 500 promotes robustness with relatively limited gains against added computational costs.The study therefore uses two FISTA iterations and 100 corrupted samples by default.Source: Table 3 · p. 4, §3.3 · p. 4
4. CONCLUSION
The paper presents information bottleneck-driven training-adaptive convolutional sparse coding, with FISTA-unfolded λ learning and label-free post-training adaptation for corrupted inputs. Its evaluation is mainly limited to ResNet classification, and the link between learned λ and information compression is primarily empirical.
4. CONCLUSION
- The framework makes λ a training variable through FISTA unfolding, jointly learning task-related representations and adaptive compression.It also adapts compression coefficients after training for corrupted inputs.Source: §4 · p. 4
- The study is mainly evaluated on ResNet architectures and classification tasks.Future work will consider more diverse visual tasks and distribution shifts.Source: §4 · p. 4
- The relationship between learned λ and information compression is supported primarily by empirical evidence.The authors propose establishing a more rigorous theoretical connection between sparse coding and the information bottleneck objective.Source: §4 · p. 4