Source-linked AI summary
UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition
Xiaohan Ding, Yiyuan Zhang, Yixiao Ge, Sijie Zhao, Lin Song, Xiangyu Yue, Ying Shan
TL;DR
Large-kernel ConvNets still lack dedicated architectural principles, and their ability to perceive across non-visual modalities remains uncertain. UniRepLKNet introduces four design guidelines and uses modality-specific preprocessing with a shared backbone. It delivers leading image-recognition results and strong audio and time-series performance, including 98.5% accuracy on Speech Commands V2.
Problem
Existing large-kernel ConvNets largely follow conventional ConvNet or transformer designs, and their universal perception ability beyond vision remains under-investigated.
Method
UniRepLKNet designs large-kernel ConvNets with four guidelines and applies modality-specific preprocessing so one architecture can process multiple modalities.
Results
UniRepLKNet achieves leading image-recognition performance and strong cross-modal results, including 98.5% accuracy on Speech Commands V2 without pretraining.
Takeaways & Limitations
Large-kernel ConvNets show universal perception across image, audio, video, point-cloud, and time-series tasks with a unified architecture.
Takeaways & Limitations
Applications to large vision-language models, cross-attention scenarios, and generation tasks remain under-explored.
Abstract
from arXiv · showhide
Large-kernel convolutional neural networks (ConvNets) have recently received extensive research attention, but two unresolved and critical issues demand further investigation. 1) The architectures of existing large-kernel ConvNets largely follow the design principles of conventional ConvNets or transformers, while the architectural design for large-kernel ConvNets remains under-addressed. 2) As transformers have dominated multiple modalities, it remains to be investigated whether ConvNets also have a strong universal perception ability in domains beyond vision. In this paper, we contribute from two aspects. 1) We propose four architectural guidelines for designing large-kernel ConvNets, the core of which is to exploit the essential characteristics of large kernels that distinguish them from small kernels - they can see wide without going deep. Following such guidelines, our proposed large-kernel ConvNet shows leading performance in image recognition (ImageNet accuracy of 88.0%, ADE20K mIoU of 55.6%, and COCO box AP of 56.4%), demonstrating better performance and higher speed than the recent powerful competitors. 2) We discover large kernels are the key to unlocking the exceptional performance of ConvNets in domains where they were originally not proficient. With certain modality-related preprocessing approaches, the proposed model achieves state-of-the-art performance on time-series forecasting and audio recognition tasks even without modality-specific customization to the architecture. All the code and models are publicly available on GitHub and Huggingface.
1. Introduction
UniRepLKNet addresses under-explored large-kernel ConvNet architecture and tests whether a unified ConvNet can generalize beyond vision. Its guidelines separate receptive-field expansion, spatial-pattern extraction, and representational capacity, supporting strong results across image, time-series, and audio tasks.
- Existing large-kernel ConvNets largely inherit conventional ConvNet or transformer architectures, leaving their architectural design under-explored.
- Large kernels provide wide receptive fields without requiring deep stacks, while small kernels can efficiently extract higher-level local spatial patterns.
- Four guidelines use efficient depth-increasing structures, Dilated Reparam Blocks, task-dependent kernel sizes, and 3×3 convolutions when scaling depth.
- UniRepLKNet achieves leading performance on ImageNet, ADE20K, and COCO while outperforming existing large-kernel ConvNets and recent architectures in accuracy and efficiency.
- UniRepLKNet applies modality-specific preprocessing but keeps one backbone architecture across audio, video, point cloud, and time-series tasks.
- UniRepLKNet outperforms a state-of-the-art transformer customized for global temperature and wind-speed forecasting, expanding large-kernel ConvNet applicability beyond vision.
2. Related Work
Prior work explored large kernels through early ConvNet designs, specialized spatial operators, and modern scaling or task extensions. However, contemporary large-kernel ConvNets generally retain architectures from other model families, leaving their own architectural principles insufficiently studied.
- Early ConvNets used 7×7 or 11×11 kernels in low-level layers, while GCN and LR-Net introduced very large or dynamic spatial aggregation operators.
- RepLKNet showed that enlarging kernels can improve performance, especially on downstream tasks, while largely following Swin Transformer architecturally.
- Recent work enlarged kernels further, generalized them to 3D scenarios, and applied them to tasks such as image dehazing and super-resolution.
- SLaK follows ConvNeXt, reinforcing that architectural design for very-large-kernel ConvNets remains under-explored.
3. Architectural Design of UniRepLKNet
UniRepLKNet organizes large-kernel ConvNets around four guidelines: efficient depth increases, dilated re-parameterization, task-dependent kernel placement, and small-kernel additions during depth scaling. The architecture preprocesses non-image modalities into embedding maps while retaining the image-model architecture.
- Architectural guidelines: The four guidelines combine efficient depth-increasing structures, dilated-kernel re-parameterization, task-dependent kernel placement, and small kernels when scaling depth.The guidelines use efficient structures for channel communication and spatial aggregation, place large kernels mainly in middle- and high-level layers, and add small-kernel blocks as depth increases.
- Dilated Reparam Block: Dilated small-kernel branches enhance a large-kernel convolution during training and can be merged into one non-dilated convolution for inference.A dilated kernel is equivalent to a larger sparse kernel, enabling the block to retain the extra branches without inference-time cost.
- Vanilla architecture: The vanilla model uses four stages with progressively expanded channels, ConvNeXt-like blocks, and 3×3 downsampling convolutions.The stages use channel widths C, 2C, 4C, and 8C; the default configuration sets C=96 and N=(3,3,9,3).
- Depth-increasing structures: +1.2 mIoU with 12% slow down is reported for an efficient bottleneck structure that increases depth.Removing the depthwise 3×3 convolution degrades performance, supporting the inclusion of nonlinear and spatial transformations.
- Kernel placement and scaling: Large kernels are most useful in later stages, while adding small-kernel blocks is favored over stacking more large-kernel blocks when increasing depth.Replacing later-stage 13×13 kernels with 3×3 kernels or reducing their size degrades results, whereas depth scaling experiments compare mixtures of LarK and SmaK Blocks.
- Beyond-image inputs: UniRepLKNet converts modalities into B × C′ × H × W embedding maps and otherwise reuses the image-oriented architecture without modality-specific customization.Time-series data are embedded and reshaped into a single-channel map; audio uses a 1 × T × F map, while point clouds use three-view projections.
4. UniRepLKNet for Image Recognition
UniRepLKNet achieves strong ImageNet recognition accuracy while maintaining high actual throughput, outperforming several recent ConvNet and transformer competitors.
- UniRepLKNet-A/F outperform ConvNeXt-V2-A/F by 0.8/0.6 in accuracy and run 19%/17% faster, respectively.Throughput is measured on the same A100 GPU with batch size 128.
- UniRepLKNet-P/N outperform FastViT-T12/S12 and ConvNeXt V2-P/N by clear margins.
- UniRepLKNet-S outperforms small- and base-level models in both speed and accuracy and runs almost as fast as InternImage-T.
- With ImageNet-22K pretraining, UniRepLKNet-S approaches RepLKNet-31L accuracy while running 3× as fast.
5. Universal Perception on other Modalities
UniRepLKNet generalizes beyond images, delivering state-of-the-art time-series forecasting and strong audio, video, and point-cloud results without modality-specific architectural customization.
- Time-series: 7.602, 1.832, 3.865, and 1.301 are UniRepLKNet’s lowest MSE and MAE errors for global temperature and wind speed forecasting.The model uses Corrformer’s embedding layer and decoder while replacing its encoder transformer with UniRepLKNet-S.
- Time-series: UniRepLKNet outperforms time-series specialists Pyraformer and Corrformer in both precision and efficiency.
- Audio: 98.5% accuracy is achieved on Speech Commands V2 without pretraining.UniRepLKNet uses fewer parameters than AST and Audio-MAE and outperforms previous audio ConvNets without structural customization.
- Video: 54.8% top-1 accuracy is reported on Kinetics-400, behind state-of-the-art architectures such as MViT.The model is evaluated as a generalist without pretraining.
- Kernel size: Table 13 compares universal perception performance across other ConvNets and UniRepLKNet with a smaller kernel.
- Point cloud: 93.2% overall accuracy and 90.3% mean accuracy surpass existing ConvNet-based models specialized for point-cloud analysis.
6. Conclusion
UniRepLKNet shows leading image-recognition performance and strong results on audio and time-series data, while its remaining applications and training efficiency present open boundaries.
- UniRepLKNet shows leading performance in image recognition and remarkable results on audio and time-series data, outperforming multiple specialist models.
- The dilated branches require more training resources, motivating simpler or gradient-based reparameterization as possible upgrades.
- Applications to large vision-language models, cross-attention scenarios, and generation tasks remain under-explored.
Appendix A: General Transformation from Dialted Convolution to Non-dilated Large-Kernel Convolution
The appendix generalizes dilated-to-non-dilated kernel conversion from single-channel cases to multi-channel grouped convolutions, including depthwise and non-depthwise settings.
- For a dilated layer with kernel size k and dilation rate r, the equivalent non-dilated kernel has size (k −1)r + 1.The conversion inserts zero entries into the kernel to represent skipped input pixels.
- For multi-channel convolutions, the kernel is represented as a 4D tensor with shape cout × cin/g × k × k.
- Multi-channel depthwise layers use a 4D identity kernel and the same transpose-convolution transformation.
- For non-depthwise grouped layers, kernels are split into slices, transformed separately, and concatenated into non-dilated slices.
Appendix B: Training Configurations
The appendix specifies training configurations for image classification, object detection, and semantic segmentation, with reproducible scripts and trained weights released publicly.
- Training configurations cover image classification, object detection, and semantic segmentation.
- Training scripts and trained weights for every model are publicly released on GitHub.
- ImageNet image classification: ImageNet-1K-only experiments use configurations detailed in Table 14, with 100 training epochs and a 0.1 drop path rate for Section 3 experiments.
Appendix C: Shape Bias
UniRepLKNet is reported to have substantially higher shape bias than existing ConvNets and vision transformers, with a 20% increase in decisions based on overall object shape. The appendix also documents implementation artifacts and training-cost caveats surrounding these results.
- Shape Bias: Higher shape bias indicates greater reliance on object shape rather than texture and behavior more similar to humans.
- Shape Bias: UniRepLKNet makes 20% more decisions based on the overall shapes of objects than the comparison models.
- Training Cost: The extra parallel dilated branches consume more training resources, although the authors consider this acceptable given performance improvements.
- Training Cost: Training-cost measurements are hardware- and implementation-dependent and should be treated as references only.
- Shape Bias: Figure 6 reports shape bias for ImageNet-22K-pretrained UniRepLKNet-L and RepLKNet-31L.
- Shape Bias: Figure 7 compares shape bias for ImageNet-1K- and ImageNet-22K-pretrained RepLKNet-31B and Swin-B models.