Source-linked AI summary
Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarios
Jiashi Li, Xin Xia, Wei Li, Huixia Li, Xing Wang, Xuefeng Xiao, Rui Wang, Min Zheng, Xin Pan
TL;DR
Vision Transformers often provide strong vision-task performance but are slower than CNNs in realistic TensorRT and CoreML deployments, motivating a better latency/accuracy trade-off. Next-ViT combines deployment-friendly NCB and NTB blocks through NHS, and reports state-of-the-art trade-offs across classification, detection, and segmentation. The paper also reports substantial detection and segmentation gains over CNN baselines under similar latency.
Problem
Vision Transformers are often slower than CNNs in realistic TensorRT and CoreML deployment scenarios, while existing CNN-Transformer hybrids provide unsatisfactory overall trade-offs.
Method
Next-ViT develops deployment-friendly NCB and NTB blocks for local and global information, then stacks them through the NHS hybrid strategy.
Results
Next-ViT achieves state-of-the-art latency/accuracy trade-offs across image classification, object detection, and semantic segmentation on TensorRT and CoreML.
Takeaways & Limitations
Next-ViT provides a reported bridge between academic vision-model design and industrial deployment on mobile devices and server GPUs.
Takeaways & Limitations
For detection and segmentation latency comparisons, the paper measures only the backbone because some Mask R-CNN and Upernet modules are not easy to deploy on TensorRT and CoreML.
Abstract
from arXiv · showhide
Due to the complex attention mechanisms and model design, most existing vision Transformers (ViTs) can not perform as efficiently as convolutional neural networks (CNNs) in realistic industrial deployment scenarios, e.g. TensorRT and CoreML. This poses a distinct challenge: Can a visual neural network be designed to infer as fast as CNNs and perform as powerful as ViTs? Recent works have tried to design CNN-Transformer hybrid architectures to address this issue, yet the overall performance of these works is far away from satisfactory. To end these, we propose a next generation vision Transformer for efficient deployment in realistic industrial scenarios, namely Next-ViT, which dominates both CNNs and ViTs from the perspective of latency/accuracy trade-off. In this work, the Next Convolution Block (NCB) and Next Transformer Block (NTB) are respectively developed to capture local and global information with deployment-friendly mechanisms. Then, Next Hybrid Strategy (NHS) is designed to stack NCB and NTB in an efficient hybrid paradigm, which boosts performance in various downstream tasks. Extensive experiments show that Next-ViT significantly outperforms existing CNNs, ViTs and CNN-Transformer hybrid architectures with respect to the latency/accuracy trade-off across various vision tasks. On TensorRT, Next-ViT surpasses ResNet by 5.5 mAP (from 40.4 to 45.9) on COCO detection and 7.7% mIoU (from 38.8% to 46.5%) on ADE20K segmentation under similar latency. Meanwhile, it achieves comparable performance with CSWin, while the inference speed is accelerated by 3.6x. On CoreML, Next-ViT surpasses EfficientFormer by 4.6 mAP (from 42.6 to 47.2) on COCO detection and 3.5% mIoU (from 45.1% to 48.6%) on ADE20K segmentation under similar latency. Our code and models are made public at: https://github.com/bytedance/Next-ViT
1. Introduction
Next-ViT addresses the deployment gap between accurate vision Transformers and faster CNNs by combining deployment-friendly convolutional and Transformer blocks. Its hybrid design targets improved latency/accuracy trade-offs across classification, detection, and segmentation.
- Motivation: ViTs often lag CNNs in deployment latency because of quadratic MHSA complexity, non-foldable layers, and frequent memory operations.These constraints are especially relevant to TensorRT and CoreML deployment scenarios.
- Method: Next-ViT develops NCB for short-term dependencies, NTB for long-term dependencies, and NHS to combine them across stages.NCB uses deployment-friendly multi-head convolutional attention, while NHS reduces the Transformer-block proportion while retaining high precision.
- Results: 5.5 mAP and 7.7% mIoU gains over ResNet are reported on TensorRT under similar latency for COCO detection and ADE20K segmentation.The comparisons are from 40.4 to 45.9 mAP and from 38.8% to 46.5% mIoU, respectively.
- Results: 4.6 mAP and 3.5% mIoU gains over EfficientFormer are reported on CoreML under similar latency for COCO detection and ADE20K segmentation.The comparisons are from 42.6 to 47.2 mAP and from 45.1% to 48.6% mIoU, respectively.
- Method: Next-ViT stacks deployment-friendly NCB and NTB modules into a CNN-Transformer hybrid architecture.The design is presented as a new hybrid strategy intended to improve performance with high efficiency.
- Results: The paper reports state-of-the-art latency/accuracy trade-offs across image classification, object detection, and semantic segmentation on TensorRT and CoreML.Next-ViT is presented as a family of vision Transformer architectures.
2. Related Work
Related work spans convolutional networks, vision Transformers, and hybrid models that combine local convolutional features with Transformer-based long-range dependencies. These approaches pursue stronger accuracy, lower computation, or improved deployment efficiency.
- Convolutional Networks: CNN families use architectural mechanisms such as residual connections, dense feature reuse, depthwise convolution, and channel shuffling to improve representation or efficiency.The cited examples include ResNet, DenseNet, MobileNets, ShuffleNet, and ShuffleNetv2.
- Vision Transformers: Vision Transformers represent images as patches or tokens and apply self-attention, with later designs introducing distillation, progressive tokenization, hierarchy, or pooling.Examples include ViT, DeiT, T2T-ViT, Swin Transformer, and PiT.
- Hybrid Models: The related designs motivate comparing Transformer-based and convolution-based blocks when balancing representation quality with deployment efficiency.Figure 3 is identified as a comparison of different Transformer-based and convolution-based blocks.
- Hybrid Models: Hybrid models combine convolution and Transformer components to capture local features and long-range dependencies.BoTNet, CvT, CMT, and MobileViT represent different placements and combinations of these components.
3. Methods
Next-ViT combines deployment-friendly convolutional and Transformer blocks with a hybrid strategy that distributes global-information modeling across stages. Its NCB, NTB, and NHS designs target local and global dependencies while improving downstream-task performance under similar TensorRT latency.
- Next-ViT Overview: Next-ViT uses a hierarchical pyramid with progressively reduced spatial resolution and expanded channel dimensions across stages.Each stage contains convolutional or Transformer blocks following patch embedding.
- Next Convolution Block: NCB captures short-term dependencies using Multi-Head Convolutional Attention built from deployment-friendly convolution operations.MHCA uses multi-head local representation learning with group and point-wise convolutions, alongside BatchNorm and ReLU.
- Next Transformer Block: NTB captures low-frequency global information with efficient self-attention and mixes multi-frequency signals in a lightweight block.It reduces channels before attention, spatially reduces keys and values, and uses BatchNorm and ReLU instead of LayerNorm and GELU.
- Next Hybrid Strategy: NHS repeatedly combines NCB and NTB within each stage so shallow stages can capture global representations needed by detection and segmentation.Its pattern is (NCB × N + NTB × 1) × L, placing NTB at each stage's end while controlling the Transformer-block proportion.
- Next Hybrid Strategy: C HN HN HN outperforms C C C T by 0.8 mAP in detection and 0.8% mIoU in segmentation under similar TensorRT latency.The results compare four-stage hybrid strategies, with C HN HN HN achieving the best overall performance reported in the passage.
- Next Hybrid Strategy: Repeating the NCB–NTB pattern improves performance over a single pattern at similar latency, whereas simply adding convolution blocks eventually reaches saturation.The authors use N = 4 as the basic configuration thereafter.
4.1. ImageNet-1K Classification
On ImageNet-1K classification, Next-ViT achieves the strongest accuracy/latency trade-off among the compared CNNs, ViTs, and hybrid networks under TensorRT and CoreML.
- Next-ViT achieves the best accuracy/latency trade-off among the compared state-of-the-art CNNs, ViTs, and hybrid networks.
- 1.7% higher accuracy than ResNet101 is achieved by Next-ViT-S at similar TensorRT latency and faster CoreML speed.CoreML latency decreases from 4.0ms to 3.5ms.
- 4.0× and 1.4× faster TensorRT inference than EfficientNet-B5 and ConvNeXt-B is achieved by Next-ViT-L at similar accuracy.On CoreML, the corresponding speedups are 3.2× and 44×.
- 0.8% higher accuracy and 1.3× faster TensorRT inference are achieved by Next-ViT-S than Twins-SVT-S.
- 0.7% higher accuracy than CMT-XS is achieved by Next-ViT-S with 1.8× and 1.4× faster TensorRT and CoreML inference.
4.2. ADE20K Semantic Segmentation
On ADE20K semantic segmentation, Next-ViT consistently improves the accuracy/latency trade-off over CNN, ViT, and hybrid backbones across Semantic FPN and UperNet evaluations.
- Next-ViT achieves the best trade-off across CNN, ViT, and hybrid backbones on ADE20K semantic segmentation.
- 7.7% and 6.8% mIoU gains over ResNet101 and ResNeXt101-32x4d are achieved by Next-ViT-S.
- 3.5% higher mIoU than EfficientFormer-L7 is achieved by Next-ViT-B with similar CoreML runtime and 38% lower TensorRT latency.
- 3.6× faster TensorRT inference than CSWin-S is achieved by Next-ViT-L with similar performance under UperNet.
4.3. Object Detection and Instance Segmentation
On COCO object detection and instance segmentation with Mask R-CNN, Next-ViT improves accuracy while maintaining or reducing deployment latency across 1× and 3× schedules.
- 5.5 APb higher performance than ResNet101 is achieved by Next-ViT-S under the 1× schedule.
- 4.0× and 3.9× faster TensorRT and CoreML runtime than PVTv2-B4 are achieved by Next-ViT-L with 0.5 APb higher performance.
- APb improves from 42.6 to 47.2 over EfficientFormer-L7 with similar CoreML latency and 39% fewer TensorRT runtime.
- 1.2 APb higher performance and 3.2× faster TensorRT inference than Twins-SVT-S are achieved by Next-ViT-S.
- 79% faster inference than CSWin is achieved by Next-ViT-L with similar object detection and instance segmentation performance.
4.4. Ablation Study and Visualization
Ablations attribute the latency/accuracy gains to NCB, NTB’s shrink ratio, and deployment-oriented normalization and activation choices; visualizations indicate broader frequency and feature coverage.
- NCB ablation: NCB achieves the best latency/accuracy trade-off across classification, detection, and segmentation under similar TensorRT latency.Against ConvNeXt blocks, it improves classification by 2.9%, detection by 4.5 APb, and segmentation by 2.8% mIoU.
- NTB ablation: r = 0.75 achieves the best NTB latency/accuracy trade-off, outperforming r = 1.0 by 0.4%, 0.5 APb, and 1.0% mIoU.These gains correspond to classification, detection, and segmentation, respectively.
- Normalization and activation: BN and ReLU achieve the best overall latency/accuracy trade-off, while LN and GELU add negligible performance improvement but significantly increase TensorRT latency.
- Visualization: Next-ViT simultaneously captures high-quality multi-frequency signals, unlike ResNet’s high-frequency emphasis and ViT’s low-frequency emphasis.
- Visualization: Next-ViT captures richer texture information and more accurate global information, including edge shape, than ResNet and Swin.
5. Conclusion
Next-ViT combines efficient convolutional and Transformer blocks in a hybrid architecture designed for deployment across mobile devices and server GPUs. Experiments show state-of-the-art latency/accuracy trade-offs across diverse visual tasks.
- Next-ViT stacks efficient Next Convolution Blocks and Next Transformer Blocks using a novel hybrid strategy.
- The architecture targets efficient deployment on both mobile devices and server GPUs while retaining powerful visual modeling.
- Experimental results demonstrate state-of-the-art latency/accuracy trade-offs across image classification, object detection, and semantic segmentation.