PaperlayerView exact revision on arXiv

Source-linked AI summary

ViT-5: Vision Transformers for The Mid-2020s

Feng Wang, Sucheng Ren, Tiezheng Zhang, Predrag Neskovic, Anand Bhattad, Cihang Xie, Alan Yuille

arXiv:2602.08071v1cs.CV

TL;DR

Although language-model architectures have evolved rapidly, the core Vision Transformer design has changed little, leaving its representational potential an open question. ViT-5 systematically modernizes components while preserving the Attention–FFN backbone, and the paper reports consistent improvements across visual tasks.

  • Problem

    The core ViT design has remained largely unchanged while language-model architectures have been refined, raising the question of whether ViT representational potential is under-optimized.

    Source: §1 · p. 1

  • Method

    ViT-5 systematically refines individual architectural components while preserving the plain ViT Attention–FFN backbone rather than proposing a radical redesign.

    Source: §1 · p. 1

  • Results

    ViT-5 improves consistently across classification, generation, and semantic segmentation; at ViT-Large and 384×384 resolution, it exceeds vision-oriented designs by 0.31% to over 1.15% top-1 accuracy on ImageNet-1k.

    Source: §4.4 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    A careful component-wise refresh can unlock substantial headroom without redesigning the overall ViT architecture.

    Source: §5 · p. 8

    Up to ViT-XL with a 1152 hidden dimension and 449M parameters, the authors conclude that LayerScale and gated MLPs should be avoided; larger scales remain to be systematically investigated.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

This work presents a systematic investigation into modernizing Vision Transformer backbones by leveraging architectural advancements from the past five years. While preserving the canonical Attention-FFN structure, we conduct a component-wise refinement involving normalization, activation functions, positional encoding, gating mechanisms, and learnable tokens. These updates form a new generation of Vision Transformers, which we call ViT-5. Extensive experiments demonstrate that ViT-5 consistently outperforms state-of-the-art plain Vision Transformers across both understanding and generation benchmarks. On ImageNet-1k classification, ViT-5-Base reaches 84.2\% top-1 accuracy under comparable compute, exceeding DeiT-III-Base at 83.8\%. ViT-5 also serves as a stronger backbone for generative modeling: when plugged into an SiT diffusion framework, it achieves 1.84 FID versus 2.06 with a vanilla ViT backbone. Beyond headline metrics, ViT-5 exhibits improved representation learning and favorable spatial reasoning behavior, and transfers reliably across tasks. With a design aligned with contemporary foundation-model practices, ViT-5 offers a simple drop-in upgrade over vanilla ViT for mid-2020s vision backbones.

1. Introduction

ViT architectures have changed relatively little, raising the question of whether modern Transformer refinements can improve them. ViT-5 systematically tests component updates while retaining the Attention–FFN backbone, reporting gains across vision tasks and identifying that effective refinements must be selected carefully.

1. Introduction

  • ViTs have seen little architectural change, motivating the question of whether refinements from language models can improve their performance and efficiency.The introduction contrasts this with the rapid refinement of language architectures and asks whether advances in normalization, activations, positional encoding, and attention transfer to vision.Source: §1 · p. 1
  • The study systematically evaluates modern design choices while preserving ViT's original Attention–FFN backbone rather than proposing a radical redesign.It aims to identify effective component variants and incorporate them in a principled, modular way.Source: §1 · p. 1
  • The authors find that ViT components remain under-optimized, but naively combining modern refinements does not necessarily yield the best performance.They identify LayerScale, RoPE, QK normalization, and register tokens among the examined refinements.Source: §1 · p. 2
  • ViT-5-Base reaches 84.2% top-1 accuracy on ImageNet-1k versus DeiT-III's 83.8%, while ViT-5 diffusion models achieve 1.84 FID versus SiT's 2.06 at nearly identical compute.Dense-prediction results also highlight advantages in spatial modeling and representation learning.Source: §1 · p. 2
  • The authors say the study improves ViT representation capacity and narrows the architectural gap between vision and language models.They hope the work will facilitate multimodal systems and inspire Transformer architectures that generalize across modalities.Source: §1 · p. 2

2. Related Work

Related work traces Transformer refinements across language and vision while distinguishing this study's focus on component-wise optimization of plain ViTs. The supplied overview and table caption identify ViT-5's component scope and a LayerScale versus post-normalization comparison.

2. Related Work

  • The ViT-5 overview lists activation scaling, normalization, positional embeddings, registers, and bias terms among the refined components.The figure caption describes an architectural overview and component analysis.Source: Figure 1 · p. 2
  • The study focuses on component-wise optimization of plain ViTs to preserve their generalizability and scalability, rather than maximizing performance on specific benchmarks.Related work contrasts plain ViTs with hierarchical and hybrid architectures that incorporate stronger inductive biases and often perform better on moderate-scale recognition benchmarks.Source: §2 · p. 2
  • Table 1 compares LayerScale with post-normalization and reports highly similar ImageNet top-1 accuracy across different model sizes.The caption identifies the metric and model-size scope but gives no individual values.Source: Table 1 · p. 3
  • Language-model refinements include RoPE and RMSNorm in LLaMA, query/key normalization in Gemma3, and bias removal in Qwen3; vision work includes LayerScale and register tokens.These changes preserve the original Transformer formulation and attention–FFN inference structure.Source: §2 · p. 3

3. Model

The section refines ViTs through modular architectural choices, retaining the Attention–FFN backbone while combining absolute and relative positional cues. It reports stronger classification results and describes component choices for stability, spatial modeling, and training behavior.

3.1. Activation Scaling

  • LayerScale and post-normalization yield highly similar performance improvements, and ViT-5 adopts LayerScale for its flexibility and lower computational overhead.LayerScale directly scales the block output; post-normalization implicitly scales both the block output and residual.Source: §3.1 · p. 3

3.2. Normalization

  • Replacing LayerNorm with RMSNorm slightly reduces computational cost and adds 0.2% top-1 accuracy on ImageNet for ViT-B.ViT-5 uses RMSNorm throughout in place of the original LayerNorm layers.Source: §3.2 · p. 3

3.3. Gated MLP

  • Combining LayerScale with SwiGLU degrades ViT performance, attributed to excessive sparsity from two channel-wise filtering mechanisms; ViT-5 retains GeLU MLPs through ViT-XL scale.The authors leave larger models for future investigation, noting that over-sparsity may be mitigated at higher scales.Source: Table 2 · p. 3, §3.3 · p. 4

3.4. Positional Encoding

  • With both APE and 2D RoPE, ViT-5 shows no noticeable accuracy degradation across test resolutions from 224 to 512, while DeiT-III degrades rapidly as resolution increases.Both models were trained at 224×224 and evaluated at different resolutions without fine-tuning; RoPE alone can make patch-level image flips equivalent.Source: §3.4 · p. 4, Figure 2 · p. 4

3.5. Register Tokens

  • ViT-5 uses four learnable registers with separate high-frequency 2D RoPE; register RoPE stabilizes register–patch interactions, and registers plus relative positions yield cleaner attention maps.The attention-map comparison describes clearer, more focused activations and improved spatial modeling.Source: §3.5 · p. 5, Table 4 · p. 5

3.6. QK-Normalization

  • QK-Norm modestly improves ViT performance and enhances training stability, reducing sharp loss spikes during optimization.The epochwise loss comparison shows smooth convergence without spikes for ViT-5-Small with QKNorm.Source: §3.6 · p. 5, Figure 5 · p. 5

3.7. Bias Terms for QKV

  • ViT-5 uses bias-free RMSNorm throughout and removes QKV projection biases, a change reported to help QKNorm operate more effectively and improve performance.Source: §3.7 · p. 5
  • ViT-5 attains the best ImageNet classification performance in each parameter-count and FLOPs group.The supplied table caption gives no cell values, so no numerical margin is stated.Source: Table 5 · p. 6

4. Experiments

Across classification, generation, and dense prediction, ViT-5 improves on vanilla or established ViT backbones, while ablations show that its component choices work together and vary in impact by model scale.

4.1. Image Classification

  • 86.0% test accuracy for ViT-5-L at 384×384 surpasses the previous ViT state of the art at 85.4% and CNN-based models at 85.5%.Models were trained from scratch on ImageNet-1k using DeiT-III’s recipe with small modifications; ViT-5’s advantage scales with model size and input resolution.Source: §4.1 · p. 6

4.2. Image Generation

  • At 7M training steps, ViT-5-XL attains the best performance across all listed image-generation metrics, and ViT-5 consistently outperforms vanilla ViT across model sizes and training lengths.The comparison uses classifier-free guidance of 1.5; Table 6 also reports better FID over vanilla ViT with similar parameter counts.Source: Table 7 · p. 6, §4.2 · p. 6, Table 6 · p. 6

4.3. Dense Prediction

  • 47.5%, 49.1%, and 52.0% mIoU for ViT-5-Small, Base, and Large exceed DeiT-III’s 45.2%, 48.0%, and 49.3% at matching parameter budgets.The models use 512×512 resolution, 160k iterations, and ImageNet-1k-pretrained backbones; the gains transfer without task-specific architectural changes.Source: §4.3 · p. 7, Table 8 · p. 7

4.4. Ablation Study

  • The ablations compare prevalent vision and language-model configurations and separately remove each of seven ViT-5 components to estimate their contributions.Exhaustively testing all possible combinations was considered prohibitively expensive.Source: §4.4 · p. 7
  • At 384×384, vision-oriented designs trail ViT-5 by 0.31% to over 1.15% ImageNet-1k top-1 accuracy, and directly transferring language-model configurations also underperforms.The comparison uses ViT-Large; the tested alternatives use subsets of refinements or language-model configurations adapted with 2D RoPE.Source: §4.4 · p. 7, Table 9 · p. 8
  • Across multiple model sizes, removing any individual component from the complete ViT-5 configuration consistently lowers accuracy; degradation magnitude varies, and no single modification dominates across all settings.Table 10 reports ImageNet-1k top-1 accuracy; Δ denotes average accuracy difference relative to complete ViT-5.Source: §4.4 · p. 7, Table 10 · p. 8
  • Component effects vary by scale: ungated GeLU-to-SwiGLU replacement without stabilization particularly hurts Small models, while LayerScale, 2D RoPE, Registers, and QK-Norm matter more at larger scales.Registers and QK-Norm cause relatively modest degradation for Small models but larger, consistent drops for Base and Large variants.Source: §4.4 · pp. 7–8
  • LayerScale initialization at 1e-4 gives 84.16 ImageNet-1k accuracy for ViT-5-B, while four learnable registers are used by default and register count has only trivial predictive impact.The reported LayerScale accuracies are 84.01 at 1e-6, 84.05 at 1e-5, and 84.16 at 1e-4; the default 1D RoPE frequency base is 1e-5 and shows the best performance.Source: §4.4 · p. 8, Table 11 · p. 8

5. Conclusion

The paper concludes that modularly modernizing ViT components improves performance across visual tasks without redesigning the architecture. It presents ViT-5 as a compatible foundation for vision and multimodal systems.

5. Conclusion

  • ViT-5’s component-wise refresh unlocks performance headroom without redesigning the overall ViT architecture.The conclusion describes gains across image classification, generation, and semantic segmentation, with transfer across tasks.Source: §5 · p. 8
  • The authors attribute cross-task gains to improved representations and spatial inductive biases, rather than benchmark-specific benefits.They connect the approach to modular, best-practice-driven refinement of large language model backbones.Source: §5 · p. 8
  • ViT-5 is proposed as a strong, compatible foundation for mid-2020s vision and multimodal systems.The authors encourage treating backbone modernization as a reproducible design process.Source: §5 · p. 8

Impact Statement

The authors state that the work’s potential societal consequences align with general progress in the field and identify no unique risks requiring specific attention.

Impact Statement

  • The authors identify no unique societal risks requiring specific highlighting beyond those associated with general progress in machine learning.They describe the contribution as foundational to advancing machine-learning methodologies.Source: Impact Statement · p. 9

A.1. Image generation examples

The paper presents qualitative image-generation examples from the SiT model with a ViT-5-XL backbone, describing coherent structure, preserved shapes, and fine texture.

A.1. Image generation examples

  • The generated examples are described as having coherent global structure, well-preserved object shapes, and fine-grained texture.The authors report no obvious artifacts or spatial inconsistencies in the examples.Source: §A.1 · p. 13, Figure 7 · p. 13
  • The authors attribute the examples’ visual quality partly to improved representational capacity and spatial awareness.They say better spatial modeling allows more accurate capture of long-range dependencies and local details.Source: §A.1 · p. 13

A.2. Architecture Design Principles

ViT-5 modernizes the plain ViT through component-wise upgrades while retaining a fixed-resolution, self-attention-based design intended for broad task applicability.

A.2. Architecture Design Principles

  • ViT-5 retains the plain ViT structure and avoids spatial downsampling after patchification to support broad applicability and integration with multimodal systems.Source: §A.2 · p. 13
  • Self-attention remains the only token-mixing mechanism, foregoing potential convolutional gains in favor of architectural simplicity and general-purpose modeling.Source: §A.2 · p. 13
  • The design keeps standard non-overlapping linear patch embeddings because patch-embedding refinements primarily benefit pixel-space understanding and do not align with its generalizability goal.The paper notes that many generation-oriented vision models instead use dedicated VAEs or learned tokenizers.Source: §A.2 · p. 13

A.3. More attention visualization

Local-token attention visualizations show cleaner, more spatially coherent patterns for ViT-5 than for DeiT-III, offering qualitative evidence of improved spatial modeling.

A.3. More attention visualization

  • The comparison uses equal-duration ImageNet-1K pretraining and fine-tuning at 384×384, attending from a selected local token rather than the class token.Source: §A.3 · p. 14
  • Together with the paper’s class-token visualizations, the local-token maps qualitatively indicate improved spatial modeling beyond numerical performance gains.Source: §A.3 · p. 14
  • ViT-5’s local-token maps are cleaner, more spatially coherent, and better aligned with semantic regions than DeiT-III’s, with background artifacts largely suppressed.Source: Figure 8 · p. 14, §A.3 · p. 14

A.4. Technical Details

The technical details describe evaluation protocols for ImageNet-1k classification, image generation, and semantic segmentation, with comparisons designed to isolate backbone effects.

ImageNet classification

  • ImageNet-1k classification uses large-scale pretraining followed by short-schedule fine-tuning, with recipe changes intended to isolate ViT-5’s architectural refinements.Pretraining uses LAMB; fine-tuning switches to AdamW and uses 224 or 384 resolution when applicable.Source: §A.4 · p. 14, Table 12 · p. 15, Table 13 · p. 16

Image generation

  • Image generation replaces SiT’s vanilla ViT backbone with ViT-5 while keeping its hyperparameters and optimization settings identical for comparison.The study includes runs up to 800K steps across model sizes and a 7M-step ViT-5-XL run.Source: §A.4 · p. 15

Semantic segmentation

  • ADE20K segmentation uses UperNet with ImageNet-1k-pretrained backbones, a shared 512×512, 160k-iteration protocol, and no task-specific backbone modifications.Source: §A.4 · p. 15

Related papers

Loading 2602.08071v1…