PaperlayerView exact revision on arXiv

Source-linked AI summary

IronViT: Toward Efficient Generalist Visual Representation Learning

Jiaxi Huang, Yueqi Hu, Xin Zhu, Xiaopeng Zhang, Huiting Qiao, Yanglin Zhang, Zefeng Ji, Rongxue Li, Yifei Xu, Huiying Yu, Wei Liu, Jiayin Zheng, Yinggan Xu, Peipeng Chen, Yin Zhang, Jian Yao

arXiv:2609.29252v1cs.CV

TL;DR

Generalist vision encoders must unify heterogeneous capabilities, while softmax attention carries quadratic costs as visual token sequences grow. IronViT consolidates specialist knowledge in a softmax bridge before transferring it to a hybrid encoder, and reports broad transfer with improved high-resolution efficiency.

  • Problem

    A generalist encoder must unify heterogeneous semantic, spatial, language-aligned, and action-relevant capabilities, while softmax attention scales quadratically with visual tokens and hybrid transfer changes the token-mixing architecture.

    Source: §1 · p. 4

  • Method

    IronViT distills complementary specialists into a softmax capability bridge, then freezes that bridge and uses it as the sole teacher for a hybrid attention encoder.

    Source: §1 · p. 4, §3.1 · p. 9

  • Results

    The two-stage strategy consistently outperforms direct multi-teacher distillation across evaluation dimensions; the VLM average increases by 7.90 points.

    Source: §4.8.2 · p. 19

  • Takeaways & Limitations

    The authors conclude that establishing a shared representation before constraining computation provides a practical path toward efficient generalist vision.

    Source: §5 · p. 20

    The retained softmax blocks still impose quadratic cost.

    Source: §5 · p. 20

Abstract

from arXiv · show

A generalist vision encoder must capture semantic, spatial, language-aligned, and action-relevant cues within a unified representation, yet softmax attention underlying today's most capable visual backbones becomes prohibitively expensive at high resolution. A natural attempt to address both challenges is to distill multiple specialist teachers directly into an efficient architecture. We find that directly coupling these objectives degrades representation quality, as the student must simultaneously reconcile heterogeneous capabilities and adapt them to a different token-mixing architecture. We introduce IronViT, built on a simple principle: consolidate capabilities before constraining computation. IronViT first distills complementary specialists into a softmax attention capability bridge, then progressively transfers the consolidated representation to a hybrid softmax-linear attention encoder. A purpose-built data pipeline further curates the distillation corpus for higher information density and broader domain coverage. Across recognition, retrieval, dense prediction, multimodal understanding, and robotic learning, IronViT is competitive with leading specialist and generalist vision encoders. The softmax bridge achieves the strongest aggregate performance in multimodal understanding and robotic learning among the evaluated backbones, while the hybrid encoder retains broad transfer performance with an efficiency advantage that grows with input resolution. Together, these results show that consolidating capabilities before architectural conversion can yield a generalist visual encoder without inheriting the prohibitive high-resolution cost of conventional softmax attention.

1 Introduction

Generalist vision encoders must reconcile complementary capabilities while avoiding the high-resolution cost of softmax attention. IronViT consolidates specialist knowledge in a softmax bridge before transferring it to a hybrid encoder, and reports broad transfer with improved efficiency in long-sequence regimes.

1 Introduction

  • A generalist must reconcile semantic, spatial, language-aligned, and action-relevant representations, while softmax attention's quadratic token cost limits high-resolution and long-sequence use.Specialist objectives yield different strengths, and transferring to hybrid attention introduces a separate architectural gap.Source: §1 · p. 4
  • IronViT first consolidates complementary specialists in a softmax attention bridge, then transfers the unified representation to a hybrid attention backbone.The curated pipeline supports broad multi-task transfer with fewer than 100M images and 2,336 H200 GPU-hours across both stages.Source: §1 · p. 4
  • The hybrid model preserves most performance with substantially improved efficiency at high resolution and long sequence lengths, while two-stage distillation beats direct distillation across ablations.The benchmark profile spans recognition, retrieval, dense prediction, multimodal reasoning, and robotic control.Source: §1 · pp. 4–5

2 Data Curation

The curation pipeline improves information density and semantic coverage by filtering image defects and redundancy, balancing visual-concept sampling, and enriching underrepresented domains. Its cumulative pretraining ablation reports consistent representation gains, especially for dense prediction.

2 Data Curation

  • The pipeline selects, enriches, and balances samples to improve information density and semantic coverage, using four modules applied selectively to the pretraining and refinement corpora.The modules are quality filtering and perceptual deduplication, semantic pruning, hierarchical balanced sampling, and target-aware enrichment.Source: §2 · p. 5

2.1 Filtering and deduplication

  • Image-level checks remove invalid, extreme-geometry, or severely degraded inputs, while perceptual hashing removes near-identical copies rather than semantically similar images.Embedding-based pruning addresses semantic redundancy separately.Source: §2.1 · p. 6

2.2 Semantic clustering and in-cluster pruning

  • Semantic pruning clusters embeddings and compares samples locally, sorting each cluster by centroid distance to prioritize less-central samples for removal.The cosine-distance threshold is ε = 0.15, corresponding to cosine similarity 0.85.Source: §2.2 · p. 6

2.3 Hierarchical balanced sampling

  • Hierarchical balanced sampling resamples cluster centroids and distributes a fixed budget top-down, respecting child capacities rather than enforcing equal counts across clusters.This is intended to mitigate overrepresentation of frequent visual concepts.Source: §2.3 · pp. 6–7

2.4 Semantic enrichment

  • Target-aware enrichment retrieves images using both image anchors from seed examples and text anchors describing desired domains, requiring candidates to meet both similarity thresholds.The image and text thresholds are calibrated separately.Source: §2.4 · pp. 7–8

2.5 Training corpora

  • The pretraining corpus is reduced from approximately 85.5 million candidates to 40 million samples, while the refinement corpus contains approximately 46.4 million samples.The refinement corpus includes 1.487 million images retrieved through semantic enrichment.Source: §2.5 · p. 8

3 Method

IronViT separates heterogeneous capability consolidation from architectural adaptation: specialist teachers first train a softmax-attention bridge, which then serves as the sole teacher for a hybrid encoder. The stages use progressive fixed- and native-resolution training curricula.

3.1 Problem Formulation

  • Direct multi-teacher distillation couples reconciling heterogeneous teacher targets with reproducing them using a different token-mixing architecture; IronViT addresses these challenges in two stages.Stage I trains a softmax capability bridge against the frozen specialists, then Stage II freezes the bridge and uses it as the hybrid encoder’s sole teacher.Source: §3.1 · p. 9

3.2 Stage I: Heterogeneous Capability Consolidation

  • Stage I learns a shared representation from heterogeneous teachers while retaining softmax attention, separating capability consolidation from adaptation to another token-mixing operator.Source: §3.2 · p. 9, §3.1 · p. 9

3.2.1 Complementary Specialist Teachers

  • The three teachers contribute language-aligned semantics, category-agnostic spatial structure, and globally discriminative image–text representations.QwenViT and DINOv3 provide dense patch features, while PE-Core-G14 supplies a pooled global representation.Source: §3.2.1 · p. 9

3.2.2 Capability Bridge

  • The capability bridge is a softmax-attention ViT with dense patch and global outputs, trained through teacher-specific alignment interfaces and normalized targets.Its objective combines cosine distance and smooth-ℓ1 matching; for dense teachers these losses act over spatial positions, with smooth-ℓ1 also averaged over feature channels.Source: §3.2.2 · p. 10, §3.2.3 · p. 10

3.2.3 Progressive Capability Learning

  • The bridge curriculum moves from fixed-resolution consolidation on 384×384 crops to native-resolution refinement using a larger token budget and the refinement corpus.The refinement phase continues from the fixed-resolution checkpoint with a lower learning rate and the same teachers, interfaces, and objective.Source: §3.2.3 · p. 11, Table 1 · p. 11

3.3.1 Hybrid Attention Student Architecture

  • The hybrid encoder preserves the bridge’s patch embedding, token layout, feature dimensions, and output interface, replacing softmax attention with linear mixers in 18 of 27 blocks.The remaining nine softmax blocks retain exact global interactions, while the linear blocks avoid materializing pairwise attention maps.Source: §3.3.1 · p. 12, §3.3 · p. 12

3.3.2 Cross-Architecture Objectives

  • Cross-architecture transfer first aligns replacement linear mixers to frozen softmax counterparts, then matches the student’s final patch and global features to the bridge.The full-model objective uses matching losses without teacher-specific alignment or target standardization, leaving intermediate features free to adapt to the new mixers.Source: §3.3.2 · p. 12

3.3.3 Progressive Architecture-Transfer Curriculum

  • The architecture-transfer curriculum proceeds through local attention alignment, fixed-resolution full-model distillation, and native-resolution refinement.The first phase freezes the rest of the transferred backbone; subsequent phases unfreeze the complete student and distill from the bridge using the pretraining and refinement corpora, respectively.Source: §3.3.3 · p. 12

4 Experiments

Experiments assess IronViT-Softmax and IronViT-Hybrid across recognition, retrieval, dense prediction, multimodal understanding, and robotic learning, alongside analyses of teacher retention, efficiency, and training choices. The results show broad transfer, with especially strong aggregate multimodal and robotic performance for the softmax bridge and increasing high-resolution efficiency advantages for the hybrid encoder.

4.1 Classification; 4.2 Retrieval

  • IronViT-Softmax ranks first in k-NN and second in linear probing, and outperforms C-RADIOv4 on four of five zero-shot datasets by 1.37 points on average; retrieval is second-best overall.IronViT-Hybrid remains competitive in classification and largely preserves text-to-image retrieval, with a more noticeable reduction in text-to-image's reverse direction.Source: Table 3 · p. 13, §4.1 · p. 13, Table 4 · p. 14

4.3 Dense vision tasks

  • IronViT-Softmax ranks second on ADE20K segmentation, remains competitive with DINOv3 on NYUv2 depth, and outperforms language-pretrained baselines on both tasks.IronViT-Hybrid closely matches segmentation and has a modest depth-estimation degradation.Source: Table 5 · p. 14, §4.3 · p. 14

4.4 VLM visual encoder; 4.5 Robotic learning

  • IronViT-Softmax achieves the best overall average in multimodal understanding and the highest overall robotic success rate; IronViT-Hybrid retains 98.4% of its robotic success rate.The softmax model leads three of four multimodal capability families and aggregate performance on both Adroit and Meta-World.Source: Table 6 · p. 15, §4.4 · p. 15, Table 7 · p. 16

4.6 Capability retention from specialist teachers

  • Against specialist teachers, IronViT-Softmax matches QwenViT overall, largely preserves PE-Core retrieval and frozen-feature recognition, and slightly exceeds DINOv3-H+/16 on ADE20K segmentation.The largest reported retention gaps are in OCR and document understanding versus QwenViT and zero-shot classification versus PE-Core, especially under distribution shift.Source: §4.6 · pp. 16–17, Table 9 · p. 17

4.7 Efficiency at high resolution

  • At 2048×2048, IronViT-Hybrid uses approximately 45% fewer FLOPs and achieves about 1.75× speedup relative to IronViT-Softmax.Its efficiency advantage grows with resolution; replacing 18 of 27 softmax blocks reduces quadratic attention cost, while shared projection and feed-forward operations limit lower-resolution gains.Source: §4.7 · p. 17, Figure 7 · p. 18

4.8.1 Data curation

  • Progressive corpus curation improves ImageNet robustness and dense prediction, with the largest dense-task gains from semantic clustering and hierarchical balancing; semantic-enriched refinement further improves recognition and dense prediction.The clustering and balancing gains on ADE20K and NYUv2 emerge early during training.Source: §4.8.1 · p. 18, Table 10 · p. 18, Figure 8 · p. 19

4.8.2 Two-stage distillation

  • Two-stage distillation outperforms direct multi-teacher distillation across all evaluation dimensions, increasing the VLM average by 7.90 points and reducing NYUv2 RMSE from 1.046 to 0.343.The reported improvements are most pronounced in zero-shot classification, depth estimation, and VLM evaluation.Source: §4.8.2 · p. 19, Table 12 · p. 20

4.8.3 Multi-phase training curriculum

  • Adding Phase I-B improves every reported evaluation; NYUv2 RMSE falls from 0.384 to 0.339 and the VLM average rises by 1.27 points.The largest VLM improvements are in OCR and vision-centric perception.Source: §4.8.3 · p. 20, Table 13 · p. 20

5 Conclusion

IronViT separates capability consolidation from architectural adaptation, using a softmax bridge to unify specialist representations before transferring them to a hybrid attention encoder. Experiments report broad downstream transfer, high-resolution efficiency gains, and improvements over direct multi-teacher distillation, while noting remaining quadratic costs and unestablished transfer to video and real-world robotic deployment.

5 Conclusion

  • IronViT separates capability consolidation from architectural adaptation to learn efficient generalist visual representations.A softmax capability bridge consolidates complementary teacher representations before transfer to a hybrid attention encoder.Source: §5 · p. 20
  • Experiments report broad downstream transfer, high-resolution efficiency gains, and consistent improvements over direct multi-teacher distillation.The conclusion frames establishing a shared representation before constraining computation as a practical path toward efficient generalist vision.Source: §5 · p. 20
  • Retained softmax blocks still impose quadratic cost, and transfer to video and real-world robotic deployment remains unestablished.The conclusion also points to mixture-of-experts and continual learning as possible future directions for scaling capabilities and adapting to deployment data.Source: §5 · p. 20

A Additional Details

The appendix documents experimental implementation and evaluation details and supplies additional benchmark-level results.

A Additional Details

A.1 Data curation details

The data-curation appendix specifies cosine similarity between L2-normalized embeddings as the default similarity measure unless otherwise specified.

A.1 Data curation details

  • Unless otherwise specified, similarity is cosine similarity between L2-normalized embeddings.Source: §A.1 · p. 25

A.2.1 Training hyperparameters

Training uses frozen teachers and mixed-precision student updates, with optimizer choices and stabilization settings varying by parameter type and distillation stage.

A.2.1 Training hyperparameters

  • Teachers remain frozen in bfloat16, while the student uses bfloat16 mixed precision with FP32 master weights.Source: §A.2.1 · p. 25
  • Two-dimensional matrices use MuonPlus, while biases, normalization parameters, and other one-dimensional parameters use AdamW.All parameter groups use linear warm-up followed by cosine learning-rate decay.Source: §A.2.1 · p. 25
  • Stage I skips updates when the combined gradient norm exceeds 5.0 after 2,000 steps; Stage II varies weight decay and uses two-step accumulation in Phases II-B and II-C.Weight decay is 0.05 in Phase II-A and 0.01 in Phases II-B and II-C.Source: §A.2.1 · p. 25

A.2.2 Per-teacher alignment

Each teacher has a learnable alignment projection, and dense supervision aligns teacher and student patch tokens on a common grid.

A.2.2 Per-teacher alignment

  • Each teacher-specific alignment interface uses a bias-free d→1536→d_k MLP with LayerNorm and GELU between its linear layers.Here d = 1152; patch tokens are aligned in row-major order, and differing grids are bicubically resampled to a common grid sized by the larger height and width.Source: §A.2.2 · p. 25

A.2.3 Backbone architectures

The capability bridge and hybrid encoder share a 27-block ViT backbone, while the hybrid model replaces linear-mixer blocks selectively and retains softmax attention in nine blocks.

A.2.3 Backbone architectures

  • Both backbones use 27 pre-LayerNorm ViT blocks with hidden width 1152, MLP width 4,304, 16 heads, 16×16 patches, and eight register tokens.Both use interpolated absolute positional embeddings and 2D RoPE.Source: §A.2.3 · p. 25
  • The bridge and hybrid backbones contain 416M and 440M parameters, respectively, excluding the MAP head and teacher-specific adapters.A single-query MAP head pools layer-normalized patch features; register tokens participate in backbone attention but are excluded from pooling.Source: §A.2.3 · p. 26
  • The bridge uses softmax attention in all 27 blocks, while the hybrid model uses linear attention in 18 blocks and retains softmax attention in nine.The retained softmax blocks are at zero-based indices {3,7,11,15,19,23,24,25,26}.Source: §A.2.3 · p. 26
  • The hybrid model’s linear-attention gate is zero-initialized, producing an initial gate value of 0.5 before output projection.The gate uses a sigmoid and element-wise multiplication; query–key normalization and 2D RoPE are omitted from the equations.Source: §A.2.3 · p. 26

A.3 Evaluation protocols and results

This section specifies evaluation protocols across data curation, classification, retrieval, dense prediction, VLM, and robotic learning, and describes the scope of the detailed VLM benchmark table.

A.3.1 Data curation ablation

  • The data-curation ablation trains independent capability bridges for 100k steps per corpus variant, holding other training and evaluation settings fixed.Checkpoints are evaluated every 20k steps under the Phase I-A configuration.Source: §A.3.1 · p. 26

A.3.2 Classification

  • Classification uses frozen encoders, with zero-shot evaluation, linear probing, and 20-NN accuracy reported under distinct feature and preprocessing protocols.IronViT uses MAP output; zero-shot inputs are resized to 384×384 without center cropping, while linear-probe images are center-cropped to 384×384.Source: §A.3.2 · p. 27

A.3.3 Image–text retrieval

  • Image–text retrieval is evaluated on COCO and Flickr30k test splits using Recall@1 in both image-to-text and text-to-image directions.The full test-set gallery is ranked by cosine similarity of L2-normalized embeddings, without retrieval-specific training.Source: §A.3.3 · p. 27

A.3.4 Dense prediction

  • ADE20K semantic segmentation trains task heads on frozen visual encoders and reports the highest validation mIoU.Patch features are combined with a broadcast global feature; inference uses overlapping 512×512 sliding-window crops.Source: §A.3.4 · p. 27
  • NYUv2 depth evaluation uses the standard BTS/Eigen split at 480×640 resolution and reports RMSE for the checkpoint with highest test-set δ1 accuracy.Depth is computed from 256 depth-bin logits as a softmax-weighted mean of bin centers spanning 10^-3 to 10 meters.Source: §A.3.4 · p. 27

A.3.5 VLM

  • VLM evaluation first trains an MLP connector on 558,000 image–caption pairs, then jointly instruction-tunes the vision encoder and language model on 739,000 examples.The connector stage freezes both encoders; the following joint stage unfreezes the vision encoder for one epoch.Source: §A.3.5 · p. 28

A.3.6 Robotic learning

  • Robotic learning uses behavior cloning on Adroit and Meta-World demonstrations and reports success rates across three training seeds.The protocols use 100 and 25 expert demonstrations per task, respectively, and evaluate 25 episodes every five epochs.Source: §A.3.6 · p. 28

A.3.7 Detailed VLM results

  • The detailed VLM table compares the QwenViT baseline, Phase I checkpoints, and direct versus two-stage hybrid models across 22 benchmarks.Its overall score averages all 22 benchmarks, with MME Perception divided by 20; the best overall score is bolded.Source: §A.3.7 · p. 28, Table 15 · p. 28

Related papers

Loading 2609.29252v1…