Source-linked AI summary
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles
Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, Christoph Feichtenhofer
TL;DR
Hierarchical vision transformers add specialized components that improve supervised models but make them slower than vanilla ViTs. Hiera removes those components and uses MAE pretraining to supply spatial bias, producing a simpler model that is more effective and faster across image and video tasks.
Problem
Specialized components in hierarchical vision transformers add complexity and make them slower, while sparse MAE pretraining is difficult to adapt to hierarchical architectures.
Method
Hiera starts from a hierarchical ViT, removes non-transformer components, and adapts MAE using mask units so sparse pretraining remains compatible with hierarchical stages.
Results
Hiera outperforms state-of-the-art models on images and video while being much faster across model scales and tasks.
Takeaways & Limitations
A pure hierarchical ViT can achieve strong image and video recognition without convolutions, shifted windows, or decomposed relative position embeddings.
Takeaways & Limitations
The method uses MViTv2 as its base architecture because its small 3 × 3 kernels are least affected by the separate-and-pad trick, though another transformer might yield a similar result.
Abstract
from arXiv · showhide
Modern hierarchical vision transformers have added several vision-specific components in the pursuit of supervised classification performance. While these components lead to effective accuracies and attractive FLOP counts, the added complexity actually makes these transformers slower than their vanilla ViT counterparts. In this paper, we argue that this additional bulk is unnecessary. By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer without losing accuracy. In the process, we create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. We evaluate Hiera on a variety of tasks for image and video recognition. Our code and models are available at https://github.com/facebookresearch/hiera.
1. Introduction
Hierarchical vision transformers improve parameter efficiency but often become slower through specialized vision modules. Hiera tests whether MAE can provide spatial bias while removing this added complexity, yielding a simpler and faster model.
- Hierarchical models use fewer channels at high-resolution early stages and more channels at low-resolution later stages, unlike vanilla ViTs.
- Specialized modules added for supervised performance make hierarchical transformers slower overall despite attractive FLOP counts.
- MAE can teach ViTs spatial reasoning while using sparse pretraining that is 4−10× as fast as normal supervised training.
- Hiera uses local attention in early stages and global attention later, with feature expansion and 2 × 2 spatial pooling at stage transitions.
- Hiera removes convolutions, shifted or cross-shaped windows, and decomposed relative position embeddings from a hierarchical ViT while training with MAE.
- Hiera outperforms state-of-the-art image models and far exceeds prior video work while being much faster across model scales, datasets, and tasks.
2. Related Work
Vision transformers have achieved strong results across image, video, and other vision tasks, while hierarchical variants combine transformer blocks with convolution-like operations and multi-stage structure. Masked pretraining improves efficiency for vanilla ViTs, but adapting sparse masking to hierarchical models is nontrivial.
- Vision transformers have been successful in image classification, video classification, segmentation, detection, video object segmentation, and 3D vision tasks.
- Hierarchical transformer variants combine transformers with convolution-like operations and CNN-style stages, but are slower than vanilla ViTs and less compatible with masked image modeling.
- Masked Autoencoders exploit variable-length vanilla ViT inputs to make masked pretraining efficient through sparse computation.
- Sparse masked pretraining is difficult to adapt to hierarchical models because deleted tokens disrupt the rigid two-dimensional grid they rely on.
3. Approach
Hiera simplifies a hierarchical transformer by adapting MAE to multi-stage models and removing specialized architectural components. The resulting model preserves or improves accuracy while substantially increasing inference and training efficiency.
- MAE adaptation: The architecture separates mask units so sparse masking remains compatible with spatial operations, then avoids padding by preventing kernels from overlapping between units.Hiera uses non-overlapping kernels instead of the separate-and-pad baseline.
- MAE adaptation: Hiera adapts sparse MAE to hierarchical models by masking 32×32-pixel mask units rather than individual 4×4-pixel tokens.Each mask unit corresponds to 8 × 8 tokens initially and covers 8^2, 4^2, 2^2, and 1^2 tokens across the four stages.
- Attention design: Mask Unit Attention replaces MViTv2’s pooled global attention in early stages with local attention inside mask units without added overhead.Global attention is retained in stages 3 and 4, while the unit grouping required for masking is reused for attention.
- Component simplification: Progressive simplification removes convolutions, shifted or cross-shaped windows, decomposed relative position embeddings, and other non-transformer components while maintaining high ImageNet-1K accuracy.The study uses MViTv2-L and reports that these components are unnecessary under MAE pretraining.
- Component simplification: MAE training makes MViTv2’s relative position embeddings unnecessary, allowing simpler absolute position embeddings that are also faster.The paper reports that relative position embeddings are not necessary when training with MAE.
- Efficiency and accuracy: Hiera is 2.4× faster on images and 5.1× faster on video than MViTv2, while Hiera-L is 3× faster to train than supervised MViTv2-L for superior image accuracy.For video, Hiera-L reaches 85.5% after 800 pretraining epochs and trains 2.1× faster than the compared MViTv2 model reporting 80.5%.
4. MAE Ablations
Hiera’s MAE pretraining is tuned through ablations of multiscale decoding, masking, reconstruction targets, regularization, decoder depth, and schedule. The experiments identify settings that improve image and video recognition.
- Multi-Scale decoder: Fusing representations from all stages in a multiscale decoder brings large gains for both images and video.
- Masking ratio: Optimal masking ratios are 0.6 for images and 0.9 for video.The paper relates the higher video ratio to greater information redundancy and notes that 32 × 32 mask units make the image task harder.
- Reconstruction target: HOG and normalized pixel reconstruction targets both produce strong performance.HOG is slightly better at the default ablation duration, matches pixel targets for video with longer training, and is slightly worse for images.
- Droppath rate: Applying drop path during MAE pretraining produces significant gains, indicating that Hiera can overfit the MAE task without it.Hiera-L’s greater depth motivates testing drop path during pretraining rather than only during finetuning.
- Decoder depth: A deeper decoder benefits video pretraining and brings its decoder depth in line with image pretraining.
- Pretraining schedule: +0.5% accuracy is gained on IN1K by extending pretraining beyond 400 epochs.At 400 epochs, Hiera is +0.7% above ViT-L MAE at the same duration, while the gap narrows to +0.2% at 1600 epochs.
- Pretraining schedule: On K400, Hiera surpasses the previous state of the art with 800 pretraining epochs, reaching 85.2%.Video gains continue less quickly with longer schedules, including a 0.9% gain from 800 to 1600 epochs and beyond.
5. Video Results
Hiera delivers strong video recognition and transfer results across Kinetics, Something-Something-v2, and AVA. It generally improves accuracy while reducing computation, model size, or inference time relative to prior systems.
- Kinetics-400, -600, -700: Hiera-L reaches 87.3% on Kinetics-400, while Hiera-H reaches 87.8%.The K400 comparison reports Hiera-L at 413×3×5 FLOPs and 213M parameters, versus ViT-L MAE at 597×3×5 and 305M.
- Kinetics-400, -600, -700: +2.1% over previous state of the art is achieved by Hiera-L on video, with approximately 45% fewer FLOPs, 43% smaller size, and 2.3× faster inference.Hiera-L also outperforms models one tier higher by +0.7% while being 3× smaller and 3.5× faster.
- Kinetics-400, -600, -700: +1.9% and +2.8% improvements over prior work are reported on Kinetics-600 and Kinetics-700.Hiera-H improves over previous state of the art by +1.2% on Kinetics-400/2242 without external data.
- Something-Something-v2: Hiera-L achieves 75.1% on Something-Something-v2, using approximately 45% fewer FLOPs and 43% fewer parameters than ViT-L.Hiera-L32 reaches 76.5%, while Hiera-L uses 16 frames at 2242 resolution versus MaskFeat’s 40 frames at 3122 resolution.
- Transferring to action detection (AVA): Hiera improves AVA action-detection transfer by +2.8% mAP over MAE-pretrained ViT-L and by +1.3% over MViTv2-L40,312 MaskFeat.Hiera-H improves over MAE-pretrained ViT-H by +3.0% mAP, while K700-pretrained Hiera-H improves by +3.2%.
6. Image Results
On images, Hiera matches or exceeds strong hierarchical and vanilla ViT baselines while remaining efficient. Transfer experiments show that its advantages extend beyond ImageNet to classification, detection, and segmentation settings.
- ImageNet-1K: 84.5% top-1 accuracy lets Hiera-B slightly outperform MViTv2-B without bells-and-whistles such as convolutions.Hiera-B+ outperforms the heavier MCMAE-B in both speed and accuracy, while smaller Hiera models remain strong.
- ImageNet-1K: 86.1% top-1 accuracy is achieved by Hiera-L on ImageNet-1K, +0.8% over MViTv2-L and +0.2% over ViT-L MAE.ViT-L MAE is 42% larger and uses 1.6× the FLOPs.
- Classification on iNaturalists and Places: Hiera consistently outperforms MAE-pretrained ViT on iNaturalist 2017, 2018, 2019, and Places 365.These transfer results indicate effectiveness beyond ImageNet.
- Object detection and segmentation on COCO: +0.6 APbox is achieved by Hiera-B over ViTDet-B with 34% fewer parameters and 15% lower inference time.Hiera-B+ further improves box AP by +1.9 while reducing inference time and model complexity versus ViTDet-B.
- Object detection and segmentation on COCO: Hiera achieves comparable results to ViTDet with faster inference and a lower operation count.For the large model, Hiera-L is consistently faster than ViTDet-L with only slightly lower accuracy.
7. Conclusion
Hiera is a simple hierarchical vision transformer that replaces architectural bells-and-whistles with spatial bias supplied through MAE pretraining. It improves image recognition, surpasses state of the art on video, and is intended to support faster future work.
- Hiera removes bells-and-whistles from an existing hierarchical transformer while supplying spatial bias through MAE pretraining.
- Hiera is more effective than current work on image recognition and surpasses the state of the art on video tasks.
- The authors hope Hiera enables future work to do more, faster.
A. Implementation Details
For video, Hiera changes the mask-unit and positional-embedding setup while keeping the encoder agnostic to spatio-temporal structure; MAE pretraining removes Q-pooling before the last stage.
- Video mask units span 2 frames × 32 px × 32 px, matching each Hiera video token’s 2-frame span.
- Image and video versions differ in mask-unit dimensions and learned spatial or separable spatio-temporal position embeddings; the remaining encoder is structure-agnostic.
- During MAE pretraining, Q-pooling is removed before the last stage to reuse prior ViT settings with minimal modifications.Stage 4 is small, so this introduces little extra computation; pretraining can also work without removing query pooling.
A.1. Video Experiments
Hiera is evaluated on video action-recognition and action-localization datasets using established pretraining, finetuning, and transfer-learning protocols.
- Hiera’s Kinetics experiments cover Kinetics-400, Kinetics-600, and Kinetics-700 using settings mainly following Feichtenhofer et al.Sampling uses 16 × 4, and epochs are reported as effective epochs accounting for repeated sampling.
- Hiera-L is evaluated on Something-Something-v2 for human-object interaction classification after pretraining on Kinetics-400 or SSv2.The reported metric is top-1 classification accuracy.
- On AVA v2.2, Hiera transfers to human action localization using ROI features, spatial max-pooling, and a linear cross-entropy classifier.The evaluation uses a center crop for Hiera.
A.2. Image Experiments
Image experiments assess Hiera across ImageNet-1K pretraining, transfer classification, and COCO detection and instance segmentation with specified training protocols.
- ImageNet-1K pretraining and finetuning settings mainly follow He et al. (2022).
- Transfer experiments cover iNaturalist2017, iNaturalist2018, iNaturalist2019, and Places365 with dataset-specific learning-rate and epoch adjustments.The experiments search layer-wise decay, drop path, and dropout rates; Hiera-H448 uses positional-embedding decay of 0.5.
- COCO experiments use Mask R-CNN in Detectron2 for object detection and instance segmentation, with shared convolutional and LayerNorm components across methods.
- COCO training follows ViTDet with 1024×1024 inputs and large-scale jittering, without layer-wise decay.
A.3. Speed Benchmarking
Speed benchmarking measures Hiera and baselines under controlled A100 fp16 settings for image and video inference, while training time is extrapolated from single-GPU passes.
- Image throughput is benchmarked on an NVIDIA A100 using fp16 at 224×224×3 resolution, with optimal batch sizes across Hiera and comparison models.Flash Attention and other attention speed-up mechanisms are not used.
- Video throughput is benchmarked on 224×224×3 clips with 16 frames and reported separately from image results.
- The benchmark tables report image throughput in im/s and video throughput in clip/s.
- Training time is estimated from a single-GPU forward-backward pass scaled by dataset size and epochs, ignoring dataloading and multi-GPU communication overheads.
B. From Scratch Supervised Training
Hiera’s architectural simplification depends on MAE pretraining: it replaces explicitly built spatial biases, whereas those modules remain necessary for supervised classification from scratch. This distinction also changes which downstream task heads are appropriate.
- B. From Scratch Supervised Training: MAE pretraining replaces the spatial biases provided by specialized hierarchical-transformer modules, allowing Hiera to remove those bells-and-whistles.The paper describes Hiera as an extremely fast and accurate vision transformer constructed through this simplification.
- B. From Scratch Supervised Training: The modules are therefore not unnecessary in general; their necessity depends on whether spatial biases are learned through MAE or built into the architecture.The authors explicitly limit their claim to the MAE-pretrained setting rather than all training regimes.
- B. From Scratch Supervised Training: When trained on ImageNet-1K classification from scratch, the removed bells-and-whistles are necessary, reversing the trend observed with strong pretraining.The experiment repeats the MViT-B-to-Hiera-B ablations using a classical supervised setting.
- B. From Scratch Supervised Training: Because Hiera acts like a ViT rather than a convnet, the authors recommend ViTDet instead of Mask R-CNN for downstream detection.Prior specialized transformers such as MViT and Swin are described as convnet-like and compatible with a normal Mask R-CNN head.