PaperlayerView exact revision on arXiv

Source-linked AI summary

Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles

Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, Christoph Feichtenhofer

arXiv:2306.00989v1cs.CVcs.LG

TL;DR

Hierarchical vision transformers add components to supply spatial biases, raising the question of whether that complexity is necessary. Hiera removes those components and uses MAE pretraining; it outperforms prior work on images and video while running faster.

  • Problem

    The paper asks whether hierarchical ViTs need added components to supply spatial biases, given that vanilla ViTs lack inductive bias after patchification.

    Source: §1 · p. 1

  • Method

    Hiera starts from an existing hierarchical ViT, removes non-essential components, and trains the simplified model with MAE pretraining.

    Source: §1 · p. 2

  • Results

    Hiera outperforms the state of the art on images and far exceeds prior work on video, while running much faster across model scales.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The authors suggest Hiera may allow future work to do more, faster.

    Source: §7 · p. 9

    The study uses MViTv2 as its base architecture, though the authors say another transformer might yield a similar result.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Modern hierarchical vision transformers have added several vision-specific components in the pursuit of supervised classification performance. While these components lead to effective accuracies and attractive FLOP counts, the added complexity actually makes these transformers slower than their vanilla ViT counterparts. In this paper, we argue that this additional bulk is unnecessary. By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer without losing accuracy. In the process, we create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. We evaluate Hiera on a variety of tasks for image and video recognition. Our code and models are available at https://github.com/facebookresearch/hiera.

1. Introduction

The introduction argues that hierarchical transformers’ specialized components add overhead, while MAE can teach spatial reasoning to a simpler hierarchical ViT. Hiera is presented as a faster, more accurate alternative.

  • Uniform resolution and channel counts make ViTs use parameters inefficiently, unlike hierarchies that shift from simpler, high-resolution features to more complex, low-resolution features.Hierarchical models use fewer channels early and more channels later as spatial resolution decreases.Source: §1 · p. 1
  • Hiera outperforms prior work on images, exceeds it on video, and is much faster across model scales and tasks.The paper reports this comparison across extensive datasets and tasks.Source: Figure 1 · p. 1, §1 · p. 2
  • MAE pretraining teaches spatial reasoning and can be 4 −10× as fast as supervised training, offering a way to learn spatial biases instead of adding specialized modules.The paper also notes that MAE-pretrained pure ViTs achieve good detection results, a task previously dominated by hierarchical models.Source: §1 · p. 1
  • Hiera uses standard ViT blocks, local attention in its first two stages, global attention afterward, and pooled stage transitions that double feature channels.At each transition, a linear layer doubles Q and skip-connection features, while a 2 × 2 maxpool reduces spatial dimensions.Source: Figure 2 · p. 2
  • Hiera removes non-transformer components—including convolutions, shifted or cross-shaped windows, and decomposed relative position embeddings—while increasing accuracy.The authors use implementation tricks and train with MAE, tuning the pretext task to the simplified architecture.Source: §1 · p. 2

2. Related Work

The related work traces Hiera’s motivation to the gap between hierarchical transformer performance and efficiency, and to the difficulty of bringing sparse masked pretraining to hierarchical architectures. It frames Hiera as a simpler architecture designed to address both issues.

2. Related Work

  • 30–40% speed-up over the best model at every scale is reported for Hiera on images; on video, it improves accuracy while being over 2× faster than popular ViT models.The figure compares Hiera with B, L, and H variants of state-of-the-art models using MAE-like pretraining.Source: Figure 3 · p. 2
  • Hybrid vision transformers combine transformers with convolution-like operations and hierarchical stages, achieving state-of-the-art results but running slower than vanilla ViTs and complicating masked image modeling.Vanilla ViTs flatten non-overlapping image patches into a sequence, whereas CNNs maintain a spatial grid across stages and introduce inductive biases such as shift equivariance.Source: §2 · p. 2
  • MAE (He et al. (2022)) uses variable-length ViT inputs to train efficiently on visible image patches, but hierarchical models make sparse training nontrivial because inputs no longer form a rigid 2D grid.MaskFeat (Wei et al., 2022) and SimMIM (Xie et al., 2022) use mask tokens, wasting computation on invisible tokens and making training slow.Source: §2 · pp. 2–3
  • Prior hierarchical masked-pretraining approaches trade off efficiency, accuracy, or simplicity, while Hiera is designed specifically for sparse MAE pretraining to avoid those restrictions.Huang et al. (2022a) add complex sparsity techniques with little accuracy gain; UM-MAE (Li et al., 2022a) restricts masking and hurts accuracy; MCMAE uses masked convolutions that reduce overall efficiency.Source: §2 · p. 3

3. Approach

Hiera simplifies a hierarchical vision transformer by using MAE pretraining to learn spatial biases rather than relying on specialized modules. The approach adapts sparse MAE to multi-stage models, then removes or simplifies MViTv2 components.

3. Approach

  • To adapt sparse MAE to hierarchical models, Hiera distinguishes 32×32-pixel mask units from smaller internal tokens and separates units during pooling to prevent convolution kernels crossing into deleted tokens.Each mask unit covers 8 × 8 tokens initially; in MViTv2 it corresponds to 8², 4², 2², and 1² tokens across stages 1–4.Source: §3 · p. 3, Figure 4 · p. 3, §3.1 · p. 4

3.1. Preparing MViTv2

  • MViTv2 is the base architecture because its small 3 × 3 kernels are least affected by the separate-and-pad trick used to handle sparse MAE.Its pooling attention aggregates features before attention, pools K and V to reduce computation in the first two stages, and pools Q at stage transitions.Source: §3.1 · pp. 3–4, Figure 5 · p. 4

3.2. Simplifying MViTv2

  • Hiera is 2.4× faster on images and 5.1× faster on video than the starting MViTv2, and is more accurate because of MAE.Hiera-L also obtains superior image accuracy while training 3× faster than supervised MViTv2-L; on video it reaches 85.5% in 800 pretraining epochs while training 2.1× faster than the comparison model reporting 80.5%.Source: Table 1 · p. 4, §3.2 · p. 5
  • With MAE, relative position embeddings are unnecessary, and replacing early-stage K/V pooling with mask unit attention has no accuracy impact while increasing video throughput by up to 32%.Mask unit attention performs local attention within the already-grouped mask units; the attention residual can also be removed after simplifying the model.Source: §3.2 · pp. 4–5
  • Replacing convolutions with maxpools drops image accuracy by over 1%, but removing extra stride=1 maxpools nearly restores it and speeds the model by 22% on images and 27% on video.Making the remaining maxpool kernels non-overlapping further increases accuracy and speeds the model by 20% on images and 12% on video.Source: §3.2 · p. 4

4. MAE Ablations

The ablations test MAE pretraining settings for Hiera-L on IN1K images and K400 video, evaluating top-1 accuracy after finetuning. They identify effective choices for multiscale decoding, masking, reconstruction targets, drop path, decoder depth, and pretraining length.

Multi-Scale decoder

  • Fusing representations from all encoder stages for the MAE decoder brings large gains for both images and video.Hiera's hierarchy permits this multiscale fusion, unlike using only the final encoder block as decoder input.Source: §4 · p. 5

Masking ratio

  • Optimal MAE masking ratios were 0.6 for images and 0.9 for video; the image ratio is below He et al. (2022)'s 0.75.The authors attribute the lower image ratio to the harder pretext task from 32 × 32 rather than 16 × 16 mask units; 0.9 is also appropriate for video in line with Feichtenhofer et al. (2022).Source: §4 · p. 6

Reconstruction target

  • Pixel targets with normalization and HOG targets both perform strongly; with longer training, HOG matches pixel targets for video but is slightly worse for images.HOG is slightly better at the default pretraining length used in the ablations.Source: §4 · p. 6

Droppath rate

  • Applying drop path during MAE pretraining yields significant gains, while omitting it can let Hiera overfit to the MAE task.Hiera-L has 48 layers versus 24 in ViT-L, motivating the authors to test drop path despite the original MAE recipes omitting it during pretraining.Source: §4 · p. 6

Decoder depth

Pretraining schedule

  • On IN1K, Hiera improves by +0.5% over 400 epochs; at 400 epochs it is +0.7% above ViT-L MAE (84.9%), narrowing to +0.2% at 1600 epochs.On K400, Hiera pretrained for 800 epochs outperforms the previous SotA result using 1600 epochs (85.2%); extending Hiera training from 800 to 1600 epochs and beyond adds 0.9%.Source: §4 · p. 6

5. Video Results

Hiera improves video-recognition results across Kinetics, SSv2, and AVA, with reported efficiency gains on Kinetics and SSv2 and higher AVA detection mAP.

Kinetics-400,-600,-700

  • +2.1%: Hiera-L improves over the previous K400 video-recognition SOTA, while Hiera-H improves over SOTA by +1.2% without external data.Hiera-L uses ∼45% fewer FLOPs, is ∼43% smaller, and is 2.3× faster; against models one tier higher it gains +0.7% while being 3× smaller and 3.5× faster, and gains +6.8% over the corresponding supervised MViTv2-L baseline. On K600 and K700, Hiera improves over SOTA by +1.9% and +2.8%, respectively.Source: Table 4 · p. 7, §5 · p. 7, Table 5 · p. 7

Something-Something-v2 (SSv2)

  • On SSv2, Hiera-L beats K400-pretrained MaskFeat by +0.6%, while Hiera-L32 reaches 76.5%, SOTA among approaches trained only on SSv2.With SSv2 pretraining, Hiera-L achieves 75.1%, outperforming MAE-pretrained ViT-L by +0.8% while using ∼45% fewer FLOPs and being ∼43% smaller. With K400 pretraining, Hiera uses 16 frames at 224² resolution versus MaskFeat’s 40 frames at 312² resolution, with 3.4× fewer FLOPs.Source: Table 6 · p. 7, §5 · p. 7

Transferring to action detection (AVA)

  • On AVA, K400-pretrained Hiera-L exceeds MAE-pretrained ViT-L by +2.8% mAP and MViTv2-L40,312 MaskFeat by +1.3% mAP, with fewer FLOPs and parameters.Hiera-H exceeds MAE-pretrained ViT-H by +3.0% mAP; K700-pretrained Hiera-H exceeds it by +3.2, establishing a new SOTA, with similar improvements for K600/K700-pretrained Hiera overall.Source: §5 · p. 7

6. Image Results

Hiera’s MAE-pretrained image models achieve strong ImageNet-1K accuracy and transfer effectively to downstream classification, detection, and segmentation, with reported COCO gains and speed advantages.

6.2. Transfer learning experiments

  • Hiera consistently outperforms MAE-pretrained ViT when transferred to iNaturalist 2017, 2018, 2019, and Places 365.These results show Hiera-L and Hiera-H are effective beyond ImageNet.Source: §6.2 · p. 8

6.1. Performance on ImageNet-1K

  • 86.1%: Hiera-L MAE reaches this ImageNet-1K accuracy, +0.8% over MViTv2-L and +0.2% over ViT-L MAE.ViT-L MAE is 42% larger and has 1.6× the FLOPs. Hiera-B reaches 84.5% versus MViTv2-B’s 84.4%, and Hiera-B+ outperforms MCMAE-B in speed and accuracy.Source: §6.1 · pp. 7–8, Table 8 · p. 8

6.2. Transfer learning experiments

  • +1.8 APbox: Hiera-L scores 55.0 versus MViTv2-L’s 53.2 on COCO, with 24% less inference time.Hiera-L also performs +1.4 APbox better than MViTv2-L with ImageNet-21K pretraining.Source: §6.2 · p. 8, Table 10 · p. 9
  • +0.6 APbox: Hiera-B exceeds ViTDet-B with 34% fewer parameters and 15% lower inference time on COCO.Hiera-B+ achieves +1.9 boxAP improvements with lower inference time and model complexity than ViTDet-B; Hiera-L is faster than ViTDet-L with slightly lower accuracy. The COCO experiments fine-tune Mask R-CNN with Hiera’s multi-scale features and an FPN.Source: §6.2 · pp. 8–9, Table 10 · p. 9

7. Conclusion

Hiera removes vision-specific bells and whistles from an existing hierarchical transformer while supplying spatial bias through MAE pretraining; it is more effective on image recognition tasks and surpasses the state of the art on video tasks.

  • Hiera is created by removing bells and whistles from an existing hierarchical vision transformer and supplying spatial bias through MAE pretraining.Source: §7 · p. 9
  • Hiera is more effective than current work on image recognition tasks.Source: §7 · p. 9
  • Hiera surpasses the state of the art on video tasks.Source: §7 · p. 9
  • The authors hope Hiera will enable future work to do more, faster.Source: §7 · p. 9

A. Implementation Details

Hiera uses the same encoder implementation for images and video, with changes limited to mask-unit size and position embeddings. For MAE pretraining, it removes Q-pooling before the final stage to match prior ViT settings with minimal modifications.

A. Implementation Details

  • Video mask units span 2 frames × 32 px × 32 px, matching Hiera’s 2-frame tokens, so Mask Unit Attention windows stay at 1 × 8 × 8 and 1 × 4 × 4 tokens across the first two stages, as in images.The paper follows Feichtenhofer et al. (2022); image mask units span 1 × 32 × 32.Source: A. Implementation Details · p. 9
  • Learned position embeddings are spatial for images and separable spatio-temporal for video; these are the only image/video differences, and the rest of the encoder is agnostic to spatio-temporal structure.Source: A. Implementation Details · p. 9
  • For MAE pretraining only, Hiera removes Q-pooling before the last stage so prior ViT MAE settings work with minimal modifications, adding little computation because stage 4 is small.As in Wei et al. (2022), this removal is optional: a 1 × 8 × 8-token mask unit would correspond to one distinct token in the last stage.Source: A. Implementation Details · p. 9

A.1. Video Experiments

The video experiments evaluate Hiera on Kinetics action recognition, SSv2 human-object interaction classification, and AVA v2.2 human action localization.

Kinetics-400, -600, -700

  • Kinetics-400, -600, and -700 experiments follow Feichtenhofer et al. (2022), use 16 × 4 sampling, and report effective epochs accounting for repeated sampling.Pretraining and finetuning settings for the main results are reported in Tab. 11.Source: §A.1 · p. 9

Something-Something-v2 (SSv2)

  • On SSv2, Hiera-L is pretrained on either Kinetics-400 or SSv2, then finetuned on SSv2, with top-1 accuracy reported.Pretraining and finetuning settings are provided in Tab. 12; accuracy is reported in Tab. 6.Source: §A.1 · p. 9

AVA v2.2

  • For AVA v2.2 localization, the framework pools last-layer Hiera ROI features by spatial max-pooling and trains a linear action classifier with cross-entropy loss.Evaluation uses the center crop and reports mAP; experiments use Kinetics-pretrained and finetuned Hiera.Source: §A.1 · p. 9

A.2. Image Experiments

The image experiments describe training and transfer settings for ImageNet-1K, iNaturalist and Places classification, and COCO detection and segmentation.

ImageNet-1K

  • ImageNet-1K settings mainly follow He et al. (2022), with pretraining and finetuning settings reported in Table 14.The passage describes the settings but does not provide their values.Source: §A.2 · p. 9

Transfer learning on iNaturalists and Places

  • Transfer learning covers iNaturalist2017, iNaturalist2018, iNaturalist2019 (Van Horn et al., 2018), and Places365.Following He et al. (2022), the experiments adjust learning rates and training epochs, search layer-wise decay, drop-path, and dropout rates, and use 0.5 positional-embedding decay for Hiera-H448.Source: §A.2 · pp. 9–10

COCO

  • COCO experiments use Mask R-CNN in Detectron2, with two hidden convolution layers for RPN and four for RoI heads, followed by LayerNorm.Following ViTDet, training uses 1024×1024 inputs with large-scale jittering and no layer-wise decay.Source: §A.2 · p. 10

A.3. Speed Benchmarking

This section specifies the setup and measurement procedure for image and video speed benchmarking, and reports benchmarking results for both modalities. Training time is estimated from a single-GPU forward-backward pass, with data-loading and multi-GPU communication overheads excluded.

A.3. Speed Benchmarking

  • Benchmarks use an NVIDIA A100 40GB GPU, PyTorch v1.12.1, and CUDA 11.4, without Flash Attention or another attention speed-up mechanism.The paper notes such mechanisms could be used to further increase speed.Source: §A.3 · pp. 10–11
  • Inference throughput is measured at the optimal batch size in fp16, using 224×224×3 inputs for images and 16-frame clips for video.The paper states that fp16 is used unless otherwise specified.Source: §A.3 · p. 11
  • Image speed benchmarking compares Hiera with ViT, ConvNeXtV2, and MCMAE, while video speed results are also reported.The image results are in Table 16 and video results in Table 17; the supplied passages do not provide table values.Source: Table 16 · p. 11, §A.3 · p. 11, Table 17 · p. 11
  • Training time is extrapolated from a single-GPU forward-backward pass using dataset size and epoch count, excluding data-loading and multi-GPU communication overheads.Source: §A.3 · p. 11

B. From Scratch Supervised Training

Unlike MAE pretraining, supervised classification from scratch requires the spatial-bias modules that Hiera removes; Hiera instead behaves like a ViT on downstream tasks.

  • The from-scratch experiment repeats the Tab. 1 ablations from MViT-B through Hiera-B, but trains on ImageNet-1K classification without pretraining.The experiment starts from MViT-B and ends at Hiera-B, using MViTv2’s supervised recipe.Source: B. From Scratch Supervised Training · p. 11, Figure 8 · p. 12
  • Under ImageNet-1K classification training from scratch, Hiera's removed bells-and-whistles are necessary, the opposite of the MAE-pretrained setting where MAE replaces explicit architectural spatial biases.Figure 8 repeats the Tab. 1 ablations without MAE pretraining, using MViTv2’s supervised recipe.Source: B. From Scratch Supervised Training · p. 11, Figure 8 · p. 12
  • Unlike MViT or Swin, which act like convnets and can use a normal Mask R-CNN detection head, Hiera acts like a ViT, motivating ViTDet for detection.Source: B. From Scratch Supervised Training · p. 12

Related papers

Loading 2306.00989v1…