PaperlayerView exact revision on arXiv

Source-linked AI summary

X3D: Expanding Architectures for Efficient Video Recognition

Christoph Feichtenhofer

arXiv:2004.04730v1cs.CV

TL;DR

Video networks are often computationally heavy, motivating better computation/accuracy trade-offs. X3D progressively expands a tiny 2D architecture along multiple axes, selecting one axis per step, and achieves competitive accuracy with substantially lower computation and parameter counts.

  • Problem

    Video architectures can be much more computationally demanding than image models, while temporal-only expansion may be sub-optimal for computation/accuracy trade-offs.

  • Method

    X3D progressively expands a tiny 2D image architecture across temporal, spatial, width, bottleneck, and depth axes, selecting one-axis changes by computation/accuracy trade-off.

  • Results

    4.8× fewer multiply-add operations and 5.5× fewer parameters deliver similar accuracy to previous work across computation regimes.

  • Takeaways & Limitations

    Thin networks with high spatiotemporal resolution can perform well for video recognition while remaining light in width and parameters.

  • Takeaways & Limitations

    Computational cost is evaluated primarily with single-clip, center-crop FLOPs, with inference cost roughly proportional under fixed clip and crop counts.

Abstract

from arXiv · show

This paper presents X3D, a family of efficient video networks that progressively expand a tiny 2D image classification architecture along multiple network axes, in space, time, width and depth. Inspired by feature selection methods in machine learning, a simple stepwise network expansion approach is employed that expands a single axis in each step, such that good accuracy to complexity trade-off is achieved. To expand X3D to a specific target complexity, we perform progressive forward expansion followed by backward contraction. X3D achieves state-of-the-art performance while requiring 4.8x and 5.5x fewer multiply-adds and parameters for similar accuracy as previous work. Our most surprising finding is that networks with high spatiotemporal resolution can perform well, while being extremely light in terms of network width and parameters. We report competitive accuracy at unprecedented efficiency on video classification and detection benchmarks. Code will be available at: https://github.com/facebookresearch/SlowFast

1. Introduction

X3D addresses the compute burden of extending image architectures to video by progressively expanding a tiny 2D model across multiple axes. This produces efficient models across computation/accuracy trade-offs, including strong results with thin, high-resolution networks.

  • Video architectures often expand temporal inputs or filters while inheriting depth, width, and spatial properties from image models, which can be sub-optimal for computation/accuracy trade-offs.
  • 27× fewer multiply-add operations can be required by an image ResNet than by a temporally extended video variant.
  • X3D progressively expands a tiny 2D image architecture across temporal duration, frame rate, spatial resolution, width, bottleneck width, and depth.
  • Each expansion step changes one axis, trains and validates the resulting model, and retains the axis with the best computation/accuracy trade-off until a target budget is reached.
  • 4.8× fewer multiply-adds and 5.5× fewer parameters are required for similar accuracy as previous work across different computation regimes.
  • X3D's most surprising finding is that very thin video architectures created by expanding spatiotemporal resolution can perform well with few width and parameter resources.

2. Related Work

Prior efficient video methods commonly extend or adapt existing image architectures, whereas X3D expands a tiny model across several axes without assuming a fixed inherited design.

  • Prior video networks commonly extend 2D architectures into spacetime, add recurrent layers, or combine 2D networks with optical-flow streams.
  • SlowFast explores temporal, spatial, and channel resolution through Slow and Fast pathways, but retains a computationally heavy Slow pathway design.
  • Efficient image architectures use channel-wise separable convolutions, expanded bottlenecks, width multipliers, and input-resolution scaling.
  • X3D trains one model per candidate axis at each step, requiring 30 models for five steps rather than the much larger searches used by prior approaches.
  • Unlike many prior efficient 3D methods, X3D expands a tiny architecture across space, time, channels, and depth to seek a favorable efficiency trade-off.

3. X3D Networks

X3D frames video design as choosing among temporal, spatial, width, bottleneck, and depth expansions under a complexity budget. Its progressive procedure starts from a tiny X2D network and selects one-axis changes by accuracy/complexity trade-off.

  • Design questions: The design questions compare long versus fast temporal sampling, finer spatial resolution, thin high-frame-rate versus wide models, bottleneck versus global width, and depth across axes.
  • Basis instantiation: X2D is the unexpanded basis model obtained when all six factors equal one, with 20.67M FLOPS and 1.63M parameters.
  • Basis instantiation: X2D uses a ResNet-style structure, Fast-pathway-inspired width, channel-wise separable convolutions, and 3×3×3 spatiotemporal convolutions.
  • Basis instantiation: 20.67M FLOPs makes X2D only 0.0097% of the computational cost of a recent state-of-the-art SlowFast network.
  • Expansion operations: X-Fast increases frame rate at fixed clip duration, whereas X-Temporal increases both sampled clip duration and frame rate.
  • Expansion operations: X-Spatial, X-Depth, X-Width, and X-Bottleneck respectively expand input resolution, layers per stage, all-layer channels, and inner bottleneck channels.
  • Progressive Network Expansion: At each progressive step, X3D changes one expansion factor, evaluates the resulting candidates by accuracy and complexity, and retains the best trade-off.
  • Progressive Network Expansion: A roughly 2× multiplicative complexity increase per step enables the procedure to reach a target budget after training relatively few models.

4. Experiments: Action Classification

X3D expands a tiny 2D architecture progressively across multiple axes to study efficient video recognition. Across Kinetics-400, Kinetics-600, and Charades, the resulting models provide strong accuracy-complexity trade-offs and generalize to held-out test data.

  • Expansion results: Each candidate-axis expansion increases accuracy, with bottleneck width selected first and temporal size selected second rather than temporal expansion leading the process.Later steps expand spatial resolution, depth, temporal resolution, duration, and eventually global width.
  • Expansion results: X2D starts at 47.75% top-1 accuracy with 1.63M parameters and 20.67M FLOPs per clip, while progressive models reach 80.0% top-1 accuracy at ≤150 GFLOPs.X3D-M, X3D-L, X3D-XL, and X3D-XXL achieve 74.6%, 76.8%, 78.4%, and 80.0% top-1 accuracy, respectively.
  • Main results: X3D-XL provides comparable performance to SlowFast while using 4.8× fewer FLOPs and 5.5× fewer parameters.It is slightly lower by 0.7% top-1 with identical top-5 accuracy against SlowFast 16×8, R101 + NL blocks.
  • Main results: X3D achieves similar or better performance than prior methods across Kinetics-600 and Charades with substantially lower computational cost.On Kinetics-600, the best model is slightly better than SlowFast with 4.8× fewer FLOPs and 5.5× fewer parameters; on Charades, X3D-XL improves mAP while using the same reductions.
  • Ablation experiments: X3D-XS improves over EfficientNet3D-B0 by 1.9% top-1 and 1.3% top-1 accuracy at comparable complexity, while larger X3D models show larger gains over corresponding baselines.X3D-M gains 2.0% top-1 and 2.1% top-5 over EfficientNet3D-B3 while using 32% fewer FLOPs and 54% fewer parameters.
  • Ablation experiments: Validation-guided expansion generalizes to K400-test, where similar or slightly higher accuracy improvements are observed for the same model comparisons.This evaluates whether selecting expansions using validation performance transfers to the test set.
  • Ablation experiments: Using three clips instead of one substantially improves video-level accuracy, whereas additional clips beyond three provide only marginal gains.Sparse clip sampling can therefore reduce inference cost when the highest accuracy is not crucial; X3D also matches several baselines with 3–20× fewer multiply-adds.

5. Experiments: AVA Action Detection

X3D is evaluated for AVA action detection using a common single-center-crop testing setup and comparisons against LFB and SlowFast backbones. It achieves comparable accuracy to strong baselines with substantially lower computational and parameter costs.

  • Evaluation setup: AVA evaluation reports mean Average Precision using single-center-crop inference, with total testing cost proportional to GFLOPs across 57k validation segments.The AVA protocol evaluates spatiotemporal localization across 60 action classes.
  • Comparison with LFB: X3D-XL matches LFB accuracy within -0.7 mAP while using 10.9×/14× fewer multiply-adds and 6.7×/11.1× fewer parameters than LFB R50/R101.The comparison is made against LFB backbones with different computational sizes.
  • Comparison with SlowFast: X3D-M is 1.5 mAP below SlowFast 4×16, R50 while requiring 8.5× fewer multiply-adds and 10.9× fewer parameters.This comparison uses the same detection architecture and varies the backbone.
  • Comparison with SlowFast: X3D-XL matches SlowFast 8×8 + NL performance with 3× fewer multiply-adds and 5.4× fewer parameters.The result supports comparable detection accuracy at substantially lower backbone cost.

6. Conclusion

The conclusion presents X3D as a progressively expanded spatiotemporal architecture that considers multiple network axes under computation/accuracy trade-offs. It highlights that thin networks with high spatiotemporal resolution can be effective for video recognition.

  • Conclusion: X3D progressively expands a tiny spatial network across space, time, width, and depth while targeting favorable computation/accuracy trade-offs.The conclusion frames these axes as candidate dimensions for spatiotemporal architecture design.
  • Conclusion: Thin channel dimensions combined with high spatiotemporal resolution can produce effective video-recognition networks.This is identified as the surprising finding of the progressive expansion approach.
  • Conclusion: X3D achieves competitive efficiency across video recognition settings.The conclusion states this as the paper’s overall efficiency outcome.

Appendix

The appendix collects additional implementation details and further empirical analyses. It covers AVA detection, Charades and Kinetics classification, plus additional Kinetics-400 results and ablations.

  • Appendix contents: Appendix section A provides implementation details for AVA action detection, Charades action classification, and Kinetics action classification.These details are organized into subsections A.1, A.2, and A.3.
  • Appendix contents: Appendix section B reports further results and ablations on Kinetics-400.The appendix supplements the main paper with additional Kinetics-400 analysis.

A.1. Details: AVA Action Detection

The AVA detection setup follows SlowFast’s architecture and training procedure for direct comparison, while using pre-computed region proposals from an off-the-shelf person detector. The detector adapts Faster R-CNN-style processing to video.

  • Detection architecture: The detection architecture exactly follows SlowFast’s design to enable direct backbone comparison.It is similar to Faster R-CNN with modifications for video, and does not increase res5 spatial resolution by 2× by default.
  • Training: AVA training follows SlowFast’s procedure and hyperparameters without modification, using Kinetics initialization and step-wise learning-rate decay.Training lasts 14k iterations with 1k iterations of linear warm-up and weight decay of 10^-7.
  • Region proposals: Region proposals are pre-computed by an off-the-shelf person detector rather than jointly trained with the action detector.The proposal model is a Faster R-CNN with a ResNeXt-101-FPN backbone, pretrained on ImageNet and COCO human keypoint images.

A.2. Details: Charades Action Classification

Charades fine-tuning uses a multilabel output and modified temporal settings to accommodate the dataset’s longer-clip preference.

  • Charades training uses a per-class sigmoid output for its multi-class labels.The model is fine-tuned from Kinetics models.

A.3. Details: Kinetics Action Classification

Kinetics experiments use large-scale synchronized training and define X3D by progressively expanding a tiny 2D base along one axis at a time. Temporal, spatial, width, bottleneck, and depth expansions are calibrated to approximately double complexity, with regularization adjusted when needed.

  • Training and regularization: Kinetics training uses synchronized SGD on 128 GPUs with 1024 total clips per mini-batch and a half-period cosine learning-rate schedule.Kinetics-600 extends the Kinetics-400 training epochs and schedule by 2×.
  • Model setup: Non-Local blocks are omitted from X3D, while the expansion details initialize all factors at one to form the X2D base model.The X2D model is expanded from a single-frame input, with the first common temporal cost increase defined by doubling the input length.
  • Expansion procedure: X3D starts from a tiny 2D base and progressively expands one network axis at a time, training and validating each candidate to select the best accuracy-complexity trade-off.The process repeats until the desired computational budget is reached.
  • Expansion procedure: Each expansion step trains one candidate per axis at approximately double complexity.The common expansion factor is set to c-hat = 2, with exact factors varying slightly due to network geometry and rounding.
  • Temporal expansions: X-Fast doubles temporal size by doubling frame rate over the same input duration, while X-Temporal increases both frame rate and input duration.X-Fast uses γτ←0.5γτ; X-Temporal uses γτ←0.75γτ and γt←2γt.
  • Network expansions: X3D defines separate depth, global-width, and bottleneck-width expansions with approximate factors of 2.2, 2, and 2.25, respectively.The exact factors can differ slightly between steps because of rounding effects in network geometry.
  • Training and regularization: Regularization expansion begins with doubled batch size and a halved learning schedule when training and validation errors diverge.Batch-normalization statistics are then computed over 16 clips, lowering regularization and improving performance on small models.

B. Additional Results

Additional experiments show that X3D maintains favorable accuracy-efficiency trade-offs across inference clip counts and test sets, while mobile components affect models differently across compute regimes. SE blocks matter more than swish, whereas separable convolutions are especially important for small budgets.

  • Inference cost: Using a logarithmic FLOPs axis reveals up to 20× lower inference multiply-adds for X3D-S and X3D-M.The comparison varies the number of temporal testing clips and measures top-1 accuracy against full video inference cost.
  • Inference cost: 3-clip X3D-S reaches 71.4% top-1 at 5.9 GFLOPs, while 10-clip CSN-50 reaches 70.8 top-1 at 119 GFLOPs.The CSN-50 comparison has 20.2× higher cost.
  • Inference cost: 10-clip X3D-S reaches 72.9% top-1 at 19.6 GFLOPs, while 10-clip CSN-101 reaches 71.8% top-1 at 159 GFLOPs.The CSN-101 comparison has 8.1× higher cost.
  • Inference cost: The same inference trends hold on the more challenging Kinetics-400 test set, indicating consistent model generalization.The test-set plots use the same accuracy-versus-inference-cost comparison.
  • Mobile components: Removing separable convolution lowers top-1 accuracy by 4% for X3D-S and 2.4% for X3D-XL when FLOPs are preserved.Keeping the expanded bottleneck ratio instead increases baseline performance by 0.3% for X3D-S and 0.8% for X3D-XL, but at higher cost.
  • Mobile components: Removing SE blocks decreases accuracy by 1.6% for X3D-S and 1.3% for X3D-XL, a larger effect than removing swish.Removing swish decreases performance by 0.9% for X3D-S and 0.4% for X3D-XL.
  • Mobile components: The observed SE-block effects align with prior findings that SE attention blocks benefit efficient video classification.The paper also notes similarity to effects reported for Non-Local attention blocks.
Loading 2004.04730v1…