Source-linked AI summary
ECO: Efficient Convolutional Network for Online Video Understanding
Mohammadreza Zolfaghari, Kamaljeet Singh, Thomas Brox
TL;DR
Video understanding methods must capture activities spanning several seconds without sacrificing efficient processing of entire videos. ECO samples redundant neighboring frames sparsely, merges long-term representations inside a 3D network, and supports online prediction. The approach achieves competitive performance across tasks while running 10x to 80x faster than state-of-the-art methods.
Problem
Long-term activities require temporal context, while existing local or window-based methods process whole videos inefficiently and often rely on suboptimal post-hoc fusion.
Method
ECO samples frames across the video, processes appearance with 2D convolutions, and learns relationships among distant frame representations with a 3D network.
Results
ECO delivers on-par or favorable performance on most action-recognition and video-captioning datasets with a superior runtime-accuracy trade-off, running 10x to 80x faster than state-of-the-art methods.
Takeaways & Limitations
The architecture captures long temporal context while processing only a small subset of frames, supporting fast whole-video and online understanding.
Takeaways & Limitations
Comparisons are restricted to raw RGB input without additional preprocessing such as optical flow or human pose.
Abstract
from arXiv · showhide
The state of the art in video understanding suffers from two problems: (1) The major part of reasoning is performed locally in the video, therefore, it misses important relationships within actions that span several seconds. (2) While there are local methods with fast per-frame processing, the processing of the whole video is not efficient and hampers fast video retrieval or online classification of long-term activities. In this paper, we introduce a network architecture that takes long-term content into account and enables fast per-video processing at the same time. The architecture is based on merging long-term content already in the network rather than in a post-hoc fusion. Together with a sampling strategy, which exploits that neighboring frames are largely redundant, this yields high-quality action classification and video captioning at up to 230 videos per second, where each video can consist of a few hundred frames. The approach achieves competitive performance across all datasets while being 10x to 80x faster than state-of-the-art methods.
1 Introduction
ECO addresses the tension between capturing long-term temporal context and processing whole videos efficiently. It integrates sampled frame representations within the network and supports fast online understanding across action recognition and video captioning.
- Long-term activities require temporal context, but computationally expensive 3D CNNs cover only short windows and post-hoc fusion is suboptimal.
- ECO samples sparse frames, processes appearance with a 2D network, and learns relationships among distant representations with a 3D network.
- The architecture captures complex actions over long temporal contexts while processing only a small fraction of video frames.
- ECO supports online understanding by providing a fast initial action guess and refining it as longer-term context accumulates.
- Experiments on action recognition and video captioning show on-par or favorable performance and a superior runtime-accuracy trade-off on most datasets.
3 Long-term Spatio-temporal Architecture
The long-term architecture samples one frame from each temporal subsection, extracts appearance features efficiently, and learns temporal relationships among the sampled representations. ECO Full adds a parallel 2D stream for static semantics, while direct prediction avoids expensive test-time aggregation.
- ECO Lite: Each video is divided into N equal subsections, one frame is randomly sampled per subsection, and shared 2D processing extracts each frame’s appearance representation.Random positions increase training diversity while runtime processes only N frames.
- ECO Lite: Stacked frame representations are fed into a 3D convolutional network that learns how appearances combine over time and outputs the final action label.
- ECO Lite and ECO Full: The architecture is trained end-to-end and can be adapted to video captioning and other video understanding tasks.
- ECO Lite and ECO Full: ECO Full adds a parallel 2D network whose average-pooled frame features represent static scene semantics alongside the 3D temporal features.
- Inference: A single forward pass predicts the whole video from N sampled frames without additional score aggregation, reducing test-time computation.
4 Online video understanding
ECO adapts its unchanged network to streaming input by maintaining a working memory of sampled frames and updating it with incoming frames. This preserves long-term context while producing predictions continuously and efficiently.
- Online processing is needed when video arrives as a stream, but sliding windows either omit long-term context or introduce long delays.
- At each update, the method samples half the frames from the incoming queue and half from the previous working memory before running ECO.
- The online modification changes only sampling: incoming video is partitioned into N-frame segments, and a working memory stores the N model inputs with timestamps.
- 675 fps with ECO and 970 fps with ECO Lite demonstrate efficient online inference on a Tesla P100 GPU.
- Keeping exactly N frames makes the online model memory-efficient and suitable for smaller hardware, including mobile devices.
5 Experiments
Across action classification and video captioning, ECO delivers competitive or favorable accuracy with a superior accuracy–runtime trade-off, including strong results on temporally demanding datasets. Its sparse sampling and integrated temporal modeling also support efficient whole-video and online processing.
- Benchmark performance: ECO achieves on-par or favorable performance against state-of-the-art approaches across action classification and video captioning using only RGB images.The evaluation covers UCF101, HMDB51, Kinetics, Something-Something, and Youtube2Text.
- Accuracy–runtime trade-off: ECO architectures match other approaches’ accuracy at much faster rates and process videos at least an order of magnitude faster.The runtime comparison uses videos per second to account for methods that sample only part of each video.
- Accuracy–runtime trade-off: 675 fps for ECO and 970 fps for ECO Lite are reported on a Tesla P100 GPU, although frames-per-second does not capture whole-video processing time.The paper therefore reports videos per second for video-level speed comparisons.
- Sampling analysis: Accuracy generally decreases with fewer sampled frames, but four samples still outperform most literature approaches because ECO models relationships between temporally distant instants.Simple short-term actions can instead perform worse with denser sampling, whereas complex actions benefit from it.
- Sampling analysis: ECO Lite is slightly less accurate but faster than full ECO, while on Something-Something it performs equally well with the same number of input samples.The dataset’s stronger reliance on temporal context reduces the relevance of raw single-image cues.
- Online understanding: In online J-HMDB experiments, ECO reaches good accuracy faster and saturates at a higher absolute accuracy than existing online methods.The online evaluation measures how many frames are needed to approach full accuracy.
- Video captioning: ECO features improve video captioning across BLEU, METEOR, and CIDEr, and concatenating ECO with ResNet features improves results further.ECO Lite is already on-par with an ImageNet-pretrained ResNet architecture.
6 Conclusions
ECO is a simple, efficient architecture that uses a small subset of video frames while exploiting temporal context, achieving strong results across multiple video-understanding tasks and running 10x to 80x faster than state-of-the-art methods.
- 10x to 80x faster than state-of-the-art methods, ECO achieves excellent results on action classification, online action classification, and video captioning.
–Supplementary Material–
The supplementary document provides additional experimental details and results for work by Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox at the University of Freiburg.
- The paper is authored by Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox.
- The authors are affiliated with the University of Freiburg.
- The supplementary document provides additional details and experimental results.
1 ECO
ECO combines parallel 2D and 3D processing: sampled frames receive per-frame visual representations, which are then processed temporally to produce video-level output.
- ECO averages pooled 2D-network features, concatenates them with 3D-network features, and produces a one-hot vector for class labels.
- Each video is divided into N equal subsections, and one frame is randomly sampled from each subsection.
- The 2D network processes sampled frames individually to provide visual semantics, while the 3D network processes their temporally stacked representations.
2 Network Architectures
ECO consists of parallel 2D and 3D components that transform sampled frames into feature maps and model their temporal relationships; its input contains N frames of size 224 × 224.
- ECO consists of three submodules, including 2D-Net, 3D-Net, and a parallel 2D-NetS component.
- 2D-Net: 2D-Net uses BN-Inception through inception-3c to create feature maps Mi for each input frame.
- 3D-Net: 3D-Net receives concatenated 2D-Net feature maps as Mϕ ∈ R^C×N×H×W, where N is the number of sampled frames and H = W = 28.
- 2D-NetS: 2D-NetS uses BN-Inception from inception-4a through the layer before the last pooling layer, operating in parallel with 3D-Net.
- The ECO architecture takes N frames of size 224 × 224 as network input.
3 Sampling Function for Online Learning
ECO samples incoming video streams to retain long-range information while emphasizing recent frames. Incremental updates keep only sampled history and the latest frame queue in memory, then average successive predictions.
- Sampling strategy: The sampling strategy considers long-range information from the incoming stream while giving greater importance to recent frames.It is designed for online video understanding.
- Initialization: At the first time step, the method uses all N incoming frames as network input.The initial sampled set is the current queue QF.
- Incremental update: At later time steps, sampling combines previously sampled frames with frames from the recent queue QF.The update keeps sampled history and newly arriving frames rather than storing the entire stream.
- Recency weighting: Recent frames contribute more than older frames because the sampled history is updated incrementally with samples from the recent queue.The method stores only SF and QF in memory.
- Prediction update: ECO updates predictions by averaging scores from the previous sampling and the current sampling.This follows feeding each updated sampled set to ECO.
4 Video Length VS Number of Samples
Sampling density should match video length: sparse sampling is preferable for shorter videos, whereas denser sampling helps longer videos up to a point. Sampling-location shifts have limited performance impact, and more samples reduce variability.
- Video length: Shorter videos favor sparse sampling, while longer videos benefit from denser sampling up to some point.Videos were grouped into five length ranges from 0–60 through 240–320 frames.
- Short videos: Action recognition is harder for videos shorter than 60 frames, and sparse sampling limits confusion in that regime.The comparison varies the number of samples across video-length categories.
- Sampling location: Shifting the temporal location of sampled frames does not excessively affect performance on UCF101 and HMDB51 split1.The experiment reports mean and standard deviation for shifted sampling locations.
- Sampling stability: Increasing the number of samples decreases the standard deviation of performance across sampling locations.This indicates more stable results under temporal shifts.
6 Early Action Recognition: UCF101
ECO performs well for early action recognition on UCF101, achieving strong predictions after observing only a few frames. Its video representation also supports qualitative captioning improvements over SCN on MSVD.
- Early action recognition: ECO performs already very well after observing only a few frames in online learning on UCF101 split1.The result is shown in the early action classification experiment.
- Early action recognition: The early-classification result indicates that ECO obtains high accuracy after seeing a short part of the video.Figure 3 reports this behavior on UCF101 split1.
- Video captioning: On MSVD, ECO improved qualitative video-captioning results over SCN.The comparison uses the same language model, while ECO benefits from a better video feature representation.