Source-linked AI summary
Fine-grained Activity Recognition in Baseball Videos
AJ Piergiovanni, Michael S. Ryoo
TL;DR
Fine-grained activity recognition in broadcast baseball is difficult because activities can look similar, while much detailed MLB tracking data is not publicly available. The paper introduces MLB-YouTube and compares temporal recognition approaches across segmented and continuous videos. It reports that learned temporal structures perform best in the two settings, with different representations leading each task.
Problem
Broadcast baseball activities can have very small visual differences, and much detailed pitch-tracking data is not publicly available.
Source: §1 · p. 1Method
Results
Learning sub-events to select temporal regions-of-interest performs best for segmented classification, while convolutional sub-events with a super-event hierarchy perform best for continuous-video detection.
Source: §7 · p. 7Takeaways & Limitations
The reported comparisons support the value of learning temporal structure for fine-grained activity recognition.
Source: §6.2.1 · p. 5, §7 · p. 7
Abstract
from arXiv · showhide
In this paper, we introduce a challenging new dataset, MLB-YouTube, designed for fine-grained activity detection. The dataset contains two settings: segmented video classification as well as activity detection in continuous videos. We experimentally compare various recognition approaches capturing temporal structure in activity videos, by classifying segmented videos and extending those approaches to continuous videos. We also compare models on the extremely difficult task of predicting pitch speed and pitch type from broadcast baseball videos. We find that learning temporal structure is valuable for fine-grained activity recognition.
1. Introduction
Sports activity recognition could automate statistics now tracked manually, but much detailed MLB tracking data is not publicly available. MLB-YouTube addresses this gap with densely annotated broadcast baseball video and comparisons of temporal feature-pooling approaches.
1. Introduction
- Much detailed data from MLB's PITCHf/x and Statcast systems is not publicly available, despite their automatic capture of pitch speed and motion.The systems use multiple high-speed cameras and radar to measure players on the field.Source: §1 · p. 1
- MLB-YouTube provides densely annotated broadcast baseball frames for fine-grained activity recognition, where activities can differ mainly in one person's motion.The paper compares temporal feature-pooling approaches for segmented-video classification and continuous-video activity detection.Source: §1 · p. 1
2. Related Works
Prior activity-recognition work spans hand-crafted and learned visual features, while temporal aggregation methods pool or model information across video intervals. Related detection approaches use short 3D CNN segments, longer-range temporal operations, or recurrent models.
2. Related Works
- Activity-recognition methods progressed from hand-crafted dense trajectories to CNNs, including two-stream RGB/flow and 3D spatiotemporal models.Large-scale datasets cited for training CNNs include Kinetics [11], THUMOS, and ActivityNet.Source: §2 · p. 1
- Temporal aggregation studies found benefits from pooling intervals of varied locations and lengths and learning important sub-event intervals [14].Ng et al. compared pooling methods and reported that LSTMs and whole-video max-pooling performed best.Source: §2 · p. 1
- Segment-based activity-detection methods use 3D CNNs that usually cover 16 frames, while other work models longer structure through pooling, wider convolutions, or recurrent networks.The cited recurrent models address activity transitions between frames.Source: §2 · p. 1
3. MLB-YouTube Dataset
MLB-YouTube comprises segmented and continuous baseball videos drawn from 2017 postseason broadcasts, where activities share camera views and can be visually similar. The dataset annotates activity labels and, for pitches, pitch type and speed.
3. MLB-YouTube Dataset
- The dataset draws on 20 postseason games from 2017, totaling over 42 hours, and includes segmented and continuous video components.The shared broadcast viewpoint and small appearance or motion differences make single-frame activity classification difficult.Source: §3 · p. 2
- Strike-versus-ball recognition can depend on whether the batter swings or the umpire signals a strike, but the umpire may be occluded and signals vary by umpire.The visual difference between a strike and ball is described as small.Source: §3 · p. 2, Figure 3 · p. 2, Figure 4 · p. 2
- The segmented dataset contains 4,290 multi-label clips with baseball activities, pitch type and speed labels for pitches, and 2,983 hard negatives.The hard negatives include crowd, field, and players-standing views before or after a pitch; Table 1 lists activity classes and instance counts.Source: §3 · p. 2, Table 1 · p. 3
- The continuous dataset comprises minute-long clips from videos, with each frame annotated for occurring baseball activities; clips average 7.2 activities, yielding over 15,000 activity instances.Source: §3 · p. 3
4. Segmented Video Recognition Approach
Segmented-video recognition compares temporal aggregation methods over CNN frame or segment features, ranging from global pooling to learned and recurrent temporal structure. Binary cross-entropy trains the multi-label classifiers.
4. Segmented Video Recognition Approach
- The base features come from two-stream CNNs using I3D or InceptionV3 to provide per-frame or per-segment representations.Segmented clips simplify classification because every frame corresponds to the activity, so the model need not determine activity boundaries.Source: §4 · p. 3
- Fixed temporal pyramid pooling preserves coarse order by max-pooling video intervals of lengths 1/2, 1/4, and 1/8, then concatenating their features for classification.This produces a K × D representation, where K is the number of intervals.Source: §4 · p. 3
- Temporal convolution applies a learned L × 1 kernel to each frame, then max-pools the outputs to aggregate nearby-frame information.Source: §4 · p. 3
- Learned sub-event filters parameterize N temporal intervals with a center, width, and stride, then map video features to an N × D representation for classification.The filters are normalized, and their matrix-multiplied output feeds a fully connected classifier.Source: §4 · p. 3
- A bidirectional LSTM with 512 hidden units uses its last hidden state as the input to a fully connected classifier.Source: §4 · p. 3
- The models formulate recognition as multi-label classification and minimize binary cross-entropy, with G(v) denoting temporal pooling and z_c the ground-truth class label.Source: §4 · p. 3
5. Activity Detection in Continuous Videos
Continuous-video detection classifies activities frame by frame, including locating their starts and ends amid sequential instances and inactive frames. The paper extends temporal aggregation approaches to this setting using sliding windows or per-frame feature processing.
5. Activity Detection in Continuous Videos
- Continuous-video detection must classify each frame while identifying activity starts and ends across sequential instances, often separated by inactive frames.A per-frame fully connected classifier serves as a baseline and adds no temporal information beyond the input features.Source: §5 · p. 3
- Sliding-window max-pooling and temporal pyramid pooling classify pooled segments, with the pyramid splitting each window into 14 intervals of lengths L/2, L/4, and L/8.Source: §5 · pp. 3–4
- For continuous-video temporal convolution, a learned kernel of length L maps T × D features to T × D outputs for per-frame classification.The operation aggregates local temporal information.Source: §5 · p. 4
- The sub-event approach sets T = L, convolves its filters with video features, and uses the resulting N × D × T representation to classify each frame.Source: §5 · p. 4
- Per-frame training uses binary classification loss over sliding-window features and ground-truth labels at each time step.Source: §5 · p. 4
- Learned temporal structure filters form super-event context, combined with per-class soft-attention weights; the representation can be concatenated with each timestep or with sub-event features.The latter creates a three-level event hierarchy.Source: §5 · p. 4
6. Experiments
The experiments compare temporal aggregation methods on segmented and continuous baseball videos, as well as pitch-speed and pitch-type prediction. Learned sub-events perform best for segmented activity classification, while combining convolutional sub-events with a super-event representation gives the best continuous-video detection performance.
6.1. Implementation Details
- The base models are pretrained I3D segment features and two-stream InceptionV3 frame features, sampled at 3 fps and 8 fps respectively.InceptionV3 features use frames sampled every three frames; I3D uses every frame with temporal stride 8.Source: §6.1 · p. 5
6.2. Segmented Video Activity Recognition
- Binary pitch/non-pitch classification is relatively easy, and the tested features and models show little difference on this task.Source: §6.2 · p. 5
6.2.1 Multi-label Classification
- Learning sub-events gives the best multi-label classification performance, while all tested temporal aggregation approaches outperform mean/max-pooling.Temporal convolution provides a larger increase but uses significantly more parameters; LSTMs require sequential processing and perform worse, likely due to overfitting.Source: §6.2.1 · p. 5, Table 4 · p. 5, Table 3 · p. 5
- Sub-event learning especially helps frame-based features and activities with post-event visual changes, including strikes, hits, foul balls, and hit by pitch.After a hit, the camera often follows the ball; after a hit by pitch, it may follow the player walking to first base.Source: §6.2.1 · p. 5, Figure 6 · p. 6, Figure 7 · p. 6
6.2.2 Pitch Speed Regression
- At 60 fps, pitch-speed prediction reaches 3.6 mph average error; the initial 8 fps and 3 fps features were insufficient to determine speed.The model predicts speed with a one-output fully connected layer trained using L1 loss.Source: §6.2.2 · p. 6, Table 6 · p. 7
6.2.3 Pitch Type Classification
- For pitch-type classification, learning sub-events helps, whereas LSTMs perform worse than the baseline, likely from overfitting small differences between pitch types.Fastballs are easiest to detect at 68% accuracy, followed by sliders at 45%; sinkers are hardest at 12%.Source: §6.2.3 · p. 6, Table 7 · p. 7
6.3. Continuous Video Activity Detection
- Continuous-video detection requires locating activity starts and ends across sequential events and intervening frames with no activity.Source: §6.3 · p. 6
- Combining convolutional sub-events with a super-event representation provides the best continuous-video detection performance.All models improve over per-frame classification; fixed temporal pyramids outperform max-pooling, while LSTMs and temporal convolution appear to overfit.Source: §6.3 · pp. 6–7
7. Conclusion
The conclusion introduces MLB-YouTube for fine-grained activity recognition and reports that learned temporal structures perform best for segmented classification and continuous-video detection.
7. Conclusion
- MLB-YouTube is a challenging new dataset for fine-grained activity recognition in videos.Source: §7 · p. 7
- Table 8 reports multi-label activity classification results on continuous videos using per-frame mAP.Source: Table 8 · p. 7
- For segmented classification, learning sub-events to select temporal regions-of-interest gives the best performance; for continuous detection, a three-level hierarchy of convolutional sub-events and super-events performs best.These are the reported best-performing approaches for the two video settings.Source: §7 · p. 7