Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images

1. INTRODUCTION

• Fine-grained actions are short and visually similar within shared contexts, while untrimmed web videos usually provide only video-level labels.These conditions make it difficult to associate the labels with action occurrences.Source: §1 · pp. 1–2
• The method iteratively transfers action classifiers between video frames and web images, filtering low-confidence examples in both domains.CNN feature activations provide a common representation, and the final transfer from web images localizes action-related video frames.Source: §1 · p. 2
• LSTM detectors model temporal evolution, with localized-action-frame scores weighting the misclassification loss for sampled video frames.The framework can also use video shots as time steps to capture local motion information.Source: §1 · p. 2
• FGA-240 introduces fine-grained localization on untrimmed videos and contains over 130,000 videos covering 240 actions across 85 sports activities.The authors state that the method can extend beyond sports, giving birthday-party actions as an example.Source: §1 · p. 2
• Combining weakly labeled video frames with noisily tagged web images lets bidirectional domain transfer identify localized action frames.Web images queried by action names often provide well-localized highlights, although some are irrelevant.Source: §1 · p. 2

2. RELATED WORK

• Existing activity datasets include pre-segmented clips, while Sports-1M offers untrimmed videos but mostly high-level activity categories; prior fine-grained localization work used kitchen videos with temporal annotation.The authors state that, to their knowledge, fine-grained localization with untrimmed training videos had not previously been studied.Source: §2 · p. 2
• Traditional action recognition aggregates hand-designed visual and motion features into fixed-length vectors; SIFT and dense trajectories with Fisher Vectors produced state-of-the-art results.Oneata et al. reported that combination.Source: §2 · p. 2
• Deep approaches jointly learn features and classifiers, including CNNs for video patches and two-stream CNNs using optical flow and raw pixels.The cited examples are Karpathy et al. and Simonyan and Zisserman [24].Source: §2 · pp. 2–3
• Prior localization approaches used spatio-temporal boxes, poselets, or tubelets, while THUMOS 2014 localization used classifiers trained on trimmed segments and temporal sliding windows.The cited box-based approaches require manually annotated spatio-temporal boxes.Source: §2 · p. 3
• Auxiliary web images have supported high-level event concepts and subcategory filtering, while one image-to-video adaptation method assumes perfect image annotation.The cited examples are, Yang et al., and Divvala et al..Source: §2 · p. 3
• LSTM networks were developed to improve on RNNs for time series by retaining or forgetting past values, and have been applied to video-level action classification.Prior applications also include speech, handwriting, and machine translation.Source: §2 · p. 3

3. OUR APPROACH

• The framework filters web images and video frames through bidirectional domain transfer, scores video frames for localized actions, and uses those scores to weight LSTM detector training.The LAF scores reduce the penalty for misclassifying unrelated frames or frames used to gather context.Source: §3 · p. 3, §3.3 · p. 5

3.1 Shared CNN Representation

• Video frames and web images share CNN activation features, enabling classifiers trained in one domain to transfer action information to the other.The method fine-tunes the top three AlexNet layers and uses fc6 activations as the shared representation.Source: §3.1 · pp. 3–4

3.2 LAF Proposal with Web Images

• Action-name image searches provide localized highlights for many actions, but their noisy results include irrelevant images, objects, and logos.The proposal model assigns video frames LAF scores based on the image-trained classifier’s response for the frame’s action label.Source: §3.2 · pp. 4–5
• A video-trained classifier filters noisy action-tagged web images, and an image-trained classifier filters video frames; this process iterates to produce the LAF proposal model.The initial frame set is randomly sampled; those frames can help filter non-video-like images such as cartoons, object photos, and logos.Source: §3.2 · pp. 4–5

3.3 Long Short-Term Memory Network

• The LSTM treats sampled video frames as time steps and uses LAF scores as penalty weights during training, emphasizing correct decisions on proposed action frames.Frames are sampled at 1 frame per second, and training uses truncated backpropagation through time.Source: §3.3 · p. 5

4.1 Data Set

• FGA-240 contains over 130,000 YouTube videos across 240 fine-grained sports actions under 85 high-level activities.Source: §4.1 · p. 5

4.2 Experiment Setup

• Classification averages frame-level softmax scores; localization averages 10-step windows, applies non-maximum suppression, and uses mean average precision at overlap ratio r.Source: §4.2 · p. 6

4.3 Video-level Classification Results

• CNN activations outperformed low-level features, and LSTM outperformed fine-tuned CNN, with temporal context helping classification.Source: §4.3 · pp. 6–7, Table 1 · p. 7
• LAF proposals raised video hit @1 by 2.3% and video hit @5 by 4.7%.Source: §4.3 · p. 7

4.4 Localization Results

• Across 45 FGA-240 categories, LSTM with LAF proposals outperformed CNN and LSTM without LAF, with the gap widening as overlap ratio increased.Source: §4.4 · p. 7, Figure 9 · p. 9

4.5 Localization Results on THUMOS 2014

• On THUMOS 2014, LSTM with LAF scores outperformed LSTM without LAF and the video-level baseline by large margins, though manually localized ground truth performed best.Source: Table 5 · p. 9, §4.5 · p. 9

• The paper studies fine-grained action localization in temporally untrimmed web videos.Source: §5 · p. 9
• Noisily tagged web images are used to discover localized action frames, and LSTM networks model temporal information.Source: §5 · p. 9
• Evaluations on FGA-240 and THUMOS 2014 show the effectiveness of the LAF proposal by domain transfer from web images.Source: §5 · p. 9