Connectionist Temporal Modeling for Weakly Supervised Action Labeling

• Because large-scale video action annotation is costly, the paper motivates weak supervision as an alternative, including supervision extracted from movie scripts or instructional videos [2].Source: §1 · p. 1
• Training provides only the order of actions, but the learned model must label each frame in unseen videos without temporal localization.Fully annotated videos with accurate per-frame action labels are costly and are required by many existing methods [48].Source: §1 · p. 1
• CTC uses dynamic programming to efficiently evaluate possible frame-to-action alignments, addressing the infeasibility of searching them naively.The work introduces CTC, originally designed for speech recognition, to video understanding.Source: §1 · p. 2
• ECTC uses frame-to-frame visual similarity to encourage consistent alignments and down-weight degenerated, visually inconsistent paths.It extends the forward-backward algorithm of [13] to evaluate alignments while enforcing their consistency with visual similarities.Source: §1 · p. 2
• For semi-supervision, ECTC incorporates labels localized in a few video frames as alignment constraints, pruning inconsistent alignments and reducing the alignment space.Sparse frame labels can be obtained from movie scripts or annotators, at lower cost than precisely marking all action boundaries.Source: §1 · pp. 2–3
• With less than 1% of supervision, the method achieves state-of-the-art performance on the Breakfast Actions Dataset and a subset of Hollywood2.Source: §1 · p. 3

2 Related Work

• Video classification assigns one action class per video, whereas dense action labeling predicts actions frame by frame and typically trains with fully annotated temporal boundaries.The related work cites UCF101, HMDB51, Sports1M, and ActivityNet as challenging video-classification datasets.Source: §2 · p. 3
• Unlike OCDC, which can use action ordering at test time, this approach targets unseen videos without action-order information and leverages learned action transitions.The paper argues that OCDC becomes a frame-by-frame classifier in this setting and thus does not exploit temporal relationships at test time.Source: §2 · p. 3
• Instructional-video and movie-script parsing [2] uses text available at test time, whereas this work models unseen test videos without associated text.Text-processing approaches could potentially provide supervision, but are not the focus of this work.Source: §2 · pp. 3–4
• Rather than classifying a whole video, this work uses video temporal structure to train a model that labels each frame in unseen videos.It integrates visual similarity, as used in unsupervised video parsing, as soft guidance alongside weak supervision rather than merely grouping frames.Source: §2 · p. 4
• RNNs capture temporal dependencies and have been used for activity recognition [8] [27] and dense video labeling [48], but often require pre-segmented data or ignore localization.ECTC enables recurrent temporal models to learn action labeling with weak supervision.Source: §2 · p. 4

3.1 Extended Connectionist Temporal Classification

• ECTC trains an RNN from ordered action sequences by summing over compatible frame-label paths and reweighting them according to consecutive-frame visual similarity.Training minimizes the negative log likelihood of the ordered label sequence, which sums probabilities over paths consistent with that order.Source: §3 · p. 4, §3.1 · p. 5
• Its binary term rewards retaining an action across visually similar consecutive frames, helping avoid visually inconsistent and trivial paths.Frames with low similarity are not penalized for retaining the same action.Source: §3.1 · p. 6

3.2 ECTC Forward-Backward Algorithm

• The ECTC forward-backward algorithm uses dynamic programming to efficiently evaluate all possible paths despite the dependence on consecutive-frame similarity.It extends the original CTC forward-backward approach to incorporate the binary term for frame-to-frame correlation.Source: §3.2 · p. 7

Optimization.

• Forward-backward gradients use posterior action probabilities over order-consistent, similarity-reweighted paths as framewise softmax targets, enabling RNN backpropagation.The cited optimization passage interprets the target as the probability of choosing an action at a frame among paths consistent with the sequence label.Source: Optimization · p. 8

• ECTC's visual-similarity constraints softly down-weight inconsistent paths, while sparse labels provide harder constraints that remove them.Source: §4 · p. 8
• Sparse frame annotations paired with action order reduce the possible alignment paths and exclude undesirable paths.Source: §4 · pp. 8–9
• The supervision is a list of frame-index/action-label pairs, obtainable from movie scripts or a few annotated frames instead of exact action boundaries.Source: §4 · p. 9
• For a six-frame video with J = [(a, 2), (b, 4)] and order [a, b], only two paths remain, excluding [a, a, a, a, a, b].Source: §4 · p. 9
• The method extends the Eq. (6) recursion with a check that rejects paths whose labels conflict with annotated frames.Source: §4 · p. 9
• The authors state that with less than 1% of frames labeled, their approach can perform comparably to a fully supervised model.Source: §4 · p. 9

5.1 Implementation Details

• The implementation uses a 1-layer BLSTM with 256 hidden units and combines adjacent-frame clustering constraints with cosine similarity to guide ECTC alignment search.Frames are over-segmented into clusters using k-means with M = 20; cosine similarity is used at cluster boundaries.Source: §5.1 · pp. 9–10

5.2 Evaluating Complex Activity Segmentation

• In ablations, clustering similarity prunes visually inconsistent paths, cosine similarity further improves unit accuracy, and their combination gives the final model.Direct CTC can produce a degenerated path dominated by one action despite preserving action order; ECTC uses visual similarity to improve localization and ordering.Source: §5.2 · pp. 10–11, Figure 5 · p. 11
• On Breakfast, final ECTC improves accuracy over its component variants and outperforms fully supervised baselines.The reported component improvements include clustering similarity, cosine similarity, and their combination in the final model.Source: Table 1 · p. 11, §5.2 · p. 11
• With frame-level supervision, ECTC’s frame accuracy drops more slowly than Uniform’s and it significantly outperforms OCDC at every labeled-frame fraction.Even with only one labeled frame per occurring action (fraction 0.0025), the method has the highest unit accuracy in the weakly supervised setting; a single frame anchor per segment reduces possible alignments and provides a strong training cue.Source: §5.2 · pp. 11–12, Figure 7 · p. 12
• On Breakfast training-set alignment, OCDC performs closer to ECTC than in the unseen-video setting, while ECTC also produces good training-set alignments.The evaluation reports frame accuracy and Jaccard measure, the latter used to assess alignment quality.Source: §5.2 · p. 13, Figure 8 · p. 13

5.3 Evaluating Action Detection

• On held-out Hollywood2 videos, both ECTC approaches outperform OCDC for action detection, while ECTC performs best on action alignment.Experiments use five random data splits; the zero-annotation fraction corresponds to ordering-only supervision for all training videos.Source: Figure 9 · p. 13, §5.3 · p. 14

• With less than 1% supervision, ECTC achieves results comparable to state-of-the-art fully supervised models and outperforms fully supervised baselines with weak supervision.Source: §6 · p. 14
• ECTC uses visual similarity between consecutive frames to weight possible alignments, evaluated efficiently with a dynamic programming algorithm.The visual-similarity weighting captures dependencies between consecutive frames.Source: §6 · p. 14
• ECTC extends to frame-level semi-supervision in a unified framework, significantly reducing the space of possible alignments.Source: §6 · p. 14
• The framework is evaluated on activity segmentation and action detection.Source: §6 · p. 14