Source-linked AI summary
SALI: Shot-Aware Late Interaction for Cross-Shot Relation Matching in Text-to-Video Retrieval using Film-Grammar Knowledge
Toya Oyama, Rainer Lienhart, Shin'ichi Satoh
TL;DR
Text-to-video retrieval often represents a clip with one embedding, which can miss relations spread across movie shots. SALI matches the full query and its subject and object against shot embeddings, using greedy max or optimal transport. It achieves the highest Rel-Multi R@1 and R@5 on Condensed Movies with overall recall on par, while short-clip MSR-VTT results include a 1.4 overall R@1 cost.
Problem
A single video embedding can miss relations between people distributed across alternating shots, a gap that matters for retrieval on movies and dramas.
Source: §1 · p. 1Method
Results
SALI+max achieves the highest Rel-Multi R@1 and R@5 on Condensed Movies with overall recall on par, while SALI+OT has the highest overall R@1.
Source: §5 · p. 5Takeaways & Limitations
Relational-query gains are attributed to greedy max at inference, while the film-grammar penalty adds a small, consistent shift.
Source: §1 · p. 2, §5 · p. 5On MSR-VTT, SALI+max is 1.4 R@1 below CLIP4Clip-meanP, and the experiments do not determine whether short clips or the 12-frame budget accounts for the drop.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Text-to-video retrieval usually represents a video clip by a single embedding. This embedding often loses important relations between people. E.g., an interaction "Anna confronts Mark" is regularly filmed as alternating shot and reverse shot of both (Fig. 1a). No single shot or averaged embedding over clip shots captures this relation. Thus, we propose SALI (Shot-Aware Late Interaction). It extracts the subject and object from a single-sentence query, and matches the query, its subject and object text embeddings against each visual shot embedding of a video clip. The matching operator is greedy max or optimal transport. A film-grammar penalty in fine-tuning adds a small, consistent shift. Built on CLIP4Clip-meanP, SALI keeps overall recall on par on Condensed Movies and ActivityNet while raising R@1 on multi-shot relation queries by 3 and 12 points, the most among all compared methods, and improves such queries on MSR-VTT at a cost of 1.4 R@1 overall.
1. INTRODUCTION
Text-to-video retrieval systems often reduce a clip to one embedding, leaving movie and drama shot structure unrepresented. SALI retains shot embeddings and matches the full query, subject, and object to shots, targeting relations spread across alternating shots.
1. INTRODUCTION
- A single clip embedding misses the shot and scene structure of movies and dramas, where two-minute clips can contain dozens of shots.The paper calls this mismatch the single-video-embedding gap.Source: §1 · p. 1
- Alternating shot/reverse-shot editing can place an interaction across different shots, so per-shot matching cannot distinguish the true video from a one-shot look-alike.Mean pooling also dilutes the relation, while late interaction can find the people separately without representing how they appear in the query.Source: §1 · p. 1, Figure 1 · p. 1
- SALI keeps one embedding per shot and matches the full query, subject, and object against shots using greedy max or optimal transport.The film-grammar penalty in fine-tuning adds a small, consistent shift.Source: §1 · pp. 1–2
2. DIAGNOSIS: DOES THE PROBLEM EXIST?
On Condensed Movies, the single-video-embedding failure is strongest for relational queries whose videos contain many shots. The diagnosis motivates evaluating relational queries at or above the median shot count.
2. DIAGNOSIS: DOES THE PROBLEM EXIST?
- Condensed Movies is the primary testbed; its two-minute scenes have captions about characters, and its relational queries are defined using interpersonal verbs and person nouns or pronouns.The diagnosis uses all 10,621 obtainable videos, with each caption as the query and its video as the only correct answer.Source: §2 · p. 2
- Relational-query R@10 falls from 28.3 in the fewest-shot quartile to 14.8 in the most-shot quartile (−13.5), versus 23.5 to 18.2 (−5.3) for non-relational queries.Rel-Multi comprises relational queries whose videos have at least the median shot count of 34 shots.Source: §2 · p. 2
3. SALI: METHOD
SALI represents a video with one embedding per shot and compares each shot with the full query and its subject and object. It offers greedy-max and optimal-transport matching, with optional film-grammar penalties during training.
3. SALI: METHOD
- SALI encodes each shot by averaging its frame embeddings and splits interpersonal queries into subject and object spans, alongside the full query.The three text embeddings share a text projection, and shot embeddings use a visual projection.Source: §3 · p. 2
- Greedy max lets each text embedding choose its most similar shot, while optimal transport distributes embedding mass across shots under a learnable per-shot capacity.The transport plan uses five entropic Sinkhorn iterations; with capacity 1 and temperature approaching zero, its score equals greedy max.Source: §3 · p. 2
- The optional film-grammar penalties discourage matching subject and object to the same shot or to shots showing the same person.Cross-shot face clustering builds the same-person matrix from training videos only; it is used in training, so inference needs no face pipeline.Source: §3 · pp. 2–3, Figure 2 · p. 3
- Training mixes the shot-level score with a clip-level score initialized to match CLIP4Clip-meanP, then fine-tunes both CLIP towers using symmetric InfoNCE.At inference, the trained model is used unchanged with either greedy max or optimal transport.Source: §3 · p. 3
4. EXPERIMENTS
Experiments evaluate SALI on Condensed Movies and the generalization datasets ActivityNet and MSR-VTT, using an expanded Condensed Movies gallery enriched for relational queries. SALI+max leads on Condensed Movies relational retrieval and raises ActivityNet Rel-Multi recall, while MSR-VTT shows an overall R@1 cost.
Datasets and protocols
- The Condensed Movies evaluation gallery contains 2,262 clips and 203 relational queries, including 113 Rel-Multi queries, with no movie shared with training.SALI uses the official frame counts, and results are averaged over three training seeds; baselines are single runs.Source: §4 · p. 3
Main results
- On Condensed Movies, SALI+max achieves the highest Rel-Multi R@1 and R@5, while overall R@1 drops by 0.29.It raises overall R@5 and R@10; SALI+OT instead has the highest overall R@1 but lower Rel-Multi R@1 than CLIP4Clip-meanP.Source: §4 · pp. 3–4, Table 2 · p. 4
Ablation
- Switching OT-trained models to greedy max at inference raises Rel-Multi R@1 by about four queries at a cost of 0.7–0.8 overall R@1.The authors interpret this as OT spreading text-embedding mass across shots, which helps ordinary queries but hurts relational queries with one shot per person; the Rel-Multi gains come from greedy max at inference.Source: §4 · p. 4, Table 3 · p. 4
Controls
- SALI+max’s gains in overall R@5 and Rel-Multi R@1 and R@5 exceed those from score ensembling and four extra CLIP4Clip-meanP training epochs.The Rel-Multi R@1 margin over the largest ensembling gain is only 0.30.Source: §4 · p. 4
Generality
- On ActivityNet, SALI+max keeps overall recall on par with CLIP4Clip-meanP while Rel-Multi R@1 rises by 12 points and R@5 by 4.On MSR-VTT, relational R@1 and R@5 rise, but overall R@1 drops by 1.4; OT is on par overall and below max on relational queries on both datasets.Source: §4 · p. 4, Table 4 · p. 4
Limitation
- On MSR-VTT’s 10–30-second clips with 12 frames per video, SALI+max is 1.4 R@1 below CLIP4Clip-meanP.The experiments do not establish whether the short clips or the 12-frame budget accounts for the drop.Source: §4 · p. 4
5. CONCLUSION
SALI matches whole-query, subject, and object representations to shot embeddings using greedy max or optimal transport. On Condensed Movies, SALI+max leads on relational R@1 and R@5 with overall recall on par; it also raises ActivityNet relational recall, while MSR-VTT incurs an overall R@1 cost.
Contribution
- SALI represents a clip with one embedding per shot and matches the query, subject, and object against shots using greedy max or optimal transport.The relational-query gains come from greedy max at inference; the film-grammar penalty adds a small, consistent shift.Source: §5 · pp. 4–5
Results
- On Condensed Movies, SALI+max achieves the highest Rel-Multi R@1 and R@5 with overall recall on par, while SALI+OT has the highest overall R@1.SALI+max also raises ActivityNet Rel-Multi recall and improves MSR-VTT relational queries at a cost of 1.4 overall R@1.Source: §5 · p. 5