Source-linked AI summary
A Dual-Transformer for Multi-Camera View Recommendation
Josep Cabacas-Maso, Carles Ventura, Ismael Benito-Altamirano
TL;DR
Multi-camera view recommendation must select the correct viewpoint while accounting for visual content, temporal history, and editing practice. The paper introduces a Dual-Transformer with cross-attention that separates temporal-memory construction from candidate evaluation, achieving a reported 69.65% Precision@0.5 with its flagship configuration.
Problem
Multi-camera editing requires selecting the correct shot while accounting for visual content, temporal context, and cinematography, but prior architectures entangle history and candidate evaluation.
Method
The Dual-Transformer builds historical memory with a dedicated temporal encoder, then uses candidate views as queries in a cross-attention module.
Results
69.65% Precision@0.5 was achieved by the flagship configuration combining the Dual-Transformer, Swin Transformer V2 backbone, and Focal Loss.
Takeaways & Limitations
The results support hierarchical window-based attention as the most discriminative visual-backbone choice for this multi-camera editing benchmark.
Takeaways & Limitations
The TVMCE dataset is not multimodal, limiting models that rely on modalities such as audio to determine when to change cameras.
Abstract
from arXiv · showhide
Multi-camera systems are foundational to modern media production, and multi-camera editing is a critical task. This involves the proper selection of the appropriate camera view at each moment. In this paper, we propose a novel Dual-Transformer architecture with Cross-Attention that heavily outperformed the current SOTA models over the TVMCE dataset (TV Shows Multicamera Editing dataset). Our model decouples these tasks: (1) a dedicated temporal encoder first processes the sequence of past frames to build a rich memory of the recent history, and (2) the candidate camera views then act as queries to this memory via a cross-attention module, allowing each candidate to independently interrogate the historical context and find the most relevant information for its own evaluation. Our approach achieved 56.60% Precision@0.5, representing a substantial improvement over the prior best result of 37.16%. We further conducted an ablation study exploring the use of lightweight backbone architectures, where the SwinV2 backbone yielded the best performance, achieving 69.65% Precision@0.5. Using this best-performing configuration, we then investigated the feasibility of adapting the model to replicate the editing style of a specific human editor. To this end, we fine-tuned the model using varying proportions of the initial segment of a target video. Our results demonstrate that even with only 20% of the video used for fine-tuning, the model exhibited measurable improvements in Precision@0.5, indicating strong potential for data-efficient personalization of editing style adapted to each individual TV show or producer.
1 Introduction
Multi-camera editing requires selecting an appropriate viewpoint at each moment while preserving production quality and narrative experience. The paper proposes decoupling historical-context modeling from candidate-view evaluation to address the limitations of prior unified Transformer architectures.
- Multi-camera editing selects the appropriate camera view at each moment and directly affects final production quality and audience narrative experience.
- Successful view recommendation must jointly consider camera-feed content, recently shown temporal context, and established cinematography practices.
- The TVMCE dataset introduced a large-scale benchmark for multi-camera linear editing and enabled deep-learning models such as the TC Transformer.
- Prior architectures entangle temporal-history understanding and future-candidate evaluation by processing them as one flat or concatenated Transformer sequence.
- The proposed Dual-Transformer separates temporal encoding from candidate evaluation through two Transformer branches connected by cross-attention.
2 Related Work
Related work spans automated video editing, multi-camera view recommendation, and Transformer-based video understanding. The literature highlights scarce realistic training data, advances enabled by TVMCE, and increasing use of cinematographic features and hierarchical visual backbones.
- 2.1 Automated Video Editing: Automated video-editing research covers video summarization, highlight detection, transition recommendation, sports broadcasts, theater recordings, and virtual-camera systems.
- 2.1 Automated Video Editing: A central training challenge is the lack of edited videos paired with corresponding raw footage, motivating pseudo-labeled datasets.
- 2.1 Automated Video Editing: Early pseudo-labeling approaches approximated raw footage from edited videos but could not properly capture the multi-camera editing problem because confuser-frame selection was flawed.
- 2.2 Multi-Camera Editing and View Recommendation: The TVMCE dataset addressed data scarcity with professionally recorded and edited footage from diverse scenarios, although it is not multimodal.
- 2.2 Multi-Camera Editing and View Recommendation: Subsequent models incorporated temporal duration, camera identity, and shot-type features to add cinematographic knowledge to view recommendation.
- Transformer Architectures: Transformer architectures became standard sequence-modeling tools, with video models including ViViT, SwinV1, and SwinV2 extending Transformer-based video understanding.
3 Proposed Method
The proposed architecture builds a temporal memory from past frames, then lets candidate views query that memory through cross-attention rather than concatenating history and candidates. Candidate interactions and prediction scoring follow this cross-attention stage.
- Feature and Positional Encoding: A shared visual backbone extracts features, which are projected into the model dimension and combined with camera and temporal encodings.
- Architecture Overview: The Dual-Transformer uses separate temporal and candidate pathways that converge through cross-attention.
- Temporal Encoder (Memory): The Temporal Encoder processes P = 16 past tokens formed by summing visual features, temporal encodings, and the current camera identity.
- Temporal Encoder (Memory): The Temporal Transformer converts past tokens into historical memory M, which supplies the keys and values for cross-attention.
- Candidate Encoding and Cross-Attention: Candidate tokens combine candidate visual features with their respective camera identities instead of concatenating candidates with historical tokens.
- Candidate Encoding and Cross-Attention: Candidate tokens first form queries, retrieve historical information through multi-head cross-attention, and then interact through a shallow self-attention layer.
- Prediction Head and Loss: An MLP head with Layer Normalization and GELU produces logits, while Focal Loss addresses class imbalance using α = 0.25 and γ = 2.0.
4 Experiments and Analysis
Experiments evaluate the Dual-Transformer on TVMCE, examining its architecture, visual backbones, thresholds, alternative metrics, and data-efficient personalization. The model reaches strong benchmark performance and adapts to human editing styles with limited video-specific fine-tuning.
- Experimental Setup: The experiments test whether decoupling temporal memory from candidate queries improves recommendation, which visual backbones are most discriminative, and how much data enables editor-specific adaptation.These questions organize the architectural, backbone, and personalization analyses.
- Experimental Setup: The evaluation uses TVMCE, containing 88 hours of raw videos and 14 hours of edited videos across concerts, sports, gala shows, and contests.The task predicts the optimal camera view at each editing boundary from P = 16 past shots and C = 6 candidate views.
- Main Results: 52.60% Precision@0.5 is achieved by the decoupled architecture with BCE loss, exceeding the previous SOTA and supporting the proposed architectural separation.The result is presented as evidence that unified encoding prematurely entangles historical context with future candidates.
- Main Results: 56.60% Precision@0.5 is achieved by the evaluated configuration, while the SwinV2-Tiny plus Focal Loss configuration reaches 69.65%.The flagship configuration combines the SwinV2-Tiny backbone with Focal Loss.
- Backbone Ablation: 69.65% is the best backbone-ablation result, while ViT-Base reaches 25.85% and ConvNeXtV2 reaches 56.23%.The ablation reports stronger performance for hierarchical models and attributes SwinV2’s result to shifted-window attention and log-spaced positional bias.
- Threshold Analysis: 81.41% validation F1-score occurs at τ = 0.3, followed by test Precision of 74.66%, Recall of 80.35%, and F1-score of 76.52%.Using τ = 0.3 recovers correct views that the default 0.5 threshold would discard under the reported Focal Loss behavior.
- Extended Evaluation: The model achieves 76.31% Recall@1, and macro-to-micro threshold tuning shifts the optimum from τ = 0.3 to τ = 0.4 while retaining comparable test F1-scores.Recall@1 measures the percentage of samples where the most confident candidate matches the ground truth.
- Data Efficiency and Adaptation: Approximately 84% precision is reached after fine-tuning on 20% of a specific video, while cross-video evaluation remains at 83.65% at the 20% mark.The reported findings support data-efficient adaptation and robustness across the two evaluated videos.
5 Conclusion
The paper addresses entangled temporal-history and candidate-view reasoning with a Dual-Transformer, achieving strong benchmark performance and demonstrating threshold tuning and adaptation to editing styles.
- 69.65% Precision@0.5 establishes a new state-of-the-art on the TVMCE benchmark using a Swin Transformer V2 backbone and Focal Loss optimization.The paper attributes the backbone result to hierarchical window-based attention providing discriminative features for cinematographic decision-making.
- The Dual-Transformer decouples temporal memory construction from candidate-view evaluation through a dedicated temporal memory and cross-attention module.This architecture explicitly separates historical-context reasoning from candidate selection.
- τ = 0.3 macro and τ = 0.4 micro thresholds recover accurate predictions, yielding a test F1-score of 76.52%.The conclusion also reports Recall@1 of 76.31% when considering only the most confident camera.
- The model demonstrates potential for rapid, high-fidelity adaptation to human editing styles from limited production data.The paper frames this adaptation as a consequence of decoupling temporal history and candidate selection.