DCA: Diversified Co-Attention towards Informative Live Video Commenting

• ALVC generates real-time comments from video frames and surrounding comments to express opinions or enable viewer interaction.Automatic comments are described as making video content easier to understand and engaging viewers.Source: §1 · p. 1
• ALVC has seen little exploration, and its sole prior approach used separate attention for video and comments without integrating their information.That approach may limit information diversity.Source: §1 · p. 1
• Surrounding comments are based on the video and can highlight important frame features, motivating interactions between the modalities.Source: §1 · p. 2
• DCA outperforms previous approaches and traditional co-attention, reaching state-of-the-art results in ALVC.Source: §1 · p. 2
• DCA learns different distance metrics to build bidirectional interactions between video frames and surrounding comments from multiple perspectives.Source: §1 · p. 2
• Parameter orthogonalization is proposed to reduce excessive overlap among information extracted from different perspectives.Source: §1 · p. 2

2 Diversified Co-Attention Model

• The ALVC input comprises video frames and surrounding comments, which are concatenated into one comment sequence.The model generates a reasonable and fluent comment from these inputs.Source: §2 · p. 2

2.1 Video Encoder & Text Encoder

• GRU encoders represent video frames and words, using a CNN to map raw images to dense vectors and word embeddings for text.Source: §2.1 · p. 3

2.2 Diversified Co-Attention

• DCA builds bidirectional interactions between video and text representations from different perspectives through metric learning.Source: §2.2 · p. 3

Single Perspective

• A shared learned transformation maps video and text representations before their inner products form similarity scores; rowwise and column-wise normalization yields opposing attention directions.The transformed inner product approximates a learnable distance metric while avoiding the need to maintain a positive-semidefinite W directly.Source: Single Perspective · p. 4

Multiple Perspectives

• DCA learns K distance metrics, each producing its own similarity matrix and co-dependent representations, which are mean-pooled across perspectives.Source: Multiple Perspectives · p. 4

2.3 Parameter Orthogonalization

• Because learned perspective matrices may become similar, the model applies post-backpropagation updates equivalent to an orthonormality constraint; simple regularization may collapse training.The constraint is intended to reduce overlap among the information captured by perspectives.Source: §2.3 · p. 5

2.4 Gated Attention Module

• The Gated Attention Module attends separately to co-dependent and original representations, then gates them into balanced textual and visual representations.A learnable vector and an outer product help form the final context vector for generation.Source: §2.4 · pp. 5–6

2.5 Decoder

• A GRU decoder uses the context vector to generate a comment word by word and is trained to maximize the ground-truth comment’s log-likelihood.Source: §2.5 · p. 6

3.1 Data and Settings

• The Live Comment Dataset contains 895,929 instances from 2,361 videos; experiments use a 34-layer pretrained ResNet and set K = 3.Training uses Adam for 50 epochs, with an initial learning rate of 3e-4 and dropout of 0.1.Source: §3.1 · p. 6

3.2 Baselines

• Baselines cover video-only, text-only, concatenation, separate-attention, and traditional co-attention models, with Transformer versions of Seq2Seq approaches.Source: §3.2 · p. 7

3.3 Evaluation Metrics

• Automatic evaluation ranks ground-truth comments among 100 candidates using Recall@k, MR, and MRR; human ratings assess fluency, relevance, informativeness, and overall quality.Source: §3.3 · pp. 7–8

3.4 Experiment Results

• DCA ranks ground-truth comments higher than other models in automatic evaluation.The table caption identifies the results as automatic evaluation and defines R@k; the prose reports the comparison without giving numerical values.Source: Table 1 · p. 8, §3.4 · p. 8
• Human judges favor DCA, with its largest margin over baselines in Informativeness.The human-evaluation table averages five annotators’ scores; the reported Informativeness margin is larger than for the other evaluated aspects.Source: Table 2 · p. 8, §3.4 · p. 8
• Results are consistent across Seq2Seq and Transformer models, suggesting the DCA modules adapt to different architectures.Source: §3.4 · p. 9

3.5 Ablation Study

• Replacing DCA with traditional co-attention significantly reduces performance; parameter orthogonalization and GAM are also critical components.Orthogonalization alleviates redundancy among perspectives, while GAM uses gates to integrate co-dependent and original representations and balance video and text.Source: §3.5 · p. 9

3.6 Visualization of DCA

• With orthogonalization, the three similarity matrices differ visibly; without it, they are highly similar, indicating redundancy across perspectives.The figure shows five surrounding frames against 20 comment words, with deeper color indicating higher relevance.Source: §3.6 · p. 9, Figure 3 · p. 10

4 Related Work

• Article-commenting work includes Chinese news dataset construction, unsupervised retrieval [16], and visual, personalized, and graph-based comment-generation approaches,,.Unlike article commenting, ALVC models interactions between text and dynamic video, whose co-dependent relationship makes the task more challenging for AI models.Source: §4 · p. 10
• Video-captioning methods generate descriptions from video,,,,, whereas ALVC also requires interaction with human viewers and modeling visual-textual dependencies.Source: §4 · p. 10
• Prior co-attention spans visual QA, dense co-attention, user-item relationships, object co-segmentation, and video QA,; DCA instead learns distance metrics and orthogonalizes perspectives.Source: §4 · pp. 10–11

5 Conclusion

• DCA uses bidirectional interactions across multiple distance metrics to capture dependencies between video frames and surrounding comments.The interactions produce diversified co-dependent representations.Source: §5 · p. 11
• Parameter orthogonalization is proposed to avoid excessive overlap among information extracted from different perspectives.Source: §5 · p. 11
• Experiments show that DCA substantially outperforms existing methods and generates comments with more novel and valuable information.Source: §5 · p. 11