Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding

Motivation

• Existing multimodal methods commonly specialize in either images or videos, creating a trade-off between fine spatial detail and temporal modeling.Image-focused methods use more tokens for spatial understanding, while video-focused methods reduce per-frame spatial detail to model temporal relationships.Source: §1 · pp. 1–2

Unified Representation

• Chat-UniVi uniformly represents images and videos with dynamic visual tokens that capture image details and video temporal relationships.Image regions receive tokens at diverse sizes, while video tokens expand across frames within events.Source: §1 · p. 2

Training and Results

• Chat-UniVi is trained on mixed image and video data and can be applied directly to both modalities without modification.The unified formulation reduces the number of visual tokens while maintaining expressive capability.Source: §1 · p. 2

Token Construction

• The method progressively merges semantically similar visual tokens using DPC-KNN clustering and averages tokens within each cluster.For videos, DPC-KNN is also applied to frame features to identify events before temporal token merging.Source: §1 · p. 2

Multi-scale Representation

• Chat-UniVi uses multi-scale representations so high-level features support semantic understanding while low-level features support detailed descriptions.This representation is intended to support varied image and video understanding tasks.Source: §1 · p. 2

Training and Results

• Chat-UniVi consistently outperforms methods designed exclusively for images or videos and achieves competitive performance across image and video tasks without fine-tuning.The introduction also reports impressive results on an object hallucination benchmark.Source: §1 · p. 2, Figure 2 · p. 2

Multimodal Models

• Multimodal models broadly use LLMs as dispatch schedulers for vision experts or integrate different modalities into end-to-end trainable systems.The related-work discussion places Chat-UniVi among models that seek broader multimodal capability.Source: §2 · pp. 2–3

Dynamic Tokens

• Prior dynamic-token methods explore token adaptation within transformers, but the cited methods cannot be directly extended to video.This limitation motivates extending dynamic tokenization across both image and video inputs.Source: §2 · p. 3

Chat-UniVi

• Chat-UniVi extends dynamic tokens to video and integrates image and video representations in a unified framework.The authors describe this as the first demonstration of reconciling image spatial detail with video temporal understanding.Source: §2 · p. 3, Figure 3 · p. 3
• Its clustering method is parameter-free, which the authors identify as central to the model’s success amid gradient instability in multimodal conversation training.The authors attribute the convergence difficulty to parameterized methods under gradient instability.Source: §2 · p. 3

Unified Visual Representation

• Chat-UniVi uniformly represents images and videos through dynamic visual tokens, bridging spatial image details with temporal video relationships.The representation is designed for a unified language sequence that LLMs can comprehend.Source: §3 · pp. 3–4

Spatial Visual Token Merging

• DPC-KNN clusters image tokens by local density and distance, then averages each cluster into one representative token.Cluster centers use relatively high ρ_i × δ_i, and merged regions are formed from the union of constituent vision regions.Source: §3.1 · p. 4

Temporal Visual Token Merging

• Temporal visual token merging clusters frames into critical events before expanding tokens only across frames within the same event.Frame representations are obtained by mean-pooling their tokens, and DPC-KNN identifies event clusters.Source: §3.1 · p. 4

Multi-scale Representation

• Multi-scale aggregation provides visual features at different merging stages for both high-level semantics and low-level detail.The method progressively merges CLIP-derived visual tokens while preserving representations at multiple scales.Source: §3.1 · p. 4

Efficiency

• 112 visual tokens are used by Chat-UniVi compared with 256 for LLaVA, despite retaining dynamic visual representations.The reported comparison emphasizes the lower token count of the proposed method.Source: §3.1 · p. 4

Multimodal Training Scheme

• Joint instruction tuning uses a mixture of multi-turn and single-turn conversations with single images, multiple images, and videos.The model is fully fine-tuned with the language model and projection matrix, enabling direct processing of images and videos without realignment.Source: §3.2 · p. 5

Video Understanding

• Chat-UniVi significantly surpasses recently proposed state-of-the-art methods focused exclusively on video in GPT-based video understanding evaluation.Evaluation uses ActivityNet-based videos and GPT-3.5 scores correctness, detail, context, temporal understanding, and consistency.Source: §4.2 · p. 6

Question-Answer Evaluation

• Chat-UniVi outperforms recently proposed methods across zero-shot video question-answering datasets, including MSRVTT-QA, MSVD-QA, TGIF-QA FrameQA, and ActivityNet-QA.The evaluation follows GPT-assisted assessment protocols.Source: §4.3 · p. 6

Object Hallucination Evaluation

• Chat-UniVi outperforms recently proposed state-of-the-art methods on POPE and even exceeds the 13B MiniGPT-4 despite using 7B parameters.The authors attribute this success to multi-scale representations combining high-level semantics with low-level visual appearance.Source: §4.4 · pp. 6–7, Table 5 · p. 6

Ablation Studies

• Joint training on image and video data improves understanding of both media, whereas training on only one medium decreases comprehension of the other.The ablation compares single-medium tuning, sequential image-then-video fine-tuning, and mixed image-and-video training.Source: §4.5 · p. 7, Table 6 · p. 7
• Fewer spatial clusters reduce fine-detail capacity, while more clusters add redundancy and can reduce overall performance.The practical configuration uses 64, 32, and 16 clusters across three spatial levels.Source: §4.5 · p. 7, Table 7 · p. 7

Human Evaluation

• Human evaluations cover 30 image and 30 video conversation scenarios, with evaluators rating multimodal conversation performance from 0 to 10.The final model score is the average rating; Chat-UniVi surpasses Flamingo-based, image-specific, and video-specific models.Source: §4.6 · pp. 7–8, Figure 4 · p. 7

5. Conclusion

• Chat-UniVi uses dynamic visual tokens and multi-scale features to represent images and videos within one unified multimodal framework.DPC-KNN progressively clusters visual tokens and supplies multi-scale representations.Source: §5 · p. 8
• Mixed image-and-video training lets Chat-UniVi apply directly to tasks involving both media without modifications.The training data encompasses both images and videos.Source: §5 · p. 8
• Chat-UniVi consistently surpasses methods designed exclusively for images or videos.The conclusion characterizes this result as supported by extensive experimental evidence.Source: §5 · p. 8

Comparison with Other Multimodal Methods

• Q-former methods extract a fixed number of tokens per image or video, making variable-length videos difficult to model effectively.Chat-UniVi instead determines temporal-cluster counts proportionally to input frame count.Source: §A.1 · p. 13
• Chat-UniVi allocates more temporal visual tokens to longer videos through variable-length video features.This contrasts with Q-former-based methods that use fixed token counts.Source: §A.1 · p. 13
• Joint training on mixed image-video datasets simultaneously enhances the model’s understanding of both images and videos.The unified representation permits direct application to tasks involving both media.Source: §A.1 · p. 13

Comparison with Clustering Transformer Methods

• Chat-UniVi extends token clustering to video inputs, integrating image and video representations with a parameter-free method.The authors report that parameterized clustering performs significantly worse and attribute this to gradient instability.Source: §A.2 · p. 13
• The clustering algorithm’s time and memory costs are negligible compared with those of the large language model.This comparison is reported in the corresponding cost analysis.Source: §A.3 · p. 13

• Pretrained language models transmit their vulnerabilities to Chat-UniVi.Source: §A.4 · p. 14
• Hallucinations remain unresolved and can produce unsupported conjectures during open multimodal conversations.Source: §A.4 · p. 14
• Long-sequence constraints cause forgetfulness in multi-turn conversations and declining performance with multiple video inputs.Source: §A.4 · p. 14
• Natural-language outputs are inconvenient for bounding boxes and dense pixel predictions, while their flexibility complicates evaluation.Source: §A.4 · p. 14
• Future work can add modalities such as audio to broaden tasks and potentially enhance video understanding.Source: §A.4 · p. 14

• Multimodal pre-training uses image-caption pairs from datasets including COCO and CC3M-595K.Source: B. Implementation Details · p. 14
• Joint instruction tuning combines multimodal in-context, visual-instruction, and video-instruction data.Source: B. Implementation Details · p. 14
• Runtime and memory complexity are analyzed using vanilla visual-token count L, feature dimension D, and frame length M.Source: Table C · p. 15
• Videos use 64 input frames, while images and frames are resized to 224×224.Source: B. Implementation Details · p. 15

• LoRA achieves competitive performance with full fine-tuning while saving more than half the GPU memory required for training.Source: C. Additional Experiments · p. 15
• EVA-CLIP performs comparably to OpenAI-CLIP at equal parameter counts, while its 1.8B version may suit larger language models.Source: C. Additional Experiments · pp. 15–16
• Multi-scale representation improves resistance to object hallucinations by combining high-level semantics with low-level visual appearance.Source: C. Additional Experiments · p. 16
• Chat-UniVi outperforms LLaVA in conversation, reasoning, and hallucination evaluations, while video instruction data especially enhances detailed captioning and reasoning.Source: C. Additional Experiments · p. 16
• A 7B Chat-UniVi model outperforms the 13B MiniGPT-4 model in object hallucination evaluation.Source: C. Additional Experiments · p. 16

• Dynamic visual-token visualizations cover portraits, sports, wildlife, art, architecture, and food.Source: D. Additional Visualization Results · p. 16
• The token-merging method is parameter-free and does not require object-outline labels.Source: D. Additional Visualization Results · p. 17
• Dynamic visual tokens generalize objects and backgrounds while capturing image spatial nuances with limited visual tokens.Source: D. Additional Visualization Results · p. 17

• Chat-UniVi provides detailed, contextually appropriate responses in conversations containing both images and videos.Source: E. Additional Qualitative Analysis · p. 17
• Chat-UniVi uses information from multiple videos to produce appropriate and coherent responses across conversational turns.Source: E. Additional Qualitative Analysis · p. 17
• Chat-UniVi leverages multiple images within context to make choices among various images during conversation.Source: E. Additional Qualitative Analysis · p. 17
• For image conversations, Chat-UniVi provides detailed descriptions and can craft compelling narratives, supporting reasoning and creative expression.Source: E. Additional Qualitative Analysis · pp. 17–18
• For video conversations, Chat-UniVi comprehends video content and offers insights inspired by the videos across conversational turns.Source: E. Additional Qualitative Analysis · p. 18

Image understanding

• Image understanding is evaluated on 90 questions from 30 COCO validation images covering conversation, detail description, and complex reasoning.Source: F. Details of Quantitative Evaluations · p. 18
• GPT-4 scores image responses for helpfulness, relevance, accuracy, and detail on a 1 to 10 scale using visual information, generated responses, and references.Source: F. Details of Quantitative Evaluations · p. 18

Video understanding

• Video understanding evaluation uses an ActivityNet-200-based test set containing dense captions and human-annotated question-answer pairs, with GPT-3.5 assigning relative scores.Source: F. Details of Quantitative Evaluations · pp. 18–19

Dynamic visual tokens

• The dynamic visual-token visualizations emphasize a parameter-free token-merging method that requires no object-outline labels.Source: F. Details of Quantitative Evaluations · pp. 19–20
• The video token-merging method imposes no restriction on the number of frames per event, indicating flexibility across video inputs.Source: F. Details of Quantitative Evaluations · p. 20

Object hallucination

• POPE evaluates object hallucination as binary Yes-or-No classification using questions generated from objects annotated in 500 COCO validation images.Source: F. Details of Quantitative Evaluations · pp. 20–21