MarineLife-16K: An Attribute-Rich Video-Text Dataset for Motion-Aware Marine Video Understanding

• Marine behaviors such as schooling and reef navigation can signal ecological functions, motivating a shift from object-centric tracking to motion-centric video understanding.Source: §1 · p. 2
• Existing benchmarks emphasize broad video summaries and lack motion diversity, while marine QA also needs fine-grained attributes and grounding.Source: §1 · p. 2
• The human-in-the-loop captioning engine uses fine-grained annotations to reduce motion hallucinations, addressing a failure left unresolved by MSC [23].The dataset is reported as 5× larger than MSC [23] and includes motion-aware attributes.Source: §1 · p. 2
• MarineLife-16K contributes 2,000 marine video-text pairs, 13 fine-grained attributes, and a 16K-pair QA benchmark, with models struggling to understand camera motion.Source: §1 · p. 2

• Figure 2 contrasts Gemini 3.1’s hallucinated camera-motion descriptions with labeled wide-angle, close-up, and bird’s-eye ground truths.Source: Figure 2 · p. 3
• Marine video-language systems integrate sequential visual tokens with language, yet fine-grained temporal comprehension remains difficult in challenging underwater scenes.Source: §2 · p. 3
• Video-QA benchmarks target long-form multimodal reasoning, fine-grained egocentric alignment, or narrative comprehension, while video-text datasets often use automatic captioning.Source: §2 · p. 3
• The caption-engine illustration shows human-annotated attributes guiding caption generation, review against those attributes, and refinement of low-confidence outputs.Source: Figure 3 · p. 3

3.1. Data engine.

• Three experts annotate marine attributes, agents draft and score captions, and a refiner revises captions scoring below 3.5 against the attributes.Source: §3.1 · p. 3

3.2. Collection and annotation

• Each video records a central object and up to three additional objects, with appearance, behavior, environment, lighting, clarity, and camera attributes.The annotation process takes around 10 minutes per video, excluding viewing time.Source: §3.2 · p. 4

3.3. Video data quality.

• Videos span 27 locations, different recording times, and seven countries, and six reviewers assessed the 2,000 captions.Source: §3.3 · p. 4

3.4. Motion-aware QA Generation

• Multiple-choice answers come from fine-grained attributes, while open-ended QA uses captions to test motion understanding and camera-motion reasoning.The resulting QA evaluates links between camera movement and object motion, appearance, and geometry.Source: §3.4 · p. 5

3.5. Statistics.

• The dataset contains 2,000 videos and 16,080 questions: 12,080 multiple-choice and 4,000 open-ended.The split comprises 1,300 training, 200 validation, and 500 test videos.Source: §3.5 · p. 5

4. Evaluation Dimensions

• The evaluation tests both broad video summaries and question grounding through multiple-choice marine QA and video captioning.Source: §4 · p. 5

4.1. Video-QA

• Video-QA dimensions include multi-object marine communities, phenotypic traits, organism behavior, environmental context, and viewpoint-dependent observations.The open-ended dimension targets camera viewpoint, object motion, central objects, and visible activity.Source: §4.1 · p. 5

4.2. Video Captioning with Motion Understanding

• Captioning predictions receive 0–5 scores for biological-entity, action, and environmental-context hallucinations.Figure 5 plots caption frequencies across object types, video quality, central-object traits, camera motion, and object motion; the human study reports agreement between human and LLM judges on open-ended QA.Source: §4.2 · pp. 5–6, Figure 5 · p. 6, Figure 6 · p. 6

5.1. Models

• The evaluation covers proprietary and open-source VLMs, using eight uniformly sampled frames per video as ordered image-sequence input.The authors report that eight frames balance camera-motion capture, accuracy, and token cost.Source: §5.1 · p. 6

5.2. Human Study

• A six-expert study found human ratings largely consistent with GPT-5.5 ratings for open-ended QA, despite some gap between judges.Six experts rated 100 questions, with three experts per question and averaged scores.Source: §5.2 · p. 7

5.3. Multiple-choice Questions with Motion Understanding

• OBK averages were 88.66% (7B-8B open-source), 87.77% (31B-78B open-source), and 95.78% (closed-source); PDO averages were 63.32%, 45.53%, and 60.8%, respectively.The authors report robust object-motion understanding but difficulty with complex camera motion.Source: §5.3 · p. 7, Table 3
• Qwen2.5-VL-7B (70%) and InternVL3-8B (70.0%) exceed Grok-4.3 (53.3%) and Claude-Opus (60.0%) on camera-motion PDO.This contrasts with the closed-source group’s lead across the overall average.Source: §5.3 · p. 7, Table 3
• Proprietary models outperform open-source models across most downstream evaluation dimensions.The comparison covers MBC-so, MBC-mo, PMT, OBK, SM-l, and OEMU.Source: §5.3 · p. 7, Table 3

5.4. Open-ended Questions

• In open-ended QA, both model groups perform better with moving cameras than static cameras, yet 7B-8B models score 2.6-2.9 across metrics under camera movement.Multiple-choice performance is about 50% on SM-l, SE-e, and OEMU with moving cameras, while MBC-so is about 90% in both settings.Source: §5.4 · p. 8, Figure 8 · p. 8, Table 4 · p. 8

6. Conclusion

• The sequential multi-trajectory expert annotation and refinement strategy uses a human-in-the-loop multi-agent data engine to curate motion-centric marine video-text data.The resulting benchmark contains 2,000 marine videos and 16,080 fine-grained question-answer pairs.Source: §6 · p. 8
• The benchmark annotates 13 attributes and evaluates open-ended and multiple-choice QA across nine dimensions; both open-source and closed-source models struggle most with camera-motion dynamics.The authors hope the benchmark advances underwater video understanding.Source: §6 · p. 9