OmniFysics-Captioner Technical Report: Grounding Omni-Modal Understanding in the Physical World for Better Captioning

• Daily-Physics 50K covers six physical-event categories and 23 observable subcategories, from brief interactions to extended multi-stage processes.Its clips are mostly shorter than one minute, with a smaller set of one- to ten-minute videos retaining long-range causal chains and slow changes.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Existing omni-modal captioners often lack perceptual granularity, spatiotemporal grounding, physical fidelity, and complete, traceable cross-modal evidence.Transient contact and deformation, longer state changes, and outcomes require localized observation and cross-modal verification.Source: §1 · p. 2
• The framework combines CATA clip discovery, active OmniFysics-Agent evidence acquisition, a PPM fine-tuned on approximately 2M image-level samples, and tool-free caption generation.Daily-Physics 50K provides approximately 50K physics-rich video–caption pairs.Source: §1 · pp. 2–3
• OmniPhysCap (OPC) evaluates recovery of physical and cross-modal evidence in captions using 1,000 audiovisual clips and 8,000 questions.Blind full-video verification and a Not Mentioned option distinguish omitted evidence from conflicting evidence.Source: §1 · p. 3
• OmniFysics-Captioner achieves state-of-the-art performance on VDC Detailed, Omni-Cloze, and caption-to-QA cascade evaluations across multiple omni-modal benchmarks.Source: §1 · p. 3
• Ablations show that explicit physical perception improves coverage of object properties, physical interactions, and state changes.Source: §1 · p. 3

• Omni-modal captioning methods improve audiovisual descriptions through data construction, preference optimization, temporal fusion, and tool use, while evaluations increasingly use QA decomposition, temporal reasoning, and information recovery.Source: §2 · p. 3
• Existing captioning methods often reduce collisions, deformation, and state changes to action labels instead of explaining underlying object interactions.Source: §2 · p. 3
• Physical-understanding benchmarks assess properties, relations, dynamics, grounding, anomalies, and mechanics, but largely use predefined questions or outcome prediction rather than open-ended caption evidence.Source: §2 · p. 3

3.1 Physical Video Dataset

• CATA retrieves physics-rich clips from nine video sources using event queries and frame–query similarities, then merges temporal anchors into segments.The resulting set contains approximately 50K clips; human evaluation found 98.2% temporally complete and 96.9% consistent with their assigned physical-event categories.Source: §3.1 · p. 4

3.2 OmniFysics-Agent

• OmniFysics-Agent iteratively plans localized ALM, VLM, and PPM observations, stores evidence, and synthesizes a caption from the resulting evidence memory.PPM is fine-tuned on approximately 2M image-level physical-instruction examples; the agent uses a proxy for global navigation and original local media for fine-grained verification.Source: §3.2 · p. 5

3.3 OmniFysics-Captioner

• The Captioner is trained on 50K agent-captioned audiovisual pairs to generate captions from raw audio and video in one forward pass without external tools.Training uses teacher forcing on final-caption tokens only, not temporal windows, tool requests, observer returns, or reflection trajectories.Source: §3.3 · p. 6

3.4 OmniPhysCap (OPC) Benchmark

• OPC evaluates caption evidence retention with 1,000 audiovisual clips and 8,000 multiple-choice probes covering eight information types, including physical interactions and audiovisual alignment.Its fixed caption-only Judge protocol includes a Not Mentioned option to distinguish missing evidence from incorrect content choices; questions are checked for consistency and video support, then deduplicated and cross-validated by annotators.Source: §3.4 · pp. 6–7

4.1 Caption Evaluation

• 57.9% Accuracy and a 2.9 VDCscore on VDC Detailed, an F1 score of 36.2 on DREAM-1K, and leading open-source performance on Omni-Cloze establish strong detailed-captioning results.The model outperforms proprietary and open-source baselines on VDC Detailed and leads open-source models across the three benchmarks.Source: Table 2 · p. 7, §4.1 · p. 8
• 64.6 on Daily-Omni is 0.5 percentage points above Gemini 3.1 Pro; OmniFysics-Captioner ranks first among open-source models on WorldSense and Video-MME.Its scores are 45.7 on WorldSense and 64.8 on Video-MME, where proprietary models remain ahead.Source: §4.1 · p. 8, Table 3 · p. 8

4.2 Results and Analysis of OPC benchmark

• On OPC, OmniFysics-Captioner scores 50.3% Total, 57.7% Phys., and 46.1% Non-Phys., beating the strongest baseline by 8.0, 9.9, and 5.3 percentage points, respectively.The strongest baselines are Gemini 3.1 Pro on Total and Phys., and AVoCaDO on Non-Phys.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 Multi-Level Analysis of OmniFysics-Agent

• 72.6 on Daily-Omni, 53.0 on OPC Total, and 63.6 on OPC Phys. show that OmniFysics-Agent outperforms standalone Qwen3-Omni and Gemini 3.1 Flash-Lite captioning.The workflow uses active evidence acquisition and cross-tool aggregation of temporally distributed, modality-specific evidence.Source: §4.3 · pp. 9–10
• Removing PPM from a matched Agent setup lowers OPC Phys. from 63.6 to 55.8, a 7.8-point drop, compared with smaller declines on Daily-Omni and OPC Total.The ablation holds the Planner, Finalizer, ALM/VLM access, call budget, caption instruction, and decoding settings unchanged.Source: §4.3 · p. 10
• PPM-enriched supervision raises the tool-free Captioner’s Daily-Omni score from 62.7 to 64.6, OPC Total from 45.7 to 50.3, and OPC Phys. from 53.1 to 57.7.The comparison uses captions generated from the same video manifest and models with identical initialization and training configuration.Source: Table 5 · p. 9, §4.3 · p. 10

5 Conclusion

• The framework combines Daily-Physics 50K, OmniFysics-Agent, OmniFysics-Captioner, and the OPC benchmark for physics-aware omni-modal captioning.The Agent acquires temporally localized audiovisual and physical evidence; the Captioner performs tool-free inference.Source: §5 · p. 10
• Experiments report state-of-the-art open-source performance on detailed audiovisual captioning benchmarks.The conclusion also reports that active evidence acquisition improves information coverage and that PPM-enriched supervision transfers gains to the Captioner.Source: §5 · p. 10
• The authors plan to release Daily-Physics 50K, OmniFysics-Captioner, and OPC to support research on physical perception and omni-modal captioning.They describe scalable physics-rich supervision and evaluation as a foundation for embodied intelligence, robotic manipulation, and world models.Source: §5 · p. 10

A.1 Source Dataset Descriptions

• The nine-source video pool spans audiovisual correspondence, human–object interaction, physical phenomena, and multi-event processes across domains and temporal scales.Examples include egocentric kitchen manipulation, long untrimmed audiovisual recordings, physics-focused videos, and long-form annotated videos.Source: §A.1 · p. 15

A.2 Event Queries and Clip Construction

• CATA caches 23 event-query embeddings, reuses sampled-frame embeddings across queries, and forms clip boundaries by a single linear scan; in 500 audited clips, 98.2% were temporally complete and 96.9% category-consistent.The category-stratified random sample came from the final training manifest and covered all six top-level physical-event categories.Source: §A.2 · p. 16

B.1 Multimodal Evidence Acquisition

• The Agent’s active loop uses a Planner to identify information-dense intervals and route localized questions to modality-specific Observers, recording source-attributed outputs in Evidence Memory.Later rounds narrow the modality, time window, and question focus to investigate missing or uncertain details.Source: §B.1 · p. 17

B.2 Coverage Completion and Caption Synthesis

• After the active loop, Coverage adds under-covered baseline evidence without invoking the Planner, and the Finalizer turns accumulated evidence into a temporally coherent caption.The Finalizer preserves uncertainty when observations cannot be reconciled; these captions supervise the end-to-end Captioner.Source: §B.2 · p. 17

C.1 Physical Perception Model (PPM)

• PPM is a dedicated Observer fine-tuned on approximately 2M bilingual image-level question–answer examples to extract object-level physical evidence from representative frames.It uses approximate normalized bounding boxes to associate properties with objects and provides cues about contact, support, deformation, motion, and state changes.Source: §C.1 · p. 17

C.2 Training Configuration

• The PPM Observer is trained for one epoch with full-parameter supervised fine-tuning, while OmniFysics-Captioner is trained for two epochs on frozen Daily-Physics 50K video–caption pairs.The caption pairs are generated by the Agent pipeline.Source: §C.2 · p. 18
• Videos are sampled at 2 fps with audio enabled, retaining up to 256 frames; reported evaluations use the checkpoint from the end of epoch two.Source: §C.2 · p. 18

D.1 Independent Evaluation-Video Curation

• OPC clips come from an independent evaluation pool with no shared clips, source videos, or overlapping segments with Daily-Physics 50K, and exact and perceptual near-duplicates are removed.This separation establishes a held-out setting for comparing evaluated systems.Source: §D.1 · p. 18

D.2 Evidence-Grounded Construction

• OPC construction checks question support, distractor exclusivity, temporal scope, and wording against the complete audiovisual video, then human annotators cross-validate retained items.The final benchmark contains 8,000 questions over 1,000 videos, including probes for physical interactions, outcomes, audio, audiovisual alignment, speech/OCR, and temporal understanding.Source: §D.2 · p. 18
• OPC’s diagnostic decomposition can reveal physical-process coverage, cross-modal grounding, temporal understanding, and information omissions that a single aggregate caption score may obscure.Source: §D.2 · p. 19

D.3 Caption-to-QA Evaluation Protocol

• Each system supplies one caption per video to a fixed GPT-5.6 Judge, which sees only the caption, question, and five options; Coverage is macro-averaged by video.Correct and Not Mentioned are also aggregated over questions, and Category Coverage is computed over videos containing each question family.Source: §D.3 · p. 19

D.4 Human Validation of OPC

• 95.1% of audited questions were reliable, and raw GPT–human scoring agreement reached 97.4% across cases from the same 200-video subset.The question audit checked answerability, answer support, mutually exclusive options, and unambiguous wording; human Judge reviewers saw the caption, question, and options but not the audiovisual input or gold answer.Source: §D.4 · p. 20

D.5 Qualitative Case Studies

• The tire-crushing and long-jump cases pair full frozen captions with diagnostic questions, marking caption spans that support recovered answers and identifying missing evidence.The examples probe physical events and cross-modal cues, including deformation, rupture, impact, commentary, and on-screen information.Source: §D.5 · p. 20

VDC Detailed

• 57.9% accuracy and a VDCscore of 2.9 on VDC Detailed were the strongest results among evaluated models.VDC Detailed tests long, information-dense descriptions using structured reference descriptions averaging approximately 501 words.Source: §E.1 · p. 20

DREAM-1K

• 36.2 F1 on DREAM-1K ranked first among compared open-source models.AutoDQ uses event extraction and bidirectional entailment to assess event coverage and unsupported content.Source: §E.1 · p. 20

Omni-Cloze

• 49.2% overall accuracy and a score of 42.8 on Omni-Cloze led all compared open-source models across reported metrics; audio–visual accuracy was 53.4%.The benchmark includes visual-only, audio-only, and audiovisual cloze questions with a Not Given option.Source: §E.1 · p. 20

Daily-Omni

• 64.6% caption-to-QA accuracy on Daily-Omni was the highest result in the comparison and 0.5 percentage points above Gemini 3.1 Pro.Daily-Omni covers temporally aligned audiovisual reasoning across six task families.Source: §E.1 · p. 20

WorldSense

• 45.7% accuracy on WorldSense led compared open-source models and approached Gemini 3.1 Pro at 49.2%.WorldSense emphasizes coupled audiovisual evidence across speech, environmental sounds, and music.Source: §E.1 · p. 21

Video-MME

• 64.8% accuracy on Video-MME ranked first among evaluated open-source models.The benchmark spans short, medium, and long videos and includes audio and available subtitles.Source: §E.1 · p. 21

E.2 Standard Benchmark Evaluation

• Evaluation follows each benchmark’s released pipeline or uses a fixed GPT-5.6 caption-only Reader for Daily-Omni, WorldSense, and Video-MME.Generation or parsing failures count as incorrect, and shared-video comparisons use paired, video-level bootstrap estimates.Source: §E.2 · p. 21

E.3 Controlled Comparisons and Ablations

• Data-construction comparisons vary one stage at a time, matching clip properties or keeping videos and Captioner configuration fixed where applicable.The Agent cascade comparison uses the same frozen Reader for direct captions and OmniFysics-Agent captions to assess active evidence acquisition under their respective inference settings.Source: §E.3 · p. 21
• The paired PPM ablation changes only physical Observer availability while holding caption-generation and training settings fixed, with duplicate checks separating training and evaluation videos.Full-PPM and no-PPM captions use the same video manifest, Planner, Finalizer, ALM/VLM access, decoding configuration, and output constraint.Source: §E.3 · p. 21

F Limitations and Future Work

• Daily-Physics 50K and OPC focus on audiovisual-supported physical processes, without requiring force sensors, calibrated geometry, or simulator states.Their scope emphasizes observable interactions, material responses, state changes, and physical outcomes.Source: F Limitations and Future Work · p. 21
• Future work could link evidence-rich captions to object tracks, depth and 3D geometry, contact graphs, material estimates, and simulation or robot-interaction trajectories.The stated aim is more structured physical world representations.Source: F Limitations and Future Work · p. 21

G The Prompt Design of Agent

• The Agent collects localized audio, visual, textual, and temporal evidence through event-centered Observer calls, matching key audio and visual evidence across the timeline.The prompt directs attention to event order, text, object trajectories, interactions, outcomes, and audiovisual alignment.Source: G The Prompt Design of Agent · pp. 22–23
• Later evidence rounds request only missing, weakly localized, incomplete, or contradictory evidence and use targeted follow-ups to repair gaps.The prompt prioritizes timeline coverage, cross-modal checks, timestamps, sequence-sensitive relations, and resolving Observer disagreements.Source: G The Prompt Design of Agent · pp. 23–24
• Audio and visual Observers are instructed to stay within their respective modalities, report timing, and state uncertainty rather than make unsupported claims.The visual Observer also transcribes visible text and tracks object states; the audio Observer reports speech, sounds, and transitions.Source: G The Prompt Design of Agent · pp. 24–25
• The Finalizer turns Observer records into chronological, evidence-grounded captions, resolving conflicts conservatively and ending with the final visible and audio state.It is instructed not to access the media or call tools during caption writing.Source: G The Prompt Design of Agent · p. 25