Source-linked AI summary
PhysMLLMs: Spatial Priors for Unified Referring Segmentation and Grounded Reasoning of Images and Videos
Siyao Yan, Bo Han, Jisheng Dang, Bimei Wang, Shude Wang, Hong Peng, Yulan Guo, Jianhuang Lai, Bin Hu, Tat-SengChua
TL;DR
Video MLLMs can produce temporally inconsistent masks, motivating methods that provide explicit spatial continuity guidance. PhysMLLMs injects such a prior by distilling global visual representations from a frozen DINOv2 teacher during training. The method improves difficult-video segmentation and consistency while preserving image-level grounding and representative general multimodal capability.
Problem
Video MLLM training provides limited direct guidance for maintaining consistent masks and stable object identity across frames, especially under occlusion and distractors.
Method
PhysMLLMs uses REPA-Global to align student global visual representations with frozen DINOv2 teacher embeddings through offline cached supervision and scheduled distillation during training.
Results
PhysMLLMs improves language-guided video segmentation and cross-frame consistency on challenging scenarios while preserving competitive image grounding and representative general VLM capability.
Takeaways & Limitations
Training-stage spatial prior injection improves temporal video stability without adding inference-time cost or clearly degrading broader grounding and multimodal performance.
Takeaways & Limitations
Dense similar distractors, persistent occlusion, and extremely small targets remain unresolved, partly because REPA-Global is global and does not explicitly model query-specific identity or physical dynamics.
Abstract
from arXiv · showhide
Video multimodal large language models support language guided video segmentation, but they often show spatio temporal inconsistencies, e.g., jitter, drift, and identity switches. These failures are more common when targets are partly hidden or when similar objects appear nearby.One likely reason is that current training lacks explicit spatial priors, which makes it difficult to maintain stable spatial identity and shape over time. We present PhysMLLMs, a training-stage prior injection architecture that injects physics-inspired spatial continuity priors into Video MLLMs. PhysMLLMs is designed to encourage more stable object-centered representations by aligning the student global visual representation with a frozen teacher model during training. Our core mechanism, Global Representation Prior Alignment (REPA-Global), distills global visual representations from a frozen DINOv2 teacher using an offline embedding cache and a scheduled distillation plan. This design keeps inference unchanged and does not add inference time cost. Across multiple video benchmarks, PhysMLLMs improves video segmentation mask quality and cross-frame consistency, with larger gains on challenging cases involving small targets, fast motion, occlusion, distractors, and reasoning queries. On single-frame referring image segmentation and representative general VLM benchmarks, PhysMLLMs maintains comparable performance, demonstrating that the injected spatial prior improves video consistency without compromising image-level grounding or general multimodal capability. These results suggest that physics-inspired spatial prior injection can improve temporal stability while preserving general capability. The code is available at https://github.com/tusu-code/20260121-icml2026-2.git.
1 Introduction
Video MLLMs can produce temporally unstable masks because training provides limited direct guidance for cross-frame consistency. PhysMLLMs injects a training-time spatial continuity prior through global representation distillation, improving difficult-video segmentation while preserving broader capability.
- Motivation: Video MLLMs often exhibit temporal jitter, identity or shape drift, and identity switches after occlusion.These failures are especially pronounced under occlusion, distractors, and appearance changes.
- Motivation: The proposed prior regularizes visual representations toward temporally coherent, object-centric features without explicit physical simulation or hand-crafted equations.It reflects object permanence, spatial continuity, and temporal coherence while allowing genuine physical changes.
- Method: REPA-Global aligns student global visual representations with cached embeddings from a frozen DINOv2 teacher during training.An offline cache and scheduled distillation make the representation-level regularizer scalable and keep inference unchanged.
- Contribution: PhysMLLMs is designed to improve temporal stability and video reasoning while preserving the underlying model’s general grounding ability.The contribution is framed as training-stage prior injection rather than an inference-time modification.
- Results: Across ReVOS, MeVIS, and Ref-DAVIS17, PhysMLLMs improves language-guided video segmentation, especially for small targets, fast motion, occlusion, distractors, and reasoning queries.It also preserves competitive single-frame referring segmentation and shows no clear degradation on representative general VLM benchmarks.
2 Related Work
Prior work supplies temporal, semantic, and geometric supervision through memory, self-supervised learning, and feature-level distillation. PhysMLLMs builds on these directions with global representation alignment, scheduled weighting, and parameter-efficient fine-tuning for stable video grounding.
- Memory and video segmentation: Memory-based systems reuse past cues to reduce short-term flicker but can struggle when representations drift or distractors resemble the query target.This motivates regularizing upstream representations rather than relying only on temporal association.
- Self-supervised visual priors: Self-supervised image and video pretraining provides transferable object-centered, invariant, and motion-related cues for temporally coherent features.DINO-style self-distillation and video objectives offer complementary priors beyond frame-by-frame segmentation supervision.
- Representation distillation: Feature-level and geometry-preserving distillation transfers structural guidance to student representations when stable spatial structure matters.For dense prediction, structured or global alignment can reduce sensitivity to local noise and support persistent identity.
- Consistency-oriented alignment: PhysMLLMs uses REPA-Global to align global visual embeddings with a frozen DINOv2 teacher during training, while excluding text embeddings from the pre-alignment global pooling branch.The teacher embedding is retrieved from an offline cache and the teacher branch is removed at inference.
- Parameter-efficient adaptation: Parameter-efficient fine-tuning limits representational drift and helps retain multimodal abilities during specialization.Vision LoRA and MaskDecoder-only calibration act as practical guardrails, although insufficient capacity may limit prior absorption.
3 Method
PhysMLLMs injects a physics-inspired spatial consistency prior during training by distilling cached global DINOv2 representations into a Video MLLM. Scheduled regularization and calibrated parameter-efficient updates preserve the original inference graph while targeting temporal instability.
- 3.1 Overview: PhysMLLMs adds representation-level distillation to encourage spatial persistence and geometric continuity without relying on explicit physical simulation.The prior is derived from object permanence, spatial continuity, and temporal coherence, rather than a physical simulator.
- 3.4 Overall Training Workflow for Cache, Scheduling, and Deployment: The training objective combines segmentation loss with a time-dependent distillation term, whose strength increases through a warmup-ramp-hold schedule.Warmup uses only the segmentation objective; the scheduled stage introduces Ldistill and updates calibrated PEFT parameters.
- 3.2 REPA-Global Distillation: REPA-Global aligns pooled student visual embeddings with frozen DINOv2 global embeddings before text-conditioned mask prediction.A lightweight projector maps the student embedding to the teacher dimension, and normalized embeddings are aligned with cosine distance.
- 3.4 Overall Training Workflow for Cache, Scheduling, and Deployment: The method keeps inference unchanged because DINOv2, the teacher cache, and prior-injection machinery are used only during training.The paper emphasizes that the novelty lies in combining cache-enabled distillation, scheduled injection, and calibrated PEFT while preserving the original student inference graph.
- 3.3 Calibrated PEFT for Prior Injection: Calibrated PEFT freezes the language model and most pretrained modules while updating Vision LoRA factors and the segmentation mask decoder.The constrained update is intended to preserve semantic grounding while allowing the injected prior to shape temporally relevant visual features.
- 3.4 Overall Training Workflow for Cache, Scheduling, and Deployment: Teacher embeddings are precomputed and cached offline, allowing deterministic distillation without repeated teacher forward passes.At inference, the teacher and cache are discarded and only the student model runs.
4 Experiments
Experiments evaluate PhysMLLMs for video segmentation, capability preservation, and the role of its teacher-prior design. Controlled analyses show gains on difficult video cases, global alignment outperforming token-level variants, and no clear collapse in image grounding or general VLM capability, while severe occlusion and tiny targets remain challenging.
- Evaluation setup: Experiments use a common Sa2VA-InternVL3-2B + SAM2 backbone for controlled comparisons, with J&F as the main video-segmentation metric and RefCOCO accuracy as an image-grounding check.The broader benchmark table also includes models with different capacities and is used for positioning rather than as the sole capacity-matched comparison.
- Main comparisons: PhysMLLMs improves consistently across MeVIS U, ReVOS, and Ref-DAVIS17 relative to representative recent Video-MLLM systems.The comparison spans different model sizes and training strategies, so it is not itself a capacity-matched evaluation.
- Prior-injection ablations: Teacher-prior injection improves ReVOS segmentation and transfers to MeVIS U when distillation is enabled under an otherwise unchanged training recipe.The MeVIS U split differs from the training distribution, providing evidence of transfer beyond in-domain performance.
- Complexity-stratified results: +2.26 J&F on small targets, +1.92 on occlusion or disappearance cases, +2.02 on distractor-heavy cases, +1.37 on fast-motion cases, +1.76 on reasoning queries, and +1.59 on the hard union subset over the official Sa2VA-InternVL3-2B model.These strata stress temporal identity preservation, spatial continuity, and robustness to distractors.
- Prior-injection ablations: 57.41 J&F for global-only REPA-Global exceeds token-only alignment at 56.26 J&F and global+token alignment at 56.07 J&F on ReVOS.The results support adopting a compact global constraint because dense token-level constraints may interfere with language-conditioned target selection.
- Capability preservation: The default configuration keeps RefCOCO and MMBench close to baseline while improving video consistency, and representative qualitative video QA shows no noticeable reasoning degradation.The unified evaluation covers image grounding, video segmentation, and MMBench under the InternVL3-2B + SAM2 backbone.
- Limitations: Dense similar distractors, persistent occlusion, and extremely small targets can still cause incorrect localization or identity ambiguity, so these scenarios are not fully solved.The limitation is linked to REPA-Global’s global regularization, which does not explicitly model physical forces, 3D dynamics, object interactions, or query-specific target identity.
5 Conclusion
PhysMLLMs injects a physics-inspired spatial continuity prior into Video MLLMs through REPA-Global, targeting temporal failures while preserving general grounding ability. Evidence supports improved difficult-video behavior, but direct feature-trajectory analysis and several challenging cases remain unresolved.
- PhysMLLMs targets jitter, drift, and identity switches by injecting a physics-inspired spatial continuity prior into Video MLLMs.
- REPA-Global aligns student global visual representations with a frozen DINOv2 teacher using offline cached supervision, scheduled weighting, and parameter-efficient fine-tuning.
- The training-time prior improves difficult video scenarios while preserving image-level grounding and representative multimodal capability.
- Current evidence relies mainly on controlled ablations, challenging-scenario analysis, and temporal stability proxies rather than direct feature-trajectory analysis.
Appendix A. Additional Experimental Results
The appendix adds quantitative evidence for general VLM capability, referring image segmentation, and distillation-weight sensitivity, while keeping the main experimental section concise.
- The appendix supplements the main experiments with general VLM checks, RefCOCO-series comparisons, and distillation-weight sensitivity analysis.
A.1 General VLM Capability Check
Additional checks use a common Sa2VA-InternVL3-2B backbone to assess whether video-oriented calibration and REPA-Global preserve general VLM capability.
- 2152.880 MME total for ReVOS calib it500 + KD dw012 is comparable to 2151.195 for the official Sa2VA-InternVL3-2B checkpoint.
- Nearby ReVOS-calibrated checkpoints remain stable on POPE and TextVQA, with no obvious collapse in representative general VLM capability.
A.2 RefCOCO-Series Referring Image Segmentation
PhysMLLMs remains competitive on single-frame RefCOCO-series referring image segmentation, which serves as a non-degradation check for image-level grounding.
- PhysMLLMs remains competitive on RefCOCO, RefCOCO+, and RefCOCOg against representative fine-tuned models.
- Table 7 reports validation accuracy for RefCOCO, RefCOCO+, and RefCOCOg, where higher values are better.
A.3 Distillation-Weight Sensitivity
Moderate distillation provides the best ReVOS J&F under the fixed warmup–ramp–hold schedule, while stronger distillation favors temporal stability at a small mask-quality cost.
- Moderate distillation achieves the best ReVOS J&F under the same warmup–ramp–hold schedule.
- Stronger distillation tends to improve the temporal stability proxy while slightly reducing mask quality.
- The sensitivity results support dw012 as the default setting for the main experiments.
A.4 Qualitative Success Cases
Qualitative LV-VIS examples show PhysMLLMs producing more coherent masks and more accurate target localization than the baseline across diverse challenging queries and scenes.
- PhysMLLMs produces more coherent masks across frames than the baseline in qualitative LV-VIS examples.The examples include distractors, partial occlusion, small objects, and reasoning-style queries.
- PhysMLLMs improves target localization for human-object interaction, similar instances, small or thin objects, and reasoning queries.
- Baseline failures include over-segmentation under distractors, missing small objects, and identity drift after partial occlusion.
A.5 Failure Cases
PhysMLLMs improves temporal consistency in many challenging scenarios but remains vulnerable to ambiguity from similar distractors, persistent occlusion, and extremely small targets.
- PhysMLLMs can still fail under dense similar distractors, persistent occlusion, and extremely small targets.
- The qualitative comparisons in Figure 4 cover human-object interaction, multiple similar instances, small or thin objects, and reasoning-style queries.
- Figure 5 compares input images, ground truth, baseline predictions, and PhysMLLMs predictions for these three failure settings.