Source-linked AI summary
Competitive Memory Readout for Robust Video Object Segmentation: 2nd Place Technical Report for the MOSEv2 Track of the 8th LSVOS Challenge
Mingqi Gao, Sijie Li, Jungong Han
TL;DR
MOSEv2 challenges video object segmentation when occlusion, disappearance, reappearance, appearance changes, and similar objects make target identity difficult to preserve. The paper augments SAM 3 with Competitive Memory Readout and adaptive restoration, achieving a 66.20 primary challenge score and second place in the track.
Problem
MOSEv2 tests whether a tracker can preserve the annotated target under severe temporal dynamics and visually similar distractors, where target-only memory retrieval can confuse instances.
Method
The method compares SAM 3 target memory with same-class competitor evidence and applies lightweight adaptive restoration when competition suppresses valid weak or reappearing target evidence.
Results
66.20 primary challenge score and 2nd place were achieved in the MOSEv2 track.
Takeaways & Limitations
The resulting system preserves the original SAM 3 tracking pipeline while improving robustness to MOSEv2’s long-term occlusion, disappearance, reappearance, and similar-object interference.
Abstract
from arXiv · showhide
We present our solution for the MOSEv2 track of the 8th Large-scale Video Object Segmentation (LSVOS) Challenge at ECCV 2026. The challenge evaluates robust video object segmentation under complex temporal dynamics, including long-term occlusion, disappearance and reappearance, large appearance changes, and strong interference from visually similar objects. Our method builds on SAM~3 and focuses on its memory readout. Standard target-only memory retrieval can confuse the annotated target with same-class non-target objects because such distractors are represented only implicitly as background. Our method introduces Competitive Memory Readout, which explicitly incorporates same-class competitor evidence when retrieving target information from memory. To prevent excessive suppression of weak or reappearing targets, we further apply a lightweight adaptive restoration rule after competition. The resulting system retains the original SAM~3 tracking pipeline while improving target identity preservation in challenging videos. Our submission achieves 66.20 on the primary challenge score and ranks 2nd in the MOSEv2 track.
1 Introduction
The MOSEv2 track targets semi-supervised video object segmentation under severe temporal and visual challenges, where preserving the annotated target’s identity is central. Competitive Memory Readout addresses target-centric retrieval by explicitly comparing target evidence with same-class competitors.
- MOSEv2 evaluates propagation of a first-frame target mask through videos with occlusion, disappearance, reappearance, appearance changes, and similar-object interference.
- These conditions make identity preservation difficult because plausible masks can gradually drift from the annotated target to a same-class distractor.
- Standard memory retrieval explicitly represents the target but treats same-class distractors as non-target context, measuring target-history similarity without comparing plausible alternatives.
- Competitive Memory Readout compares target evidence against same-class competitor evidence during memory retrieval to address identity confusion after occlusion or reappearance.
2 Related Work
Memory-based video object segmentation evolved from direct propagation toward storing and retrieving historical features and masks. The paper’s framing highlights a limitation of target-only retrieval: visually similar non-targets can induce identity drift.
- Semi-supervised video object segmentation propagates specified target objects from first-frame masks through the remaining video.
- Memory-based methods became dominant by storing previous features and masks and retrieving them for current-frame prediction.
- The standard target-only readout does not model visually similar non-target objects as competitors, allowing distractor-supported target responses and identity drift.
3 Method
The method retains SAM 3’s backbone, memory encoder, and mask decoder while replacing target-only retrieval with competitive target-versus-distractor evidence. Adaptive restoration then recovers useful target evidence when competition suppresses it too strongly.
- 3.1 Overview: Competitive Memory Readout augments SAM 3 target-conditioned retrieval with same-class non-target competitors while leaving the backbone, memory encoder, and mask decoder unchanged.
- 3.2 Competitive Memory Readout: SAM 3 concept-aware detections provide same-class non-target hypotheses, which are filtered for strong overlap with the tracked target before competitor-token encoding.
- 3.2 Competitive Memory Readout: For each current-frame token, target and competitor attention logits are aggregated separately with log-sum-exp to form competing evidence.
- 3.2 Competitive Memory Readout: A competition gate is applied to target foreground logits before original memory-attention normalisation, suppressing target evidence when competitors provide stronger support.
- 3.2 Competitive Memory Readout: When no valid competitor exists, Competitive Memory Readout reduces to the original target-only readout.
- 3.3 Adaptive Restoration: Adaptive restoration measures suppression between pre- and post-competition target responses and restores part of the target evidence.
- 3.3 Adaptive Restoration: The restoration factor ranges from 1.0 to 1.5, increasing recovery as target responses are more heavily suppressed without requiring an additional learned policy.
4 Experiments
Experiments evaluate the method on the official MOSEv2 challenge setting, using first-frame masks and difficult videos with temporal changes and similar objects. The submission ranks second with a primary score of 66.20, while qualitative results show more stable segmentation than SAM 3 on a challenging long sequence.
- 4.1 Challenge Setting: MOSEv2 experiments follow the semi-supervised protocol, providing first-frame target masks and predicting the same targets in remaining frames.
- 4.1 Challenge Setting: The challenge videos contain frequent occlusion, disappearance and reappearance, large target variations, and multiple similar objects.
- 4.1 Challenge Setting: Across frames sampled from the beginning to the end of a challenging long sequence, the method maintains more stable target segmentation than the SAM 3 baseline under appearance variation and distractor interference.
- 4.3 Challenge Result and Qualitative Analysis: 66.20 primary challenge score and 2nd place were achieved on the MOSEv2 leaderboard.
- 4.3 Challenge Result and Qualitative Analysis: 81.57 was achieved on the disappearance metric, while reappearance remained substantially more challenging.
5 Conclusion
The 2nd-place MOSEv2 solution augments SAM 3 with Competitive Memory Readout and adaptive restoration to preserve target identity under challenging video conditions. It achieves a primary challenge score of 66.20 while retaining the original SAM 3 tracking pipeline.
- Competitive Memory Readout compares target memory against same-class competitor evidence to reduce identity drift.It explicitly incorporates plausible non-target alternatives during memory retrieval.
- Adaptive restoration recovers valid target evidence when competition suppresses weak or reappearing targets too strongly.
- 66.20 primary challenge score and 2nd-place MOSEv2 ranking demonstrate the submission’s reported challenge outcome.The method preserves the original SAM 3 tracking pipeline while improving robustness to difficult temporal dynamics and similar-object interference.