Source-linked AI summary
Competitive Memory Readout for Robust Video Object Segmentation: 2nd Place Technical Report for the MOSEv2 Track of the 8th LSVOS Challenge
Mingqi Gao, Sijie Li, Jungong Han
TL;DR
MOSEv2 challenges video object segmentation when occlusion, disappearance, reappearance, appearance changes, and similar objects make target identity difficult to preserve. The paper augments SAM 3 with Competitive Memory Readout and adaptive restoration, achieving a 66.20 primary challenge score and second place in the track.
Problem
MOSEv2 tests whether a tracker can preserve the annotated target under severe temporal dynamics and visually similar distractors, where target-only memory retrieval can confuse instances.
Source: §1 · p. 2Method
The method compares SAM 3 target memory with same-class competitor evidence and applies lightweight adaptive restoration when competition suppresses valid weak or reappearing target evidence.
Source: §3.1 · p. 3, §3.3 · p. 4Results
66.20 primary challenge score and 2nd place were achieved in the MOSEv2 track.
Source: §5 · p. 6Takeaways & Limitations
The resulting system preserves the original SAM 3 tracking pipeline while improving robustness to MOSEv2’s long-term occlusion, disappearance, reappearance, and similar-object interference.
Source: §5 · p. 6
Abstract
from arXiv · showhide
We present our solution for the MOSEv2 track of the 8th Large-scale Video Object Segmentation (LSVOS) Challenge at ECCV 2026. The challenge evaluates robust video object segmentation under complex temporal dynamics, including long-term occlusion, disappearance and reappearance, large appearance changes, and strong interference from visually similar objects. Our method builds on SAM~3 and focuses on its memory readout. Standard target-only memory retrieval can confuse the annotated target with same-class non-target objects because such distractors are represented only implicitly as background. Our method introduces Competitive Memory Readout, which explicitly incorporates same-class competitor evidence when retrieving target information from memory. To prevent excessive suppression of weak or reappearing targets, we further apply a lightweight adaptive restoration rule after competition. The resulting system retains the original SAM~3 tracking pipeline while improving target identity preservation in challenging videos. Our submission achieves 66.20 on the primary challenge score and ranks 2nd in the MOSEv2 track.
1 Introduction
The MOSEv2 track targets semi-supervised video object segmentation under severe temporal and visual challenges, where preserving the annotated target’s identity is central. Competitive Memory Readout addresses target-centric retrieval by explicitly comparing target evidence with same-class competitors.
- MOSEv2 evaluates propagation of a first-frame target mask through videos with occlusion, disappearance, reappearance, appearance changes, and similar-object interference.Source: §1 · p. 2
- These conditions make identity preservation difficult because plausible masks can gradually drift from the annotated target to a same-class distractor.Source: §1 · p. 2
- Standard memory retrieval explicitly represents the target but treats same-class distractors as non-target context, measuring target-history similarity without comparing plausible alternatives.Source: §1 · p. 2
- Competitive Memory Readout compares target evidence against same-class competitor evidence during memory retrieval to address identity confusion after occlusion or reappearance.Source: §1 · p. 2
2 Related Work
Memory-based video object segmentation evolved from direct propagation toward storing and retrieving historical features and masks. The paper’s framing highlights a limitation of target-only retrieval: visually similar non-targets can induce identity drift.
- Semi-supervised video object segmentation propagates specified target objects from first-frame masks through the remaining video.Source: §2 · p. 2
- Memory-based methods became dominant by storing previous features and masks and retrieving them for current-frame prediction.Source: §2 · p. 2
- The standard target-only readout does not model visually similar non-target objects as competitors, allowing distractor-supported target responses and identity drift.Source: Figure 1 · p. 3
3 Method
The method retains SAM 3’s backbone, memory encoder, and mask decoder while replacing target-only retrieval with competitive target-versus-distractor evidence. Adaptive restoration then recovers useful target evidence when competition suppresses it too strongly.
3.1 Overview
- Competitive Memory Readout augments SAM 3 target-conditioned retrieval with same-class non-target competitors while leaving the backbone, memory encoder, and mask decoder unchanged.Source: §3.1 · p. 3
3.2 Competitive Memory Readout
- SAM 3 concept-aware detections provide same-class non-target hypotheses, which are filtered for strong overlap with the tracked target before competitor-token encoding.Source: §3.2 · pp. 3–4
- For each current-frame token, target and competitor attention logits are aggregated separately with log-sum-exp to form competing evidence.Source: §3.2 · p. 4
- A competition gate is applied to target foreground logits before original memory-attention normalisation, suppressing target evidence when competitors provide stronger support.Source: §3.2 · p. 4
- When no valid competitor exists, Competitive Memory Readout reduces to the original target-only readout.Source: §3.2 · p. 4
3.3 Adaptive Restoration
- Adaptive restoration measures suppression between pre- and post-competition target responses and restores part of the target evidence.Source: §3.3 · p. 4
- The restoration factor ranges from 1.0 to 1.5, increasing recovery as target responses are more heavily suppressed without requiring an additional learned policy.Source: §3.3 · p. 4
4 Experiments
Experiments evaluate the method on the official MOSEv2 challenge setting, using first-frame masks and difficult videos with temporal changes and similar objects. The submission ranks second with a primary score of 66.20, while qualitative results show more stable segmentation than SAM 3 on a challenging long sequence.
4.1 Challenge Setting
- MOSEv2 experiments follow the semi-supervised protocol, providing first-frame target masks and predicting the same targets in remaining frames.Source: §4.1 · p. 4
- The challenge videos contain frequent occlusion, disappearance and reappearance, large target variations, and multiple similar objects.Source: §4.1 · p. 4
- Across frames sampled from the beginning to the end of a challenging long sequence, the method maintains more stable target segmentation than the SAM 3 baseline under appearance variation and distractor interference.Source: Figure 2 · p. 5
4.3 Challenge Result and Qualitative Analysis
- 66.20 primary challenge score and 2nd place were achieved on the MOSEv2 leaderboard.Source: §4.3 · p. 5
- 81.57 was achieved on the disappearance metric, while reappearance remained substantially more challenging.Source: §4.3 · p. 5
5 Conclusion
The 2nd-place MOSEv2 solution augments SAM 3 with Competitive Memory Readout and adaptive restoration to preserve target identity under challenging video conditions. It achieves a primary challenge score of 66.20 while retaining the original SAM 3 tracking pipeline.
- Competitive Memory Readout compares target memory against same-class competitor evidence to reduce identity drift.It explicitly incorporates plausible non-target alternatives during memory retrieval.Source: §5 · p. 6
- Adaptive restoration recovers valid target evidence when competition suppresses weak or reappearing targets too strongly.Source: §5 · p. 6
- 66.20 primary challenge score and 2nd-place MOSEv2 ranking demonstrate the submission’s reported challenge outcome.The method preserves the original SAM 3 tracking pipeline while improving robustness to difficult temporal dynamics and similar-object interference.Source: §5 · p. 6