Source-linked AI summary
SAM3Dual: A 3rd Place Solution to the MOSEv2 Track, 8th LSVOS Challenge
JeongRae Kim, Chaehyun Kim, Changwon Lim
TL;DR
Long-term VOS must preserve target identity while adapting to changing appearances and handling occlusions, disappearance, and reappearance. SAM3Dual reorganizes pretrained SAM 3’s temporal memory into short- and long-term branches at inference time, combining them through deterministic fusion and confidence modulation. The complete system achieved 64.37 J&F and ranked third in the MOSEv2 track.
Problem
Long-term VOS requires balancing recent, error-prone observations with stable but potentially outdated historical information.
Method
SAM3Dual separates recent and interval-sampled historical representations into two memory branches, combines their responses deterministically, and keeps all pretrained SAM 3 parameters frozen.
Results
64.37 J&F and third place were achieved by the complete SAM3Dual system in the MOSEv2 track.
Takeaways & Limitations
The result supports the feasibility of a competitive training-free long-term VOS system that reorganizes temporal memory at inference time while preserving pretrained parameters.
Takeaways & Limitations
Confidence modulation scales both memory branches equally, while propagated predictions can introduce unreliable representations into subsequent memory.
Abstract
from arXiv · showhide
We present SAM3Dual, our third-place solution to the MOSEv2 track of the 8th Large-scale Video Object Segmentation (LSVOS) Challenge at ECCV 2026. SAM3Dual is a training-free inference extension of pretrained SAM 3 that explicitly separates temporal memory into a short-term branch for recent observations and a long-term branch for interval-sampled historical representations. The two memory responses are combined using a deterministic sequence-relative fusion schedule and conservatively modulated by the previous-frame object confidence. All pretrained SAM 3 parameters remain frozen, requiring no task-specific training, fine-tuning, test-time training, or online parameter optimization. The complete system achieved an official J&F score of 64.37 and ranked third in the MOSEv2 track. This result highlights the potential of reorganizing temporal memory entirely at inference time to obtain competitive long-term VOS performance while preserving the pretrained model.
1 Introduction
Long-term VOS must balance recent adaptation against stable historical information under motion, occlusion, disappearance, reappearance, and appearance variation. SAM3Dual addresses this tension by separating temporal memory within frozen SAM 3 and achieves third place in MOSEv2.
- Long-term VOS remains difficult because motion, deformation, occlusion, disappearance, reappearance, and clutter can cause accumulated errors, drift, and identity switches.
- A single streaming memory must balance recent observations, which support local adaptation but may contain propagated errors, against distant observations, which preserve history but may be less current.
- SAM3Dual separates recent observations and interval-sampled historical representations into short-term and long-term branches within a frozen SAM 3 pathway.
- The two memory responses are combined with a deterministic sequence-relative fusion schedule and bounded confidence-guided modulation without additional learnable parameters.
- 64.37 J&F and third place were achieved by the complete SAM3Dual system in the MOSEv2 track.The result is reported for the complete challenge submission rather than as a component-wise performance claim.
2 Related Work
Memory-based VOS relies on organizing, selecting, and retrieving temporal information, especially for long-term propagation. SAM3Dual retains SAM 3’s frozen memory-attention mechanism while reorganizing memory into temporal roles within a single inference pathway.
- 2.1 Memory-Based Video Object Segmentation: Memory-based propagation retrieves information from previously segmented frames and remains a dominant paradigm in semi-supervised VOS.
- 2.1 Memory-Based Video Object Segmentation: Long-term VOS methods assign different roles to historical representations, including sensory, working, and long-term memory stores.
- 2.1 Memory-Based Video Object Segmentation: Long-term VOS depends on how historical information is organized, selected, and retrieved, not only on whether it is retained.
- 2.2 Foundation Models and Training-Free Long-Term VOS: SAM 2 and SAM 3 extend promptable segmentation to video through streaming or memory-based temporal propagation using pretrained representations.
- 2.2 Foundation Models and Training-Free Long-Term VOS: SAM2Long modifies memory usage at inference time, whereas SAM3Dual uses one frozen SAM 3 pathway with explicitly separated short-term and long-term branches.
3 Method
SAM3Dual extends frozen SAM 3 with separate short-term and long-term temporal memory branches. It modulates their responses by previous-frame confidence and combines them through a deterministic sequence-relative schedule.
- Dual-Memory Architecture: SAM3Dual propagates video masks with all pretrained SAM 3 parameters frozen, requiring no task-specific or online parameter updates.The image encoder, memory encoder, memory-attention modules, and mask decoder remain frozen.
- Dual-Memory Architecture: Short-term memory retains recent observations, while long-term memory stores interval-sampled historical representations alongside a persistent first-frame reference.Each branch maintains a bounded 1 + 6 memory structure, with up to six temporal entries.
- Dual-Memory Architecture: Both branches independently use the same frozen SAM 3 memory-attention operation, so their distinction comes from temporal memory composition rather than learned modules.The current-frame representation serves as the query for both memory banks.
- Confidence-Guided Memory Modulation: Previous-frame object confidence scales both memory responses between 0.9 and 1.1, attenuating low-confidence responses and slightly amplifying high-confidence responses.The shared scale changes overall memory magnitude without changing the branches’ relative weighting.
- Sequence-Relative Temporal Fusion: The short-term fusion weight decreases linearly from 1.0 to 0.5, while the long-term weight increases from 0 to 0.5 over the sequence.The schedule uses α = 0.5 and normalizes time by N −1, requiring the total sequence length N to be known beforehand.
4 Experiments
SAM3Dual was evaluated on the MOSEv2 track of the 8th LSVOS Challenge under the official semi-supervised long-term VOS protocol. The submission used frozen pretrained SAM 3 with fixed inference settings.
- Evaluation Setting: MOSEv2 evaluates semi-supervised VOS under long-term conditions including occlusion, disappearance, reappearance, object interactions, and substantial appearance variation.Targets are initialized from the first-frame ground-truth annotation and propagated without additional ground-truth supervision.
- Evaluation Setting: The official evaluation uses the J &F metric, averaging region similarity J and contour accuracy F.All leaderboard scores reported are official challenge results.
- Implementation Details: The official submission used the publicly released pretrained SAM 3 checkpoint with BF16 mixed precision on a single NVIDIA GeForce RTX 4090 GPU.All pretrained SAM 3 parameters remained frozen during inference.
- Implementation Details: The submitted inference settings were fixed and are not claimed to be individually optimal.The configuration is summarized in Table 1.
5 Results and Analysis
SAM3Dual ranked third on the official MOSEv2 leaderboard, while qualitative examples show stable propagation in challenging cases and failures involving ambiguity, displacement, and reappearance.
- Official Challenge Results: 64.37 J&F ranked SAM3Dual third on the official MOSEv2 leaderboard.The first- and second-ranked submissions scored 69.82 and 66.20, respectively.
- Qualitative Analysis: Successful examples show stable target propagation through motion, partial occlusion, disappearance, and appearance variation.Green, red, and blue denote true-positive, false-positive, and false-negative regions.
- Qualitative Analysis: The two memory branches provide complementary temporal information, retaining recent observations and dispersed historical memories when recent evidence becomes less informative.The complete system can maintain masks across temporally separated target appearances.
- Failure Case Analysis: Failure cases include visually similar distractors, large target displacement, and target reappearance.These cases are illustrated as representative failures on the MOSEv2 training set.
- Failure Case Analysis: Appearance ambiguity can cause memory retrieval to select a plausible distractor, producing false positives on the distractor and false negatives on the target.Preserving historical appearance information does not guarantee object identity preservation.
- Failure Case Analysis: Large displacement after disappearance or occlusion can associate stored target representations with the wrong object, especially alongside similar distractors.SAM3Dual does not add explicit long-range spatial correspondence or target relocation.
6 Discussion and Limitations
The challenge result supports competitiveness at the complete-system level, but the design has unmeasured efficiency costs, fixed temporal choices, propagated-memory errors, and identity and online-use limitations.
- Interpretation of the Challenge Result: 64.37 J&F demonstrates competitiveness of the complete training-free SAM3Dual configuration, not of any individual component.The original SAM 3 configuration and SAM3Dual components were not evaluated under an identical official setting, and no controlled ablation was conducted.
- Computational and Practical Considerations: The dual-memory design adds a long-term bank and second memory-attention operation relative to single-branch SAM 3.Confidence modulation and temporal fusion use scalar rescaling and weighted combination, but the additional retrieval increases inference-time computation and memory usage.
- Computational and Practical Considerations: No controlled runtime, throughput, or peak GPU-memory measurements were performed against the original SAM 3 configuration.Consequently, the paper makes no quantitative efficiency claim.
- Limitations and Future Directions: The submitted memory configuration fixes capacity at 1 + 6, long-term sampling at ∆= 10, and fusion at α = 0.5.These choices are bounded and predictable but cannot adapt to sequence-specific motion, visibility, appearance, or scene complexity.
- Limitations and Future Directions: Confidence modulation changes overall memory-response magnitude rather than the relative reliability of the two branches, while propagated predictions can enter temporal memory.The first-frame ground-truth representation is retained, but later memories derive from propagated predictions.
- Limitations and Future Directions: Appearance-based memory does not explicitly model long-range spatial correspondence, target relocation, or re-identification.Identity ambiguity remains possible when targets reappear far away near visually similar objects.
- Limitations and Future Directions: The fusion schedule assumes that total sequence length N is known before propagation, limiting direct use in strictly online applications.A causal or event-driven rule independent of final sequence length would be needed for online use.
- Limitations and Future Directions: SAM3Dual is a challenge solution rather than a comprehensive study of long-term memory design.Controlled comparisons of memory capacities, sampling strategies, fusion mechanisms, and confidence usage remain future evaluation directions.
7 Conclusion
SAM3Dual reorganizes frozen SAM 3 temporal memory into short-term and long-term branches for training-free inference and achieved a third-place MOSEv2 result. The conclusion also notes that appearance history alone is insufficient for reliable identity preservation.
- Conclusion: SAM3Dual separates pretrained SAM 3 temporal memory into short-term and long-term branches with deterministic fusion and confidence-guided modulation.The framework operates entirely at inference time with all pretrained parameters frozen and no task-specific or test-time training.
- Conclusion: 64.37 J&F ranked the complete SAM3Dual system third in the MOSEv2 track.The result is reported as the official challenge outcome.
- Conclusion: Qualitative failures under appearance ambiguity, large displacement, and reappearance show that historical appearance alone is insufficient for reliable target identity preservation.Future directions include adaptive memory selection and fusion, uncertainty-aware updates, and stronger identity-preserving mechanisms.