Source-linked AI summary
Learning to Reason with Persistent Object States for Video Instance Segmentation
Yongxue Xu, Boxue Yang, Ziqian Liu, Shaoqiu Zhang, Rui Qian, Haopeng Chen
TL;DR
Video segmentation identity can drift when occlusion, reappearance, or similar-instance interactions make observations unreliable. POSReasoner verifies state-transition hypotheses before writing evidence to persistent identity states, and experiments report consistent improvements across evaluated VOS/VIS hosts, especially under occlusion.
Problem
Identity continuity remains brittle under prolonged occlusion, reappearance, and similar-instance interactions, where mistaken associations can affect later predictions.
Source: §1 · pp. 1–2Method
POSReasoner uses persistent identity states and Propose–Verify reasoning to refine associations and transitions, admitting evidence to memory only through verified transitions.
Source: §1 · p. 2, §3.2 · p. 4Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Video segmentation models maintain object identities by carrying instance information across frames. Under prolonged occlusion, reappearance, or interactions between similar instances, however, an unreliable update can overwrite a valid history and cause persistent identity drift. We introduce POSReasoner, a trainable, plug-and-play framework that explicitly decides when an observation should change an object's state. Each persistent state records identity, confidence, and absence history. A sparse state-observation graph supports Propose-Verify reasoning: provisional associations are revisited using object history, predicted presence, and competition among identities. The verified decisions determine whether to retain, update, reactivate, or suppress each state, while a learned gate controls the evidence written back to memory. Only verified transitions update the persistent state used in subsequent frames. POSReasoner uses standard video annotations and keeps the base model frozen, enabling integration with diverse VOS and VIS architectures. Experiments across long-term VOS and VIS benchmarks show consistent improvements over strong baselines, with the largest gains under occlusion and object reappearance.
1 INTRODUCTION
Video segmentation must maintain object identity across frames, but continuity remains brittle when occlusion, reappearance, or similar-instance interactions make observations unreliable. POSReasoner uses persistent identity states and verified transitions to improve identity maintenance across VOS and VIS benchmarks, especially under occlusion and reappearance.
Motivation
- Identity continuity remains brittle under prolonged occlusion, reappearance, and interactions between similar instances, when mistaken associations can affect the rest of a video.VOS specifies target objects, while VIS must also discover and classify them.Source: §1 · p. 1
Related approaches
- Existing approaches carry features or queries, retrieve prior-frame features, or propagate and refine tracks, while recent systems also manage memory and filter distractors.These approaches span memory-based VOS and query-based or decoupled VIS designs.Source: §1 · p. 1
Research gap
- Three coupled gaps remain: admitted errors can alter future identity references, absence signals are not jointly reasoned with association and state admission, and provisional matches are not revisited using lifecycle state.The shared issue is treating the current observation as the next state rather than evidence for a state transition.Source: §1 · p. 2
Approach
- POSReasoner maintains identity-indexed states and uses a sparse state–observation graph with Propose–Verify reasoning to refine associations and predicted transitions.It records identity evidence, geometry, visibility history, and confidence; only a final verified transition writes candidate evidence into persistent state.Source: §1 · p. 2
Contributions and results
- Across long-term VOS/VIS benchmarks, POSReasoner consistently improves strong hosts, with the clearest gains under occlusion and reappearance.The framework keeps the host frozen and uses standard video annotations for reasoning targets.Source: §1 · p. 2
2 RELATED WORK
Prior VIS, memory-based VOS, and object-centric approaches improve association, memory use, or persistent representations, but the paper distinguishes its focus as revising state claims from candidates supplied by a frozen host.
2.1 VIDEO INSTANCE SEGMENTATION
- VIS approaches aggregate frame or clip queries, propagate queries or embeddings online, and strengthen temporal association with prototypes, memory banks, decoupled refinement, dynamic anchors, synchronized queries, or contextual matching.The cited examples include GenVIS, CTVIS, DVIS, and DVIS++.Source: §2.1 · p. 2
- VIS association and state update are often coupled, so a matched observation becomes part of the future reference and a local error can alter subsequent identity evidence.LOMM separates existing-object association from new-object allocation, yet instability remains in long, crowded, and heavily occluded videos.Source: §2.1 · p. 3
2.2 MEMORY-BASED VIDEO OBJECT SEGMENTATION
- Memory-based VOS methods improve which observations are stored or retrieved within particular memory designs, whereas POSReasoner reasons over competing claims on persistent identity states from an existing segmenter's candidates.The surveyed systems include dense correspondence, multiple-object identification embeddings, multi-stage memory, object queries, streaming memory, and memory-selection extensions.Source: §2.2 · p. 3
2.3 OBJECT-CENTRIC VIDEO LEARNING
- Object-centric video models propagate persistent slots, filter identity updates, gate updates during invisibility, or defer object admission and consolidate tracks.Their objectives concern scene decomposition or open-world discovery; POSReasoner instead starts from candidates of a frozen host and carries verified state transitions forward.Source: §2.3 · p. 3
3 METHOD
POSReasoner treats each host observation as a hypothesis about a persistent identity state, not as an automatic memory update. Propose–Verify reasoning uses identity history and competition to refine associations and actions, and only the final verified transition advances the state.
3 METHOD
- POSReasoner separates persistent identity states from frame-local observations and verifies competing hypotheses before admitting evidence to memory.The method reframes association as deciding whether and how a match should alter a persistent state.Source: §3 · p. 3, Figure 2 · p. 4
3.1 PERSISTENT OBJECT STATES
- Each state stores a latent identity feature, confidence, and time since its last reliable observation, retaining identity features through absence.VOS states begin at annotated first appearance; VIS can open a free slot for an unassigned foreground observation.Source: §3.1 · p. 4
- A sparse bipartite graph connects states to globally confident and top-K identity-compatible observations, plus a learned null observation for unmatched states.The null edge allows occlusion without forcing a match.Source: §3.1 · p. 4
3.2 PROPOSE–VERIFY STATE REASONING
- Two shared-parameter reasoning steps refine associations: Propose forms hypotheses, while Verify uses contextualized states and observations to address competition among identities.The graph masks invalid edges throughout cross-attention and association.Source: §3.2 · p. 4, Figure 3 · p. 5
- The column probability mass above one acts as a soft duplicate-demand penalty, discouraging multiple states from claiming the same observation without enforcing hard one-to-one assignment.The learned null observation has a zero identity-prior term.Source: §3.2 · p. 5
- Verify can change either observation ownership or the action, since the same match may imply UPDATE, REVIVE, or KEEP depending on state history.The transition predictions include visibility and existence, a transition distribution, and transition advantage.Source: §3.2 · pp. 5–6
3.3 VERIFIED STATE TRANSITION AND LEARNING
- Verified KEEP, UPDATE, REVIVE, or SUPPRESS decisions govern a separate write gate, and only the final transition advances persistent memory.Intermediate reasoning steps use a temporary workspace anchored to the previous state, preventing multiple commits from the same frame.Source: §3.3 · p. 6
- Training targets use frozen-host predictions and annotations to label associations and transitions, while the objective combines association, transition, belief, state, advantage, and refinement losses.At validation and test time, the reasoner uses host outputs, persistent state, and task-provided prompts.Source: §3.3 · p. 6
4 EXPERIMENTS
Experiments evaluate POSReasoner with frozen hosts across video-instance and long-term video-object segmentation benchmarks. Results show improvements across hosts, while component analyses highlight the roles of persistent identity history, reactivation, and cross-trajectory reasoning.
4.1 EXPERIMENTAL SETTINGS
- The evaluation covers YouTube-VIS, OVIS, LVOS v1, and LVOS v2, pairing POSReasoner with diverse VIS and VOS hosts while keeping host weights frozen.The VOS readout uses disjoint fitting, calibration, and holdout videos, and architecture and candidate-budget choices are fixed before evaluation.Source: §4.1 · p. 7
4.2 COMPARISON WITH STATE-OF-THE-ART METHODS
- With larger backbones on OVIS, DAQ improves from 49.6 to 50.6 AP with Swin-L and from 53.9 to 55.2 AP with ViT-L.Both AP50 and AP75 also improve.Source: §4.2 · p. 7
- POSReasoner improves AP across all three YouTube-VIS benchmarks and yields 2.7, 2.1, and 1.6 AP gains for CTVIS, DVIS++, and DAQ on OVIS.The OVIS comparison is against matched hosts; YouTube-VIS gains include 1.0, 1.1, and 2.1 AP for CTVIS on 2019, 2021, and 2022.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 8
4.3 LONG-HORIZON STATE ANALYSIS
- On LVOS v1, state reasoning improves J &F by 2.1 points, reactivation adds 1.2 points, and cross-trajectory reasoning brings the full model 3.5 points above SAM3.Cross-trajectory reasoning lets competing identities redistribute support before persistent states are updated.Source: §4.3 · p. 8, Table 3 · p. 9
- On LVOS v2, the complete model improves J &F from 87.2 to 89.0, with gains in both region similarity and contour accuracy.Qualitative examples show state-based recovery of a host-missed target and reactivation from a railing prediction back to the returning target.Source: §4.3 · pp. 8–9, Figure 5 · p. 9
4.4 FURTHER ANALYSIS
- The largest post-reappearance gains occur after 11–30-frame gaps: 13.1 and 4.5 points in J on LVOS v1 and v2, respectively.On OVIS, the full model reaches 37.31 AP, versus 34.67 for shuffled states and 34.82 for host cues alone.Source: §4.4 · p. 9
- The five-member VIS ensemble uses 0.90M parameters and takes 5.64–5.80 ms per inference on an A100 for 10–40 candidates.Measured latency is similar across those candidate budgets.Source: §4.4 · p. 9
5 CONCLUSION
POSReasoner maintains video-object identities through persistent states and verified transitions, with experiments reporting improvements across evaluated hosts. The paper also analyzes how its association, verification, and state-transition rules handle competing and uncertain evidence.
5 CONCLUSION
- POSReasoner is a trainable, plug-and-play framework that maintains object identities through persistent state history and Propose–Verify reasoning.It supports different VOS and VIS architectures while keeping host models frozen.Source: §5 · p. 9
- Experiments on YouTube-VIS, OVIS, and LVOS show consistent improvements across evaluated hosts, with particularly strong gains under occlusion.Ablations and qualitative comparisons highlight state persistence and reactivation in long-term identity maintenance.Source: §5 · p. 9
A MATHEMATICAL PROPERTIES OF STATE REASONING
- The mathematical-properties analysis characterizes how association, verification, and state-transition rules handle competing and uncertain evidence; segmentation performance is evaluated separately in Section 4.Source: A MATHEMATICAL PROPERTIES OF STATE… · p. 14
A.1 SPARSE ASSOCIATION AND COMPETITION
Sparse legal candidates, including a null option, are normalized into state-to-observation associations. A competition penalty reduces real-candidate odds and, with compatibility logits fixed, increasing real-candidate penalties cannot decrease null probability.
A.1 SPARSE ASSOCIATION AND COMPETITION
- Each state’s legal candidate set includes null, and its normalized matched evidence lies in the convex hull of legal observation features, including the learned null feature.When no real candidates are legal, null receives probability one.Source: §A.1 · p. 14
- With fixed compatibility logits, increasing real-candidate penalties cannot decrease null probability.The competition term acts on relative preferences; the separate output adapter selects at most one retained claimant per real observation.Source: §A.1 · p. 14
- Occupancy combines within-state concentration with cross-state overlap, whereas association entropy depends only on each individual state’s candidate distribution.The null contribution contextualizes unmatched states rather than representing a real-observation conflict.Source: §A.1 · p. 14
A.2 CONTROLLED REFINEMENT OF TRANSITION DECISIONS
A gated correction shifts proposed action logits, with a bound on how much the action distribution can change. The bound concerns correction size, not correction accuracy.
A.2 CONTROLLED REFINEMENT OF TRANSITION DECISIONS
- The gated logit correction bounds the change between proposed and corrected action distributions by its size, without requiring a sign or interval constraint on the gate.The bound also notes that two probability distributions are at most 2 apart in ℓ1.Source: §A.2 · p. 15
- The distribution bound follows by varying logits continuously and bounding the log-sum-exp curvature by the squared maximum correction magnitude.The stated bound uses the correction’s infinity norm and the universal ℓ1 distance bound between probability distributions.Source: §A.2 · p. 15
- Small gated logit corrections preserve a proposal’s action distribution, but the bound does not establish that the correction is accurate.For association probabilities, changes in the competition penalty must also be included in the effective logit correction.Source: §A.2 · p. 15
A.3 SELECTIVE ADMISSION TO PERSISTENT MEMORY
The memory update scales candidate-state movement by a gate shaped by action preference, conditional concentration, and real-evidence mass. Refinement supervision measures deterioration in target-association probability, while the host interface keeps host weights frozen.
A.3 SELECTIVE ADMISSION TO PERSISTENT MEMORY
- When real-association mass is at most ε, the write gate is at most ε², limiting feature movement relative to candidate displacement.At null probability at least 0.9, the gate is at most 0.01; this is a relative bound unless candidate displacement is also bounded.Source: §A.3 · p. 15
- Across a realized trajectory, retained coefficients are algebraic mixture weights, and Propose and Verify affect persistent memory only through the final gate and candidate.With a uniform ε bound, the retained coefficient is at least (1 − ε²)^L; this is not a contraction claim because gate and candidate dependencies are uncontrolled.Source: §A.3 · pp. 15–16
A.4 MEANING OF REFINEMENT SUPERVISION
- The refinement loss controls the proportion of supervised states whose target-association probability deteriorates beyond a chosen threshold.The transition-advantage target separately specifies whether a candidate improves continuation quality relative to retaining the state; exact optimization and perfect advantage prediction are not assumed.Source: §A.4 · p. 16
B.1 HOST INTERFACE
- POSReasoner consumes frozen host trajectory outputs and does not change host weights or request additional image-level proposals.Matched comparisons hold the host checkpoint, input resolution, candidate export, and evaluator fixed.Source: §B.1 · p. 16
B.2 TRAINING AND EVALUATION
Training and evaluation use video-disjoint procedures and fixed host–reasoner configurations. Validation annotations are not read during inference, and the main benchmarks use single runs.
B.2 TRAINING AND EVALUATION
- The VIS reasoner uses a 96-dimensional hidden space, four attention heads, two relation layers, and 180 training epochs.Five video-grouped folds select the executor from out-of-fold predictions, whose models are averaged at inference; candidate budgets are fixed before validation.Source: §B.2 · p. 16
- The VOS reasoner replays frozen identity-indexed masks and constructs events from visibility, geometry, and cross-trajectory changes.Persistent-state, reactivation, and sparse multi-trajectory policies are fitted or calibrated on video-disjoint training splits, then frozen before validation; readout videos are disjoint from calibration and holdout evaluation.Source: §B.2 · p. 16
- Validation annotations are not read during inference, and main benchmarks are single runs of fixed host–reasoner pairs.The OVIS mechanism study repeats three seeds; hyperparameters and executor choices are selected using grouped out-of-fold training performance and fixed before validation.Source: §B.2 · p. 16
B.3 CONTROLLED COMPARISONS AND EFFICIENCY
Controlled comparisons support structured interaction beyond state retention, while efficiency measurements report the reasoner’s parameter count, latency, and memory use.
B.3 CONTROLLED COMPARISONS AND EFFICIENCY
- Across three seeds, the full model achieves 37.203 ± 0.099 AP, a 2.570 ± 0.099 gain.Intact states without a graph reach 36.55 AP, while the full model reaches 37.31 AP.Source: §B.3 · p. 17, Table 4 · p. 17
- Shared two-step reasoning reaches 36.97 AP versus 36.13 AP for one-step reasoning, consistent with benefits from revisiting initial decisions.Host cues alone and shuffled states remain near the frozen host; controls use the same frozen candidates and training protocol.Source: §B.3 · p. 17
- The five-member VIS reasoner uses 0.90M parameters across 10–40 candidates, with 5.64–5.80 ms median latency and 12.71–14.03 MB peak CUDA memory.Latency is measured over 500 runs after 100 warm-ups.Source: §B.3 · p. 17, Table 5 · p. 17
C REAPPEARANCE ANALYSIS
The reappearance analysis examines recovery by absence duration, distinguishing long-term recovery from ordinary frame-to-frame matching. Intermediate gaps show the clearest gains when local identity is ambiguous but persistent evidence remains.
C REAPPEARANCE ANALYSIS
- Intermediate absence gaps show the clearest gains when local identity is ambiguous but persistent evidence remains.The analysis groups reappearance events by absence duration; gaps are measured in absent frames.Source: C REAPPEARANCE ANALYSIS · p. 17, Table 6 · p. 17
- Table 6 reports region similarity, event and video counts, and paired bootstrap intervals over videos; H/T/D counts events that improve, remain unchanged, or degrade.The table groups post-reappearance results by absence duration.Source: C REAPPEARANCE ANALYSIS · p. 17, Table 6 · p. 17
- Figure 6 illustrates identity recovery, reappearance, similar-object interaction, and crowded motion, with yellow references and teal unmodified POSReasoner outputs.The post-hoc diagnostics do not affect policy selection.Source: Figure 6 · p. 17, C REAPPEARANCE ANALYSIS · p. 17
D ADDITIONAL BENCHMARK RESULTS
Across the additional benchmark comparisons, POSReasoner improves evaluated hosts on OVIS and YouTube-VIS. The paired comparisons use frozen hosts and candidate sets, isolating the added reasoner from published benchmark rows.
D ADDITIONAL BENCHMARK RESULTS
- All five evaluated OVIS host–backbone pairs improve in AP, AP50, and AP75; ResNet-50 AP gains are 2.7 for CTVIS, 2.1 for DVIS++, and 1.6 for DAQ.Each OVIS gain exceeds the corresponding host’s gain on any of the three YouTube-VIS versions.Source: Table 7 · p. 18, D ADDITIONAL BENCHMARK RESULTS · p. 19
- Across YouTube-VIS 2019, 2021, and 2022, AP and AP75 improve in all 15 reported host–version comparisons, including the provisionally marked LOMM/ResNet-50 configuration.CTVIS gains 1.0, 1.1, and 2.1 AP across the versions; DAQ gains 0.9, 0.9, and 0.8 AP.Source: Table 8 · p. 18, D ADDITIONAL BENCHMARK RESULTS · p. 19
- Each paired host and POSReasoner comparison shares the host checkpoint, candidate masks, input resolution, and evaluator; only POSReasoner is trained.The paired host, rather than another published implementation of the same architecture, is the appropriate reference for a plug-in gain; the table retains offline-refinement and provisional-result markers.Source: D ADDITIONAL BENCHMARK RESULTS · pp. 18–19, Table 8 · p. 18
- DAQ AP rises from 38.3 to 39.9 with ResNet-50, 49.6 to 50.6 with Swin-L, and 53.9 to 55.2 with ViT-L.DAQ AP75 gains are 1.8, 1.1, and 1.4 points, respectively.Source: D ADDITIONAL BENCHMARK RESULTS · p. 19
- AP75 requires closer mask overlap than AP50, and fixed candidate masks make the paired comparisons about selection, association, and temporal carry-through rather than training a new mask generator.Per-threshold columns complement the state and component analyses of temporal behavior.Source: D ADDITIONAL BENCHMARK RESULTS · p. 19
- Table 4 controls connect benchmark improvements to identity-specific history and structured interaction; the LVOS study separately examines retention, reactivation, and cross-trajectory reasoning.Appendix C groups recovery events by absence duration.Source: D ADDITIONAL BENCHMARK RESULTS · p. 19