Source-linked AI summary
Selective Synergistic Learning for Video Object-Centric Learning
WonJun Moon, Jae-Pil Heo
TL;DR
Video object-centric learning must reconcile encoder attention maps that are sharp but noisy with decoder object maps that are coherent but blurry. SSync selectively distills boundary and interior cues, and its combined alignment with transitive merging reaches 79.4 FG-ARI and 39.5 mBO in ablation results.
Problem
VOCL encoder and decoder maps remain misaligned because sharp, noisy encoder assignments contrast with coherent decoder maps having blurred boundaries.
Method
SSync selectively distills encoder boundary cues and decoder interior cues through pseudo-labeling, then merges redundant slots using spatio-temporal activation consistency.
Results
79.4 FG-ARI and 39.5 mBO are achieved after combining selective alignment losses with transitive merging in the ablation study.
Takeaways & Limitations
Qualitative results show SSync maintains stable object identities and clearer, more semantically aligned boundaries in challenging real-world videos.
Takeaways & Limitations
SSync can under-fragment object identities in early frames and over-fragment large objects with strong intra-object variation.
Abstract
from arXiv · showhide
Typical video object-centric learning (VOCL) approaches employ slot-based frameworks that rely on reconstruction-driven encoder-decoder architectures, where learning is mediated by two spatial maps: attention maps from the encoder and object maps from the decoder. As these two distinct maps exhibit different properties, a recent dense alignment strategy attempted to reconcile this discrepancy by enforcing agreement across all spatio-temporal patches via contrastive learning. However, this indiscriminate alignment inadvertently propagates the inherent weaknesses of each module, such as noisy encoder predictions and blurred decoder boundaries. Moreover, computing dense similarities across all pairs incurs a computational cost quadratic in the total number of spatio-temporal patches, severely limiting scalability. Motivated by this, we propose Selective Synergistic Learning (SSync). Instead of exhaustive patch-to-patch alignment, SSync prevents error propagation by selectively distilling only the most reliable cues: leveraging the encoder strictly for boundary refinement and the decoder for interior denoising. This is realized via a pseudo-labeling with linear complexity, eliminating the need for quadratic spatial comparisons. Also, to prevent the reinforcement of architectural biases like slot redundancy, we introduce a transitive pseudo-label merging that consolidates overlapping slots based on spatio-temporal activation consistency. Extensive studies demonstrate that SSync improves decomposition quality and serves as a versatile, plug-and-play module while also exhibiting exceptional robustness to slot configurations. Code is available at github.com/wjun0830/SSync.
1 Introduction · 2 Related Work · 3 Selective Synergistic Learning
SSync addresses encoder–decoder mismatch in video object-centric learning by selectively transferring reliable boundary and interior cues rather than enforcing dense global agreement. It combines linear-complexity pseudo-labeling with transitive slot merging to refine maps while reducing noise, blur, and redundant slot identities.
- 1 Introduction: Object-centric learning decomposes scenes into discrete object representations, while video extensions use temporal cues for consistent object discovery and tracking.Slot Attention established the iterative grouping and shared-decoder framework underlying these approaches.
- 1 Introduction: Encoder attention maps are spatially sharp but noisy, whereas decoder object maps are coherent but have blurry boundaries, creating a structural mismatch.Both maps assign patches to slots but inherit different error characteristics from their modules.
- 2.1 Object-Centric Representation Learning: SRL [14] applies dense contrastive alignment across all spatio-temporal patches, but uniform supervision propagates branch-specific weaknesses and incurs quadratic complexity O((T · H · W)2).This approach assumes every patch is equally reliable, limiting scalability to long or high-resolution videos.
- 3.1 Preliminaries: SSync selectively distills encoder boundary cues to refine decoder masks and decoder interior consistency to denoise encoder assignments.Local spatial and temporal consistency identifies reliable boundary and interior regions for asymmetric cross-module supervision.
- 2.1 Object-Centric Representation Learning: Related video slot-learning methods use motion, depth, transformers, motion-consistent grouping, or temporal contrastive learning to improve dynamic-scene representations.These methods include SAVi, SAVi++, STEVE, Videosaur, and SlotContrast.
- 3.3 Transitive Pseudo-Label Merging: Pseudo-labeling can propagate early errors and over-fragment objects, so SSync merges redundant slots using spatio-temporal activation overlap.Connected components of the redundancy graph consolidate transitively overlapping slots under a dominant representative identity.
- 3.2 Structuring Pseudo-Labels: Boundary candidates use encoder disagreement with at least one agreeing neighbor, while interior patches use decoder disagreement below n_nbd.These criteria filter isolated noisy assignments and provide complementary regions for cross-distillation.
- 3.4 Training Objective: The training objective uses MSE cross-distillation, supervising the decoder on boundary patches and the encoder on interior patches.SSync is added to the reconstruction and temporal slot contrastive base objective after a warm-up covering the first 30% of iterations.
4 Experiments
Across video and image benchmarks, SSync consistently improves decomposition quality, generalizes across architectures and visual domains, and substantially reduces memory usage. Ablations and qualitative analyses attribute these gains to complementary selective alignment, transitive merging, and improved boundary and interior consistency.
- Video Benchmarks: SSync achieves superior or highly competitive performance across MOVi-C, MOVi-E, and YTVIS, addressing fragmentation and boundary-precision challenges.On MOVi-C, selective alignment consolidates fragmented identities; on MOVi-E, it improves capture of numerous small-object boundaries.
- Video Benchmarks: SSync achieves competitive FG-ARI and the highest mBO on YTVIS, while qualitative results show more consistent separation of secondary objects and background regions.YTVIS annotations cover only a sparse set of salient foreground objects, limiting what the reported metrics capture.
- Video Benchmarks: SSync improves two RandSF.Q variants built on VideoSAUR and SlotContrast, demonstrating plug-and-play applicability.The reported improvements indicate compatibility across different underlying VOCL architectures.
- Image Benchmarks: SSync achieves a SOTA FG-ARI of 86.0 on MOVi-E and an ARI of 47.9 and mBO of 33.1 on COCO2017, substantially outperforming SRL.These image-level results support spatial consistency independent of motion signals.
- Memory Efficiency: 27GB versus 70GB at batch size 32 per GPU reduces memory usage by roughly 60% for SSync compared with SRL.SRL has quadratic memory growth, O((T ·H ·W)2), from dense patch-to-patch comparisons, whereas SSync scales approximately linearly with patch count.
- Ablation and Robustness Analysis: 77.1 FG-ARI and 38.0 mBO result from combining selective alignment losses, rising to 79.4 FG-ARI and 39.5 mBO with transitive merging.The results show boundary calibration and interior denoising are complementary, while merging further stabilizes redundant slot identities.
5 Conclusion
SSync is proposed as a selective and efficient approach for VOCL that mitigates discrepancies between encoder attention maps and decoder object maps. It aligns the maps only in regions where each has relevant expertise, using pseudo-labeling.
- 5 Conclusion: SSync mitigates discrepancies between encoder slot attention maps and decoder object maps while achieving performance gains.The method is designed to address the differing properties of the two maps in VOCL.
- 5 Conclusion: SSync selectively aligns attention and object maps only where each map has its respective expertise.This selective alignment is enabled by an efficient pseudo-labeling scheme.
6 Training Details
The experiments use uniform optimization settings and largely consistent training configurations across datasets and benchmarks, with hardware adjusted for YouTube-VIS’s higher resolution.
- Training configuration: All datasets use batch size 128 and learning rate 8e-4, with experiments run on two NVIDIA RTX A6000 GPUs except YouTube-VIS.YouTube-VIS uses NVIDIA RTX PRO 6000 Blackwell GPUs because its 518×518 inputs require more VRAM.
- Training configuration: For Tab. 3, all experiments process 224 × 224 images with DINOv2-Small and train for 50000 iterations.The Tab. 3 setup follows RandSF.Q [20].
- Training configuration: The hyperparameters nbd, nnbd, λSSync, and τmerge remain consistent across benchmarks, using the MOVi-D value established for MOVi-C.This consistency is used to demonstrate framework robustness.
7 Detailed Comparison with SRL
Compared with SRL, SSync simplifies mutual learning through a projector-free, linear-complexity design that improves scalability and parameter efficiency. Its adaptive redundancy mitigation and selective cross-distillation avoid uniform dense alignment and reduce error propagation from noisy or blurred predictions.
- Overall comparison: Compared with SRL, SSync eliminates auxiliary projectors and complex contrastive objectives while achieving linear complexity and higher parameter efficiency.These design choices improve scalability, efficiency, and robustness.
- Architecture and efficiency: SSync operates directly on native slot attention maps, eliminating SRL’s two MLP projectors and introducing no additional parameters.This makes SSync a plug-and-play module for existing slot-based architectures.
- Computational complexity: SSync reduces alignment complexity from SRL’s O((T · H · W)2) to O(T · H · W) by applying MSE only to reliable regions.This supports higher-resolution inputs and longer video sequences where SRL becomes memory-intensive.
- Redundancy mitigation: SSync uses transitive pseudo-label merging to dynamically resolve slot redundancy throughout training, unlike SRL’s regularization and complex warm-up schedule.SRL’s warm-up enforces exactly half of the slots to remain vacant during the initial stage.
- Selective supervision: SSync replaces SRL’s dense ternary contrastive alignment with selective cross-distillation, letting each branch supervise only regions where it is most reliable.This avoids propagating encoder noise and decoder blur, improving object decomposition quality.
8 Qualitative illustration of SSync Mechanisms.
SSync’s synergistic objectives activate after a 30% warm-up, addressing noisy attention maps, blurry object boundaries, and object over-fragmentation. Transitive merging fuses redundant slots through spatiotemporal overlap to produce semantically coherent groupings before global convergence.
- Training mechanism: SSync activates its synergistic objectives after a warm-up covering 30% of total training iterations.The mechanism is illustrated as an evolving training process.
- Training mechanism: At activation, attention maps contain noisy patches, while object maps have blurry boundaries and both maps show object over-fragmentation.These imperfections motivate extracting supervision selectively from the two maps.
- Training mechanism: Transitive merging identifies and fuses redundant slots using spatiotemporal overlap, yielding semantically coherent groupings before global convergence.The merging operates on imperfect attention and object maps to obtain reliable supervision.
9 Boundary & Non-Boundary (Interior) Analysis
SSync separates boundary patches from semantically consistent interior regions, locating them along object transitions and within stable object cores. Its criterion also captures temporal boundaries caused by motion-driven changes across frames.
- Visualization: SSync effectively identifies object boundary regions and semantically consistent interior regions in representative frames from 10 example videos.The visualizations show the method’s boundary and interior classifications across the examples.
- Spatial Analysis: Boundary patches concentrate along object transitions, while interior patches cluster within stable object cores.This indicates that the local consistency criterion captures high-frequency structural transitions without disrupting interior coherence.
- Temporal Analysis: Some patches inside objects are classified as temporal boundaries when semantic content changes across consecutive frames due to object motion.The mechanism therefore adapts beyond spatial edges to spatio-temporal changes and dynamic transitions.
10 Impact of Denoising and Deblurring
SSync selectively denoises encoder attention maps and deblurs decoder object-map boundaries. It reduces slot-mask fragmentation and outside leakage while increasing high-coverage matches.
- Denoising: SSync substantially lowers fragmented components compared with SlotContrast and SRL, demonstrating stronger denoising of predicted slot masks.FCC measures frame-averaged connected components under 4- and 8-neighbor connectivity; SlotContrast exceeds 33 and SRL is approximately 21.
- Deblurring: SSync achieves the lowest outside leakage at both 75% and 90% GT-coverage thresholds while increasing the number of high-coverage matches.Outside leakage measures predicted-slot area spilling into the background for matched GT-slot pairs.
- Complementary refinement: The analyses show that SSync denoises encoder attention representations and deblurs spatially inflated decoder object-map boundaries by selectively distilling complementary module strengths.This interpretation links the fragmentation and leakage analyses to SSync’s selective distillation strategy.
11 Evolution of Synergistic Refinement
Training evolution shows selective synergistic refinement turning encoder–decoder mismatches into mutual improvement: the decoder sharpens boundaries while the encoder suppresses interior noise. The boundary–interior supervision sets also become more separated over training.
- Deblurring: Decoder outside leakage falls from approximately 93.96 × 10^3 patches at 30% training to 71.03 × 10^3 by training’s end under encoder boundary guidance.This indicates progressively sharper decoder boundaries through the selective alignment loss Lbd.
- Denoising: Encoder FCC decreases from 41.48 at 30% iterations to 8.90 as decoder interior cues are distilled through Lnbd, removing isolated noisy patches.The decoder’s FCC is 13.15 at 30% of total iterations, providing the less noisy interior supervision.
- Overall evolution: Together, declining leakage and FCC confirm that selective synergistic refinement converts module mismatch into mutual improvement and more coherent object discovery.Outside leakage measures boundary blurring, while FCC measures spatial denoising.
- Supervision-set evolution: At 30% of iterations, the overlap between encoder boundary patches Pbd and decoder non-boundary patches Pnbd is 0.277 IoU, indicating imperfect initial separation.The overlap ratio tracks the spatial evolution of the two selective supervision sets during training.
12 Quality of Learned Slots: Object Dynamics Prediction
The learned slots are evaluated for transferability to object dynamics prediction using a frozen object-centric model and SlotFormer. SSync improves dynamics prediction consistently across datasets, including fine-grained dynamics for small objects on MOVi-E.
- Evaluation setup: SlotFormer autoregressively predicts future slots from frozen object-centric models for F rollout steps after B burn-in frames.The settings are (B, F) = (14, 10) for MOVi-C, (5, 10) for MOVi-E, and (10, 5) for YouTube-VIS.
- Results: SSync achieves consistent object dynamics prediction improvements across MOVi-C, MOVi-E, and YouTube-VIS compared with previous methods.SlotFormer is used to evaluate the learned slots from each method.
- Results: On MOVi-E, SSync models fine-grained dynamics effectively despite prior models struggling with its prevalence of small objects.The reported advantage is attributed to capturing precise semantic boundaries.
- Conclusion: These results indicate that SSync enhances static object discovery while producing robust representations for downstream dynamics prediction.The conclusion follows from the consistent cross-dataset improvements and MOVi-E performance described in the passage.
13 Ablation on the SSync Warmup Ratio · 14 Ablation on Activation Thresholding for Transitive Merging · 15 Ablation on Patch Selection Variants
SSync remains robust across warmup schedules, while activation-threshold and patch-selection ablations clarify practical tradeoffs. Quantile tuning and enhanced spatio-temporal selection can improve performance, but fixed quantiles may limit cross-dataset generalization.
- 13 Ablation on the SSync Warmup Ratio: SSync is generally robust when the selective alignment loss begins after 5%–50% of training iterations, compared with the default 30% schedule.The warmup-ratio ablation varies warmup length from 5% to 50% of total training steps.
- 14 Ablation on Activation Thresholding for Transitive Merging: The default transitive merging criterion marks patches active when attention exceeds each slot’s mean activation.This slot-wise mean threshold defines active patches for transitive merging.
- 14 Ablation on Activation Thresholding for Transitive Merging: Quantile-based thresholding retains each slot’s top 10%, 20%, or 30% activation values to construct the overlap graph.These quantile thresholds replace the default mean-based activation criterion.
- 14 Ablation on Activation Thresholding for Transitive Merging: Quantile thresholding slightly enhances performance, but fixed quantiles introduce dataset-dependent hyperparameters tied to object-size distributions.The optimal quantile depends on objects’ average scale and scale variance, limiting generalization across domains.
- 15 Ablation on Patch Selection Variants: Patch-selection comparisons exclude transitive merging to isolate the contribution of the boundary and interior selection process.The alternatives are evaluated as selection variants described in Table 21.
- 15 Ablation on Patch Selection Variants: The enhanced selection method extends relaxed morphological erosion with noise filtering and spatio-temporal comparisons, outperforming standard erosion.These additions detect motion edges and improve temporal coherence; standard erosion nevertheless validates the core selection philosophy.
- 15 Ablation on Patch Selection Variants: The straightforward selection design consistently outperforms entropy-based alternatives across thresholds ranging from the average entropy to different percentiles.The entropy variant treats low-entropy patches as reliable for distillation.
16 Additional Ablation Studies on SSync Design · 17 Additional Qualitative Results. · 18 Failure Case Analysis (Limitation).
Additional ablations show that SSync’s default hard-target design and encoder–decoder supervision assignment outperform softened or reversed alternatives. Qualitative results demonstrate more coherent decompositions, while failures remain in early identity separation and grouping visually heterogeneous parts of large objects.
- 16 Additional Ablation Studies on SSync Design: SSync’s default formulation outperforms soft-target, relaxed-selection, and reversed-supervision alternatives.Soft distributions retain uncertainty that can propagate structural ambiguity between modules, while reversed SSync swaps encoder boundary and decoder interior supervision.
- 17 Additional Qualitative Results.: On MOVi-C, SSync produces temporally coherent decompositions with tighter, sharper, and less over-fragmented object masks than SlotContrast and SRL.It also assigns a stable background slot while preserving compact foreground masks.
- 17 Additional Qualitative Results.: On YouTube-VIS 2021, SSync maintains stable person instances and consistently separates visually similar chairs throughout clips.SlotContrast splits a person across slots and, alongside SRL, struggles with the two-chair separation in the described sequence.
- 18 Failure Case Analysis (Limitation).: SSync exhibits two failure modes: early-frame identity under-fragmentation and part-level over-fragmentation for large objects with strong intra-object variation.These limitations are analyzed on MOVi-C, MOVi-E, and YouTube-VIS.
- 18 Failure Case Analysis (Limitation).: On MOVi-C and MOVi-E, insufficient early motion cues can cause objects entering from similar locations or overlapping objects to remain assigned to one slot.In the example, two small objects remain covered by one slot until their motion trajectories diverge spatially.
- 18 Failure Case Analysis (Limitation).: On YouTube-VIS, large objects with markedly different visual characteristics across parts can trigger part-level over-fragmentation.A cargo truck’s driver’s cabin and container have substantially different textures.
- 18 Failure Case Analysis (Limitation).: Even with transitive pseudo-label merging, spatio-temporal overlap alone may not group semantically related parts when intra-object appearance variance is high.The analysis suggests stronger part-to-whole grouping priors as a future direction.