PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconstruction-Guided Slot Curriculum: Addressing Object Over-Fragmentation in Video Object-Centric Learning

WonJun Moon, Hyun Seok Seong, Jae-Pil Heo

arXiv:2603.22758v1cs.CVcs.LG

TL;DR

Video object-centric models often over-fragment objects because reconstruction encourages cooperation among all available slots. SlotCurri uses a reconstruction-guided curriculum, structure-aware loss, and cyclic inference to allocate capacity selectively and stabilize representations; it reports substantial FG-ARI gains, while showing limitations on under-fragmentation and fixed scheduling.

  • Problem

    Existing slot-based video object-centric models can use multiple slots for one object because reconstruction quality improves with larger slot budgets, introducing redundancy and incomplete object representations.

  • Method

    SlotCurri begins with few coarse slots, spawns perturbed children for high-reconstruction-error regions, adds structure-aware reconstruction, and applies forward-then-backward cyclic inference.

  • Results

    SlotCurri substantially improves FG-ARI while maintaining SlotContrast’s mBO on synthetic datasets and improving mBO on YouTube-VIS.

  • Takeaways & Limitations

    The method provides a practical coarse-to-fine training paradigm for reducing object over-fragmentation and improving temporally consistent object representations.

  • Takeaways & Limitations

    SlotCurri is less suited to MOVi-E’s under-fragmentation challenge and relies on a predefined curriculum schedule that may require tuning for new datasets.

Abstract

from arXiv · show

Video Object-Centric Learning seeks to decompose raw videos into a small set of object slots, but existing slot-attention models often suffer from severe over-fragmentation. This is because the model is implicitly encouraged to occupy all slots to minimize the reconstruction objective, thereby representing a single object with multiple redundant slots. We tackle this limitation with a reconstruction-guided slot curriculum (SlotCurri). Training starts with only a few coarse slots and progressively allocates new slots where reconstruction error remains high, thus expanding capacity only where it is needed and preventing fragmentation from the outset. Yet, during slot expansion, meaningful sub-parts can emerge only if coarse-level semantics are already well separated; however, with a small initial slot budget and an MSE objective, semantic boundaries remain blurry. Therefore, we augment MSE with a structure-aware loss that preserves local contrast and edge information to encourage each slot to sharpen its semantic boundaries. Lastly, we propose a cyclic inference that rolls slots forward and then backward through the frame sequence, producing temporally consistent object representations even in the earliest frames. All combined, SlotCurri addresses object over-fragmentation by allocating representational capacity where reconstruction fails, further enhanced by structural cues and cyclic inference. Notable FG-ARI gains of +6.8 on YouTube-VIS and +8.3 on MOVi-C validate the effectiveness of SlotCurri. Our code is available at github.com/wjun0830/SlotCurri.

1. Introduction

SlotCurri addresses object over-fragmentation in video object-centric learning by expanding slot capacity progressively where reconstruction remains poor. It combines reconstruction-guided spawning, structure-aware reconstruction, and cyclic inference to improve semantic separation and temporal consistency.

  • Problem: Video object-centric models can split one object across multiple slots because reconstruction objectives implicitly favor using the full slot budget.This creates redundant representations and prevents individual slots from capturing complete objects.
  • SlotCurri: SlotCurri starts with a minimal coarse slot set and progressively increases capacity during training.New slots are initialized by duplicating high-error slots and perturbing them with scaled noise, directing capacity toward under-represented regions.
  • Cyclic inference: Cyclic inference propagates slots forward through the sequence and then backward to improve the earliest-frame representations.The strategy aggregates contextual information and produces more stable, temporally consistent encodings without additional training.
  • Results: The method reports state-of-the-art performance on YouTube-VIS, MOVi-C, and MOVi-E while substantially reducing targeted over-fragmentation.FG-ARI is used to validate over-fragmentation mitigation because it penalizes splitting one ground-truth object across multiple slots.
  • Structure-aware reconstruction: A structure-aware loss complements MSE to preserve local structure and sharpen boundaries during coarse-to-fine decomposition.Sharper parent-slot boundaries support more coherent subsequent slot expansion.

4. Experiments

Experiments show that SlotCurri improves object discovery and reduces over-fragmentation across real-world and synthetic video datasets, with gains supported by ablations and object-level metrics. Its advantage is strongest for over-fragmentation, while MOVi-E exposes a limitation on under-fragmentation.

  • Comparison with the State-of-the-art Methods: SlotCurri improves FG-ARI on real-world and synthetic datasets while maintaining SlotContrast’s mBO on synthetic data and improving mBO on YouTube-VIS.The authors interpret the FG-ARI gains as direct evidence of reduced over-fragmentation.
  • Limitations: SlotCurri’s performance gain on MOVi-E is relatively modest because the dataset’s primary challenge is characterized as under-fragmentation rather than over-fragmentation.The authors identify fine-scale partitioning of many small objects as a mismatch with SlotCurri’s central design.
  • Component Ablation: The reconstruction-guided curriculum achieves gains of up to 6.5 points in FG-ARI over the reproduced SlotContrast baseline.The ablation attributes the gain to early broad grouping, error-guided capacity allocation, structure-aware loss, and cyclic inference.
  • Hyperparameter Sensitivity: Three curriculum stages yield the highest FG-ARI of 44.8, compared with 41.5 for two stages; four stages reduce performance.The result suggests that three stages provide an effective coarse-to-fine schedule in this setting.
  • Object Discovery Quality: Object Identification Recall improves by +4.2, +5.4, and +2.7 points at ρ = 0.3, 0.5, and 0.7, respectively.OIR measures whether at least one slot reaches the specified IoU threshold for each foreground ground-truth object.
  • Object Discovery Quality: SlotCurri reduces over-fragmentation from 1.38 to 1.26 slots per detected object at ρ = 0.5.Degree of over-fragmentation is computed only over ground-truth objects detected by at least one slot; values closer to 1 indicate fewer splits.

5. Conclusion

SlotCurri addresses VOCL over-fragmentation by expanding slots where reconstruction error remains high, sharpening boundaries with structural loss, and improving early-frame consistency through cyclic inference. The paper presents this as a practical training paradigm for compact, consistent video object slots.

  • 5. Conclusion: SlotCurri starts with a minimal slot budget, expands capacity in high-error regions, and uses distance-aware noise to explore under-represented regions.The method is designed to avoid redundantly splitting regions already modeled well.
  • 5. Conclusion: An SSIM-driven structural loss sharpens semantic boundaries during slot expansion.The loss preserves structural cues so newborn slots can inherit more precise coarse-level semantics.
  • 5. Conclusion: Cyclic inference strengthens early-frame representations by aggregating contextual cues from both temporal directions with negligible overhead and no additional training.The strategy is applied during inference rather than requiring another training procedure.
  • 5. Conclusion: Together, the components substantially reduce object over-fragmentation and deliver state-of-the-art performance on YouTube-VIS.The conclusion also identifies scene-adaptive slot schedules and multi-scale slot hierarchies as future work.

A. Datasets

The evaluation benchmarks SlotCurri on YouTube-VIS 2021, MOVi-C, and MOVi-E using the same data splits and preprocessing methodology as SlotContrast.

  • A. Datasets: Experiments use YouTube-VIS 2021, MOVi-C, and MOVi-E to evaluate SlotCurri against existing video object-centric learning approaches.YouTube-VIS 2021 is described as a large-scale real-world video instance segmentation benchmark.
  • A. Datasets: The evaluation adopts the identical data splits and preprocessing methodology used by SlotContrast.This design is stated as ensuring a fair comparison.

B. Object Identification Recall

Object Identification Recall measures whether each foreground ground-truth object is covered by at least one predicted slot above an IoU threshold. It reports the fraction of valid objects meeting that criterion.

  • B. Object Identification Recall: OIR@ρ is the fraction of foreground ground-truth instances for which at least one slot attains IoU≥ρ with the instance mask.The metric isolates object coverage irrespective of fragmentation.
  • B. Object Identification Recall: For each valid ground-truth object, OIR checks whether the maximum IoU across predicted slots reaches threshold ρ.Gvalid contains non-zero-area ground-truth objects and excludes the background class.

C. Degree of Over-Fragmentation

The Degree of Over-Fragmentation (DOF) measures how many slots are assigned to each detected ground-truth object, with values closer to 1 indicating fewer unnecessary splits.

  • DOF assigns a slot to a ground-truth object when at least a fraction ρ of the slot area lies within that object’s mask.The assignment condition is |m_k ∩ g| / |m_k| ≥ ρ.
  • DOF averages the number of slots assigned to each detected ground-truth object.Only identified ground-truth objects are included in the average.
  • Lower DOF values, especially values closer to 1, indicate fewer unnecessary splits of the same object.Unidentified objects are excluded so the metric does not conflate fragmentation with detection recall.

D. Degree of Under-Fragmentation

The Degree of Under-Fragmentation (DUF) measures how many distinct ground-truth objects are merged into individual slots. SlotCurri achieves lower DUF scores than the baseline across all evaluated thresholds.

  • DUF evaluates under-segmentation, where one slot erroneously merges multiple distinct objects.A slot captures a ground-truth object when their Intersection-over-Union exceeds threshold ρ.
  • SlotCurri achieves lower DUF scores than the baseline across all evaluated thresholds.This indicates improved distinctness and quality of object disentanglement rather than merely shifting errors from over- to under-fragmentation.

E. Qualitative Ablation Study

The qualitative ablation study examines how individual components affect SlotCurri’s representations. The curriculum, structure-aware objective, and cyclic inference each address different representation or temporal-consistency issues.

  • The R-G curriculum mitigates over-fragmentation in the qualitative visualizations.
  • The SSIM objective sharpens object boundaries and helps separate distinct entities through compositional structures.The visualizations specifically show clearer separation among pink, red, and orange masks.
  • Cyclic inference incorporates long-range contextual cues to enhance slot consistency across the video.

F. Analysis of the Accelerated Slot Schedule

The accelerated slot schedule increases newly added slots progressively across curriculum stages, unlike linear and decelerated alternatives. Its ablation supports allocating limited capacity early for coarse semantics and more capacity later for fine-grained refinement.

  • SlotCurri uses an accelerated schedule in which the number of new slots added increases quadratically at each curriculum stage.The ablation compares this schedule with linear and decelerated schedules, starting at K_init = 2 and ending at the same K_final.
  • The decelerated schedule yields the smallest gain because it introduces high capacity too early, before coarse-level semantics stabilize.The linear schedule performs better but remains significantly worse than the accelerated approach.
  • The accelerated approach outperforms the linear schedule in the YouTube-VIS scheduling ablation.Table A2 reports the comparison among the scheduling strategies.
  • The coarse-to-fine hypothesis separates learning into stable broad-region anchoring followed by higher-capacity partitioning of fine-grained details.Later refinement requires a larger representational budget because it must capture many smaller entities.

G. Analysis of the Reconstruction-Guided Slot Spawning Criterion

SlotCurri allocates new slots according to reconstruction error, favoring actively used, poorly reconstructed regions over idle or already well-modeled regions. The analysis also identifies robustness and streaming boundaries for this strategy.

  • Total Error Mass is compared with an Area-Normalized Error criterion for spawning slots on YouTube-VIS.
  • New slots duplicate parent slots with the highest accumulated, decoder-weighted reconstruction error.
  • Idle, weakly used, or perfectly reconstructed slots have negligible error mass and are ineligible for splitting.
  • Area normalization can make weakly used slots appear arbitrarily erroneous because near-zero reconstruction mass amplifies minor noise.
  • Total error mass is more robust in this setup, although hybrid criteria may better serve domains dominated by small, dense objects such as MOVi-E.
  • Full cyclic inference becomes impractical for long streaming videos, motivating chunk-wise cycles that bound latency and avoid buffering the entire video.

I. Necessity of Structure Loss in Slot Curriculum Learning

Structure-aware reconstruction loss sharpens boundaries needed for slot curriculum learning, while spatio-temporal consistency and curriculum allocation improve performance under varying slot capacities. The analysis also documents failure modes for small objects and fixed schedules.

  • +2.3 FG-ARI and +1.0 mBO result when structure-aware loss is combined with reconstruction-guided curriculum learning.
  • 3D-SSIM is used instead of frame-wise 2D-SSIM because it preserves structural consistency across time.
  • MSE blurs spatial details and object boundaries, making early coarse slots ambiguous and prone to later subdivision.
  • When K increases from 11 to 15, SlotContrast FG-ARI falls from 69.3 to 61.8, whereas SlotCurri reaches 74.8 at K = 15, a +13.0 margin.
  • MOVi-E small objects can remain spatially entangled or lack clear boundaries, limiting SlotCurri under under-fragmentation.
  • The fixed curriculum schedule may require manual tuning on datasets with substantially different characteristics.

L. SlotCurri on Object Dynamics Prediction

SlotCurri improves object-centric representations in downstream dynamics prediction and static-image evaluation while reducing over-fragmentation in qualitative comparisons. Its gains are strongest on MOVi-C, whereas improvements are less pronounced on MOVi-E.

  • Object dynamics prediction: SlotCurri improves object dynamics prediction over existing baselines, with especially strong FG-ARI performance on MOVi-C.The authors interpret this as evidence that the learned slots capture distinct object identities and handle complex dynamics more robustly.
  • Object dynamics prediction: On MOVi-E, SlotCurri remains competitive with state-of-the-art methods, but gains are less pronounced because the dataset primarily challenges under-fragmentation.This differs from SlotCurri’s primary target of mitigating over-fragmentation.
  • Static-image evaluation: On MS COCO, SlotCurri substantially increases Image-ARI over a reconstruction-based baseline by resolving over-fragmentation without temporal cues.The static-image adaptation uses reconstruction-guided slot curriculum with a 2D SSIM loss and omits cyclic inference.
  • Qualitative comparisons: Qualitative comparisons show SlotCurri assigning one slot to each semantic entity, whereas SlotContrast splits objects such as deer and bears across multiple slots.The more compact groupings persist across both YouTube-VIS and MOVi-C examples.
Loading 2603.22758v1…