PaperlayerView exact revision on arXiv

Source-linked AI summary

iSEE: Object Permanence Through Self-Supervision

Pramish Paudel, Ajad Chhatkuli, Luc Van Gool, Danda Pani Paudel

arXiv:2610.01201v1cs.CV

TL;DR

Self-supervised object-centric slots can lose object identity during occlusion, while object permanence methods commonly rely on labels. iSEE uses temporal evidence, separate appearance and position streams, and a reappearance-trained walker; it returns objects to their slots after long occlusions and localises them while hidden.

  • Problem

    Self-supervised slot representations are occlusion-sensitive, while object permanence requires tracking an object's identity and position when hidden.

    Source: §1 · pp. 1–2

  • Method

    iSEE detects occlusion from a slot's temporal attention, separates appearance from position, and trains a walker from reappearance to follow hidden positions without labels.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    77.5% of occlusions longer than four seconds under a static camera: iSEE returned the target to its slot; it also returns objects after long occlusions and localises them while hidden without labels.

    Source: §4.2 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    Separating appearance and position gives world models a natural action through which a planner reaches its goal by gradient descent.

    Source: §5 · p. 9

    iSEE inherits slot-model binding: a slot may latch onto a similar object or be released before the target returns, and the model uses a fixed number of slots per dataset.

    Source: §5 · p. 9

Abstract

from arXiv · show

Object permanence, keeping track of an object's identity and position while it is occluded, is central to video representations that track, predict and plan. Trackers that achieve it learn from boxes, track identities and visibility labels. On the other hand, self-supervised object-centric methods discover objects without labels: through slot attention, it represents a video as slots that bind to objects and follow them across frames. However, these slots are lost under occlusion, making the desired permanence impossible. Reasoning permanence is a hard problem because it requires to detect when an object becomes occluded, re-identify when object reappears, and keep the object's hidden position continuous, using reapperance as the only learning cue. To address this, we propose iSEE, a novel framework that offers all three aforementioned requirements, without any labels whatsoever. We built iSEE using the following three proposed components: (i) Object evidence modelling: a slot's attention, compared with its own past, reveals when its object is hidden. (ii) Appearance-position separation: two slot streams let the appearance be held for re-identification while the position keeps changing. (iii) Permanence from reappearance: a walker follows the hidden object's position, trained only on where the object reappears. On LA-CATER static, iSEE returns a reappearing object to its own slot after 86% of occlusions, against 32% for SlotContrast, and localises it while hidden within 4.1 mAP of the label-trained SoTA RAM. The two streams also allow downstream planning, with the position stream as the action of a world model. Project page: https://insait-institute.github.io/iSEE/

1 INTRODUCTION

Self-supervised object slots are occlusion-sensitive, and existing permanence approaches rely on extra supervision. iSEE combines temporal evidence, separate appearance and position streams, and a walker trained from reappearance to maintain object identity through occlusion.

1 INTRODUCTION

  • Self-supervised slot representations can lose an object's identity during occlusion, while previous permanence approaches use visible-object boxes or clean scene backgrounds.Existing methods update slots every frame, which can rewrite a hidden object's slot with a visible patch.Source: §1 · p. 2
  • TEN compares a slot's attention with its own past to detect occlusion and hold the slot until the object reappears.It supplies an explicit visibility cue for pausing updates.Source: §1 · p. 2
  • Separate appearance and position streams hold an occluded object's appearance while its position keeps changing.Figure 2 depicts the two-stream slot and its training setup.Source: §1 · p. 2, Figure 2 · p. 3
  • A walker follows an occluded object's position and is trained only on where the object reappears, without labels.The method models object permanence from reappearance, consistent with the stated principle that observers learn the distinction between occlusion and disappearance from reappearances.Source: §1 · p. 2

2 RELATED WORK

Related work spans self-supervised object-centric video representations, appearance-position separation, and object permanence. The cited approaches vary in their treatment of occlusion and the supervision they require.

Unsupervised object-centric video representation learning

  • Object-centric video models use slots bound to image features and have expanded from synthetic scenes to real images and video, but occlusion remains open.VideoSAUR reports slot reassignment to other objects in longer videos with occlusions.Source: §2 · p. 3

Separating appearance from position

  • Scene and particle models separate appearance from position, but particles may span multiple objects or split one object, and cited methods name camera motion as a limitation.STAITUS separates appearance, position, and scale on DINO features, which carry position.Source: §2 · p. 3

Object permanence

  • Most cited object-permanence methods learn from labels; Loci-Looped and OC-MOT instead use a clean background image or detection labels on at most 6.25% of frames.RAM uses visible-frame boxes and track identities, then localises hidden objects with a reappearance-supervised random walk.Source: §2 · p. 3

3 METHOD

The method models scenes as slots with slowly changing appearance and frame-varying position, including objects hidden for contiguous spans. iSEE separates those streams, detects occlusion from temporal evidence, and trains a walker on reappearance to continue hidden positions.

3 METHOD

  • A video is modeled as a scene of at most K objects, each with slowly changing appearance and position that may change every frame, including during contiguous occlusion.Occlusion may occur behind an occluder, inside a container, or when another object carries the target.Source: §3 · p. 4

3.1 TWO STREAMS

  • A slot is split into appearance and position streams; the appearance grid uses features without positional encoding, while the position grid contains fixed patch-coordinate codes.The decoder places each object from position alone, and the streams are updated from their respective grids.Source: §3.1 · p. 4
  • The two-stream attention combines appearance and position matching, while the position state is a recurrent location-and-extent representation on the grid.The position predictor reads appearance through a stop-gradient, and the slot–slot contrastive loss acts only on appearance.Source: §3.1 · p. 4

3.2 TEMPORAL EVIDENCE NORMALISATION AND THE WALKER

  • Temporal evidence normalization compares slot attention with that slot's earlier values to detect when to hold and release it.Local evidence supports starting a hold, while evidence anywhere supports release; a partial glimpse does not end the hold.Source: §3.2 · p. 5
  • During occlusion, a walker follows content across appearance-feature patches and shifts the held slot's position along the inferred path.It is trained on the encoder's position just after release, with no ground-truth position in the loss; the encoder is trained first, and TEN and the walker run together at inference.Source: §3.2 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate iSEE on synthetic and real video for hidden-object localization, re-identification, discovery, and planning. Results show label-free permanence near supervised RAM on static LA-CATER and improved same-slot return over label-free baselines.

4.1 SETUP

  • Evaluation spans LA-CATER’s static and moving cameras, MOVi-C validation, and KITTI video, comparing SlotContrast and other alternatives with supervised RAM as a reference.Source: §4.1 · p. 6

4.2 OBJECT PERMANENCE AND RE-IDENTIFICATION

  • 80.9 vs 85.0 mAP@[0.1, 0.3] puts label-free iSEE within 4.1 points of supervised RAM on static-camera LA-CATER hidden frames, ahead of every other model.The table reports mAP on hidden frames; the comparison is for the static-camera setting.Source: §4.2 · p. 6, Table 1 · p. 7
  • Carried frames are hardest: iSEE scores 78.7 while frozen and Loci-Looped fall to 41.9 and 32.9; moving-camera performance drops upstream at encoder discovery.The moving-camera drop is attributed to weaker object discovery and fewer targets returning to their slot.Source: §4.2 · p. 7
  • 77.5% of static-camera and 64.2% of moving-camera LA-CATER occlusions longer than four seconds return to iSEE’s slot, versus at most 25.4% for baselines; KITTI rates are 48.4% vs 29.0% for SlotContrast.Re-identification is measured as same-slot return (SAME).Source: §4.2 · p. 7, Table 2 · p. 8

4.3 OBJECT DISCOVERY

  • iSEE improves discovery over SlotContrast on every dataset; with TEN on both encoders, KITTI gains are 9.5 FG-ARI and 10.5 mBO, versus 2.1–3.5 FG-ARI on rendered datasets.Source: §4.3 · p. 8, Table 3 · p. 9

4.4 APPLICATION: PLANNING WITH A WORLD MODEL

  • iSEE’s predictor is 3.9 FG-ARI below its oracle versus 9.4 for SlotContrast, while gradient-based planning moves mask centres from 25.1 to 5.2 px over 20 validation clips.The position stream supplies the predictor’s action input.Source: §4.4 · p. 8, Table 5 · p. 9

4.5 ABLATIONS

  • TEN improves SlotContrast re-identification; the appearance-position split helps only with DINOv2 NoPE features, and the walker tracks the object while hidden.Splitting hurts while features carry position; without positional encoding, it improves re-identification and most notably mAP.Source: §4.5 · p. 9, Table 4 · p. 9

5 CONCLUSION

The conclusion describes inherited slot-binding limitations, reports AI-assistant use, and states that code and related artifacts will be released.

5 CONCLUSION

  • A slot not assigned to the target when occlusion begins cannot follow it, and iSEE inherits a fixed slot count per dataset.The authors also note that similar-looking objects can attract a slot or that a slot can be released before the target returns.Source: §5 · p. 9
  • AI assistants helped write code, check related work, and edit the text; the authors say they checked the claims against code and cited papers.The authors attribute the ideas, experiment design, and claims to themselves.Source: AI USE STATEMENT · p. 9
  • The authors state that code, configurations, occlusion lists, and trained walkers will be released.The encoder changes only the keys of the released SlotContrast configuration; other constants, metrics, baselines' checkpoints, and the dynamics model are in appendices.Source: REPRODUCIBILITY STATEMENT · pp. 9–10

A METHOD

iSEE combines three stages, including temporal evidence normalization and a walker that writes position into held slots; the method uses no box, mask, visibility, or occlusion labels.

A METHOD

  • The three-stage method runs temporal evidence normalization and the walker together at inference, with the walker's position written into held slots and no labels used.The table identifies the three stages as iSEE's method components.Source: Table S1 · p. 14

A.1 TWO STREAMS

  • The position stream carries patch-centre coordinate codes, and the decoder reads position and scale from that stream alone.The painting MLP has no access to absolute position.Source: §A.1 · p. 14

A.2 TEMPORAL EVIDENCE NORMALISATION

  • The evidence window uses the slot's last unheld-frame attention, max-scaled and spatially dilated; held slots retain dimensions [0, 60) from the previous committed state.The references update on held as well as updated frames, and a hold does not start while local evidence is rising.Source: §A.2 · p. 14

A.3 WALKER

  • The walker uses a frozen-corrector-derived position field, interpolated between patches, and restarts from a position recorded during visible updates.It moves every second frame and is trained on clips with up to eight pseudo-occlusions each.Source: §A.3 · pp. 14–15

B.1 IMPLEMENTATION DETAILS

The implementation uses frozen DINOv2-derived patch features and trains on KITTI clips; pseudo-occlusions provide walker targets without labels.

B.1 IMPLEMENTATION DETAILS

  • The encoder uses frozen DINOv2 ViT-S/14 features, fine-tuned on COCO to reproduce its original patch features, then maps 384-dimensional features to 64.The learned positional embedding is deleted; resolutions vary by dataset.Source: §B.1 · p. 15
  • KITTI training uses 8-frame windows from 70 training clips, while discovery is scored over 50-frame validation clips.The crop is fixed across each window, and the crop is horizontally flipped with probability 0.5.Source: §B.1 · p. 15
  • Because KITTI provides only 39 car occlusion–reappearance events, pseudo-occlusions replace their use for walker training, with targets set to the slot's own later position.Occluded patch features are replaced with features from patches the slot does not own, and no label enters the features, anchor, or target.Source: §B.1 · p. 15

B.2 DATASETS

The section specifies LA-CATER and KITTI evaluation data, the scored-occlusion protocol, and how several baseline predictions and checkpoints are selected.

B.2 DATASETS

  • LA-CATER provides 9,300/3,327/1,371 static-camera train/validation/test clips and 9,299/2,152/1,371 moving-camera clips.The scored-occlusion table covers the first 1,200 test clips of each camera and measures target-centre displacement from the first hidden frame to its furthest occlusion point.Source: §B.2 · p. 16, Table S2 · p. 16
  • Scorable LA-CATER occlusions last at least five frames, have visible target frames on both sides, and are scored when the amodal centre moves at least 10 pixels; the remainder are capped at 50.The same sampled list is used for every row; scored occlusions span 5–245 frames for static and 5–240 for moving camera.Source: §B.2 · p. 16
  • KITTI occlusion analysis covers 39 occlusions of 29 cars from 21 labelled tracking videos, lasting 0.1 to 2.2 seconds.Car occlusions are identified from KITTI-MOTS masks using visible-area thresholds, with cars visible before and after.Source: §B.2 · p. 16

B.3 BASELINES

  • RAM's published LA-CATER results include post-processing, so the comparison reports its raw output; SAME checks whether the target's track identity matches across reappearance.The ground-truth box selects which RAM track is read but does not enter a prediction.Source: §B.3 · p. 16
  • Loci-Looped receives a background image built from ground-truth instance masks, while its moving-camera input is rebuilt by inpainting visible-object boxes.Hidden-object boxes are left untouched so no hidden position enters the input.Source: §B.3 · p. 16
  • iSEE and SlotContrast are evaluated at step 100k, while RandSF.Q is evaluated at convergence and Loci-Looped uses different checkpoints by camera setting.For moving-camera Loci-Looped, the checkpoint is 170k, before training collapsed at 171.5k without recovering.Source: §B.3 · p. 17

B.4 PERMANENCE METRIC

The permanence metric scores predicted object boxes over hidden frames at IoU thresholds from 0.10 to 0.30. SAME separately tests whether an object returns to its original slot after an occlusion.

B.4 PERMANENCE METRIC

  • Permanence mAP averages IoU thresholds 0.10 to 0.30 over hidden frames, using boxes centred on predicted positions and sized from the followed slot’s pre-occlusion mask.The grid resolution limits the metric above IoU 0.30; the pre-occlusion boxes overlap ground truth above IoU 0.1 on 99.5% of static and 96.9% of moving occlusions.Source: §B.4 · p. 17, Figure S1 · p. 17

B.5 SAME-SLOT RETURN (SAME)

  • SAME asks whether the slot owning an object before an occlusion also owns it afterward, comparing only the last and first frames when it is at least half visible.It requires no ground-truth identities or labels while the object is hidden.Source: §B.5 · pp. 17–18
  • SAME reads ownership from the slot with the greatest attention mass on the object’s visible pixels, without an overlap threshold.LA-CATER scores 183 of 195 static and 360 of 388 moving occlusions; KITTI uses car boxes because it lacks amodal masks.Source: §B.5 · pp. 17–18

C MORE EXPERIMENTS

Across longer LA-CATER occlusions, iSEE and RAM retain permanence while frozen-position and Loci-Looped scores fall. Whole-clip identity results vary by camera setting and object set.

C.1 PERMANENCE AND THE LENGTH OF THE OCCLUSION

  • iSEE’s paired lead over frozen grows from +13.3 to +28.3 points across shortest-to-longest occlusions; iSEE and RAM lose less than 3 points, while frozen loses a quarter and Loci-Looped almost half.Table S3 reports mAP@[0.1, 0.3] over hidden frames, split into four occlusion-length bins.Source: §C.1 · p. 18, Table S3 · p. 19

C.2 IDENTITY OVER WHOLE CLIPS

  • With static cameras, iSEE and SlotContrast + TEN are level on occluded objects, while iSEE leads in AssA over all objects; with moving cameras, iSEE leads in AssA on both sets.TEN raises association accuracy on both encoders but costs one to three points of detection accuracy.Source: §C.2 · p. 19, Table S4 · p. 19
  • Without TEN, iSEE is below SlotContrast on static-camera clips and level with it on moving-camera clips.Table S4 reports whole-clip HOTA, DetA, and AssA.Source: §C.2 · p. 19, Table S4 · p. 19

C.3 SEPARATION OF APPEARANCE AND POSITION

These experiments examine how iSEE separates appearance from position, how design choices affect permanence, and how its slots perform in dynamics, discovery, and occlusion examples.

C.3 SEPARATION OF APPEARANCE AND POSITION

  • 47 points of headroom accrue to the dynamics model using iSEE’s position-delta action, versus five for SlotContrast using its whole-slot delta.The position stream is four-dimensional, while SlotContrast carries position in its 64-dimensional slot; the probe measures how readily position can be used, not whether it is encoded.Source: §C.3 · pp. 19–20, Table S5 · p. 20

C.4 CONTRASTIVE LOSS ON THE WHOLE SLOT

  • Applying contrastive loss to the whole slot degrades position-stream encoding and permanence mAP, while other scores stay similar.The loss asks the position code to differ across sequences and batches.Source: §C.4 · p. 20, Table S6 · p. 20

C.5 APPEARANCE INPUT OF THE POSITION PREDICTOR

  • Without sg(a), the position predictor loses permanence mAP, consistent with object motion depending on shape.Table S7 evaluates permanence mAP@[0.1, 0.3] over all 195 static-camera LA-CATER occlusions, with TEN and without the walker.Source: §C.5 · p. 20, Table S7 · p. 21

D DYNAMICS MODEL

  • The action-conditioned dynamics model predicts the next frame’s slots from a 14-frame window of frozen slots, using each slot’s one-step position change as its action.The action reaches the predictor through the position stream.Source: D DYNAMICS MODEL · pp. 20–21, Figure S2 · p. 21

E.1 OBJECT DISCOVERY

E.2 RE-IDENTIFICATION ON LA-CATER

  • Figure S5 follows one LA-CATER target through an occlusion; only iSEE has the target back in its original slot when it reappears.The target is hidden behind another object.Source: Figure S5 · p. 24, §E.2 · p. 24

E.3 OBJECT PERMANENCE ON LA-CATER

  • Figures S7 and S8 show best-case LA-CATER occlusions for iSEE over other label-free rows, on static- and moving-camera clips.Figure S9 shows two moving-camera cases where TEN releases the slot while the target remains hidden.Source: §E.3 · p. 25, Figure S7 · p. 25, Figure S8 · p. 26

E.4.1 RE-IDENTIFICATION

  • Figure S10 illustrates two ways iSEE loses a car through a KITTI occlusion.The examples are from a single tile, with columns showing unevenly spaced frames.Source: §E.4.1 · p. 27, Figure S10 · p. 28

E.4.2 OBJECT PERMANENCE

The examples catalogue five modes in which iSEE’s predicted position fails on LA-CATER occlusions.

E.4.2 OBJECT PERMANENCE

  • The five failure modes are wrong-way motion, phantom motion, a box too small for IoU above 0.1, the wrong slot at occlusion onset, and a frozen position despite target motion.Figure S11 uses one occlusion per mode.Source: §E.4.2 · p. 29, Figure S11 · p. 29
  • Each Figure S11 example is the median occlusion for its failure mode, not the worst case.The examples therefore illustrate representative modes rather than their maximum severity.Source: §E.4.2 · p. 29

Related papers

Loading 2610.01201v1…