PaperlayerView exact revision on arXiv

Source-linked AI summary

OccJEPA: Occlusion as Masking for Amodal Video Segmentation

Genpei Zhang

arXiv:2610.29061v1cs.CVcs.LG

TL;DR

Amodal video segmentation must recover complete object masks over time, including parts hidden by other objects, while leading diffusion methods require a depth network and many denoising steps. OccJEPA drops tokens not covered by the visible mask, predicts their features with the pre-trained V-JEPA 2.1 predictor, and decodes observed and predicted features into amodal masks. OccJEPA is more accurate than Diffusion-VAS on SAIL-VOS, TAO-Amodal, and Kubric, and is 33.5× faster.

  • Problem

    Amodal video segmentation must recover complete object masks over time, including parts hidden by other objects, while leading diffusion methods require a depth network and many denoising steps.

    Source: §1 · p. 1

  • Method

    OccJEPA drops tokens not covered by the visible mask, predicts their features with the pre-trained V-JEPA 2.1 predictor, and decodes observed and predicted features into amodal masks.

    Source: §1 · p. 2

  • Results

    OccJEPA is more accurate than Diffusion-VAS on SAIL-VOS, TAO-Amodal, and Kubric, and is 33.5× faster.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Controlled experiments and feature visualisations show the completion gain comes from pre-trained weights of the completion module for world models of every size and for VideoMAE.

    Source: §1 · p. 2

Abstract

from arXiv · show

Amodal video segmentation asks for the complete mask of an object in every frame, including the parts hidden behind other objects. The best current methods generate these masks with fine-tuned video diffusion models, which need many denoising steps and an extra depth network, and give a different answer for every sample. Self-supervised video world models of the JEPA family are trained on a closely related task: predicting the representation of masked space-time regions from the visible ones. We therefore treat occlusion as masking. The tokens that the object's visible mask does not cover are dropped, the pre-trained V-JEPA 2.1 predictor fills them in, and a light head decodes the observed and predicted features into amodal masks. The resulting model, OccJEPA (Occlusion-as-masking JEPA), is deterministic and 33.5x faster than the diffusion-based state of the art, and more accurate on SAIL-VOS, on TAO-Amodal, including a held-out test split (+4.00 AP50) and visible masks tracked by SAM2, and on Kubric. Controlled experiments trace the gain to the predictor's pre-trained weights rather than to its capacity, for world models from 87M to 1.85B parameters. Code is available at https://github.com/zgp110/occjepa.

1. Introduction

Amodal video segmentation must recover objects’ visible and hidden parts over time, while leading diffusion methods require depth and many denoising steps. OccJEPA treats occlusion as masking and uses a pre-trained video-world-model predictor to complete masks, improving benchmark accuracy with deterministic, faster inference.

Motivation

  • Amodal video segmentation predicts each object’s complete mask, including hidden parts, from visible masks supplied in every frame.Trackers, robots, and video editors need this ability.Source: §1 · p. 1
  • Diffusion-VAS fine-tunes video diffusion to generate amodal mask videos from visible masks and estimated depth, requiring a depth network and many denoising steps.Different samples can produce different answers.Source: §1 · p. 1
  • JEPA predictors infer representations of masked space-time regions from visible regions, a pre-training task closely related to completing occluded objects.The predictor is normally discarded after pre-training.Source: §1 · p. 1

Approach

  • OccJEPA drops tokens outside the object’s visible mask, uses the pre-trained V-JEPA 2.1 predictor to fill their features, and decodes observed and predicted features into amodal masks.It uses a second parameter-free re-crop pass and needs no occluder masks, depth, or sampling.Source: §1 · p. 2

Results

  • 33.5× faster than Diffusion-VAS, OccJEPA improves results on SAIL-VOS, TAO-Amodal, and Kubric, including a +4.00 AP50 gain on a held-out TAO-Amodal test split.The lead also holds when visible masks are tracked by SAM2.Source: §1 · p. 2

Contributions

  • Controlled experiments attribute the gain to pre-trained completion-module weights across world-model sizes and to VideoMAE’s pre-trained pixel decoder.The latent V-JEPA predictor gives the strongest model.Source: §1 · p. 2

2. Related Work

Prior amodal segmentation work spans annotated and synthetic training, video cues, and priors from large pre-trained models. OccJEPA differs by repurposing a predictive world-model component for amodal segmentation.

Amodal segmentation

  • Amodal video methods include learning from motion, object-centric bird’s-eye-view representations, amodal-aware video instance segmentation, and trackers that retain objects through occlusion.SAIL-VOS and TAO-Amodal provide annotations for synthetic and real clips.Source: §2 · p. 2
  • Diffusion-VAS generates amodal masks from visible masks and pseudo-depth, while TACO completes hidden RGB content with a video diffusion model.Visible masks may be supplied by a video object segmenter.Source: §2 · p. 2

Self-supervised video models and world models

  • JEPAs predict masked-region representations in latent space; V-JEPA applies this to video, and V-JEPA 2.1 adds dense objectives.Prior downstream uses of world models usually retain only the encoder; OccJEPA uses the released video predictor for amodal segmentation.Source: §2 · p. 2

Repurposing pre-trained models

  • Unlike prior work repurposing generative models, OccJEPA repurposes a predictive model, using a light head and low-rank adapters for deterministic inference.The predictor is used for the operation it was trained on.Source: §2 · p. 2

3. Method

OccJEPA uses visible object masks to define which video tokens are context and which are missing, then completes latent features and decodes them into amodal masks. Its method combines a pre-trained V-JEPA predictor, a light mask head, and a one-time re-crop for objects extending beyond the initial crop.

3.1. Problem setup

  • Amodal segmentation predicts each target’s full mask in every frame from supplied visible masks, including parts hidden by other objects.In practice, visible masks can come from a video object segmenter.Source: §3.1 · p. 2

3.2. Background: V-JEPA

  • V-JEPA encodes kept video tubelets and predicts masked-region features from them; OccJEPA uses the released V-JEPA 2.1 model and predictor.The model uses a 48-block, 1.85B-parameter encoder and a 24-block predictor.Source: §3.2 · pp. 2–3

3.3. Occlusion as masking

  • The clip is cropped around the visible mask, and tubelets with visible-mask coverage below 0.25 are dropped; the dropped set includes hidden object parts, occluders, and background.The model is left to infer where the object continues without an occluder mask.Source: §3.3 · p. 3
  • The predictor fills dropped token positions from visible-object context, pairing predicted features with full-clip features so the head can distinguish the occluder from likely object continuation.At occluded positions, observed features describe the occluder while predicted features can describe the hidden object; the paper’s explanation links completion to the pre-trained predictor’s learned context.Source: §3.3 · pp. 3–4
  • A mask head decodes concatenated observed and predicted features into amodal logits, and overlapping windows cover clips longer than 24 frames.The model is trained with a 50/50 mixture of SAIL-VOS and MOVi-D clips; the added synthetic data does not raise accuracy on SAIL-VOS or TAO-Amodal.Source: §3.3 · p. 4

3.4. Test-time re-crop

  • A second pass recomputes crops from the first prediction combined with the visible mask, allowing objects whose amodal extent leaves the initial crop to be predicted on a larger crop.This parameter-free re-crop is used once and doubles inference cost.Source: §3.4 · p. 4

4. Experiments

OccJEPA is evaluated against diffusion-based methods on real and synthetic benchmarks, using shared protocols and controlled comparisons. The experiments report its accuracy, inference cost, and the roles of pre-trained completion, re-cropping, and occlusion level.

4.1. Setup

  • On SAIL-VOS and TAO-Amodal, the evaluation follows Diffusion-VAS protocols and uses ground-truth visible masks; TAO-Amodal test sequences were held out from model choices.SAIL-VOS reports official mIoU and occluded-region mIoU, plus fixed-frame occluded IoU; TAO-Amodal reports box AP at multiple thresholds.Source: §4.1 · p. 4

4.2. Comparison with the state of the art

  • +3.70 [ +1.38, +5.40] mIoUfixed occ over Diffusion-VAS on the same SAIL-VOS frames, alongside +4.11 AP50 on TAO-Amodal val and +4.00 on its held-out test split.OccJEPA also leads with SAM2-tracked visible masks and on Kubric under a common scorer.Source: §4.2 · pp. 4–5, Table 2 · p. 5

4.3. Cost

  • 33.5× faster than Diffusion-VAS, OccJEPA takes 0.35 s instead of 11.77 s per object and uses 25× fewer FLOPs per clip.A distilled ViT-L world model uses 3.6 TFLOPs and still exceeds Diffusion-VAS on TAO-Amodal AP50 by +1.83.Source: §4.3 · p. 5

4.4. Where does the gain come from?

  • The pre-trained predictor accounts for 75% of the SAIL-VOS gain and 81% of the TAO-Amodal gain; the same architecture with random initialization recovers much less.Without the predictor, OccJEPA falls below Diffusion-VAS on SAIL-VOS and is level with it on TAO-Amodal.Source: §4.4 · pp. 5–6, Table 4 · p. 8
  • Re-cropping adds +0.36 mIoUfixed occ on SAIL-VOS and +1.31 AP50 on TAO-Amodal val, with +11.33 AP50 on test frames where visibility is below 0.5.The second pass matters most when the object is mostly hidden and the initial crop is too small.Source: §4.4 · p. 6
  • OccJEPA leads on SAIL-VOS from 20% occlusion onward, by 6.2 to 9.2 points; below 10%, pixel-level generation better fits hidden slivers finer than the token grid.On TAO-Amodal, its AP50 is 77.7 vs. 43.4 at visibility 0.25–0.5, and 41.5 vs. 24.5 below 0.25 visibility.Source: §4.4 · pp. 6–7

5. Conclusion

OccJEPA treats occlusion as masked-region prediction, using a pre-trained video-world-model predictor for amodal completion. It improves on diffusion-based methods across the reported benchmarks, with deterministic inference and lower cost.

5. Conclusion

  • OccJEPA outperforms the diffusion-based state of the art on SAIL-VOS, TAO-Amodal, and Kubric while running 33.5× faster.Controlled experiments attribute the gain to the pre-trained predictor rather than extra capacity.Source: §5 · p. 8, Table 5 · p. 8
  • The predictor's pre-training task—inferring masked-region representations—closely matches the task of completing occluded objects.The conclusion also reports a lead with visible masks tracked by SAM2, without a depth network or sampling.Source: §5 · p. 8
  • With a video object segmenter supplying visible masks, OccJEPA forms a complete amodal video segmenter without depth estimation or sampling.The conclusion frames world-model predictors as useful for perception as well as pre-training.Source: §5 · p. 8

Supplementary Material

The supplement collects implementation details, evaluation protocols, full results, analyses, temporal behavior, and additional qualitative examples.

Supplementary Material

  • The supplement documents implementation details and evaluation protocols, including the TAO-Amodal test split and SAM2-tracked visible masks.It also describes how Diffusion-VAS is run and reports statistics.Source: Supplementary Material · p. 11
  • It provides full results with confidence intervals and analyses behind Figure 5, including temporal behavior.Source: Supplementary Material · p. 11
  • Further qualitative results include random draws and failure cases.Source: Supplementary Material · p. 11

A. Implementation details

The implementation uses per-frame crops and overlapping temporal windows, then predicts features for masked tokens and decodes them into masks. Training adapts the encoder with LoRA and combines SAIL-VOS and MOVi-D data.

A. Implementation details

  • Crops are square regions sized from visible-mask boxes, resampled to 256×256; overlapping 25-frame windows use the later window's outputs on shared frames.For frames without visible pixels, crop centre and side are interpolated from other frames.Source: A. Implementation details · p. 11
  • Tokens are kept when visible-mask coverage is at least 0.25, and the head combines four-level target features with predicted fills to decode masks.If all tokens would be dropped, the most-covered token is retained; the decoder upsamples a 16×16 grid and fuses the visible mask.Source: A. Implementation details · p. 11
  • Training uses LoRA on the last 12 encoder blocks, 6,000 optimization steps, and batches drawn equally from SAIL-VOS and MOVi-D.SAIL-VOS training uses 15,688 sequences; the world model was selected using held-out training videos, not benchmark results.Source: A. Implementation details · p. 11

B. Evaluation protocols

The protocols define SAIL-VOS and TAO-Amodal scoring, establish the held-out test split and tracked-mask setting, and specify checks on baseline reproduction and statistical comparisons.

SAIL-VOS

  • SAIL-VOS reports official mIoUocc on frames where a method predicts hidden area, so mIoUfixed_occ additionally scores all 83,245 occluded frames and counts misses as zero.Diffusion-VAS predicts no hidden part on 27.5% of occluded frames.Source: B. Evaluation protocols · pp. 11–12

TAO-Amodal and the test split

  • TAO-Amodal AP25, AP50, and AP75 measure the fraction of frames whose predicted amodal box exceeds the respective IoU threshold; the reconstructed test rule yields 1,947 sequences from 772 videos.The split and comparison were fixed before evaluating any method on the test annotations.Source: B. Evaluation protocols · p. 12

Running Diffusion-VAS

  • The Diffusion-VAS port uses its released checkpoint and settings, seeds sequences separately, and reproduces released validation boxes within the pre-set acceptance criterion.On 40 validation sequences, AP50 is 79.5 versus 82.1 and median box IoU is 0.971.Source: B. Evaluation protocols · p. 12

Predicted visible masks

  • All methods receive the same annotated visible masks; SAM2.1 tracking instead uses the first-frame annotation as a prompt and supplies tracked masks on subsequent frames.On frames 2–25, SAM2 masks have mean IoU 0.826 and median IoU 0.919 with annotations.Source: B. Evaluation protocols · p. 12

MOVi and Kubric

  • MOVi uses EoRaS test lists and specified frame-counting and crop settings; Kubric uses TACO test sets at 256×256 scored by a reimplementation applied to every computed row.Statistical comparisons use 95% paired bootstrap intervals over videos; design choices use validation data, not the TAO-Amodal test split, and results are from single training runs.Source: B. Evaluation protocols · p. 12

C. Additional results

Across the additional benchmarks and evaluation settings, OccJEPA outperforms comparison methods, with gains tied to pretrained completion features and retained across held-out and tracked-mask evaluations. Ablations and feature visualizations examine that contribution and the model’s design choices.

SAIL-VOS

  • On SAIL-VOS, OccJEPA leads on every metric scored on the same frames, including +1.52 mIoU and +3.70 mIoUfixed_occ over Diffusion-VAS.Its un-eroded occluded IoU advantage is +7.89; the fixed-frame margin remains positive under sequence, mission, and video-prefix resampling.Source: C. Additional results · p. 12, Table S1 · p. 13

TAO-Amodal

  • On TAO-Amodal, OccJEPA leads across val and test frame types; when less than half the object is visible, AP50 rises by +29.44 on test, and its lead with SAM2 masks is +4.32 AP50.The SAM2 comparison uses the same tracked masks as input for every method.Source: C. Additional results · p. 13, Table S2 · p. 13, Table S3 · p. 14

Best of three samples

  • OccJEPA returns one answer without sampling or selection, and on TAO-Amodal it exceeds the ground-truth-selected best of three diffusion samples in AP25 and AP50.The diffusion selection uses ground truth for each frame.Source: C. Additional results · p. 13, Table S4 · p. 14

Completion module

  • The pretrained completion module adds +9.83 mIoUocc on SAIL-VOS, while the same architecture trained from scratch recovers only +2.15 and +0.74 on TAO-Amodal.The reported pretrained-weight shares of the gain are 75% and 81%; the decomposition is similar in one pass and with re-crop.Source: C. Additional results · pp. 13–14, Table S5 · p. 15

Predictor ablation and feature maps

  • Across three example sequences, the pretrained predictor has the highest sequence-level occluded IoU in two; filled features continue hidden object regions in the bottle and thin-object examples.The third example varies: the model without a predictor covers more of the hidden lower region.Source: C. Additional results · p. 14

World models and completion modules

  • The predictor improves mIoUfixed_occ by +8.19 to +8.67 across world-model sizes; its TAO-Amodal gain grows from +1.95 AP50 (ViT-B) to +3.24 (ViT-G).At matched size, latent prediction gives the better model than VideoMAE on both benchmarks; pretrained weights matter for both model families.Source: C. Additional results · p. 14, Table S6 · p. 15

MOVi and Kubric

  • On MOVi, OccJEPA is best in all five columns under each of four counting settings; on Kubric, it is best in all four occluded-region columns.On Kubric-Dynamic, a single pass scores 51.78 versus 44.33 for Diffusion-VAS.Source: C. Additional results · pp. 14–15, Table S7 · p. 15

Design components

  • With the predictor, re-crop adds +1.24 AP50 on TAO-Amodal; the high-resolution refiner adds +0.37 on SAIL-VOS with the predictor and +1.92 without it.Mixing MOVi-D into training leaves SAIL-VOS essentially unchanged (+0.04) and TAO-Amodal within half a point (−0.36 AP50), while one model serves all benchmarks.Source: C. Additional results · p. 15

D. Additional analyses

The additional analyses quantify OccJEPA’s inference cost, performance across occlusion levels, and temporal behavior. They also identify thinner-than-token-grid hidden regions and frame-to-frame smoothness as relevant trade-offs.

Cost

  • With re-crop, OccJEPA requires 35.6 TFLOPs and 0.35 seconds per object and clip, versus 903 TFLOPs and 11.77 seconds for Diffusion-VAS.OccJEPA uses 25× fewer FLOPs; Diffusion-VAS spends 3.2 seconds on depth estimation alone and 8.7 seconds on denoising.Source: D. Additional analyses · pp. 15–16

Occlusion level

  • On SAIL-VOS, OccJEPA leads by 6.2 to 9.2 points from 20% occlusion onward, but below 10% occlusion pixel-level generation scores 43.3 versus 30.9.The 20%-and-higher bins contain 72% of the fixed frame set; very thin hidden slivers can be finer than the token grid.Source: D. Additional analyses · p. 16

Temporal behaviour

  • Diffusion-VAS masks change less between frames than OccJEPA’s (3.96 versus 7.15), while OccJEPA is trained for per-frame accuracy and recomputes its crop per frame.The authors note that temporally smoothed crops or a temporal loss term could be added directly.Source: D. Additional analyses · pp. 16–17

E. Qualitative results

Qualitative examples show OccJEPA completing heavily occluded objects where Diffusion-VAS often predicts little or no hidden region, alongside failures on tiny occlusions. Examples across SAIL-VOS, TAO-Amodal, and Kubric illustrate both the gains and their limits.

Largest and median gain

  • In the largest-gain SAIL-VOS example, OccJEPA fills the hidden band between a traffic cone’s visible tip and base, scoring 0.96 occluded IoU versus 0.00 for Diffusion-VAS.In the median-gain example, both methods complete a person’s legs, but OccJEPA follows the ground truth more closely.Source: E. Qualitative results · pp. 17–18, Figure S3 · p. 18

Random draws

  • At 70% to 97% mean occlusion, OccJEPA’s sequence occluded IoU is 0.29 to 0.55, versus at most 0.04 for Diffusion-VAS; Diffusion-VAS predicts no hidden part on 27.5% of occluded frames.The examples are three random SAIL-VOS sequences shown at the first, middle, and last selected frames.Source: E. Qualitative results · p. 18, Figure S4 · p. 19

Failure cases

  • At 1.8% and 9.1% hidden area, OccJEPA predicts no hidden part and scores zero occluded IoU, while Diffusion-VAS scores 1.00 and 0.91.These two largest losses occur in the lowest occlusion bin; the per-frame metric weights each frame equally, even when little of the object is hidden.Source: E. Qualitative results · p. 18, Table S11 · p. 18, Figure S5 · p. 19

TAO-Amodal

  • +4.54 AP50 is OccJEPA’s lead over Diffusion-VAS across frames where the object leaves the image; in the shown in-frame occlusions, its boxes follow hidden extents more closely in every frame.Both methods extend boxes beyond the image border when objects leave the frame, and their per-frame IoUs in the two examples are mostly close.Source: E. Qualitative results · pp. 18–19, Figure S6 · p. 20

Kubric

  • +7.71 is OccJEPA’s margin on Kubric-Dynamic, where its completion is closer to ground truth in most frames, including when hidden parts change shape; on Kubric-Static, the margin is +0.85 and per-frame differences go both ways.The figure shows two random instances per benchmark at the first, middle, and last selected frames.Source: E. Qualitative results · p. 19, Figure S7 · p. 20

Related papers

Loading 2610.29061v1…