SWT: Self-Supervised Video Object Segmentation via Sliding, Wavelet and Transportation

• Sample scarcity and limited receptive fields hinder correspondence learning and global spatio-temporal consistency, complicating segmentation of rapidly moving or deforming objects.Source: §1 · p. 1
• SWT reuses overlapping groups of independently sampled images across successive updates, while cascaded wavelet decomposition and frequency-specific small-kernel convolutions enlarge the effective receptive field.Source: §1 · pp. 1–2
• Optimal transport with differentiable Sinkhorn iterations computes globally optimal matching plans to support smooth, consistent correspondence under spatial distortions or occlusions.Source: §1 · p. 2
• The components jointly provide VOS correspondence supervision through a static-image pipeline, without requiring video annotations during training.Source: §1 · p. 2
• 1.6% on DAVIS16-val and 1.3% on DAVIS17-val: SWT improves J&F over the baseline while training solely on static images.Source: §1 · p. 2

• Self-supervised VOS learns without training annotations, while contrastive proxy tasks pull same-class embeddings together and push different-class embeddings apart.Source: §2 · pp. 2–3
• Video correspondence methods use color-frame reconstruction, temporal cycle consistency, frame-level similarity, or pixel-level consistency across image views [27], [26], [54], [23], [59], [58].Source: §2 · p. 3
• In VOS, temporal consistency concerns persistent object identities and accurate mask propagation; cited video-generation methods also address temporal consistency [68], [72], [46], [66].Source: §2 · p. 3
• Wavelet-based methods span signal classification, image processing, and neural representations; WTConv uses cascaded wavelets to enlarge the effective receptive field with minimal additional trainable parameters,, [14], [45], [13].Source: §2 · p. 3
• Optimal transport matches probability distributions with minimal cost; Sinkhorn computes approximate entropy-regularized transport couplings, and prior work applies transport to multiple matching tasks, [47], [40], [4], [5], [70].Source: §2 · p. 3

3.1 Method Overview

• SWT learns correspondences from two differently cropped views of static images, using spatial and pseudo-temporal matching during annotation-free training.At inference, it uses the first-frame mask and memory-based propagation across query frames.Source: §3 · p. 3, §3.1 · p. 5, §3.5 · pp. 9–10

3.2 Sliding-Window Sample Buffer

• The rolling buffer reuses overlapping groups of independently sampled mini-batches across successive updates.With buffer length 4 and mini-batches of 16, each update uses 64 samples: 48 retained and 16 new; the buffer is cleared after each epoch.Source: §3.2 · pp. 5–6

3.3 Wavelet Transform

• Wavelet convolution decomposes features into low- and high-frequency components, applies small-kernel depth-wise convolutions, then reconstructs the result.The second-level transform processes only the low-frequency channel; the operation lets smaller kernels cover larger receptive fields of the original input.Source: §3.3 · p. 6, Figure 3 · p. 7

3.4 Optimal Transport

• Optimal transport finds a minimum-cost matching plan between feature distributions under marginal constraints, encouraging globally balanced assignments between cropped views.SWT uses the plan as a training-time spatial matching loss, not as an additional inference-stage matching module.Source: §3.4 · pp. 7–9

3.5 Details

• SWT combines spatial and temporal matching losses, weighting both equally by default, to form its hybrid training loss.Temporal matching comes from pseudo-dynamic representations produced by applying the projector to concatenated features in different orders.Source: §3.5 · p. 9
• At inference, masked attention retrieves the top-K similar memory frames, whose masks are weighted and fused to predict each query-frame mask.The predicted mask is added to memory for subsequent propagation.Source: §3.5 · pp. 9–10

4.1 Results for Video Object Segmentation

• 73.3% J&F on DAVIS17-val, surpassing all compared methods; SWT improves J&F by 2.9% over recent methods[17].On the more challenging DAVIS17-test set, SWT also maintains leading performance.Source: Table 2 · p. 11, §4.1 · pp. 11–12, Table 3 · p. 11
• On DAVIS16-val, SWT improves J&F by 1.7% over SPGO[38] and outperforms some semi-supervised methods.The compared semi-supervised methods include OSVOS[1] and SiamMask[53].Source: Table 4 · p. 12, §4.1 · p. 12
• 2.7% improvement in mean score over LIIR[29] on YouTubeVOS18-val, with advanced performance also reported on YouTubeVOS19-val.The authors report high scores on both seen and unseen categories.Source: §4.1 · p. 12, Table 5 · p. 13, Table 6 · p. 13

4.2 Results for Body Part Propagation

• On VIP body-part propagation, SWT exceeds LIIR[29] and SCC[48] by 1.9% and 2.3% mIoU, respectively, and ATEN[73] by 5.2%.VIP evaluates propagation of 19 body parts across 50 validation videos.Source: §4.2 · p. 12, Table 7 · p. 14

4.3 Ablation Studies

• Haar, Euclidean transport cost, a (4, 16) buffer setting, refresh size N=16, and α=1 are the best tested sensitivity settings.At α=1, results are 73.3% and 80.5% on DAVIS17-val and DAVIS16-val; intermediate α settings improve on α=0, but the trend is not monotonic.Source: §4.3 · pp. 13–14, Table 10 · p. 15, §4.4 · p. 19
• SWT raises parameters from 0.753M to 1.160M, FLOPs from 1.865G to 2.064G, and memory from 16.0GB to 21.4GB, while inference falls from 48 FPS to 43 FPS.The OT module is a training-time matching loss and adds no inference parameters; qualitative analyses also report an enlarged effective receptive field and cross-image correspondences.Source: Table 11 · p. 15, §4.3 · p. 18, §4.4 · pp. 18–19
• The full S+W+T model reaches 73.3% and 80.5% on DAVIS17-val and DAVIS16-val; OT alone provides the largest single-module gain.The buffer improves J&F by 0.3% and 0.4%, while wavelets improve it by 0.6% and 0.7%, and OT by 1.1% and 1.3%, on the same datasets, respectively.Source: §4.3 · pp. 14–16, Table 8 · p. 14
• Detection-guided crops underperform random cropping, while cross-correlation initialization slightly outperforms uniform initialization.The authors attribute the crop result to loss of background context and unrealistic motion patterns; OT and wavelet-domain modeling are retained over simpler alternatives.Source: Table 9 · p. 15, §4.3 · pp. 16–18

5 CONCLUSION

• SWT requires training on static images rather than extensive pixel-level annotated video data.The conclusion reports advanced results on multiple VOS benchmarks and competitive performance against some fully supervised VOS approaches.Source: §5 · p. 20
• A rolling sample buffer reuses samples, cascaded wavelet transforms expand the effective receptive field, and optimal transport provides cross-frame global feature matching.The wavelet transform adds moderate computational cost; the matching component addresses rapid object motion and deformation.Source: §5 · p. 20
• The approach achieves advanced results on multiple VOS benchmarks and competitive performance against some fully supervised VOS approaches.Source: §5 · p. 20