PaperlayerView exact revision on arXiv

Source-linked AI summary

Object Concepts Emerge from Motion

Boshi Li, Xiaohui Wang, Xiaoyang Wu, Zhichao Li, Ya Yang, Naiyan Wang

arXiv:2609.04348v1cs.CV

TL;DR

Existing visual pretraining often captures semantic categories without preserving individual-instance identity and coherence. This paper learns object-centric representations from raw video using motion-derived pseudo-instance supervision, achieving competitive transfer across four physical-world tasks and especially strong transfer on geometry- and instance-sensitive tasks.

  • Problem

    Existing visual pretraining methods often capture semantic categories without preserving the identity and coherence of individual instances, limiting object-centric representation learning for physical-world perception.

  • Method

    The framework converts optical flow and clustering into category-agnostic pseudo-instance supervision, trains a single-image encoder with dense pairwise metric learning, and expands supervision through Motion-Verified Self-Training.

  • Results

    Across monocular depth estimation, 3D object detection, 3D occupancy prediction, and end-to-end planning, the learned representations transfer competitively with supervised and self-supervised alternatives, especially on geometry- and instance-sensitive tasks.

  • Takeaways & Limitations

    Motion provides a distinct supervision signal that encourages object unity and instance separation without specifying semantic categories, complementing category-oriented and temporal objectives.

  • Takeaways & Limitations

    The framework depends on optical-flow quality and cannot recover object boundaries when neither the flow field nor the Cycle-1 encoder provides sufficient evidence.

Abstract

from arXiv · show

Object-centric visual representations are important for physical-world perception, but existing visual pretraining methods often capture semantic categories without preserving the identity and coherence of individual instances. We present a biologically inspired framework that learns object-centric representations for single images from raw videos. Our approach uses motion boundaries as a source of object-level grouping: off-the-shelf optical flow and clustering produce pseudo-instance masks, which supervise a single-image encoder with pixel-level pairwise metric learning. The framework requires neither human annotations nor camera calibration. We first obtain 195 million pseudo-labeled frames from 7,163 hours of driving and web videos, then expand the supervision to 421 million frames with Motion-Verified Self-Training, which combines model proposals with motion evidence. We train encoders up to Swin-H and distill the learned representations into a family of Swin backbones. Across monocular depth estimation, 3D object detection, 3D occupancy prediction, and end-to-end planning, the resulting models achieve competitive or superior performance relative to supervised and self-supervised pretraining baselines, with particularly strong transfer on geometry- and instance-sensitive tasks. These results show that motion-derived supervision can teach static image encoders to represent visual instances, providing a complementary direction for scalable visual pretraining.

1 INTRODUCTION

The paper develops a biologically inspired framework that uses motion-derived grouping to learn object-centric representations for static images without human annotations or camera calibration. It scales this supervision and model capacity substantially, then demonstrates competitive or superior transfer across four downstream tasks.

  • Motivation: Object perception is presented as a learned capacity in which common or coherent motion provides a powerful signal for inferring object boundaries and unity.The motivation connects developmental neuroscience findings with the design of the visual pretraining framework.
  • Framework: Motion boundaries are converted into pseudo-instance masks through optical-flow estimation and clustering, then used to supervise a single-image encoder with contrastive learning.The framework targets object-level semantics while avoiding human annotations and camera calibration on the video corpus.
  • Scaling: 195M frames from 7,163 hours of heterogeneous driving and web videos receive motion-derived pseudo-labels through a unified processing pipeline.This expands the preliminary framework beyond its earlier 48M-frame driving-video version.
  • Scaling: 421M pseudo-labeled frames are obtained with Motion-Verified Self-Training, which combines initial model proposals with independent motion evidence.A controlled Swin-T experiment isolates the added supervision from simply continuing pretraining on the original labels.
  • Evaluation: Swin-H training and distillation into a family of Swin backbones support evaluation on monocular depth, 3D detection, 3D occupancy, and end-to-end planning.The resulting models achieve competitive or superior performance relative to supervised and self-supervised baselines.

2 RELATED WORK

Related work spans object discovery, motion-based grouping, video self-supervision, visual foundation models, and self-training. The paper differs by using motion to create intermediate category-agnostic supervision for a static-image encoder, with motion also verifying self-training proposals.

  • Object discovery: Object discovery methods localize objects without instance-level annotations using repeated occurrences, proposal optimization, or pretrained-image features with graph or spectral clustering.These approaches generally treat object localization or segmentation as the target output.
  • Motion-based supervision: Motion-based discovery separates objects from surroundings using motion consistency, whereas this paper uses optical flow and clustering to construct pseudo-instance masks for representation learning.The masks are supervision for a static-image encoder rather than the final deployed output.
  • Motion-based supervision: Motion-derived grouping has previously transferred to static recognition, while this framework evaluates the learned features through downstream applications rather than predicted masks directly.Related methods include ego-motion supervision, unsupervised motion segmentation, and dynamic-static bootstrapping.
  • Video pretraining: VideoMAE and V-JEPA learn spatio-temporal representations from masked-video reconstruction or prediction, whereas this method explicitly converts motion into object-level grouping for single-image inference.The distinction is between implicit temporal-context learning and explicit motion-derived object supervision.
  • Visual foundation models: Unlike category-oriented or promptable segmentation objectives, the method uses motion as a category-agnostic signal to preserve within-object coherence and distinguish adjacent instances.This inductive bias complements objectives dominated by category invariance or appearance reconstruction.
  • Self-training: Motion-Verified Self-Training uses model proposals for coverage but retains proposal-independent motion evidence as an external verification and refinement constraint.This distinguishes it from self-training based only on confidence filtering or teacher-student agreement.

3 METHOD

The method uses motion boundaries from raw videos to generate pseudo-instance supervision, then trains static-image encoders to capture object unity and instance separation. A two-cycle pipeline scales and verifies this supervision before distilling the resulting representations into deployed backbones.

  • Cycle-1: Image Encoder Training: Dense image features are trained with pairwise metric learning that pulls same-instance pixels together and pushes different-instance pixels apart.This objective learns object-level grouping from static images without explicitly supervising category identity.
  • Cycle-1: Pseudo-label Generation: Cycle-1 converts optical flow into category-agnostic pseudo-instance labels by clustering reliable pixels with coherent motion.The pipeline estimates flow, checks validity with forward-backward consistency, clusters 4-connected pixels by flow similarity, and filters small clusters.
  • Motion-Derived Object Supervision: Motion boundaries provide a practical, though not one-to-one, proxy for object boundaries through depth discontinuities and independent motion.The geometric relationship follows from optical flow’s dependence on depth, camera motion, and intrinsics.
  • Cycle-1: Pseudo-label Generation: Cycle-1 retains only highly reliable motion regions, leaving incomplete or noisy motion cues unused despite their potential value for object separation.Missing boundaries, local noise, or connections to nearby regions can prevent clean closed masks from forming.
  • Cycle-2: Motion-Verified Self-Training: Motion-Verified Self-Training uses Cycle-1 encoder proposals together with optical-flow evidence to refine and verify masks, expanding supervision from 195M to 421M frames.The Cycle-1 Swin-H encoder is frozen while a MaskFormer head generates candidate masks; verified Cycle-2 labels then train a Swin-H teacher for distillation.

4 EXPERIMENTS

The experiments evaluate motion-derived representations across depth estimation and qualitative feature geometry. Results show strong transfer, including improvements over DINO baselines and localized instance-level similarity.

  • Implementation Details: 7,163 hours of driving and web videos provide the pretraining corpus for Cycle-1, with training implemented using large-scale distributed optimization.The corpus combines four video sources and supports the motion-derived pretraining pipeline.
  • Representation Analysis: Reference-point similarity responses remain concentrated within coherent object regions rather than spreading across pixels with similar appearance.This qualitative analysis indicates instance-level affinity in the learned features.
  • Monocular Depth Estimation: 0.042 Abs Rel, 1.715 RMSE, and 0.988 δ1 are achieved by Swin-L, compared with 0.044, 1.787, and 0.986 for DINOv3 ViT-L.The comparison is reported on the KITTI Eigen depth-estimation split.
  • Monocular Depth Estimation: Swin-S and Swin-B outperform corresponding DINOv3 ViT-S and ViT-B models across Abs Rel, RMSE, and δ1.The models achieve the best available results at their respective scales across all three reported metrics.
  • Monocular Depth Estimation: 6.36 Abs Rel, 1.02 Sq Rel, 8.53 iRMSE, and 7.91 SILog are obtained by Swin-H on the official KITTI online benchmark.These results are reported under the official evaluation protocol.
  • Geometric Qualities: Swin-L produces more coherent surfaces and less foreground–background leakage than DINOv2 ViT-L in 1.5 m lateral novel-view depth renderings.The rendering exposes local depth inconsistencies as distorted, fragmented, or spurious surfaces.

4.4 3D Object Detection

The representation is evaluated for 3D object detection and occupancy prediction across backbone scales and input resolutions. Its strongest relative advantages occur with compact and medium backbones, while larger DINOv3 models can lead at some settings.

  • 3D Object Detection: NDS generally increases from Swin-T to Swin-H for 3D object detection at 1600 × 900 resolution, while mAP follows the same trend with a small Swin-L plateau.Swin-L remains close to Swin-H at this resolution.
  • 3D Object Detection: At matched Swin scales, the representation is competitive with or better than ImageNet-22K and SimMIM pretraining for 3D object detection.The comparison uses BEVFormer V2 on the nuScenes validation set.
  • 3D Object Detection: At 704 × 256 resolution, Swin-S clearly outperforms DINOv2 and DINOv3, while Swin-B exceeds DINOv2 and remains competitive with DINOv3.At the L scale, Swin-L outperforms DINOv2, but DINOv3 achieves the strongest performance.
  • 3D Occupancy Prediction: 40.04 RayIoU and 33.99 RayIoU1m are reached by Swin-L in 3D occupancy prediction, with nearly identical performance from Swin-H.Scaling from Swin-T to Swin-L generally improves occupancy performance, with limited additional gain at Swin-H.
  • 3D Occupancy Prediction: Swin-S and Swin-B outperform both DINOv2 and DINOv3 on RayIoU and RayIoU1m, whereas DINOv3 leads at the L scale by 0.98 RayIoU and 0.94 RayIoU1m.Relative gains are most pronounced at the S and B scales.

4.6 End-to-End Planning

The Swin-L representation transfers to end-to-end planning when used as the visual backbone in a fixed planning setup. It outperforms DINOv2-L and nearly matches DINOv3-L, while DA-ViT-L remains higher.

  • Planning Results: 88.9 EPDMS is achieved by Swin-L, exceeding DINOv2-L by 1.7 points and remaining within 0.1 points of DINOv3-L.All compared variants use the same DriveSuprim planning decoder and training configuration and differ only in the visual backbone.
  • Planning Results: 90.5 is the highest aggregate score, achieved by DA-ViT-L and leaving a 1.6-point gap to Swin-L.The passage suggests explicit depth supervision may partly explain this difference.

4.7 Effectiveness of the Complete Cycle-2 Pipeline

The complete Cycle-2 pipeline improves downstream transfer over Cycle-1 and benefits from expanded, motion-verified pseudo-labels. Controlled experiments distinguish improved label quality and coverage from simply extending Cycle-1 optimization.

  • Complete Cycle-2 Comparison: SILog decreases from 5.90 to 5.69, NDS increases from 56.80 to 56.92, and RayIoU increases from 39.71 to 40.03 under the complete Cycle-2 pipeline.The comparison uses identical downstream architectures, schedules, and evaluation protocols.
  • Cycle-2 Supervision Control: 10 epochs of Cycle-2 pretraining improve every non-saturated depth metric, whereas extending Cycle-1 from 50 to 75 epochs produces no consistent improvement.This control excludes the final Swin-H feature-distillation stage.
  • Pseudo-label Coverage: 2.4 pseudo-instances and 16.6% reference-foreground coverage are obtained in Cycle-2, compared with 1.6 and 13.8% in Cycle-1.On paired frames, the fraction containing at least one matched instance rises from 45.0% to 54.0%, while IoU changes from 83.8% to 81.6%.
  • Feature Complementarity: 6.622 SILog is achieved by fusing Ours with DINOv2, improving over 6.714 for the motion-derived representation alone.Under the same two-encoder fusion design, Ours+DINO also outperforms ImageNet+DINO.

5 DISCUSSION

The framework’s supervision remains constrained by the quality and coverage of optical-flow evidence, although Cycle-2 verification improves use of incomplete cues. Scaling model capacity also shows diminishing gains, motivating better data and jointly learned motion representations.

  • Fast motion, occlusion, blur, weak texture, non-rigid deformation, and incomplete boundaries can prevent reliable pseudo-instance generation.
  • Reliable-flow filtering introduces selection bias against small, heavily occluded, and motion-ambiguous objects, which Cycle-2 may inherit from Cycle-1 supervision.
  • The performance gain from scaling Swin-L to Swin-H seems to diminish, which the authors attribute to insufficient data amount and diversity.
  • Jointly learning low-level motion and high-level instances could improve flow in difficult regions while strengthening instance supervision.

6 CONCLUSION

The paper presents a scalable, annotation-free framework that converts motion into object-centric supervision for static-image encoders. Expanded data and model capacity yield competitive transfer across geometry- and instance-sensitive downstream tasks.

  • The framework learns object-centric representations from raw videos without human annotations on the target corpus or camera calibration.
  • 421M frames are obtained after Motion-Verified Self-Training expands the initial motion-derived supervision.
  • The representation scales to Swin-H and is distilled into a family of Swin backbones.
  • Across monocular depth estimation, 3D object detection, 3D occupancy prediction, and end-to-end planning, transfer is competitive with supervised and self-supervised alternatives.
  • Motion provides distinct supervision that encourages object unity and instance separation without specifying semantic categories.

APPENDIX A PSEUDO-CODE FOR PIXEL CLUSTERING

The pixel-clustering procedure groups valid optical-flow pixels into motion-induced clusters using breadth-first search and retains clusters meeting a minimum area threshold.

  • The algorithm takes optical flow, a forward-backward-consistency valid-flow mask, and thresholds θf and θs as inputs.
  • Breadth-first search explores 4-connected valid pixels and groups neighbors whose flow difference is at most θf.
  • The output is a set S of retained clusters after valid pixels are visited and grouped.

APPENDIX B CYCLE-2 LABEL GENERATION HYPERPARAMETERS

Cycle-2 label generation refines and verifies proposals using motion-derived boundary and interior criteria at the Cycle-2 label resolution.

  • The refinement and verification hyperparameters include the image domain Ω, boundary-support threshold τbdry, and interior-conflict threshold τint.
  • A refined proposal is accepted when its boundary-support ratio reaches τbdry and its interior-conflict ratio does not exceed τint.
  • The erosion radius is measured at the 512 × 288 Cycle-2 label resolution.

APPENDIX C ADDITIONAL FEATURE VISUALIZATIONS

The appendix visualizes dense features with three-component PCA mapped to RGB, showing representative cases and failure modes across images.

  • Three-component PCA visualizations map dense features to RGB for representative images from the final Cycle-2 Swin-H model.PCA colors are defined independently for each image and indicate only within-image feature similarity.
  • The visualizations cover diverse visual domains and present each input image alongside its corresponding PCA feature map.
  • Heavy occlusion, overlapping instances, motion blur, and weak appearance cues can fragment object features or merge neighboring instances.
Loading 2609.04348v1…