PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Pretraining for Dense Spatial Perception

Zelin Fu, Bin Tan, Changjiang Sun, Shaohui Liu, Kecheng Zheng, Yinghao Xu, Xing Zhu, Yujun Shen, Nan Xue

arXiv:2607.05247v1cs.CV

TL;DR

Modern visual foundation models remain comparatively weak at fine-grained spatial understanding, despite its importance for physical intelligence. This paper introduces masked boundary modeling and LingBot-Vision, which achieve leading dense spatial perception and depth-completion performance across tasks.

  • Problem

    Modern visual foundation models favor semantic invariance and remain comparatively weak at the fine-grained spatial understanding required for dense physical-world perception.

  • Method

    Masked boundary modeling discovers boundary-bearing tokens online and forces them into masked self-distillation targets to learn spatially structured visual representations.

  • Results

    LingBot-Vision matches or surpasses visual foundation models up to 7× larger across dense spatial tasks and delivers leading depth-completion performance after initialization transfer.

  • Takeaways & Limitations

    Boundary-oriented pretraining is presented as a practical route toward robotic spatial perception and scalable learning of spatially structured visual representations.

  • Takeaways & Limitations

    The method assumes coherent boundary segments can be recovered from teacher-generated fields without human boundary annotations or pretrained boundary targets.

Abstract

from arXiv · show

Dense spatial perception is essential for physical intelligence, where visual systems are expected to recover structured, metric, and actionable representations from pixel observations. Modern visual foundation models tend to prioritize semantic invariance, often at the expense of detailed spatial understanding. In this work, we study vision pretraining through a boundary-centric lens, motivated by the premise that boundaries and shape discontinuities offer essential cues for perceiving geometric properties. Concretely, we propose masked boundary modeling, a self-supervised paradigm that dynamically learns sub-pixel boundary representations and subsequently leverages the discovered boundary-bearing tokens as masked targets to facilitate dense visual token learning. By scaling this framework, we develop LingBot-Vision and demonstrate its efficacy across a diverse set of downstream vision tasks with DINOv3 as a strong baseline. Remarkably, LingBot-Vision drives the progression from LingBot-Depth 1.0 to LingBot-Depth 2.0 for depth completion, and thereby yields enhanced depth estimation, a key pillar for embodied artificial intelligence. Our findings reveal that boundary modeling goes beyond simple line segments and instead serves as a scalable pretraining principle for learning spatially structured visual representations.

1 Introduction

The introduction frames dense spatial perception as essential for physical intelligence and argues that shapes and boundaries should become native self-supervised learning signals. It presents masked boundary modeling and categorical boundary-field reparameterization, scaled as LingBot-Vision for dense spatial tasks.

  • Motivation: Dense spatial structure makes images measurable, navigable, and actionable, with shapes and boundaries revealing masks, geometry, motion regions, and occluded surfaces.
  • Problem: Modern foundation models rarely use shapes and boundaries as native learning signals because they are typically task-specific outputs requiring costly, ambiguous, or unavailable annotations.
  • Method: Masked boundary modeling uses teacher-discovered boundary-bearing tokens as forced student targets, turning information-dense boundaries into masked reconstruction challenges.
  • Method: Categorical reparameterization converts continuous boundary fields into discretized per-pixel classification distributions, stabilizing dense self-distillation and enabling parameter-free boundary validation.
  • Scaling and Results: LingBot-Vision is a 1B-parameter self-supervised Vision Transformer that rivals or surpasses models up to 7× larger across dense spatial perception tasks.

2 Related Work

Prior vision pretraining follows several self-supervised and predictive paradigms, but dense spatial quality is often indirect or retrofitted. Boundary-centric pretraining instead makes dense spatial structure the objective and uses categorical attraction fields as its self-supervised medium.

  • Self-Supervised Vision Pretraining: Self-supervised vision learning includes contrastive learning, masked image modeling, self-distillation, and joint-embedding predictive architectures.These methods aim to learn visual representations from data without labels.
  • Self-Distillation: Self-distillation methods train student representations against EMA-teacher distributions, with DINO revealing object layouts and boundaries in unsupervised attention maps.iBOT extends agreement learning to masked tokens, while DINOv2 scales the approach on curated data.
  • Spatial-Semantic Trade-off: Masked image modeling is spatially faithful through pixel-level reconstruction, whereas language supervision favors recognition but underperforms on dense prediction.The two families therefore occupy opposite ends of a spatial-semantic trade-off.
  • Predictive Architectures and World Models: JEPA methods predict in latent space to avoid pixel-level nuisance, while frozen encoders increasingly serve as perception substrates for world models and robot policies.In these settings, geometric errors can produce wrong actions rather than merely reduce benchmark scores.
  • Dense Spatial Representation: Dense spatial quality is commonly obtained indirectly and protected through feature upsamplers, corrective regularization, or multi-teacher distillation.LingBot-Vision makes dense spatial structure the objective of pretraining itself.
  • Boundary Modeling: Boundary modeling builds on classical edge and contour operators, but this work turns a categorical attraction field into the medium of self-supervised pretraining.Detection-quality boundaries emerge as a byproduct of representation learning rather than its goal.

3 Boundary-centric Masked Representation Modeling

This section introduces boundary-centric masked representation modeling, which replaces content-agnostic masking with boundary-guided token selection and geometric reconstruction. Online, label-free boundary discovery bootstraps coherent sub-pixel representations and complements semantic self-distillation.

  • Motivation: Boundary-centric masking lets image boundaries determine which tokens to hide and reconstruct, addressing random masking’s content-agnostic treatment of structurally unequal tokens.The method builds on DINO-style self-distillation and adds a geometric supervision channel.
  • Boundary-forcing mask and geometry routing: Teacher-predicted boundaries are rasterized into a pixel map, and tokens containing boundaries are routed for boundary supervision and masked representation learning.A token is designated a boundary token when a boundary falls within its patch.
  • Boundary emergence: Corner points seed boundary fields whose redundant support votes decode coherent, corner-anchored line fragments before the field becomes pixel-accurate.Direction guidance from parameter-free image-gradient level-line orientation is required for coherent segments; without corner points, decoding fails.
  • Boundary representation: The method models line segments rather than isolated edge pixels because multi-pixel support enables validation against a no-structure null hypothesis.Segment hypotheses are supported by many pixels, unlike individual thresholded filter responses.
  • Boundary representation: Boundary prediction is reparameterized categorically because continuous teacher–student regression targets drift and collapse, while classification supports centering and sharpening.Orientation bins use circular arc distance, and narrow labels preserve the signal by avoiding over-smoothing.
  • Main comparison: 81.6% to 82.4% ImageNet-1K k-NN top-1, 81.4% to 84.9% NYUv2 δ1, and 0.474 to 0.440 RMSE show simultaneous semantic and dense-geometric gains over DINO+iBOT.The categorical boundary target supplies essentially the dense improvement, while dual supervision adds +0.2 k-NN points and +0.3 δ1 points.

4 LingBot-Vision: Pushing the Boundaries in Vision Pretraining

LingBot-Vision scales boundary-centric masked representation learning to a 1.1-billion-parameter vision foundation model trained on a curated 161M-image corpus. Its sparse, fused boundary computation and batched target generation keep the added training cost small, while the resulting system supports efficient training and distillation to smaller backbones.

  • Scaling the Data and Model: LingBot-Vision trains a ViT-g/16 backbone with roughly 1.1B parameters on roughly 161M images selected from 2B raw images.The corpus combines curated public datasets with retrieval-curated web pools, while distributed systems and a scale-stable optimization recipe support training.
  • Scaling the Data and Model: The curation pipeline uses open-source datasets as seeds for retrieval from an in-house pool of roughly 2B web images, with DINOv2 ViT-B embeddings guiding selection.Seeds include ImageNet-1k, ImageNet-21k, GLDv2, Mapillary-SLS, and twenty additional datasets spanning fine-grained recognition.
  • Efficient Boundary Computation: The boundary head processes only boundary tokens, while fused kernels and batched GPU target generation avoid quadratic memory use and host round-trips.In the final recipe, validated boundary-target generation adds only a small fraction of total step time, and throughput remains close to the semantic-only baseline under identical settings.
  • Model Distillation: After training the flagship ViT-g, distillation produces 300M ViT-L, 86M ViT-B, and 21M ViT-S backbones that inherit semantic and boundary-aware representations.The frozen ViT-g replaces the EMA teacher in the same training pipeline, using the pretraining-stage budget of 300k iterations.

5 Results

Section 5 evaluates LingBot-Vision through frozen-feature probing across dense prediction, classification, global, and video tasks. Its performance is compared against leading vision foundation models on public benchmarks.

  • Evaluation Protocol: LingBot-Vision is evaluated using frozen features across various downstream tasks.The protocol includes linear probing for dense prediction and classification, and non-parametric probing for global and video tasks.
  • Evaluation Protocol: Linear probing is applied to dense prediction and classification tasks.
  • Evaluation Protocol: Non-parametric probing uses k-NN retrieval and label propagation for global and video tasks.
  • Benchmark Comparison: Performance is assessed on public benchmarks against DINOv2, DINOv3, SigLIP 2, and V-JEPA 2.1.

5.1 Robust Dense Representation of Images

LingBot-Vision produces coherent, sharply bounded frozen patch features that support robust dense perception under a single-linear-layer probing protocol. It achieves strong depth and segmentation performance, including the best NYUv2 RMSE in the comparison and competitive or superior mIoU across benchmarks.

  • Feature structure: LingBot-Vision resolves objects and fine structures as coherent, sharply bounded regions in PCA visualizations of frozen patch features.The maps show cars and highway lanes, hen silhouettes against a fence, snake contours, and fine structures as distinct regions.
  • Evaluation protocol: A single linear layer on frozen patch tokens isolates representation quality by avoiding added decoding capacity and feature aggregation.The protocol freezes the ViT backbone and trains only the decoder on each target dataset’s training split.
  • Depth estimation: 0.296 NYUv2 RMSE is LingBot-Vision’s best result in the table, surpassing 7B-parameter DINOv3 at 0.309 and 2B-parameter V-JEPA 2.1 at 0.307.The 1B-parameter ViT-g/16 backbone uses 7× and 2× fewer parameters than those respective models.
  • Depth estimation: Boundary-oriented features support accurate depth by remaining smooth within surfaces and changing sharply across occlusion boundaries.The linear decoder can only read out information already encoded in frozen features, making boundary-sensitive representation important for depth discontinuities.
  • Semantic segmentation: LingBot-Vision trails DINOv3 ViT-H+ by 1.3 mIoU on ADE20K, matches it on Cityscapes, and leads on VOC12.Against equal-size DINOv2, it improves by at least 4 mIoU on all three benchmarks, including 53.5 versus 49.5 on ADE20K.
  • Semantic segmentation: Segmentation benefits from both nearby features within semantic regions and crisp feature transitions at class boundaries, including contours and thin structures.The PCA maps exhibit coherent object interiors and sharp transitions, while language-aligned pretraining is characterized separately by its ADE20K performance.

5.2 Video Understanding

LingBot-Vision exhibits strong temporal consistency in frozen features, achieving competitive training-free video object segmentation and supporting boundary-token tracking across diverse videos. Its performance remains strong without fine-tuning and surpasses DINOv2 at equal size.

  • Evaluation protocol: Training-free video object segmentation propagates the first-frame ground-truth mask using top-k attention over frozen patch features, without fine-tuning.Evaluation uses DAVIS-2017 and YouTube-VOS, reporting J & F-Mean with resolution determined by patch size.
  • Results: LingBot-Vision reaches 70.0 and 73.5 J & F on DAVIS and YouTube-VOS, respectively.These results are on par with DINOv3 ViT-H+ and 7B DINOv3, and are best among the remaining models at any scale.
  • Results: At equal size, LingBot-Vision exceeds DINOv2 by 6.1 and 7.9 points on DAVIS and YouTube-VOS, respectively.V-JEPA 2.1 ViT-g trails by 1.9 and 1.2 points despite being trained on video.
  • Temporal token consistency: Individual boundary tokens are tracked through robot-camera, cat, and home-pan videos by local-window cosine similarity matching over each frame’s boundary tokens.The qualitative probe visualizes queried-token similarity responses over time.

5.3 Strong Distilled Models on both Global and Dense Tasks

LingBot-Vision’s distilled models preserve the teacher’s strong dense-task profile across deployment sizes, while the remaining gap to leading baselines is concentrated in image-level recognition. At ViT-L, the model substantially improves NYUv2 depth accuracy relative to DINOv2 and remains competitive on global and segmentation metrics.

  • Giant-scale recognition: At giant scale, LingBot-Vision reaches 86.32 linear and 83.39 k-NN top-1 accuracy, trailing DINOv3-7B and SigLIP 2 on image-level recognition.Its dense-task comparisons favor LingBot-Vision, consistent with boundary-oriented pretraining emphasizing localized structure.
  • Distillation setup: The giant teacher is distilled into ViT-L, ViT-B, and ViT-S students evaluated with the same global and dense protocols.The setup targets different deployment budgets while preserving comparable token-aligned evaluation.
  • ViT-L results: At ViT-L, LingBot-Vision matches DINOv2 on linear accuracy, 86.38 vs. 86.43, while reducing NYUv2 RMSE by 25%, 0.310 vs. 0.411.Against DINOv3-L, it concedes 0.9 points of linear accuracy but leads NYUv2 depth by 12% and stays within 0.5 to 2.3 mIoU across segmentation benchmarks.
  • Efficient dense perception: The 0.3B student attains NYUv2 RMSE of 0.310, on par with DINOv3’s 7B model at 0.309 under the same protocol.This result demonstrates that the distilled family retains strong dense spatial performance at substantially smaller scale.
  • Analysis: Distillation transfers the teacher’s dense advantage essentially intact, while residual differences versus DINOv3 concentrate on image-level accuracy.Across deployment budgets, the analysis describes a practical accuracy-compute frontier in which LingBot-Vision students provide the best available depth accuracy in their class.

6 Meet LingBot-Depth 2.0

LingBot-Depth 2.0 upgrades depth completion by replacing the encoder initialization with LingBot-Vision and scaling curated training data from 3M to 150M RGB-D samples, while retaining masked depth modeling. It improves quantitative and qualitative completion across masked, sparse, real-sensor, transparent, reflective, and glass scenes.

  • Method upgrades: LingBot-Depth 2.0 retains masked depth modeling while upgrading encoder initialization and curated-data scale.The DINOv2 ViT-L/14 encoder is replaced with LingBot-Vision at ViT-L/16 and ViT-g/16, while the curated corpus grows from 3M to 150M RGB-D samples.
  • Quantitative results: LingBot-Depth 2.0 with ViT-L achieves the best RMSE on seven of eight block-masked and sparse benchmarks.It wins all four block-mask settings and three of four sparse settings; ETH3D remains with LingBot-Depth 1.0.
  • Quantitative results: 0.132 to 0.062 RMSE on block-masked DIODE-Indoor and 3.404 to 2.440 on DIODE-Outdoor mark large gains over LingBot-Depth 1.0.The DIODE-Indoor RMSE is halved, while DIODE-Outdoor also improves substantially.
  • Quantitative results: LingBot-Depth 2.0 is best on six of eight real-sensor configurations and is particularly strong on transparent-object ClearGrasp captures.The ClearGrasp RMSE values are 0.010 and 0.012; ViT-g further reaches 0.228, 0.345 and 0.375 across the three LingBot cameras, while ClearGrasp favors ViT-L.
  • Qualitative results: On mirror and glass sequences, LingBot-Depth 2.0 reconstructs missing regions as flat, contiguous, temporally stable surfaces with sharp boundaries.It recovers window panes, a glass balustrade, and reflective floors where raw sensor depth returns no measurements.

7 Conclusion

LingBot-Vision is presented as a spatial-perception-native vision foundation model built around boundary structure in self-supervised learning. Its initialization improves LingBot-Depth 2.0 depth completion, with the advantage widening as training data grows.

  • Core contribution: LingBot-Vision makes boundary structure central to self-supervised learning by representing image boundaries as dense categorical boundary fields.Boundary-bearing tokens are forced into the masking pattern, while training targets are bootstrapped online from the teacher’s predictions and validated with a parameter-free a-contrario test.
  • Core contribution: Boundary-bearing tokens are forced into the masking pattern, and training targets are bootstrapped online from teacher predictions.The targets are validated by a parameter-free a-contrario test.
  • Downstream impact: LingBot-Depth 2.0 achieved leading depth completion performance across masking patterns and real depth cameras after changing only encoder initialization and curated-data scale.The advantage of LingBot-Vision initialization widened as training data grew.

Appendix · A Sampling Boundary Fields

This appendix demonstrates that line segments can be decoded from boundary fields carrying no learned information. It specifies the sampling setup and decoding procedure, showing how random and parameter-free guidance affect segment proposals and stability.

  • A Sampling Boundary Fields: The appendix demonstrates decoding line segments from boundary fields that carry no learned information.The demonstration supports Finding 1 and Fig. 3.
  • A Sampling Boundary Fields: Images are resized to 512 × 512, while the boundary field uses output stride 2 and channels normalized to [0, 1].The distance channel uses a 5 px support threshold at field resolution, and angular channels use their angular ranges.
  • A Sampling Boundary Fields: Corner points come from the frozen detector used throughout pretraining and remain fixed across all draws.This keeps the corner anchors constant while the boundary-field channels vary.
  • A Sampling Boundary Fields: Each draw samples every field channel independently from U(0, 1), with sub-pixel refinement offsets set to zero.The randomization is applied per position.
  • A Sampling Boundary Fields: Randomly oriented chord proposals rarely agree, so decoded segments stay short while remaining anchored on corner points.The direction channel θ is randomized in the bottom rows of Fig. 3.
  • A Sampling Boundary Fields: Decoding forms chords from (d, θ, ϕ1, ϕ2), attaches endpoints to nearest corners, accumulates votes over corner pairs, and selects pairs with sufficient votes.A-contrario validation is disabled, and Fig. 3 sweeps the minimum vote count from 2 to 10 across independent draws.

B A-contrario Validation

The a-contrario validation accepts candidate segments when their false-alarm probability is sufficiently low under a uniform-orientation null model. The proposed pipeline decouples dense candidate formation from validation by anchoring chord proposals to corner points.

  • A-contrario criterion: A candidate segment is accepted iff NFA ≤ ε, with ε = 1 allowing at most one false alarm per image on average.The test is therefore essentially parameter-free.
  • A-contrario criterion: The null hypothesis assumes uniformly distributed orientations, matching the uniform distribution over bins of the categorical boundary field.
  • Parallelizable pipeline: Unlike LSD’s sequential region growing, the proposed method decouples candidate formation from validation through dense chord proposals anchored by corner points.LSD consumes pixels during greedy region growing, preventing later seeds from reusing them; the proposed pipeline obtains candidates without grouping.
  • Implementation constants: Candidates use a fixed 3 px rectangle and compare orientations modulo π with tolerance π/16, corresponding to p = 1/16.Pixels with negligible gradient magnitude are undefined and never count as aligned.
Loading 2607.05247v1…