PaperlayerView exact revision on arXiv

Source-linked AI summary

In Pursuit of Pixel Supervision for Visual Pre-training

Lihe Yang, Shang-Wen Li, Yang Li, Xinjie Lei, Dong Wang, Abdelrahman Mohamed, Hengshuang Zhao, Hu Xu

arXiv:2512.15715v1cs.CV

TL;DR

Visual pre-training often uses latent-space objectives, although pixels contain information from low-level attributes to high-level concepts. Pixio enhances MAE and trains on 2 billion web-crawled images with soft self-curation. It performs competitively across diverse downstream tasks, matching or outperforming DINOv2/v3 on several tasks.

  • Problem

    Visual pre-training commonly uses latent-space objectives, while pixels themselves contain information from low-level attributes to high-level concepts.

    Source: §1 · p. 1

  • Method

    Pixio enhances MAE with larger masking blocks, a deeper decoder, more class tokens, and soft self-curation of 2 billion web-crawled images.

    Source: §1 · pp. 1–2

  • Results

    Pixio outperforms or matches DINOv2/v3 on depth estimation, 3D reconstruction, and semantic segmentation, and delivers promising robot-learning results.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results suggest pixel-space self-supervision can be a competitive alternative and complement to latent-space approaches.

    Source: §1 · p. 2, §5 · p. 8

    Static-image masking entails a trade-off: low ratios risk ground-truth leakage, while high ratios can leave insufficient context and create train-inference distribution shift.

    Source: §6.2 · p. 13

Abstract

from arXiv · show

At the most basic level, pixels are the source of the visual information through which we perceive the world. Pixels contain information at all levels, ranging from low-level attributes to high-level concepts. Autoencoders represent a classical and long-standing paradigm for learning representations from pixels or other raw inputs. In this work, we demonstrate that autoencoder-based self-supervised learning remains competitive today and can produce strong representations for downstream tasks, while remaining simple, stable, and efficient. Our model, codenamed "Pixio", is an enhanced masked autoencoder (MAE) with more challenging pre-training tasks and more capable architectures. The model is trained on 2B web-crawled images with a self-curation strategy with minimal human curation. Pixio performs competitively across a wide range of downstream tasks in the wild, including monocular depth estimation (e.g., Depth Anything), feed-forward 3D reconstruction (i.e., MapAnything), semantic segmentation, and robot learning, outperforming or matching DINOv3 trained at similar scales. Our results suggest that pixel-space self-supervised learning can serve as a promising alternative and a complement to latent-space approaches.

1. Introduction

Pixio revisits pixel-space self-supervision, arguing that pixels carry visual information from low-level attributes to high-level concepts. It updates MAE and trains on minimally curated web-scale data, with competitive results across diverse downstream tasks.

1. Introduction

  • Self-supervised representation learning uses unannotated data, with methods differing in whether their objective is defined in pixel or latent space.The introduction contrasts pixel-predicting MAEs with latent-space approaches such as DINO and JEPA.Source: §1 · p. 1
  • The paper argues that pixel prediction can train representations that compress and reorganize visual information across low-level attributes and high-level concepts.The figure illustrates masked inputs, reconstructions, and ground-truth images; its caption says MAE learns visual properties through masking and pixel reconstruction.Source: §1 · p. 1, Figure 1 · p. 2
  • Pixio makes MAE pre-training more challenging with larger masking blocks and strengthens the model with a deeper decoder and more class tokens.The authors report these changes facilitate representation learning by giving capable models a sufficiently challenging task.Source: §1 · p. 1
  • The training pipeline uses 2 billion web-crawled images and soft self-curation based on each image’s reconstruction loss.The data is intended to reduce benchmark-distribution bias while supplying diverse training examples.Source: §1 · pp. 1–2
  • Pixio outperforms or matches DINOv2/v3 on depth estimation, 3D reconstruction, and semantic segmentation, and delivers promising robot-learning results.The authors conclude pixel-space self-supervision can complement or compete with latent-space approaches.Source: §1 · p. 2

2. Related Work

The section traces visual supervision from human labels and captions to self-supervised objectives, noting that each provides only a partial account of the visual world. It positions direct pixel supervision as a less human-directed alternative.

2. Related Work

  • ImageNet class labels provide explicit supervision but describe images with a single human-defined category, creating scalability and transfer limitations.The section reports only marginal gains when transferring this pre-training paradigm to downstream tasks.Source: §2 · p. 2
  • Web image-text pairs scale supervision and support open-vocabulary applications, but captions can be partial, irrelevant, and constrained by language.The section also notes paired data is not infinitely scalable and may retain human biases that omit difficult-to-verbalize visual phenomena.Source: §2 · p. 2
  • Self-supervised methods avoid explicit labels but retain human priors through designed distortions and objectives in pixel or latent space.For latent-space approaches, the section describes inductive biases such as multi-view consistency as helping prevent collapse and guide learning.Source: §2 · pp. 2–3
  • The figure motivates four Pixio changes: a deeper decoder, larger mask blocks, more class tokens, and web-scale training data.Its caption associates these with decoder capacity, reconstruction shortcuts and context, global visual properties, and training-data diversity, respectively.Source: Figure 2 · p. 3
  • Pixio advocates pixel supervision as grounded in observable reality, with less human intervention than categories, text, or designed visual priors.The authors describe it as a scalable, less biased path for learning capable vision models.Source: §2 · p. 3

3. Pixio

Pixio retains MAE’s core masked pixel-reconstruction design while revisiting its model components and training data. Its changes target decoder capacity, masking context, global representations, and data diversity.

3. Pixio

  • Pixio builds on MAE with algorithm and data updates, preserving its three core designs while revisiting components considered important for performance.The section first reviews MAE, then presents the algorithm changes and data update.Source: §3 · p. 3, §3.2 · p. 3

3.1. Preliminary of MAE

  • MAE masks part of the input and reconstructs it with direct pixel supervision, an asymmetric encoder-decoder, and a high masking ratio.The encoder drops masked tokens and the lightweight decoder appends them back; high masking ratios help avoid ground-truth leakage in redundant visual inputs.Source: §3.1 · p. 3

3.2. MAE Redesign

  • MAE’s decoder may lack capacity for pixel regression, so increasing its depth can improve representations while retaining lightweight overhead.The authors caution that an excessively powerful decoder may memorize details or make the encoder lazy; their account also notes that generic features need not emerge from the final encoder block.Source: §3.2 · p. 4
  • Larger mask blocks preserve richer local context and reduce copying from nearby visible patches, but excessively large masked regions become unpredictable.The section examines masking granularity alongside masking ratio; its ablation passage reports poor results for 8×8 patches because the context is overly large.Source: §3.2 · p. 4
  • Multiple class tokens are intended to capture diverse global visual properties; downstream tasks can average or concatenate them.Unlike register tokens that are discarded during evaluation, these class tokens serve as global representations for tasks such as classification and robot learning.Source: §3.2 · p. 4

3.3. Web-Scale Data and Curation

  • Pixio trains on 2 billion web-crawled images, using reconstruction-loss soft sampling and low-color-histogram-entropy filtering to counter product and text-heavy image dominance.The authors describe the combined strategies as preserving rich, diverse visual content with minimal curation bias.Source: §3.3 · p. 4

4. Experiments

Experiments evaluate Pixio on dense visual tasks, robot learning, and ImageNet fine-tuning, alongside ablations of its framework and training data. Across these settings, results include strong comparisons with DINOv3 and improvements from Pixio’s design updates.

4.1. Monocular Depth Estimation

  • On NYUv2, Pixio reduces RMSE from 0.320 to 0.268 and improves δ1 from 93.2 to 95.5 versus DINOv3.The comparison uses a frozen encoder with a trainable DPT or linear regression head.Source: Table 1 · p. 5, §4.1 · p. 5
  • Pixio outperforms or matches DINOv3 on most zero-shot relative-depth benchmarks, but trails DINOv2/v3 on KITTI.Evaluation trains on five synthetic datasets and transfers to unseen datasets.Source: Table 2 · p. 5, §4.1 · p. 5

4.2. Feed-Forward 3D Reconstruction

  • Across indoor, outdoor, and synthetic scenes, Pixio consistently beats MAE, DINOv2, and DINOv3 on reconstruction and pose estimation.Pixio is pretrained on single-view images, while DINOv3 explicitly uses multiple views.Source: §4.2 · p. 5, Table 3 · p. 6

4.3. Semantic Segmentation

  • Pixio matches or outperforms DINOv3 on semantic segmentation benchmarks despite using a simpler pretraining objective and no benchmark-specific data curation.It uses 200M fewer parameters and is distilled from a model with 1.3B fewer parameters.Source: §4.3 · pp. 5–6, Table 4 · p. 6

4.4. Robot Learning

  • On CortexBench, Pixio scores 1.2% above R3M and 3.1% above DINOv3.Averaged class tokens provide Pixio’s global embedding and avoid the CNN policy network’s computational overhead.Source: Table 5 · p. 6, §4.4 · p. 6

4.5. Ablation Study

  • Increasing decoder depth from 8 to 32 attention blocks improves IN-1K k-NN accuracy from 35.3 to 55.8; changing masking from 1×1 to 2×2 patches improves it by 19.0.For the decoder change, NYUv2 depth error falls from 0.431 to 0.410 and ADE20K mIoU rises from 35.8 to 40.4; 8×8 masking performs poorly because its context is unpredictable.Source: §4.5 · p. 6, Figure 4 · p. 7, Figure 5 · p. 8
  • Combining the three framework updates improves IN-1K classification from 37.9 to 59.5, while increasing class tokens from 1 to 4 raises k-NN accuracy from 63.3 to 75.1.The combined updates also reduce NYUv2 depth error from 0.392 to 0.321 and improve ADE20K mIoU from 37.2 to 46.8; class-token gains on dense prediction are minor.Source: Table 6 · p. 7, §4.5 · p. 7, Figure 6 · p. 8
  • For dense prediction, IN-21K and YFCC100M outperform IN-1K, while simple curation substantially improves the uncurated 2B-image source, though it remains below highly curated IN-21K.The compared sources include IN-1K, IN-21K, YFCC100M, and 2B web-crawled images.Source: Table 7 · p. 7, §4.5 · pp. 7–8

5. Conclusion

Pixio combines minimal MAE updates with 2B web-crawled images and simple self-curation, demonstrating that pixel supervision can support competitive visual pre-training.

5. Conclusion

  • Pixio introduces three minimal algorithm updates to MAE and trains on 2B web-crawled images using a simple self-curation strategy.The paper reports results better than or comparable to DINOv3.Source: §5 · p. 8
  • Pixio delivers results better than or comparable to the state-of-the-art DINOv3 model.The authors present this as evidence that pixel supervision can serve as promising supervision for large-scale visual pre-training while retaining simplicity, stability, and efficiency.Source: §5 · p. 8
  • The authors conclude that pixels can serve as promising supervision for large-scale visual pre-training while preserving simplicity, stability, and efficiency.Source: §5 · p. 8

6. Failure Attempts, Limitations, and Future Directions

The authors retain a simple MAE design after several alternatives failed to improve performance consistently. They identify trade-offs in static-image masking and point to video prediction as a future direction.

6.1. Failure Attempts

  • Inpainting and outpainting added tuning complexity and restricted masking-pattern diversity without outperforming patch-block masking; hybrid masking ratios showed no clear gains, and fixed granularity performed best.Outpainting also limited learning long-range dependencies by preserving a large contiguous visible region.Source: §6.1 · p. 12
  • Koleo loss on multiple class tokens slightly improved dense prediction in preliminary experiments but harmed ImageNet classification and destabilized training.Source: §6.1 · p. 12
  • Reconstructing visible patches substantially degraded downstream performance, and reconstructing only a random subset of masked patches consistently did so as well.The latter provided only marginal training speedup.Source: §6.1 · p. 13
  • Multi-stage decoder features gave marginal, inconsistent improvements, while decoder cross-attention yielded only marginal speedup and no downstream gains.The final framework retains the simpler single-stage features and standard self-attention design.Source: §6.1 · p. 13

6.2. Limitations and Future Directions

  • The authors retain three minimal MAE modifications and propose scaling pixel supervision to web-scale video, where temporal prediction can use natural event progression without artificial spatial masking.They frame the work as a pioneering validation that pixel supervision alone can produce strong representations competitive with more complex pre-training paradigms.Source: §6.1 · p. 13, §6.2 · p. 14

6.1. Failure Attempts

  • Online hard-example mining produced noisy early difficulty estimates and unpredictable distribution shifts, so the authors used offline pre-computation for more stable estimates.Source: §6.1 · p. 13
  • Canny edge density overreacted to low-level patterns and repetitive textures that offered limited semantic diversity.Source: §6.1 · p. 13

6.2. Limitations and Future Directions

  • Static-image masking trades off target leakage at low ratios against insufficient context and train–inference distribution shift at high ratios; masked image modeling never trains on natural, complete images.The authors nevertheless say masking or other artificial distortions appear necessary for image-based pre-training.Source: §6.2 · pp. 13–14

7. Implementation Details

The implementation scales MAE pre-training to web data, distills a large teacher into smaller encoders, and evaluates them using task-specific protocols.

7.1. Pre-training

  • Pre-training extends MAE to 1.3M iterations and batch size 16,384, uses 256×256 inputs, and lowers peak learning rate from 2.4e-3 to 8e-4 for stable convergence on diverse web data.The pre-training configurations are listed in Table 9.Source: §7.1 · p. 14, Table 9 · p. 14

7.2. Distillation

  • Pixio-5B serves as teacher for Pixio-1B, Pixio-H, Pixio-L, and Pixio-B; student features are aligned to teacher features with a projection head and cosine-similarity loss.Capable students receive masked inputs, while the teacher processes unmasked images.Source: §7.2 · pp. 14–15, Table 10 · p. 14

7.3. Downstream Evaluation

  • Downstream evaluation follows task-specific protocols, including k-NN and fine-tuning for ImageNet-1K, frozen-encoder heads for depth and segmentation, and official implementations for Depth Anything, MapAnything, and CortexBench.The authors state that they open-sourced the downstream evaluation code to facilitate reproducibility.Source: §7.3 · p. 15

8. Ablation Studies

The ablations examine encoder features, decoder capacity, masking, and class tokens. They support a deeper decoder and larger masking granularity while showing that the best encoder features and design choices depend on the tested configuration.

8.1. Block-Wise Performance of MAE and Pixio

  • 3.3 mIoU separates the last from the optimal intermediate MAE-H encoder block on ADE20K; Pixio’s final block is within 0.006 RMSE of its best intermediate block on NYUv2.The reported comparisons probe MAE-H and Pixio encoders, respectively.Source: §8.1 · p. 15, Table 11 · p. 15, Table 12 · p. 16

8.2. Decoder Design

  • Excessively shallow decoders are suboptimal because they lack capacity for the pixel-reconstruction objective.Table 13 ablates decoder design on a ViT-H encoder with single-patch masking and one class token.Source: §8.2 · p. 16, Table 13 · p. 16

8.3. Masking Design

  • Ablations under the 384×32 decoder setting further support larger masking granularity for learning better representations.Table 14 evaluates masking ratio and granularity, with granularity measured in patches.Source: §8.3 · p. 16, Table 14 · p. 17

8.4. Number of Class Tokens

  • Including class tokens in the decoder input yields slightly better performance, suggesting that their participation in reconstruction helps learn more informative global representations.Table 15 varies the number of class tokens and their inclusion in the decoder, using 2×2 patch blocks.Source: §8.4 · p. 16, Table 15 · p. 17
Loading 2512.15715v1…