PaperlayerView exact revision on arXiv

Source-linked AI summary

CanViT: Toward Active-Vision Foundation Models

Yohaï-Eliel Berreby, Sabrina Du, Audrey Durand, B. Suresh Krishna

arXiv:2603.22570v2cs.CV

TL;DR

Active vision has struggled to match passive models in accuracy, efficiency, flexibility, and dense prediction. CanViT introduces a task- and policy-agnostic architecture with a persistent scene-wide canvas and sets a new active-vision state of the art across segmentation and classification.

  • Problem

    Active vision lacks general-purpose models that combine strong perception and memory across tasks, while existing models often lag passive systems, especially on dense prediction.

  • Method

    CanViT combines a ViT glimpse-processing stream with a persistent scene-wide canvas, linked bidirectionally through Canvas Attention for task- and policy-agnostic active vision.

  • Results

    CanViT-B sets a new active-vision state of the art on ADE20K segmentation and ImageNet-1k classification, reaching 45.9% ADE20K mIoU.

  • Takeaways & Limitations

    CanViT demonstrates that task- and policy-agnostic active-vision pretraining can deliver strong performance, efficiency, and generalization across viewing policies and temporal horizons.

  • Takeaways & Limitations

    CanViT was trained and evaluated on static scenes, leaving adaptation to real-time video and embodied active perception for future work.

Abstract

from arXiv · show

Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses, but lacks scalable general-purpose architectures and pretraining pipelines, leaving Active-Vision Foundation Models (AVFMs) underexplored. We introduce CanViT, the first task- and policy-agnostic AVFM. CanViT uses scene-relative RoPE to bind a retinotopic Vision Transformer backbone and a spatiotopic scene-wide latent workspace, the canvas. Efficient interaction with this high-capacity working memory is supported by Canvas Attention, a novel asymmetric cross-attention mechanism. We decouple thinking (backbone-level) and memory (canvas-level), eliminating canvas-side self-attention and fully-connected layers to achieve fast sequential inference and scalability to high output resolutions. We propose a label-free active vision pretraining scheme, policy-agnostic passive-to-active dense latent distillation: reconstructing scene-wide DINOv3 embeddings from sequences of low-resolution glimpses with randomized locations, zoom levels, and lengths. We pretrain CanViT-B from a random initialization on 13.2 million ImageNet-21k scenes--an order of magnitude more than previous active models--and 1 billion random glimpses, in 166 hours on a single H100. On ADE20K segmentation, a frozen CanViT-B achieves 38.5% mIoU in a single low-resolution glimpse, outperforming the best active model's 27.6% with 20x fewer inference FLOPs as well as its FLOP- or input-matched DINOv3 teacher. Given additional glimpses, CanViT-B reaches 45.9% ADE20K mIoU. On ImageNet-1k classification, CanViT-B also sets a new active-vision state of the art, with 84.5% top-1 accuracy after fine-tuning. CanViT generalizes to longer rollouts, larger scenes, and new policies. Our work narrows the wide gap between passive and active computer vision, demonstrating the potential of task- and policy-agnostic AVFM pretraining.

1 Introduction

The introduction argues that passive frame-wise vision lacks active, recurrent processing, memory, and strategic viewing, motivating a task- and policy-agnostic Active-Vision Foundation Model. It presents CanViT as a recurrent canvas-based architecture pretrained through passive-to-active distillation and evaluated across tasks and viewing conditions.

  • Motivation: Active computer vision uses sequential localized glimpses to model strategic planning, temporal working memory, and recurrent top-down processing, but has lagged passive models in accuracy, efficiency, flexibility, and representation richness.The gap is especially pronounced for dense prediction.
  • Motivation: Active vision separates instantaneous vision, persistent scene memory, and action selection, with the latter defining the sensory policy.The first two axes concern understanding and recalling visual inputs, while action selection determines where and at what zoom level to look next.
  • Motivation: The proposed AVFM targets spatial and semantic understanding across arbitrary glimpse sequences while transferring rich representations across tasks and viewing policies.Policy agnosticism separates how to see from where to look and avoids dependence on reinforcement learning.
  • Contributions: CanViT is introduced as a recurrent Vision Transformer built around a latent scene-wide canvas and pretrained with passive-to-active distillation.The introduction identifies the canvas as the model’s persistent scene-wide representation and previews evaluation on ADE20K segmentation and ImageNet-1k classification.
  • Contributions: CanViT-B is reported to set a new active-vision state of the art on ADE20K and ImageNet-1k while transferring across policies, temporal horizons, and canvas resolutions.These claims summarize the introduction’s stated evaluation outcomes and generalization targets.

2 Related work

Prior active-vision research established sequential glimpse processing but focused mainly on simple tasks, while dense prediction remained underexplored. CanViT relates this work to latent distillation, cross-attention architectures, and recurrent latent reasoning while differing in how it organizes inputs and memory.

  • Active vision: Active vision models process fixed- or variable-scale image or video glimpses, but the field historically emphasized simple tasks before expanding toward ImageNet-1k recognition.The Recurrent Attention Model initiated this line, and Saccader reached 75% ImageNet-1k top-1 accuracy after introducing intermediate pretraining.
  • Dense prediction: Dense prediction has remained underexplored because most active-vision architectures cannot directly produce scene-wide, spatially dense outputs.AME and AdaGlimpse instead use post-hoc expansion with a Transformer decoder, encoded glimpse tokens, learnable mask tokens, and self-attention.
  • Latent distillation: Dense latent distillation transfers visuospatial representations from extensively pretrained self-supervised vision transformers into randomly initialized smaller models.Prior work distilled DINOv2 and DINOv3 into smaller ViTs, while Proteus used 100× less data than pretraining with CLS-token and dense-feature matching.
  • Cross-attention architectures: CanViT builds on cross-attention methods that decouple dimensionality and computation, alternating read and write cross-attention across depth and time.Unlike Perceiver and RINs, CanViT places the external glimpse on the few-token side and the canvas latents on the many-token side.
  • Recurrent latent reasoning: Latent-space recurrent reasoning reuses computation over fixed external inputs with tied weights, separating representational capacity from computational depth and enabling flexible test-time compute.This paradigm has attracted renewed interest in both large language models and compact capable models.

3 Preliminaries

CanViT is motivated by active vision over bounded scenes, where sequential fixed-resolution glimpses support progressively refined, spatially grounded scene understanding. The proposed solution is a recurrent architecture centered on a scene-wide latent representation called the canvas.

  • Definitions: Scenes, Glimpses, Viewpoints: A scene is modeled as a bounded 2D coordinate domain mapping continuous (x, y) positions to RGB values; experiments use static scenes with ψ_t = ψ_0.The scene function is ψ_t : [−1, +1]2 → R3.
  • Definitions: Scenes, Glimpses, Viewpoints: A glimpse is a fixed-resolution crop centered at viewpoint v_t = (x_t, y_t, s_t), with center coordinates in the scene and scale s_t ∈ (0, 1].Its scene-coordinate extent is [x_t − s_t, x_t + s_t] × [y_t − s_t, y_t + s_t].
  • Definitions: Scenes, Glimpses, Viewpoints: Regardless of scale, each crop is resized to H_g × W_g pixels, so s_t trades spatial coverage against perception of detail under fixed information capacity.The crop covers a fraction s_t^2 of the scene surface area.
  • General-Purpose, Spatially-Grounded Active Vision: The target is general-purpose scene understanding that accumulates through sequential glimpses and remains decodable at every timestep for global and dense spatial tasks.The motivating examples are object classification and semantic segmentation.
  • General-Purpose, Spatially-Grounded Active Vision: CanViT is introduced as a recurrent vision architecture built around a scene-wide latent representation called the canvas.The architecture is named the Canvas Vision Transformer, or CanViT.

4 The Canvas Vision Transformer (CanViT)

CanViT combines an ephemeral ViT glimpse stream with a persistent, scene-wide canvas memory, enabling bidirectional interaction through Canvas Attention. Scene-relative positional encoding and asymmetric projections preserve spatial-semantic capacity while reducing the cost of processing fine-grained canvas grids.

  • Dual-stream architecture: CanViT couples compact localized-glimpse processing in a ViT backbone with a high-capacity persistent canvas memory through bidirectional Canvas Attention.The backbone pulls information from the canvas and sends updates to it.
  • Canvas memory: The canvas stores scene-wide spatio-semantic information in non-spatial registers and an H × W grid of patches covering normalized scene coordinates.At rollout start, the grid is broadcast to the desired size from a single learnable initial patch.
  • Scene-relative positioning: Scene-Relative RoPE computes 2D positional embeddings from glimpse- and canvas-patch centers in shared [−1, +1]2 scene coordinates, encoding viewpoint position and zoom.The same scene-relative coordinates are used in backbone self-attention and Canvas Attention.
  • Canvas Attention: Canvas Attention alternates efficient Read and Write cross-attention operations, avoiding quadratic self-attention over the large canvas token set.In CanViT-B, Read and Write operations alternate every 2 ViT blocks.
  • Asymmetric projections: 2.8 to 37.3 GFLOPs: adding canvas-side QKVO projections would raise each Canvas Attention Read/Write pair’s cost for CanViT-B’s 64 × 64 canvas grid.The asymmetric design keeps projections on the smaller glimpse stream while retaining a high-capacity canvas.
  • Viewpoint encoding: The optional VPE token concentrates the viewpoint triplet (xt, yt, st) into one glimpse-side token to support future end-to-end policy learning.It can facilitate decoding the next viewpoint from a transformation of the current viewpoint.

5 Policy-agnostic passive-to-active dense latent distillation

CanViT is pretrained through policy-agnostic, label-free reconstruction of dense DINOv3 scene representations from randomized glimpse rollouts. The scheme combines spatially informative teacher targets with rollout diversity and temporally dense supervision.

  • Distillation objective: Policy-agnostic passive-to-active dense latent distillation teaches scene understanding and persistent spatial representation without task labels, while remaining robust to the viewing policy.The method combines informative reconstruction targets with rollout randomization.
  • Distillation objective: DINOv3 ViT-B supplies high-resolution, scene-wide spatio-semantic patch and CLS embeddings that CanViT reconstructs from sequences of glimpses.The reconstruction target is defined in DINOv3 latent space rather than pixel space.
  • Distillation objective: CanViT decodes updated canvas patches, registers, and CLS tokens into DINOv3-space reconstructions through token-wise linear projections, with patch- and CLS-level losses averaged across space and time.This provides supervision for both dense scene structure and global scene representation.
  • Rollout randomization: 1 F-IID + 1 R-IID rollout accelerates convergence compared with 2 R-IID rollouts, including evaluation on held-out data with an R-IID policy.F-IID guarantees an initial full-scene, zoomed-out view, whereas R-IID begins with a random view.
  • Rollout randomization: K = 2 glimpse truncated BPTT with pstop = 0.5 yields an average sequence length of T = 4 glimpses while occasionally exposing CanViT to longer sequences.Temporally dense, per-timestep supervision enables this short-chunk training scheme.

6 Experiments

Experiments evaluate a general-purpose CanViT-B checkpoint across segmentation and classification, testing task, policy, temporal-horizon, and canvas-resolution generalization. CanViT-B substantially improves accuracy–efficiency tradeoffs over prior active models while benefiting from finer output canvases.

  • Experimental setup: Experiments assess frozen-weight feature decodability on ADE20K semantic segmentation and ImageNet-1k classification, plus peak ImageNet-1k performance through full fine-tuning.ADE20K uses linear probes without fine-tuning, while ImageNet-1k uses linear probing and LP-FT fine-tuning.
  • Pretraining: 166 hours on a single H100 produced CanViT-B from random initialization using approximately 1 billion glimpses from 13.2 million ImageNet-21k scenes.Pretraining used average sequence length T = 4 and a 32² canvas resolution, corresponding to 1024 patches.
  • Benchmark results: 45.9% ADE20K mIoU and 84.5% IN1k top-1 accuracy surpass prior bests of 27.6% and 82.2%, respectively.The comparison covers leading active vision models including Saccader, AME, AdaGlimpse, and AdaptiveNN.
  • Benchmark results: 38.5% mIoU in a single glimpse uses 20x fewer FLOPs than AME’s 27.6% mIoU on ADE20K.CanViT-B therefore combines higher single-glimpse segmentation accuracy with substantially lower computation.
  • Generalization: CanViT-B generalizes across policies, temporal horizons, and canvas resolutions even with frozen weights, while C2F improves ADE20K accuracy and efficiency over R-IID and F-IID.EG-C2F provides additional ADE20K efficiency gains, and ADE20K mIoU continues improving through T = 21 glimpses.
  • Canvas resolution: Finer canvases improve frozen CanViT-B ADE20K segmentation from low-resolution full-scene glimpses, outperforming coarser canvases and the input-matched DINOv3-ViT-B teacher.The dual-stream design decouples glimpse resolution from canvas and output resolution, with finer canvases especially beneficial for small ground-truth regions.

7 Conclusion … B.1 Definitions: scene, viewpoint, crop

CanViT advances active vision with a task- and policy-agnostic foundation model whose architecture and learning signal support strong, efficient inference across tasks and viewing policies. The paper also identifies static-scene training as a limitation and formalizes its attention, viewpoint, and crop definitions.

  • 7 Conclusion: CanViT applies the foundation-model playbook to active vision, working across tasks and viewing policies without complex reinforcement-learning pipelines.The results attribute this advance to pairing a carefully designed architecture with an informative learning signal.
  • 7 Conclusion: CanViT is trained and evaluated on static scenes, while its constant-memory recurrent design and computational efficiency motivate future adaptation to real-time video and embodied perception.A lightweight gating mechanism could facilitate forgetting, and reinforcement-learning-based policy learning is identified as a natural extension.
  • A Canvas Attention pseudocode: Canvas Attention reads project backbone queries and outputs while leaving canvas keys and values unchanged; writes project backbone keys and values while leaving canvas queries and outputs unchanged.This implements asymmetric cross-attention in opposite directions.
  • A Canvas Attention pseudocode: Canvas Attention alternates backbone-to-canvas reads and canvas-to-backbone writes across network depth, using scene-relative rotary embeddings for glimpse tokens and fixed embeddings for the canvas.The pseudocode shows backbone processing between successive reads and writes.
  • B Viewpoint Encoding (VPE): A viewpoint encoding represents (x, y, s) as (x/s, y/s, log s), then maps it through 64-dimensional Random Fourier Features and layer normalization.The parameterization provides scale invariance, same-scale translation invariance, and planar isotropy.
  • B.1 Definitions: scene, viewpoint, crop: The scene is a finite 2D domain whose coordinates span [−1, +1]2.This domain underlies the formal viewpoint and crop definitions.
  • B.1 Definitions: scene, viewpoint, crop: A valid viewpoint is (x, y, s) ∈ Vraw, with s ∈ (0, 1] and center constraints |x| ≤ 1 − s and |y| ≤ 1 − s.These constraints ensure the corresponding square crop remains within the scene.
  • B.1 Definitions: scene, viewpoint, crop: The full-scene viewpoint is (0, 0, 1), while (0.5, 0.5, 0.5) covers [0, 1]2; viewpoints whose centers or crop borders exceed the scene are invalid.Thus (2, 2, 0.5) and (0.5, 0.5, 1) are invalid for different boundary violations.

B.2 Finding a scale-invariant representation · B.3 Properties of u

The paper replaces the raw viewpoint triplet with a smooth, injective embedding designed to preserve scale-invariant distances. Its Euclidean geometry also supports same-scale translation invariance and planar isotropy for valid viewpoints.

  • B.2 Finding a scale-invariant representation: The raw (x, y, s) viewpoint representation fails to capture scale invariance, especially because small crops become artificially insensitive to positional and scale shifts.
  • B.2 Finding a scale-invariant representation: The desired representation makes pairwise distances unchanged under global rescaling, while also preserving same-scale translation invariance and planar isotropy.
  • B.2 Finding a scale-invariant representation: The proposed embedding u maps raw viewpoints into a subset of R3, with smooth component functions for s > 0 and an injective inverse.
  • B.3 Properties of u: For globally rescaled valid viewpoints, the Euclidean distance between their embedded representations remains exactly unchanged.
  • B.3 Properties of u: This scale invariance follows because the additive log c term cancels in viewpoint differences, making the difference vectors and their norms equal.
  • B.3 Properties of u: Distances between same-scale viewpoints are unchanged when the same valid planar offset is added to both viewpoints.
  • B.3 Properties of u: Distances are also unchanged under valid planar orthogonal transformations, establishing planar isotropy through linearity and norm preservation.

C CanViT-B pretraining details … D.4 Task: ImageNet-1k classification

The paper specifies CanViT-B’s architecture and pretraining implementation, viewing policies, FLOP accounting, and evaluation procedures for ADE20K segmentation and ImageNet-1k classification. The evaluation protocols span multiple timesteps, canvas resolutions, policies, and frozen or fine-tuned model settings.

  • C CanViT-B pretraining details: CanViT-B uses LayerScale in its ViT backbone, with architectural and ImageNet-21k pretraining hyperparameters documented in Tables 2 and 3.
  • C CanViT-B pretraining details: DINOv3 ViT-B teacher features are precomputed for all 13.2 million ImageNet-21k images at 512^2-pixel resolution and a 32 × 32 patch grid.Images are resized, center-cropped, and processed without augmentation; the export used approximately 8 H100-equivalent capacity.
  • D.1 Viewing policies: F-IID starts with a full-scene viewpoint before random crops, whereas R-IID samples random crops at every timestep, including t = 0.The pretraining setup uses one F-IID and one R-IID rollout per step, with TBPTT chunks of K = 2 glimpses and stop probability p_stop = 0.5.
  • D.1 Viewing policies: C2F traverses scene tiles from coarse to fine, while F2C reverses that order to isolate processing-order effects under identical image coverage.EG-C2F instead visits high-entropy, segmentation-probe-uncertain tiles first, and RFS repeats the full-scene view as a recurrence-only control.
  • D.2 FLOP counting: Each multiply-add counts as two FLOPs, and architecture parameters and fine-grained FLOP curves are sourced or analytically computed using standard PyTorch flop-counter formulas.Total counts were validated against traced and previously reported FLOP counts.
  • D.3 Task: ADE20K segmentation: ADE20K evaluation resizes images and masks to 512 × 512 and reports mIoU across up to T = 21 timesteps by policy, timestep, and canvas grid.Results are reported at 32 × 32 and 64 × 64 grids, with best-t mIoU additionally evaluated for grids c × c where c ∈ {8, 16, 32, 64}.
  • D.4 Task: ImageNet-1k classification: ImageNet-1k classification uses linear probes for five smaller DINOv3 ViT variants and initializes CanViT-B’s recurrent CLS-token head by fusing affine transforms.Because CanViT-B predicts DINOv3 representations, a DINOv3-fitted probe can also be used with CanViT-B.
  • D.4 Task: ImageNet-1k classification: ImageNet-1k fine-tuning follows LP-FT: a frozen classification head is fit first, then all CanViT and head parameters are jointly fine-tuned for 20 epochs.Evaluation uses 512-pixel shortest-side resizing and center cropping, reporting top-1 accuracy across up to T = 21 timesteps and canvas grids c × c where c ∈ {8, 16, 32, 64}.

D.5 Methodology for Figure 5 analyses · D.6 DINOv3 ImageNet-1k linear classification probes

The appendices detail evaluation procedures for canvas-resolution analyses and establish linear-probe protocols for smaller DINOv3 ViT checkpoints on ImageNet-1k. Figure 5 analyses compare frozen CanViT-B and DINOv3 across resolutions, timesteps, mask areas, and IoU, while DINOv3 probes use cached CLS features and hyperparameter search.

  • D.5 Methodology for Figure 5 analyses: Figure 5 A–B evaluates how canvas resolution affects ADE20K segmentation IoU as a function of ground-truth mask area.The analysis uses all validation-set (scene, class) pairs with non-empty masks and computes per-mask IoU for frozen DINOv3 and CanViT-B.
  • D.5 Methodology for Figure 5 analyses: 1282 px input images are used for both models, with CanViT-B evaluated at canvas resolutions 82, 162, 322, and 642, while DINOv3 produces an 82 feature map.DINOv3’s output resolution follows its input resolution and 162 px patch size; each model-resolution pair is evaluated independently.
  • D.5 Methodology for Figure 5 analyses: The pipeline trains four CanViT-B linear probes and one DINOv3 probe, producing one datapoint per ground-truth mask per model across scene, class, mask area, timestep, and IoU.CanViT timesteps span t = 0 to t = 20 under EG-C2F, with t = 0 corresponding to the passive comparison in Table 9.
  • D.5 Methodology for Figure 5 analyses: Figure 5 trends use locally weighted scatter-plot smoothing with fraction 0.25, no iterative reweighting, and 1,000 bootstrap resamples for 95% confidence intervals.The smoothing is applied after selecting the first timestep or a timestep difference, depending on the panel.
  • D.5 Methodology for Figure 5 analyses: CanViT-B’s EG-C2F glimpse budget is swept from t = 0 to t = 20 at canvas resolutions 82, 162, 322, and 642, while DINOv3 ViT-B/16 is evaluated at seven input resolutions.The DINOv3 resolutions are 128, 144, 160, 192, 256, 384, and 512 px, with an independently trained linear probe per resolution.
  • D.6 DINOv3 ImageNet-1k linear classification probes: DINOv3’s smaller ViT checkpoints lack released ImageNet-1k linear heads and standard ImageNet-1k top-1 validation results, motivating probes for all five models.The models are ViT-S/16, ViT-S+/16, ViT-B/16, ViT-L/16, and ViT-H+/16; the released head was limited to the 7B flagship model.
  • D.6 DINOv3 ImageNet-1k linear classification probes: DINOv3 probe features are CLS tokens from the final transformer layer extracted at 512 × 512 resolution, stored in float32 after bfloat16 mixed-precision inference.The extraction uses 1024 patch tokens with 16 × 16 px patches, Inception-crop augmentation for training, and disk caching before probe optimization.
  • D.6 DINOv3 ImageNet-1k linear classification probes: Optuna sweeps probe loss, optimizer, learning rate, batch size, weight decay, and AdamW β parameters to maximize ImageNet-ReAL top-1 validation accuracy.The searched losses are softmax or sigmoid cross-entropy; optimizers are AdamW or SGD, and the reference learning rate spans 10^-7 to 10^-2 log-uniformly.

E Pretraining ablations

Pretraining ablations show that CanViT benefits from frequent bidirectional canvas–backbone interaction, dense latent supervision, F-IID rollouts, and a wide backbone, while the VPE token has minimal impact. Removing canvas-side projections trades individual attention expressiveness for lower overhead and greater interaction or canvas capacity.

  • Capacity–expressiveness trade-offs: Removing canvas-side QKVO projections lowers Canvas Attention overhead, enabling more frequent interactions, larger canvas embeddings, and more canvas patches, but reduces each operation’s expressiveness at fixed dimensionality.The ablation exposes a capacity–expressiveness trade-off between efficient interaction and the expressiveness of individual cross-attention operations.
  • Frequency and directionality of canvas–backbone interaction: Three reads and three writes per glimpse, evenly distributed across 12 backbone blocks, demonstrate the importance of frequent bidirectional Canvas Attention operations.Writes update the canvas and support dense outputs, while removing reads causes a large drop in patch-level and other reconstruction outcomes.
  • Dense latent supervision: Omitting dense supervision reversely affects the expected objective trade-off, despite leaving architecture and raw expressiveness unchanged.Dense supervision is aligned with using frozen canvas features for dense tasks, while its omission could theoretically specialize representations for CLS reconstruction.
  • F-IID rollouts: −9.4% spatial reconstruction follows removal of the F-IID rollout, which also dramatically slows R-IID loss reduction.F-IID guarantees at least one full-spatial-coverage glimpse and exposes the full-scene viewpoint during training.
  • Backbone embedding dimension: −21.1% CLS reconstruction results from reducing backbone width from Dbb = 768 to Dbb = 384, while patch reconstruction declines by −8.4%.The narrower ViT-S-equivalent backbone also produces the largest reductions in parameter count and per-glimpse computational footprint across ablations.
  • VPE token: −0.1% spatial and −0.2% CLS changes result from removing the VPE token, the smallest impact across both policies and loss types.This supports interpreting VPE primarily as an affordance for future end-to-end policy learning rather than a core architectural component.

F Inference latency … H.2 Broader impact

CanViT’s constant-memory design supports fast, resolution-scalable inference, while PCA visualizations expose canvas updates across glimpses. The broader-impact discussion highlights potential applications alongside surveillance-related societal risks.

  • F Inference latency: Figure 8 reports latency scaling with output resolution across individual iterations.The figure presents latency behavior as output resolution changes.
  • F Inference latency: Latency and peak-memory scaling are evaluated across hardware, precision, and output resolution.Benchmarks use an NVIDIA GeForce RTX 4090 and an AMD Ryzen 9 7950X 16-Core Processor, with CUDA float32 or AMP bfloat16 and CPU float32.
  • F Inference latency: Measurements use batch size B = 1, synchronization, warmups, repeated runs, and pooled per-iteration latencies.Each configuration is repeated 3 times, with at least 5 measured iterations and limits of 20 seconds or 500 iterations.
  • G Interpretability: PCA maps groups of three token-space principal components to RGB for visualizing glimpse and canvas patch grids.Min-max scaling across the visible region preserves local detail when visualizing canvas subsets.
  • G Interpretability: Figure 9 visualizes CanViT’s multiple Canvas Attention Write operations and their residual canvas updates within and across glimpses.Intermediate residuals and canvases are captured after each Write operation within a 1282 px glimpse.
  • H Declarations: CanViT’s declarations emphasize a constant-memory recurrent architecture, fast sequential inference, and scalability to large scenes.No separate declaration passage is provided in the supplied material.
  • H.2 Broader impact: These properties could support real-time, high-FPS video processing, embodied active perception, high-resolution image processing, and domain applications.Examples include edge/physical AI, medical imaging, and camera- or satellite-based environmental monitoring.
  • H.2 Broader impact: Derivatives could be incorporated into surveillance and security technology, motivating attention to societal implications in specific applications.The paper invites researchers building on active-vision foundation models to engage with these implications.

H.3 Compute reporting · H.4 Statistical analysis · H.5 Licenses and attribution

The paper reports substantial compute use across pretraining, development, evaluation, and fine-tuning, with statistical results based on 95% bootstrap confidence intervals. It also documents dataset usage and image attributions under specified licenses.

  • H.3 Compute reporting: 2500 H100-equivalent hours were used on the SLURM-based compute platform, including failed runs, preliminary experiments, and hyperparameter sweeps.Pretraining required under 24 GB of GPU memory, including 18.2 GB for the 2M-step CanViT-B run.
  • H.3 Compute reporting: 2000 RTX 4090 GPU-hours were estimated for interactive development, probe training, and downstream evaluations.Fine-grained usage tracking was unavailable for the RTX 4090 workstation.
  • H.3 Compute reporting: Under 800 USD was spent on ImageNet-1k fine-tuning experiments using Google Cloud TPU v6e-4 spot instances.The reported fine-tuning run took under 15 wall-clock hours on TPUv6e-4 with SPMD data parallelism.
  • H.4 Statistical analysis: 95% bootstrap CIs were used for all confidence intervals reported in the paper.For stochastic policies, evaluation runs used fresh policy seeds and bootstrapped means with 10,000 percentile-method resamples.
  • H.4 Statistical analysis: The four stochastic policies were evaluated through full passes over the evaluation set under fresh policy seeds, with confidence intervals plotted as shaded bands.The policies were F-IID, R-IID, C2F, and F2C; the bands were often too tight to be visible in Figures 4 and 6.
  • H.5 Licenses and attribution: CanViT-B was pretrained on ImageNet-21k’s winter21_whole split and evaluated on ImageNet-1k and ADE20K.The cited datasets are identified in the paper’s pretraining and evaluation attribution statement.
  • H.5 Licenses and attribution: The Cat03.jpg example image was sourced from Wikimedia Commons and attributed to Fir0002/Flagstaffotos under the CC BY-NC 3.0 license.The image appears in figures throughout the paper.
  • H.5 Licenses and attribution: Other example images came from the Places365 dataset and were used solely for illustration purposes.This attribution applies to the paper’s additional example images.
Loading 2603.22570v2…