PaperlayerView exact revision on arXiv

Source-linked AI summary

Atomizer-IO: Beyond Pixels, Patches and Grids

Hugo Riffaud de Turckheim, Sylvain Lobry, Nicolas Houdré, Damien Robert, Roberto Interdonato, Diego Marcos

arXiv:2609.40320v1cs.CV

TL;DR

Vision models commonly assume regular grids, while sensing data varies in channels, sampling, timing, and geometry. Atomizer-IO represents measurements with acquisition metadata and derives structure from their physical relationships. It remains competitive across raster and LiDAR benchmarks while supporting inference-time compute control.

  • Problem

    Vision architectures assume regular grids, whereas sensing data varies in physical measurements, spatial resolution, acquisition time, channel availability, and geometry.

    Source: §1 · p. 1

  • Method

    Atomizer-IO represents observations with measurement and acquisition metadata and uses spatially anchored latents whose number scales with input observations.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5

  • Results

    Atomizer-IO remains competitive with grid-based and flexible Earth-observation architectures across eight benchmarks and two LiDAR datasets spanning classification, regression, and 2D/3D segmentation.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Competitive performance under a competitive compute budget does not require computing on a regular grid.

    Source: §5 · p. 9

    Probe agreement is a heuristic rather than a guarantee when reducing inference cost through coarser latents.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Most vision architectures assume that observations lie on a regular grid, an effective abstraction for natural images but a restrictive one for sensing data whose channels, temporal sampling, spatial resolution, and geometry can vary. Generic set-based architectures remove the grid, but also remove useful spatial inductive biases. We introduce Atomizer-IO, an architecture that places observations first and derives structure from their physical relationships. Building on top of an atomic representation of the data, each observation is described by its measurement and acquisition metadata, while local cross-attention maps observations to anchor points that can be arbitrarily placed. We evaluate this design by progressively relaxing the grid assumption, from varying input raster configurations and incomplete channel sets to flexible output density and, ultimately, inputs without a raster grid. Atomizer-IO is competitive with flexible EO-specific architectures on most tasks, while offering post-training control over inference cost and competitive compute--performance trade-offs. The same formulation extends without architectural redesign to unordered 3D point clouds, showing that the atomic interface generalizes beyond regular raster inputs. These results suggest that pixels, patches, and grids do not need to define the interface of a sensing architecture.

1 INTRODUCTION

Fixed-grid vision interfaces fit natural images but constrain sensing data whose measurements, sampling, timing, channels, and geometry vary. Atomizer-IO represents observations first and derives spatial structure from their physical relationships, then evaluates the design as grid assumptions are progressively relaxed.

1 INTRODUCTION

  • Sensing data varies in measurement identity, resolution, acquisition time, channel availability, and geometry, making fixed tensor interfaces restrictive.In Earth observation, channel position does not provide a stable physical identity across sensors.Source: §1 · p. 1
  • Atomizer-IO represents each observation through its measurement and acquisition metadata, then defines locality and relative geometry in physical coordinates.Its atomic encoder–processor–decoder aggregates observations into spatial latents and decodes local latent neighborhoods at requested output locations.Source: §1 · p. 2
  • The evaluation progressively relaxes fixed-grid assumptions, from conventional raster tasks and varying channels to output-query density and inputs without a raster grid.The tested irregular inputs include unordered LiDAR and joint image–point observations.Source: §1 · p. 2
  • A single architecture spans raster and irregular geometries across classification, regression, and 2D and 3D segmentation, with models trained from scratch under matched conditions.The atomic interface makes measurement identity explicit rather than tying it to tensor position.Source: §1 · p. 2

2 RELATED WORK

Related work adapts grid-based models to varying sensing properties, while set-based approaches relax fixed layouts but may lack spatial structure. Atomizer-IO aims to retain geometry while supporting irregular observations and query-defined outputs.

2 RELATED WORK

  • Grid-based architectures exploit locality and regular geometric relationships, but Earth-observation data varies across spatial resolution, spectral bands, and acquisition time.Existing models add dedicated mechanisms for these physical properties, with RAMEN and UniverSat jointly accommodating spectral, spatial, and temporal variation.Source: §2 · pp. 2–3
  • Its architecture locally aggregates atoms into spatial latents, processes them globally, and decodes through spatial output queries using physical coordinates.The same local context-to-query operation is used for encoding and decoding.Source: Figure 2 · p. 3
  • Set-based approaches process unordered observations, while Perceiver IO uses a compact latent bottleneck and query-defined outputs.Presto models individual pixel time series; Atomizer retains atomic measurements as an unordered set.Source: §2 · p. 3
  • Atomizer-IO addresses the tension between grid models’ spatial structure and set models’ flexibility by reasoning over arbitrary Voronoi tessellations.Sensor-specific grid structures are confined to interfaces where observations enter or predictions are queried out.Source: §2 · p. 3

3 METHODOLOGY

Atomizer-IO represents observations as atoms and derives spatial structure from physical relationships. Local cross-attention builds spatial latents and maps them to requested outputs, while global latent processing connects local neighborhoods.

3.1 TOKEN CONSTRUCTION [

  • Each atom combines a measured value with acquisition metadata, while its spatial position is kept separate; raster and LiDAR observations can share this interface.Raster metadata can describe ground sampling distance, spectral support, and date; LiDAR metadata can describe sensor and return properties.Source: §3.1 · p. 4, Figure 3

3.2 SPATIAL SAMPLER

  • The sampler defines receptive fields by spatial proximity, encodes query-relative offsets, and uses local cross-attention; encoder assignments form Voronoi regions, while decoder queries gather nearest contexts.Encoder queries are spatial latents; decoder queries are requested output locations.Source: §3.2 · pp. 4–5, Figure 4 · p. 4

3.3 ENCODER

  • Spatial latents scale with input observations and are uniformly placed; shared MLP and local cross-attention encode assigned atoms, sampling at most m atoms per dense neighborhood.The MLP and cross-attention weights are shared across latents, so parameter count does not depend on the number of latents.Source: §3.3 · p. 5

3.4 PROCESSOR

  • Global self-attention propagates information across spatial and global latents, using RoPE for spatial anchors; optional temporal aggregation encodes timesteps separately before combining them with a learned-token transformer.Alternatively, atoms from all acquisitions can be encoded jointly with time supplied as metadata.Source: §3.4 · pp. 5–6

3.5 DECODER

  • The decoder uses local cross-attention to map latent neighborhoods to requested locations, avoiding uniform evaluation; dense prediction can first condition each query on observations at its target.For dense prediction, target-location observations provide a local pathway alongside information from surrounding latents.Source: §3.5 · p. 6

4 EXPERIMENTS

The experiments progressively relax regular-grid assumptions, from raster Earth-observation tasks to variable input composition and output density, unordered LiDAR, and unseen input geometries. Across these settings, Atomizer-IO remains competitive while exposing flexible input and inference choices.

4.1 PERFORMANCE ACROSS DIVERSE EO TASKS

  • On eight raster tasks, Atomizer-IO ranks first on four spatially explicit tasks and second on two, matching or exceeding grid-based and flexible EO architectures.It has no edge over Perceiver-IO on non-spatially explicit classification tasks such as EuroSAT and ForestNet.Source: Table 2 · p. 7, §4.1 · p. 7

4.2 FLEXIBLE INPUT COMPOSITION: HANDLING MISSING CHANNELS

  • Atomizer-IO achieves the best average performance on Sen1Floods11 and BioMassters and leads in most input configurations, with its advantage nearly disappearing under RGB-only inputs.All models use the same band-dropout protocol; fixed-channel models zero-pad unavailable channels, while Atomizer-IO and UniverSat omit them.Source: Table 3 · p. 7, §4.2 · p. 7

4.3 FLEXIBLE OUTPUT DECODING: INFERENCE-TIME COMPUTE CONTROL

  • A single trained checkpoint spans a range of inference costs: Dense decoding is competitive with flexible EO baselines at high compute, while adaptive queries reduce cost.Zone-probe uses regions associated with latents; Quadtree recursively refines regions when probe predictions disagree, and probe agreement is a heuristic rather than a guarantee of exact dense evaluation.Source: Figure 6 · p. 8, §4.3 · p. 8

4.4 FLEXIBLE INPUT GEOMETRY: UNORDERED LIDAR POINT CLOUDS

  • Atomizer-IO is competitive on DALES and FRACTAL without 3D-specific design beyond its tokenizer, ahead of three DALES 3D baselines and best on FRACTAL.On FRACTAL, orthophoto RGB pixels are additionally encoded as atoms alongside LiDAR points.Source: Table 4 · p. 8, §4.4 · p. 8

FRACTAL

  • 78.4 mIoU on FRACTAL for Atomizer-IO compares with 78.1 for Perceiver-IO and 77.5 for RandLA-Net.The paper notes that image modality alone may already be a strong predictor for this benchmark.Source: FRACTAL · p. 8, Table 4 · p. 8, §4.4 · p. 8

4.5 CONTROLLED ANALYSIS OF SPATIAL STRUCTURE

  • Removing RoPE reduces MNIST accuracy from 99.37 to 29.48, but Sen1Floods11 mIoU only from 93.17 to 92.42.The authors relate the stronger MNIST effect to the task’s reliance on spatial arrangement, while local multispectral and radar observations are already predictive on Sen1Floods11.Source: Table 5 · p. 9, §4.5 · p. 9
  • At 25% retained background pixels, Atomizer-IO has 94.93% MNIST accuracy versus ViT’s 72.17% under unseen input geometries.Both models were trained on complete images, tested with the same surviving observations, and retained all digit pixels.Source: Table 6 · p. 9, §4.5 · p. 9

5 CONCLUSION

Atomizer-IO remains competitive across raster Earth-observation benchmarks and LiDAR datasets while relaxing assumptions about input and output structure. Its atomic interface describes measurements explicitly and introduces locality and geometry through physical relationships between observations.

5 CONCLUSION

  • Across eight Earth-observation benchmarks and two LiDAR datasets, Atomizer-IO remains competitive with grid-based and flexible EO architectures as it relaxes input and output structure.The same encoder processes multispectral rasters and unordered LiDAR without modality-specific processing branches.Source: §5 · p. 9
  • Each atom represents a measurement at a particular place, time, and sensor, without requiring a fixed shape, channel count, or neighboring measurement.Locality and geometry are introduced afterwards through physical relationships between observations.Source: §5 · p. 9
  • The authors suggest Atomizer-IO could be a candidate architecture for a multimodal EO foundation model.They identify reconciling observations that differ in what they measure and how they are sampled as a challenge for large-scale pretraining.Source: §5 · p. 9
  • Pixels, patches, and grids can be treated as particular discretizations of observations, introduced when useful rather than assumed from the start.Source: §5 · p. 10

A SUPPLEMENTARY MATERIAL

  • The encoder aggregates local input-atom neighborhoods into spatial latents, the processor applies latent self-attention with optional temporal aggregation, and the decoder maps local latent neighborhoods to output queries.The decoder can optionally condition first on raw observations at the target location; relative position is introduced locally and weights are shared across spatial neighborhoods.Source: Figure 7 · p. 16

B ABLATION STUDIES

The ablations examine how temporal, spatial, and target-local information contribute to prediction across tasks with differing sensing characteristics.

B ABLATION STUDIES

  • The study ablates temporal organization across acquisitions to assess its contribution to prediction.Source: B ABLATION STUDIES · p. 16
  • The study ablates spatial relationships between latents to assess their contribution to prediction.Source: B ABLATION STUDIES · p. 16
  • The study ablates direct target-local observations in the decoder to assess their contribution to prediction.Source: B ABLATION STUDIES · p. 16

B.1 TEMPORAL AGGREGATION

Temporal aggregation encodes each acquisition into its own spatial latent set, then merges corresponding latents across time. Its benefit on PASTIS grows as more acquisitions are used.

B.1 TEMPORAL AGGREGATION

  • Without temporal aggregation, atoms from all acquisitions are encoded jointly into one temporally mixed latent set; with it, each acquisition gets its own spatial latent set before aligned latents are merged by a temporal transformer.The comparison varies input acquisition count while keeping the remaining training setup fixed.Source: §B.1 · pp. 16–17
  • 0.47 mIoU at three timesteps, 4.85 at six, and 8.58 at twelve: temporal aggregation’s benefit on PASTIS increases with acquisition count.Both conditions process the same observations; the difference is how latent capacity is organized.Source: §B.1 · p. 17, Figure 8 · p. 17

B.2 SPATIAL ROPE ABLATION

Spatial RoPE, target-local observations, and temporal aggregation contribute differently across tasks. The section also details how atomic features, relative positions, and physical scales are encoded.

B.2 SPATIAL ROPE ABLATION

  • 70.3% on MNIST, 13.0% on Cashew, 7.8% on PASTIS, 3.0% on BurnScars, and 0.8% on Sen1Floods11: removing spatial RoPE degrades performance most when local observations are less informative.On PASTIS, spatial RoPE gives a moderate gain, while the temporal aggregation ablation shows a much larger dependence on temporal structure.Source: §B.2 · p. 17, Table 7 · p. 17

B.3 LOCAL OBSERVATION CONDITIONING

  • Removing target-local observation conditioning reduces mIoU by 3.91 points on Sen1Floods11, 1.43 on Cashew, 0.80 on PASTIS, and 0.61 on BurnScars.Sen1Floods11 shows the greatest dependence on target-local evidence among these datasets.Source: §B.4 · p. 18, Table 8 · p. 18, Figure 10 · p. 18

B.4 ATOMIC FEATURE ENCODING

  • The relative importance of target-local evidence, spatial relationships, and temporal organization is task dependent; PASTIS benefits strongly from temporal aggregation but only modestly from target-local observations.The same pattern of task-dependent spatial contribution is reflected in the RoPE results.Source: §B.2 · p. 17, §B.4 · p. 18
  • Continuous quantities use Fourier features, and raster spatial scale is represented by compressing ground sampling distance before encoding it.The raster feature encoding is built from value, spatial-scale, spectral, and time encoders.Source: §B.4 · p. 18
  • Optical bands are represented by central wavelength and bandwidth, while channels without meaningful spectral support use learned embeddings; acquisition time uses periodic day-of-year features or a zero vector when unavailable.The spectral vector depends on both central wavelength and bandwidth.Source: §B.4 · pp. 18–19

B.5 RELATIVE POSITION ENCODING

  • Horizontal coordinates are kept separate from the atomic feature, with each context-to-query offset encoded using Fourier features and concatenated across components.The reference scale is the cross-attention spatial scale.Source: §B.5 · p. 19

B.6 PROPERTIES OF THE COMPRESSION FUNCTION

  • The compression function maps signed inputs to (−1, 1) and non-negative inputs to [0, 1), with magnitude saturating toward one for large inputs.Its reference scale sets the characteristic physical scale before Fourier encoding; the magnitude reaches 0.5 when |w| = s.Source: §B.6 · p. 19

B.7 SCALABILITY OF THE SPATIAL SAMPLER

The spatial sampler assigns each atom to a nearby latent and can limit per-latent attention context to control cost. Adaptive decoding reuses a trained checkpoint to reduce the number of output locations fully decoded, relying on probe agreement as a heuristic.

B.7 SCALABILITY OF THE SPATIAL SAMPLER

  • For Sen1Floods11, matching a global attention budget of 500 context atoms per latent would require discarding 99.987% of atoms; local sampling instead assigns each atom to a spatial cell and subsamples dense cells.Without subsampling, Voronoi assignment reduces input-to-latent interactions from LN to N; the per-latent budget m bounds interactions by min(N, Lm).Source: §B.7 · p. 19, Table 9 · p. 20

B.8 IMPLEMENTATION DETAILS OF THE SPATIAL SAMPLER

  • The complete context for a latent is its Voronoi cell; when the cell exceeds budget m, training samples up to m atoms uniformly without replacement at each step.Repeated sampling exposes the model to different subsets rather than permanently restricting a latent to a fixed subset.Source: §B.7 · pp. 19–20, §B.8 · p. 20

C.1 ADAPTIVE DECODING STRATEGIES

  • Dense decoding independently queries each output location, while Zone-probe and Quadtree use one trained checkpoint and differ in where they evaluate and propagate predictions.Zone-probe uses latent-associated Voronoi regions; Quadtree recursively subdivides regions where probes disagree.Source: §C.1 · p. 21
  • Probe agreement is a heuristic, not a guarantee: small or thin objects may be missed, and savings depend on prediction homogeneity at the relevant region scale.The strategies reduce to dense decoding in the worst case when no region can be treated as homogeneous.Source: §C.1 · p. 21, Table 10 · p. 21

C.2 COMPUTE–PERFORMANCE SENSITIVITY

Compute–performance sensitivity varies with architectural and decoding choices. The section also describes an encoding for LiDAR returns that is independent of sensor-specific maximum return counts and preserves the original return attributes.

C.2 COMPUTE–PERFORMANCE SENSITIVITY

  • Increasing atoms per latent eventually degrades mIoU across Dense, Zone-probe, and Quadtree decoding; increasing encoder attention budget improves performance until saturation, while larger decoder neighborhoods add computation once local context is sufficient.The shared degradation across decoding policies reflects reduced latent representational capacity; Quadtree can reach more aggressive low-cost operating points before quality is affected.Source: §C.2 · pp. 21–22, Figure 11 · p. 22

D ECHO ENCODING FOR LIDAR TOKENS

  • LiDAR return number r and total returns R locate an observation within its pulse’s return sequence, where multiple returns can distinguish surfaces such as buildings or ground from vegetation canopies.Canopies can produce several returns at different depths along the beam path.Source: D ECHO ENCODING FOR LIDAR TOKENS · p. 22

D.1 ENCODING

  • The LiDAR encoding normalizes preceding and subsequent returns by the per-pulse return count, so it works across sensors with different maximum return counts and can be inverted to recover (r, R).The two normalized quantities lie in [0, 1).Source: §D.1 · p. 23, Figure 12 · p. 23

D.2 TRAINING DETAILS

The appendix specifies matched training and capacity choices, Atomizer-IO’s latent and output-query settings, and the MNIST evaluation protocol for sparse inputs.

D.2 TRAINING DETAILS

  • All architectures share task-specific training conditions and, within each dataset, batch size, learning-rate schedule, and epoch budget.Sen1Floods11 and BioMassters use 150 epochs; DALES and FRACTAL use 100.Source: §D.2 · pp. 23, 25, Table 13 · p. 25
  • Model sizes are chosen within similar task-family parameter regimes; Atomizer-IO’s dense single-temporal and temporal-aggregation variants have 33.5M and 34.8M parameters, respectively.The number of spatial latents changes intermediate representation size, not learned parameter count, because weights are shared across latents.Source: §D.2 · p. 23, Table 11 · p. 24
  • Training randomizes target atoms per spatial latent k and maximum neighborhood samples m, varying latent density and local encoder attention budget without changing learned parameters.Latent anchors are placed on a hexagonal lattice over the input’s horizontal extent.Source: §D.2 · p. 25, Table 14 · p. 25
  • Dense-prediction training supervises at most 100,000 spatial output queries per sample, sampling uniformly when the output contains more locations.A 512 × 512 map uses 100,000 sampled queries, while a 128 × 128 output is supervised densely.Source: §D.2 · p. 25

D.3 MNIST SPARSIFICATION PROTOCOL

  • MNIST compares capacity-matched Atomizer-IO and ViT models at pixel-level granularity, then progressively removes only pixels with value v ≤0.5 during evaluation.Both models train on complete images and are not retrained or finetuned for sparse geometries; removed pixels are absent rather than zero-valued tokens.Source: §D.3 · p. 26, Figure 13 · p. 26

D.4 LIDAR TRAINING AND INFERENCE PROTOCOL

The LiDAR protocols adapt Atomizer-IO to DALES and FRACTAL inputs. On DALES, its reported mIoU exceeds Perceiver IO but remains below the strongest task-specific point-cloud methods.

D.4 LIDAR TRAINING AND INFERENCE PROTOCOL

  • For FRACTAL, all co-registered VHR image atoms and 40,000 sampled LiDAR points are concatenated and processed through one atomic encoder, without modality-specific branches.The protocol follows the FRACTAL benchmark’s RandLA-Net training protocol for the LiDAR stream.Source: §D.4 · p. 26
  • DALES scenes are evaluated in overlapping 50 m × 50 m windows at 25 m stride, with predictions for shared points averaged before argmax.Per-class IoU and mean IoU are computed from a global confusion matrix accumulated across scenes.Source: §D.4 · pp. 26–27
  • Atomizer-IO reaches 66.7% mIoU on DALES, substantially above Perceiver IO but below the strongest task-specific point-cloud methods.Table 17 reports per-class IoU alongside point-cloud baselines.Source: §D.4 · p. 27, Table 17 · p. 27
Loading 2609.40320v1…