PaperlayerView exact revision on arXiv

Source-linked AI summary

Translating Images into Maps

Avishkar Saha, Oscar Mendez Maldonado, Chris Russell, Richard Bowden

arXiv:2110.00966v2cs.CV

TL;DR

Instantaneous semantic BEV mapping is difficult because images must be converted into a spatial overhead representation for dynamic, previously unseen scenes. The paper formulates this conversion as scanline-to-ray sequence translation and implements it with a constrained end-to-end transformer network. It reports state-of-the-art results across three large-scale datasets, including relative gains of 15% on nuScenes and 30% on Argoverse over prior best methods.

  • Problem

    Instantaneous semantic BEV maps are needed for autonomous-driving scenarios involving moving objects and scenes visited only once.

  • Method

    The paper uses a physically grounded transformer formulation that translates vertical image scanlines into polar BEV rays within an end-to-end image-to-map architecture.

  • Results

    15% relative improvement over STA-S on nuScenes and 30% relative improvement over PON on Argoverse are reported, alongside state-of-the-art results across three large-scale datasets.

  • Takeaways & Limitations

    The constrained, physically grounded formulation achieves state-of-the-art instantaneous mapping across three established datasets.

Abstract

from arXiv · show

We approach instantaneous mapping, converting images to a top-down view of the world, as a translation problem. We show how a novel form of transformer network can be used to map from images and video directly to an overhead map or bird's-eye-view (BEV) of the world, in a single end-to-end network. We assume a 1-1 correspondence between a vertical scanline in the image, and rays passing through the camera location in an overhead map. This lets us formulate map generation from an image as a set of sequence-to-sequence translations. Posing the problem as translation allows the network to use the context of the image when interpreting the role of each pixel. This constrained formulation, based upon a strong physical grounding of the problem, leads to a restricted transformer network that is convolutional in the horizontal direction only. The structure allows us to make efficient use of data when training, and obtains state-of-the-art results for instantaneous mapping of three large-scale datasets, including a 15% and 30% relative gain against existing best performing methods on the nuScenes and Argoverse datasets, respectively. We make our code available on https://github.com/avishkarsaha/translating-images-into-maps.

I. INTRODUCTION

The paper frames instantaneous semantic BEV mapping as an image-to-world translation problem. A physically grounded, constrained transformer formulation directly predicts static and dynamic BEV maps from monocular images.

  • BEV maps provide a convenient low-dimensional representation for navigation and obstacle-aware autonomous driving.The paper emphasizes on-the-fly generation because objects move and scenes may be visited only once.
  • Existing approaches either use dense depth and segmentation intermediates or resolve depth and semantics implicitly, without explicitly learning image-to-BEV interactions.
  • The proposed formulation aligns vertical image scanlines with polar BEV rays and treats mapping as image-to-world translation.This makes projective geometry implicit while allowing attention to model scanline–ray interactions.
  • The end-to-end model takes a monocular image and its intrinsic matrix as input and predicts semantic BEV maps for static and dynamic classes.
  • The paper contributes a 1D sequence-to-sequence formulation and a horizontally convolutional, spatially aware transformer designed for data efficiency.

II. RELATED WORK

Prior BEV methods transform image information through detection, projection, compression, or lifting strategies. These approaches provide useful priors but can lack global 3D reasoning, spatial awareness, or efficient contextualization.

  • Early BEV detection methods independently generated 3D object proposals, limiting global scene reasoning.OFTNet instead projected a 3D voxel grid into the image and performed detection over the resulting 3D features.
  • Semantic BEV mapping methods progressed from flat-world homographies to approaches using depth and segmentation to lift objects into BEV.The flat-world assumption creates artifacts for dynamic objects such as cars and pedestrians.
  • Other methods reason about depth and semantics implicitly, either using camera geometry or learning the transformation directly.
  • Compression methods condense vertical image features into a bottleneck, while lift methods expand pixels into frustums with learned depth distributions.Compression can overlook small distant objects, whereas lift gives each pixel the entire image as context and may increase redundancy-driven overfitting.
  • Transformers use attention for sequence alignment, but BEV ray sequences require spatial awareness and permutation equivariance because their order is spatially meaningful.

III. METHOD

The method learns an end-to-end mapping from monocular images and camera intrinsics to semantically segmented BEV maps. Its pipeline separates image representation, image-to-BEV transformation, and BEV segmentation.

  • The model takes a monocular image I and intrinsic matrix C, then predicts binary BEV variables Y_k for each semantic class.
  • The neural network Φ is trained to resolve both semantic and positional uncertainties.
  • The end-to-end architecture constructs image-plane features, transforms them into BEV, and semantically segments the resulting BEV representation.The three subtasks respectively encode semantics and depth knowledge, perform the transformation, and decode BEV features into semantic maps.

A. Image-to-BEV Translation

Image-to-BEV translation exploits the geometric correspondence between vertical image scanlines and polar BEV rays. Inter-plane attention assigns image context to radial positions, followed by self-attention that reasons globally along each ray.

  • Image-to-BEV Translation: Camera geometry provides a one-to-one correspondence between each vertical image scanline and an associated BEV polar ray.
  • Image-to-BEV Translation: Each image column is translated into a BEV ray containing discretized radial depths up to r metres from the camera.The column and ray are treated as input and output sequences whose positions represent semantic objects along the ground plane.
  • Image-to-BEV Translation: The model learns scanline-to-ray alignment with inter-plane attention and then applies polar-ray self-attention for global positional reasoning.
  • Inter-plane attention: Soft alignment assigns each radial BEV element a context vector formed from a combination of image-column features and radial position.Soft assignment preserves features useful for resolving semantics and depth instead of forcing a premature hard assignment.
  • Inter-plane attention: The encoded image memory and radial positional queries are projected before attention computes alignment between image elements and radial positions.
  • Inter-plane attention: Each radial slot independently gathers relevant image-column information, producing an initial assignment of image components to BEV locations.Because assignment represents a distribution of depths, image context is decoupled from distance to the camera.
  • Polar ray self-attention: A nonlinear function g operates across the entire polar ray to enforce globally coherent spatial feature distributions.Independent context vectors can otherwise yield scattered object-part suggestions rather than shapes that are locally or globally congruent.
  • Transformer extension: The attention formulation extends to transformer encoder-decoder layers by replacing key projections with value projections of the memory or context.

B. Infinite lookback monotonic attention

The model constrains image-to-BEV alignment with monotonic attention and infinite lookback, reflecting the typical ordering between image height and scene depth. This combines a hard monotonic assignment with soft attention over preceding image features.

  • B. Infinite lookback monotonic attention: Monotonic attention enforces the typical urban relation that depth increases as image height increases.This constrains radial-depth intervals according to spatial ordering.
  • B. Infinite lookback monotonic attention: MAIL combines a hard monotonic alignment with soft attention extending from the selected source position to earlier memory entries.The soft extension preserves context that a hard assignment alone could neglect.
  • B. Infinite lookback monotonic attention: For each polar-ray position, the decoder scans memory entries from the previous selected index and assigns probabilities to stopping or advancing.Training uses the expected context because hard assignment is not differentiable.
  • B. Infinite lookback monotonic attention: Infinite lookback represents a distribution over image elements below a point, while flipping the image column provides the corresponding above-point distribution.The context vector is then computed from the selected directional distribution.

C. Model architecture

The architecture translates multi-scale image features into polar BEV features and then decodes them into semantic maps. Its restricted one-dimensional attention and horizontal convolution structure preserve spatial organization while supporting efficient training.

  • Core architecture: The model combines a CNN backbone, encoder-decoder transformers, and a BEV segmentation network.These components extract image features, translate them into BEV, and decode semantic occupancy maps.
  • Image-plane processing: A feature pyramid produces multi-scale image representations to handle scene elements at varying depths and scales.The backbone provides feature maps at multiple resolutions.
  • Image-plane processing: One-dimensional transformer encoders reshape each feature map into vertical image-column sequences and add fixed sinusoidal positional encodings.Self-attention models long-range vertical dependencies at each feature scale.
  • BEV transformation: Each decoder converts an encoded image column into a BEV polar ray using positional queries and multi-head attention.Replacing soft attention with monotonic attention yields the constrained alignment variant.
  • BEV transformation: Shared projection matrices across translations make the transformer convolutional along the horizontal x-axis and data-efficient during training.The resulting polar sequences are concatenated and converted into a rectilinear BEV grid.
  • Polar adaptation: Polar-adaptive positional encodings add angular information to the image-plane encoder or BEV-plane decoder.The design addresses the otherwise polar-angle-agnostic spatial relationship modeled by the vertical transformation.
  • BEV segmentation: A convolutional BEV encoder-decoder produces semantic occupancy grids and is trained with multi-scale Dice loss.The convolutional decoder helps connect discontinuities between adjacent polar rays after rectilinear resampling.
  • Spatial and temporal context: Horizontal context is provided through convolutional locality, while optional axial attention adds temporal context across multiple timesteps.The temporal module can be omitted for a purely spatial model.

IV. EXPERIMENTS AND RESULTS

The experiments evaluate the image-to-BEV translation formulation on nuScenes and compare the approach with existing methods across three large-scale datasets. nuScenes also supports ablations because it is larger and contains more object categories.

  • IV. EXPERIMENTS AND RESULTS: The study evaluates the translation formulation on nuScenes and compares it with state-of-the-art methods on nuScenes, Argoverse, and Lyft.The evaluation includes ablations of lookback direction, horizontal context, and polar positional information.
  • Datasets: nuScenes contains 1000 20-second clips from Boston and Singapore with 3D bounding boxes and vectorized road maps.The authors follow prior data-generation, class, and split choices for fair comparison.
  • Implementation: The implementation uses a pretrained ResNet-50 feature-pyramid frontend and 100×100 BEV feature maps with 0.5m² per pixel.The spatiotemporal model processes four images sampled at 6Hz and trains end-to-end with Adam for 40 epochs.

A. Ablation studies

Ablations examine directional lookback, long-range horizontal context, and polar positional encoding. Looking downward helps depth reasoning, whole-image context is unnecessary, and polar encodings improve performance across classes.

  • Lookback direction: Looking downward from an image point outperforms looking upward, while looking in both directions produces the highest accuracy.The authors relate downward context to where objects intersect the ground plane.
  • Long-range horizontal dependencies: Long-range horizontal context does not improve the model and is slightly detrimental.The constrained formulation already receives sufficient context through frontend convolutions, while longer attention sequences may be harder to train.
  • Polar-adaptive transformers: Adding polar encodings to either plane improves over a polar-agnostic transformer, with the largest dynamic-class gains from single-plane additions.Adding encodings to both planes further improves results, especially for static classes.

B. Comparison to state-of-the-art

The method is compared with compression and lift baselines across nuScenes, Argoverse, and Lyft, with stronger reported performance on the evaluated datasets. The comparisons include qualitative nuScenes results and dataset-specific evaluation settings.

  • Baselines: The evaluation compares against compression approaches on nuScenes and Argoverse and lift approaches on nuScenes and Lyft.The baselines use the train/validation splits specified for prior work; Lyft lacks a canonical split, preventing an exact comparison.
  • nuScenes: 15% mean relative improvement over STA-S is reported on the nuScenes validation set.The largest gains occur for smaller dynamic classes, including buses, trucks, trailers, and barriers, which improve by 35–45% relative.
  • Evaluation protocol: Qualitative nuScenes comparisons apply the ground-truth visibility mask to predictions and follow the prior work’s colour scheme.This procedure is used for fair comparison with the reported baselines.
  • Argoverse: 30% relative improvement over PON is reported on Argoverse.The passage describes similar improvement patterns to those observed on nuScenes.
  • Lift comparisons: The spatial model outperforms LSS and FIERY on nuScenes and Lyft.FIERY uses the lift approach; the authors attribute the difference partly to constrained, spatially aware translations between scanlines and rays.

V. CONCLUSION

The paper presents transformer-based mapping from images and video sequences to overhead maps or BEV representations. Its physically grounded, constrained formulation is supported by ablations and achieves state-of-the-art instantaneous mapping results across three established datasets.

  • Approach: The paper maps images and video sequences to overhead maps or bird’s-eye-view representations using transformer networks.The approach is described as a novel use of transformers for this mapping task.
  • Analysis: The formulation combines physical grounding and architectural constraints with monotonic-attention ablations.These ablations examine whether context above or below an image point is more important for map generation.
  • Results: The formulation obtains state-of-the-art results for instantaneous mapping on three established datasets.The conclusion reports this result without specifying dataset names in the supplied passage.
Loading 2110.00966v2…