Source-linked AI summary
Learning Multi-Scene Absolute Pose Regression with Transformers
Yoli Shavit, Ron Ferens, Yosi Keller
TL;DR
Absolute pose regressors are fast and lightweight but generally less accurate than hierarchical methods and usually embed only one scene per model. The paper introduces Transformers that separately aggregate positional and orientational cues while encoding scene-specific information through decoder queries. It reports new state-of-the-art accuracy for both multi-scene and single-scene absolute pose regression across indoor and outdoor benchmarks.
Problem
Absolute pose regressors are fast and lightweight but less accurate than hierarchical pipelines, and most require a separate model for each scene.
Method
The method uses separate positional and orientational Transformer Encoders to aggregate CNN activation maps and Transformer-Decoders to encode scene-specific information.
Results
The approach achieves new state-of-the-art accuracy for both multi-scene and single-scene absolute pose regression across indoor and outdoor benchmarks.
Takeaways & Limitations
Self-attention aggregates positional and rotational cues while the architecture embeds multiple scenes in parallel.
Abstract
from arXiv · showhide
Absolute camera pose regressors estimate the position and orientation of a camera from the captured image alone. Typically, a convolutional backbone with a multi-layer perceptron head is trained with images and pose labels to embed a single reference scene at a time. Recently, this scheme was extended for learning multiple scenes by replacing the MLP head with a set of fully connected layers. In this work, we propose to learn multi-scene absolute camera pose regression with Transformers, where encoders are used to aggregate activation maps with self-attention and decoders transform latent features and scenes encoding into candidate pose predictions. This mechanism allows our model to focus on general features that are informative for localization while embedding multiple scenes in parallel. We evaluate our method on commonly benchmarked indoor and outdoor datasets and show that it surpasses both multi-scene and state-of-the-art single-scene absolute pose regressors. We make our code publicly available from https://github.com/yolish/multi-scene-pose-transformer.
1. Introduction
Absolute pose regressors offer fast, lightweight camera localization but typically sacrifice accuracy and require a separate model for each scene. This work introduces a Transformer formulation for multi-scene regression and reports state-of-the-art accuracy across indoor and outdoor benchmarks.
- Motivation: Absolute pose regressors estimate camera pose in one forward pass, making them faster and lighter than hierarchical localization pipelines.Hierarchical pipelines achieve state-of-the-art accuracy but require hundreds of milliseconds, substantial memory, and client-server connectivity.
- Problem: Most absolute pose regressors use a CNN global latent vector and train one model per scene.A dataset containing N scenes therefore requires N models to be trained, deployed, and selected during inference.
- Approach: The proposed formulation uses Transformers for multi-scene absolute pose regression.The contribution replaces the conventional single-scene paradigm with a Transformer-based multi-scene formulation.
- Approach: Self-attention aggregates positional and rotational image cues for localization.The method uses separate attention mechanisms for position- and orientation-informative features from convolutional activations.
- Results: The approach achieves new SOTA accuracy for both multi-scene and single-scene APRs across indoor and outdoor benchmarks.The reported result covers contemporary localization benchmarks rather than a single dataset or scene.
2. Related Work
Camera pose estimation methods differ in their inference inputs and algorithmic trade-offs. Retrieval, 3D-based, and attention-based approaches improve localization through additional data, correspondence estimation, or learned feature weighting, while many still use scene-specific models.
- Image Retrieval: Image retrieval methods find nearby database images using global descriptors and support hierarchical and relative pose estimation.Retrieved images can provide vicinity estimates or support subsequent pose estimation procedures.
- 3D-Based Methods: DSAC and DSAC++ estimate image-pixel 3D coordinates to establish 2D-3D correspondences for PnP-RANSAC.These methods require only the query image at inference time and achieve state-of-the-art accuracy, but still require one model per scene.
- Attention-Based APRs: Attention-based absolute pose regression applies self-attention to CNN backbone outputs before pose regression with an MLP head.This changes the representation used for regression while retaining the broader CNN-plus-head paradigm.
3. Multi-Scene Absolute Camera Pose Regression with Transformers
The proposed multi-scene APR uses separate Transformer encoder-decoder streams for position and orientation, aggregating activation maps and producing scene-specific pose predictions in parallel. Scene classification selects the relevant latent outputs before MLP regression, while joint pose and scene losses train the model.
- Network architecture: Separate positional and orientational Transformer Encoders adaptively aggregate flattened activation maps from a convolutional backbone.The model samples the backbone at two resolutions, using distinct activation maps for position and orientation regression.
- Network architecture: Separate Transformer Decoders query learned scene embeddings and output latent position and orientation representations for all scenes in parallel.The decoder design replaces autoregressive prediction with parallel scene-wise outputs using encoder-decoder attention.
- Scene selection and regression: The model classifies the query scene by concatenating positional and orientational outputs, then selects the highest-probability scene before pose regression.The selected scene representations are passed to separate MLP heads that regress position or orientation.
- Network architecture: Learned two-dimensional spatial encodings preserve activation-map locations after projection and flattening into Transformer-compatible sequences.Independent one-dimensional encodings represent the X and Y axes and are concatenated for each spatial position.
- Training objective: Training combines position and orientation losses with a learned balance and adds negative log likelihood for ground-truth scene classification.The pose loss uses learned parameters to control the relative weighting of position and orientation, while the additional NLL term supervises scene selection.
- Implementation: The implementation uses a pre-trained EfficientNet-B0 backbone and six-layer, four-head Transformer encoders and decoders.The Transformer input dimension is 256, with GELU nonlinearities, dropout p = 0.1, and two-layer MLP components.
4. Experimental Results
The experiments evaluate MS-Transformer on indoor and outdoor benchmarks, compare it with multi-scene and single-scene APRs, and analyze attention maps and architectural choices. Results show consistent comparative gains, reliable scene selection, interpretable attention, and sensitivity to backbone and activation-map design.
- Experimental Setup: The evaluation uses Cambridge Landmarks for outdoor localization and 7Scenes for indoor localization, reporting median position and orientation errors.Cambridge Landmarks contains six urban scenes, while 7Scenes contains seven small-scale office scenes.
- Comparative Analysis of APRs: MS-Transformer consistently outperforms MSPN across outdoor and indoor scenes, reducing both position and orientation errors.The comparison reports median position/orientation errors for Cambridge Landmarks and 7Scenes.
- Comparative Analysis of APRs: When trained across both datasets, MS-Transformer remains competitive and outperforms a single-scene model despite some degradation from combining distinct environments.The model also achieves 98.9% average scene-classification accuracy, supporting decoder-output selection.
- Attention Maps Visualization and Interpretation: Training on seven scenes improves attention to corner-like positional cues and elongated orientational edges compared with training on three scenes.Encoder attention visualizations use the Chess scene from 7Scenes and show more informative cues with more training scenes.
- Attention Maps Visualization and Interpretation: Decoder activations are strongest for the query image’s corresponding scene, as illustrated by the OldHospital example.Each activation corresponds to a scene, and the OldHospital activation is significantly stronger for an OldHospital input.
- Ablation Study: EfficientNet backbones outperform ResNet50, EfficientNetB1 performs best, and combining coarse and fine activation maps yields the best reported configuration.The backbone comparison notes a trade-off: deeper models may improve accuracy at the expense of memory and runtime.
5. Conclusions
The paper presents a transformer-based formulation for multi-scene absolute pose regression that separates general image information from scene-specific information. It achieves state-of-the-art localization accuracy for both single- and multi-scene regression across indoor and outdoor datasets.
- Two Transformer Encoders use self-attention to aggregate positional and orientational cues from CNN activation maps.The encoders attend separately to information useful for position and orientation.
- Transformer-Decoders encode scene-specific information and query it separately for each scene.
- The approach achieves new state-of-the-art localization accuracy for single- and multi-scene absolute pose regression across indoor and outdoor datasets.