Source-linked AI summary
BA-T: An Iterative Transformer for Two-View Bundle Adjustment
Ganlin Zhang, Weirong Chen, Daniel Cremers, Xi Wang
TL;DR
Feed-forward 3D reconstruction models rely on heavy attention stacks and lack structured geometric refinement, while classical bundle adjustment is iterative but relies on explicit geometry and costly optimization. BA-T addresses this gap with a lightweight iterative Transformer layer that performs BA-inspired updates in implicit token space. It progressively improves pose and reconstruction accuracy, strengthens cross-view consistency, and remains compact relative to larger decoders.
Problem
Feed-forward models lack structured geometric refinement, while classical optimization relies on explicit geometry and incurs inference overhead that complicates integration with feed-forward architectures.
Method
BA-T uses a single repeatable Transformer layer to refine camera-pose and geometry tokens through BA-inspired information propagation in implicit latent space.
Results
BA-T progressively improves pose and reconstruction accuracy across iterations, achieves stronger multi-view consistency than conventional decoders, and delivers high-accuracy reconstruction efficiently.
Takeaways & Limitations
BA-T provides a compact alternative to deep attention stacks by combining feed-forward prediction with structured iterative geometric refinement.
Takeaways & Limitations
The current implementation and training setup are scoped to two-view reconstruction despite the architecture's stated extension to multiple views.
Abstract
from arXiv · showhide
Feed-forward models for 3D reconstruction have achieved strong performance using deep cross-view attention to exchange information across images. However, these approaches often depend on heavy decoder stacks and lack a structured mechanism for geometry refinement, resulting in poor multi-view consistency. We address this by drawing inspiration from classical bundle adjustment (BA), which can be viewed as an iterative information propagation process between poses and local geometry. Inspired by BA, we propose BA-T, an iterative Transformer that implements BA-style structured updates as a repeatable layer in implicit token space. Instead of relying on deep attention stacks, BA-T refines predictions based on latent residual by a single lightweight layer. Experiments demonstrate that BA-T progressively improves pose and reconstruction accuracy across iterations, achieves stronger cross-view consistency than conventional decoders, and matches or surpasses substantially larger models while using only 16% of their decoder parameters. BA-T provides a compact, efficient, and structural alternative to depth-heavy attention, enabling accurate 3D reconstruction within a lightweight architecture. The code will be made publicly at https://github.com/zhangganlin/BA-T.
1 Introduction
BA-T addresses the gap between feed-forward models with expressive latent tokens but limited geometric refinement and classical optimization that is iterative but depends on explicit geometry. It introduces a compact, reusable Transformer layer for structured pose and geometry refinement in implicit token space.
- Feed-forward reconstruction methods use deep cross-view attention decoders to exchange information across images but lack a mechanism for self-correcting misalignments.
- Classical bundle adjustment iteratively refines camera poses and scene structure through explicit geometric optimization.
- BA-T implements BA-style information propagation as repeatable learnable updates directly in implicit token space.
- BA-T uses a single structured iterative refinement layer instead of deep cross-view attention stacks, enabling compact updates of camera poses and 3D geometry.
- BA-T refines relative camera poses and local geometry at the token level while retaining latent-token expressiveness and feed-forward efficiency.
2 Related Work
Existing reconstruction systems either use iterative classical optimization with substantial inference overhead or deep feed-forward attention stacks that implicitly approximate geometric constraints. BA-T combines feed-forward prediction with structured, repeatable refinement without explicit solver-based optimization.
- Classical bundle adjustment refines camera trajectories and 3D structure iteratively, but solver-based methods require large second-order Hessian construction and inversion during inference.
- Classical optimization can also be less robust in texture-less or low-co-visibility scenarios because it depends on optimization dynamics.
- Feed-forward models use increasingly deep cross-view attention stacks to enforce geometric consistency implicitly, yet lack iterative correction after a forward pass.
- BA-T replaces heavy attention stacks with structured, repeatable refinement steps inspired by bundle adjustment.
- BA-T combines efficient feed-forward prediction with iterative geometric refinement without explicit solver-based optimization.
3 BA-T
BA-T adapts bundle adjustment’s iterative information flow into a learnable Transformer that repeatedly refines camera and local geometry tokens in implicit latent space.
- BA-T: BA jointly refines camera poses and scene geometry by computing residuals and repeatedly propagating information between them.The classical process uses iterative optimization and updates geometry after aggregating coupled residual information into camera updates.
- BA-T: BA-T replaces explicit linear-system solving with a reusable learnable refinement layer that preserves BA’s camera–geometry dependency structure.The layer operates on latent tokens rather than explicit geometry, Jacobians, and matrix inversions.
- BA-T: BA-T initializes camera tokens learnably and geometry tokens from image features, then applies its token update function repeatedly for a variable number of iterations.Each iteration computes latent residuals and updates the camera and geometry tokens accordingly.
- BA-T: Camera-conditioned geometry is transformed into a shared space, matched across views with attention, and converted into latent residuals.The soft correspondence mechanism supports sub-token matches and occlusions, while the residual represents discrepancy between corresponding geometry representations.
- BA-T: The camera update aggregates multiple residual tokens through one-to-many cross-attention, implicitly modeling the Schur-complement information flow.The geometry update uses symmetric cross-view attention so local structure is refined in coordination with the relative camera parameters.
- BA-T: Training supervises point maps, relative pose, and cross-view geometric consistency at every refinement iteration.Later iterations receive higher loss weights to encourage gradual refinement rather than exclusive focus on the final output.
4 Experiments
Experiments show that BA-T improves pose and geometry through iterative refinement, outperforming comparable conventional decoders and correcting cross-view inconsistencies over successive updates. Ablations support the importance of its structured components, while multiview results indicate extensibility beyond two views.
- 4.1 Evaluation of Pose Estimation and 3D Reconstruction: BA-T is evaluated against feed-forward reconstruction models, using pose and geometry metrics on 7Scenes and BundleFusion.The evaluation also analyzes iterative refinement, component ablations, and multiview reconstruction.
- 4.2 Performance under Iterative Refinements: 44% versus 27%: BA-T reduces 3D correspondence error more than ViSTA† with iterative training.Both pose and geometry metrics improve across iterations, with most gains occurring within the first 3–4 iterations before convergence.
- 4.2 Performance under Iterative Refinements: BA-T corrects first-iteration geometric misalignments and progressively improves relative pose estimates through later iterations.Qualitative results show improved alignment and robustness in low-overlap or low-visibility cases.
- 4.3 Ablation: Effectiveness of Components in BA-T: The full BA-T model performs best in ablations, while iterative supervision, correspondence tokens, and geometry transformation each support pose or geometry refinement.Removing geometry transformation degrades both pose and geometry by making cross-view coordinate alignment more difficult.
- 4.4 Extension: BA-T for Multiview Reconstruction: BA-T extends naturally to multiview inputs because cross-view interactions occur in each view’s local space.The current implementation and training mainly focus on two-view reconstruction, leaving broader multiview configurations unexplored.
5 Conclusion
BA-T jointly refines camera poses and 3D geometry with a compact, structured iterative layer. It improves reconstruction across iterations and provides stronger consistency and efficient high-accuracy estimation than conventional stacked-attention decoders.
- BA-T jointly refines camera poses and 3D geometry through an implicit, structured, learnable bundle-adjustment-inspired layer.
- BA-T progressively improves pose and geometry estimates, achieves stronger consistency than standard stacked-attention decoders, and performs efficiently within a compact architecture.
A Architecture and Training Details
The architecture represents each two-view pair with camera and geometry tokens and uses latent-space transformations to support bidirectional refinement. A lightweight learned inverse transformation avoids redundant camera tokens while preserving bidirectional geometric consistency.
- BA-T uses 768-dimensional camera and geometry tokens, with camera-conditioned normalization and cross-view correspondence matching.
- Each two-view pair uses one camera token encoding the relative transformation, although local geometry refinement requires both transformation directions.
- A lightweight MLP learns the inverse transformation in latent space, avoiding redundant camera tokens while preserving bidirectional geometric consistency.
C Solver-based BA Comparison
BA-T outperforms classical bundle adjustment in two-view reconstruction while requiring fewer iterations and faster runtime, despite Droid-SLAM receiving ground-truth intrinsics.
- C Solver-based BA Comparison: BA-T achieves stronger reconstruction performance than classical bundle adjustment, particularly for geometry reconstruction under limited co-visibility.Droid-SLAM uses reprojection-error residuals and Gauss–Newton optimization, while BA-T benefits from expressive latent representations and learned priors.
- C Solver-based BA Comparison: Approximately 5× faster inference and 45% lower memory consumption distinguish BA-T from Droid-SLAM.
- C Solver-based BA Comparison: BA-T reaches stronger performance with fewer iterations and faster runtime than Droid-SLAM, even though Droid-SLAM uses ground-truth intrinsics.
D Running Time and Peak Memory
BA-T provides the most compact and computationally efficient architecture among the compared two-view methods, while maintaining competitive accuracy.
- D Running Time and Peak Memory: 38M parameters make BA-T the most compact architecture among the compared multiview and two-view methods.
- D Running Time and Peak Memory: 1.32 GB is BA-T’s lowest peak GPU memory requirement among the compared methods.
- D Running Time and Peak Memory: BA-T delivers faster inference while maintaining competitive accuracy under identical input images and resolution.The comparison measures decoder size, peak GPU memory, and inference time.
E.1 Comparison with Unrolling and Recurrent Methods
BA-T differs from prior iterative methods by refining implicit latent representations through BA-inspired updates, including camera-conditioned transformations of local geometry.
- E.1 Comparison with Unrolling and Recurrent Methods: BA-T updates implicit latent tokens, unlike BA-Net’s explicit geometric variables, RAFT’s correspondence fields, and ReSplat’s recurrent representations.
- E.1 Comparison with Unrolling and Recurrent Methods: Camera-conditioned geometry transformation maps local geometry tokens into another view’s coordinate frame using the corresponding camera token.
- E.1 Comparison with Unrolling and Recurrent Methods: The transformed and directly regressed point clouds align closely under the pose transformation encoded in the camera token.
E.2 Pose and Geometry Behavior in Experiments
BA-T’s refinement improves pose estimation more consistently than geometry reconstruction, while component visualizations support the intended camera-conditioned transformation behavior.
- E.2 Pose and Geometry Behavior in Experiments: Pose estimation improves more consistently than geometry reconstruction because residual tokens directly influence camera updates, whereas geometry updates follow camera refinement indirectly.
- E.2 Pose and Geometry Behavior in Experiments: Per-view geometry reconstruction forms final scene geometry by combining local predictions with estimated poses rather than jointly optimizing one global representation.
- E.2 Pose and Geometry Behavior in Experiments: Camera-conditioned geometry predictions roughly align under the provided transform, supporting the intended functional role of this learned component.The component is not explicitly trained to exactly match its bundle-adjustment counterpart.
F Multiview Extension Details
BA-T extends its two-view refinement layer to multiview inputs by changing token connectivity according to a view graph, without altering network structure. On four-view BundleFusion data, it maintains comparable accuracy while substantially reducing computational requirements.
- Each view-graph node carries geometry tokens, while each edge carries a relative camera token for multiview refinement.
- Residual tokens are computed per edge from transformed geometry and latent correspondences, then used to update camera tokens independently.
- Local geometry updates aggregate information from all neighboring views after transforming their geometry into a shared coordinate frame.
- The multiview extension reuses the same refinement layer and parameters, with capability emerging solely from changed view-graph token connectivity.
- 39% of MapAnything's peak memory is used by BA-T on four-view BundleFusion inputs while achieving comparable accuracy and faster inference.The evaluation uses 547 four-view samples and includes trajectory, rotation, depth, accuracy, inference-time, and memory metrics.