PaperlayerView exact revision on arXiv

Source-linked AI summary

From Alignment to Fusion in 3D Vision-Language

Xueqi Qiu, Xingyu Miao, Jingjing Deng, Haoran Duan, Yang Long, Ling Shao

arXiv:2609.28222v1cs.CV

TL;DR

Unified 3D vision-language systems combine heterogeneous scene representations, but shared prediction architectures do not guarantee latent compatibility, and unconstrained adaptation may distort feature geometry. The paper aligns corresponding features before orthogonal re-parameterisation and Adaptive Fusion. It reports competitive or state-of-the-art performance across tasks on eight datasets, while noting limits in training scope, efficiency, and evaluation coverage.

  • Problem

    Representation-specific encoders can leave heterogeneous features without guaranteed latent compatibility, and adapting them unconstrained may alter their encoded geometry.

    Source: §I · p. 1

  • Method

    The framework aligns corresponding point-cloud, voxel, and multi-view features, applies geometry-preserving orthogonal re-parameterisation, and combines them through Adaptive Fusion.

    Source: §I · pp. 1–2

  • Results

    The model achieves competitive or state-of-the-art performance across tasks on eight datasets spanning segmentation, grounding, question answering, and dense captioning.

    Source: §VII · p. 12

  • Takeaways & Limitations

    Ablations support distinct roles for alignment and orthogonal re-parameterisation, with their combination strongest in visual grounding, though neither component dominates every metric.

    Source: §VI · p. 11

    The method relies on pretrained representation-specific encoders, adds inference cost, and is evaluated on eight largely indoor-scene datasets, leaving robustness to outdoor scenes, missing representations, and sensor corruption untested.

    Source: §VI · p. 12

Abstract

from arXiv · show

Unified 3D vision-language systems must combine complementary geometry, scale, and appearance cues while supporting tasks from instance segmentation to language-guided reasoning. Existing methods often process point clouds, voxel grids, and multi-view images independently; directly combining these heterogeneous representations may leave substantial feature discrepancy unresolved, while subsequent unconstrained adaptation may distort their internal geometry. We propose an align-then-fuse framework that first applies triple pairwise cosine alignment to establish segment-level correspondence across the three representations and then retrieves task-conditioned features with a prompt-guided query decoder. Before fusion, representation-specific query features are transformed by learnable mappings constrained to the special orthogonal group. These mappings preserve inner products and Euclidean distances within each representation, permitting controlled representation-specific re-parameterisation without arbitrarily distorting its internal geometry. The transformed features are subsequently combined through Adaptive Fusion under downstream task supervision. Experiments cover eight datasets for instance segmentation, visual grounding, question answering, and dense captioning. Compared with PQ3D, the model improves average precision by 3.2 points on ScanNet200 and grounding accuracy by 2.9, 10.6, 4.6, and 4.1 points on ScanRefer, Nr3D, Sr3D, and Multi3DRefer, respectively, while also improving performance on ScanQA, SQA3D, and Scan2Cap. Ablations further support the complementary roles of alignment and orthogonal re-parameterisation and the effectiveness of Adaptive Fusion.

I. INTRODUCTION

Unified 3D vision-language research spans tasks from segmentation to language-conditioned reasoning, but combining heterogeneous representations does not itself ensure latent compatibility. This paper proposes aligning point-cloud, voxel, and multi-view features before controlled adaptation and fusion.

I. INTRODUCTION

  • Shared prediction architectures do not guarantee latent compatibility among features produced by representation-specific encoders.The paper identifies representation heterogeneity and latent-basis discrepancy as related challenges in unified 3D vision-language learning.Source: §I · p. 1
  • Triple Alignment establishes pairwise correspondence among spatially matched point-cloud, voxel, and multi-view features before prediction.Representation-specific orthogonal re-parameterisation then preserves intra-representation geometry before Adaptive Fusion.Source: §I · pp. 1–2
  • Evaluation covers eight datasets spanning instance segmentation, visual localization, question answering, and dense description.The authors report that their method outperforms the baseline across these experiments.Source: §I · p. 2

A. Query-Based 3D Vision-Language Learning

  • Query-based prediction uses latent queries conditioned by prompts as a shared interface for diverse downstream outputs.Prior work extends query-based approaches from object prediction to mask prediction, language-guided segmentation, and multimodal prediction.Source: A. Query-Based 3D Vision-Language… · p. 2

B. 3D Multimodal Fusion

Point clouds, voxels, and multi-view images encode complementary scene information, but independent encoders produce feature spaces with no inherent guarantee of compatibility. Prior fusion methods combine representations through projection, augmentation, or interaction, while this section also describes the encodings and prompt-conditioned query decoder used in the framework.

B. 3D Multimodal Fusion

  • Point clouds preserve geometric observations, voxel grids impose regular spatial structure, and multi-view images provide dense appearance information.These representations arise from different sampling processes and are typically handled by independently designed encoders.Source: B. 3D Multimodal Fusion · p. 2
  • Earlier fusion methods project image features, augment point features, or use cross-modal attention and deformable interaction.Their interaction mechanisms are typically designed around a particular representation pair or downstream setting.Source: B. 3D Multimodal Fusion · pp. 2–3

III. PRELIMINARIES

  • The framework encodes point clouds by segmenting and sampling points, voxels with a sparse convolutional U-Net, and multi-view images through back-projected pixel embeddings.Voxel and image features are pooled or averaged into segment-level representations.Source: §III · p. 3
  • A Transformer-style query decoder aggregates scene information, interacts with task prompts, and sends refined queries to task-specific prediction heads.Textual prompts include categories, referring expressions, and questions; numerical prompts include 3D boxes and locations.Source: §III · p. 3

IV. METHOD

The method separates cross-representation correspondence from representation-specific adaptation before fusion. It aligns features for corresponding scene segments, then combines them through controlled re-parameterisation and Adaptive Fusion.

A. Problem Formulation •

  • Independently encoded features can have distinct distributions and latent parameterizations, while unconstrained adaptation may distort their internal geometry.The paper calls the remaining mismatch after alignment latent-basis discrepancy.Source: A. Problem Formulation • · pp. 3–4
  • The align-then-fuse strategy aligns point-cloud, voxel, and multi-view features, applies orthogonal re-parameterisation, and combines them through Adaptive Fusion.The fused representation is passed to multi-task prediction heads.Source: A. Problem Formulation • · p. 4, Figure 3 · p. 4

B. Alignment for Representation Heterogeneity

  • Triple Alignment applies pairwise cosine alignment to corresponding scene-segment features from image, point-cloud, and voxel representations.The objective encourages compatible directions before task-conditioned decoding without requiring identical representation distributions.Source: B. Alignment for Representation… · pp. 4–5

C. Orthogonal Fusion with Re-parameterisation

The section applies geometry-preserving, representation-specific transformations to aligned query features, then learns how to combine them for downstream prediction.

C. Orthogonal Fusion with Re-parameterisation

  • SO(D)-constrained adapters transform each representation's query features while preserving within-stream distances and inner products.The adapters avoid the arbitrary scaling, shearing, and anisotropic distortion possible with unconstrained linear mappings.Source: C. Orthogonal Fusion with… · p. 5
  • A differentiable QR-based projection supplies orthogonal adapters, with each modality's feature then rotated by its mapping.The QR factor is converted to a proper rotation, preserving the representation's internal geometry.Source: C. Orthogonal Fusion with… · p. 5
  • Triple Alignment precedes independently learned transformations, whose outputs are jointly fused and optimized under the same task objective.This combines explicit pre-fusion correspondence with implicit task-driven coordination among the transformed representations.Source: C. Orthogonal Fusion with… · p. 5
  • Adaptive Fusion learns the relative contributions of the three transformed streams under downstream supervision, rather than assigning equal weights.Its convex-combination weights avoid unbounded feature-scale amplification.Source: C. Orthogonal Fusion with… · p. 6

A. Datasets, Metrics, and Implementation

The evaluation covers eight public benchmarks across four 3D vision-language task families, using benchmark protocols and a two-stage training procedure.

A. Datasets, Metrics, and Implementation

  • Eight benchmarks cover instance segmentation, visual localization, question answering, and dense description, with metrics following each benchmark's protocol.ScanNet200 uses AP measures; grounding uses standard accuracy splits; ScanQA, SQA3D, and Scan2Cap use their listed language and answer metrics.Source: A. Datasets, Metrics, and Implementation · p. 7
  • Training uses two stages: 800 epochs of ScanNet200 instance segmentation, then 50 epochs on the full training set with all objectives.Ground-truth mask guidance is used for the first 200 epochs of stage one; the model uses 120 queries and runs on four NVIDIA A100 GPUs.Source: A. Datasets, Metrics, and Implementation · p. 7
  • Optimization uses AdamW with a 1 × 10^-4 learning rate, batch size 16, β1 = 0.9, β2 = 0.98, and cosine decay.Training also follows PQ3D's geometric and color augmentations.Source: A. Datasets, Metrics, and Implementation · p. 7

B. Qualitative Results

Qualitative examples illustrate language-conditioned scene understanding across instance segmentation, grounding, question answering, and dense captioning.

B. Qualitative Results

  • For instance segmentation, predictions respond to explicit category prompts and functional descriptions across the illustrated scenes.Examples include “Printer,” “Drawer,” and “I want to wash fruit.”Source: B. Qualitative Results · p. 7
  • For spatially ambiguous grounding expressions, predictions align more closely with ground-truth regions than PQ3D's.The examples include “the black trash can under the table” and “the first chair from the right.”Source: B. Qualitative Results · pp. 7–8
  • Question-answering examples more consistently identify answer-relevant objects for viewpoint-dependent directions, while captions better preserve object identity, attributes, and spatial relations.The caption examples distinguish a vending machine from a refrigerator and describe a washing machine's appearance and room location.Source: B. Qualitative Results · p. 8

C. Quantitative Results

The reported results improve over prior methods on instance segmentation, visual grounding, question answering, and dense captioning, with gains varying across benchmarks and metrics.

a) Instance Segmentation

  • ScanNet200 Average Precision rises from 27.0 to 30.2, with AP50 increasing from 38.9 to 41.6 and AP25 from 46.3 to 48.8.Head-class AP reaches 52.7, surpassing PQ3D by 16.9 points; common and tail AP rise from 24.2 and 20.0 to 26.7 and 21.4.Source: C. Quantitative Results · pp. 8–9

b) Visual Grounding

  • Grounding accuracy improves over PQ3D by 2.9, 10.6, and 4.6 points on ScanRefer, Nr3D, and Sr3D, while Multi3DRefer average accuracy rises from 50.1 to 54.2.On Multi3DRefer, ST and MT increase from 43.6 and 40.9 to 48.2 and 46.5; ZT reaches 59.3, above PQ3D’s 57.7 but below 3DJCG’s 66.9.Source: C. Quantitative Results · p. 9

c) Question Answering

  • ScanQA achieves the best overall results across exact-match and generation metrics, while SQA3D average accuracy reaches 48.7 versus the previous best of 48.5.On ScanQA’s test-with-object split, EM@1 rises from 27.0 to 28.2; SQA3D also leads on ‘Is’ and ‘Other’ categories at 63.9 and 50.8, without improving every question category uniformly.Source: C. Quantitative Results · pp. 9–10

d) Dense Captioning

  • Scan2Cap outperforms PQ3D on all four metrics, including BLEU-4 at 39.4 versus 36.0 and ROUGE at 63.5 versus 57.9.CIDEr rises from 80.3 to 81.2 and METEOR from 29.1 to 30.4.Source: C. Quantitative Results · p. 10

D. Ablation Study

The ablations support complementary roles for Triple Alignment, geometry-preserving Orthogonal Fusion, and Adaptive Fusion, while showing that no fusion rule wins every metric.

a) Ablation on Triple Alignment and Orthogonal Fusion

  • Triple Alignment alone improves grounding over baseline by 2.1, 5.1, 0.8, and 3.0 points across ScanRefer, Nr3D, Sr3D, and Multi3DRefer, with further gains when Orthogonal Fusion is enabled.With Orthogonal Fusion enabled, adding Triple Alignment improves the four benchmarks by 1.3, 1.9, 0.6, and 2.3 points; feature distributions also become more consistent.Source: D. Ablation Study · p. 10
  • Orthogonal Fusion alone improves ScanNet200 AP, AP50, and AP25 by 2.8, 2.4, and 1.3 points, respectively, with gains also reported across grounding, QA, and captioning.Its transformations adapt representations while preserving within-representation pairwise distances and inner products; neither component consistently dominates every individual metric.Source: D. Ablation Study · p. 11

b) Ablation for Fusion Methods

  • Adaptive Fusion performs best overall across evaluated tasks, but full concatenation leads on Multi3DRefer at 55.1 and on Scan2Cap CIDEr and ROUGE at 81.6 and 63.8.Adaptive Fusion uses normalized learnable coefficients optimized under downstream task supervision; full concatenation generally exceeds element-wise addition.Source: D. Ablation Study · p. 11

E. Model Complexity Comparison.

The proposed model has modestly higher measured complexity and inference time than PQ3D, with the added cost attributed mainly to Orthogonal Fusion and Adaptive Fusion.

E. Model Complexity Comparison.

  • Compared with PQ3D, the model uses 254.95M rather than 247.88M parameters and 26.94 rather than 25.81 GFLOPs, with forward-pass time rising from 0.1340 s to 0.1648 s.The additional inference cost mainly arises from representation-specific orthogonal re-parameterisation and Adaptive Fusion operations.Source: E. Model Complexity Comparison · p. 11

VI. DISCUSSION AND LIMITATIONS

The discussion distinguishes alignment from geometry-preserving re-parameterisation and bounds the findings to tested indoor-scene benchmarks and pretrained feature pipelines. The conclusion presents an align-then-fuse architecture and reports competitive or state-of-the-art performance across eight datasets.

VI. DISCUSSION AND LIMITATIONS

  • Alignment improves pre-decoding correspondence, while Orthogonal Fusion preserves within-representation distances and inner products; their combination is strongest in the visual-grounding ablation.Ablations across other tasks show that neither component dominates every individual metric.Source: §VI · p. 11
  • The method relies on pretrained, representation-specific encoders and does not establish fully end-to-end cross-representation learning; evaluation is limited to eight largely indoor-scene datasets.Robustness to outdoor scenes, missing representations, and sensor corruption remains untested.Source: §VI · p. 12

VII. CONCLUSION

  • The conclusion describes a unified architecture integrating point clouds, voxel grids, and multi-view images, with competitive or state-of-the-art results across tasks from segmentation to reasoning.Triple Alignment establishes segment-level correspondence; orthogonal re-parameterisation preserves within-representation pairwise geometry before Adaptive Fusion under task supervision.Source: §VII · p. 12

Related papers

Loading 2609.28222v1…