Source-linked AI summary
SeqAlign3DVG: A Sequence-Aligned Benchmark and Voxel Reasoning Framework for 3D Visual Grounding
Yi Zhang, Yi Wang, Yueting Wu, Kaiyue Yang, Yuejiao Su, Lap-Pui Chau
TL;DR
Existing benchmarks inadequately represent embodied grounding because they rely on global 3D scenes or loosely aligned visual observations, despite the need to ground language in partial visual inputs. SeqAlign3DVG introduces strictly verified single-view and ordered-sequence data with a voxel-based ROVM–PLVF pipeline, achieving state-of-the-art depth-free results, including 51.30%/22.39% Acc@0.25/0.50 on observation sequences.
Problem
Existing benchmarks inadequately represent embodied grounding because they rely on global 3D scenes or loosely aligned visual observations.
Method
SeqAlign3DVG combines strictly verified single-view and ordered-sequence observations with voxel-based ROVM and PLVF reasoning.
Results
51.30%/22.39% Acc@0.25/0.50 is achieved on observation sequences under the depth-free protocol, exceeding BIP3D by 1.25/2.27 percentage points.
Takeaways & Limitations
SeqAlign3DVG enables evaluation of image-based grounding under strict observation alignment and temporal ordering.
Abstract
from arXiv · showhide
Image-based 3D visual grounding is critical for embodied agents, yet existing benchmarks suffer from loose text-observation alignment and neglect temporal ordering. We introduce SeqAlign3DVG, a novel benchmark dedicated to temporally ordered and strictly observation-aligned image-based 3D visual grounding. Unlike prior works using order-agnostic views or global point clouds, SeqAlign3DVG ensures all expressions are human-verified and strictly grounded in the provided RGB observations (single frames or ordered observation sequences). It comprises 9,622 single-view and 14,493 sequence samples featuring rich descriptions, complex relations, and multi-instance ambiguities. To tackle this benchmark, we propose a unified voxel-based pipeline featuring Relevance-Ordered Voxel Memory (ROVM) and Progressive Language-Voxel Fusion (PLVF). ROVM dynamically ranks and aggregates multi-view evidence via a conservative memory to mitigate noisy observations, while PLVF performs coarse-to-fine spatial-linguistic reasoning for precise disambiguation. Our approach achieves state-of-the-art performance under the depth-free protocol, significantly improving localization for targets defined by complex relations and appearance cues.
1 Introduction
Existing 3D visual grounding benchmarks inadequately reflect embodied perception because they rely on global 3D scenes or loosely aligned views. SeqAlign3DVG addresses this with strictly observation-aligned data and a voxel-based framework that achieves state-of-the-art depth-free performance.
- Motivation: Existing benchmarks often assume complete point clouds or meshes, unlike embodied agents’ partial, viewpoint-dependent observations.Image-based grounding instead infers 3D locations from visual observations.
- Motivation: Recent image-based benchmarks commonly attach language to scene-level targets or relations before selecting supporting views, weakening text–observation alignment.They also fall short of representing realistic temporal dynamics.
- Benchmark: SeqAlign3DVG human-verifies expressions against exact single-view or ordered-sequence inputs, while combining appearance cues, spatial relations, and same-class distractors.These properties require fine-grained target disambiguation.
- Method: ROVM ranks views by query relevance and conservatively preserves high-quality voxel evidence while suppressing noisy observations.The pipeline lifts per-view 2D features into 3D voxel volumes and regresses 3D boxes.
- Results: 51.30%/22.39% Acc@0.25/0.50 surpasses BIP3D by 1.25/2.27 percentage points on observation sequences under the depth-free protocol.The method uses neither target phrase-token supervision nor a pretrained 3D detector prior.
- Contributions: The paper contributes the SeqAlign3DVG benchmark, a unified voxel-based pipeline, ROVM, and PLVF for spatial-linguistic disambiguation.The contribution summary also reports state-of-the-art performance under the depth-free protocol.
2 Related Work
Prior 3D grounding benchmarks emphasize reconstructed geometry, while image-based datasets increasingly study partial observations but often lack exact language–observation alignment and temporal ordering. SeqAlign3DVG is positioned as an indoor RGB benchmark that unifies strict alignment with ordered-sequence and single-view grounding.
- Reconstructed-scene benchmarks: Established 3D grounding benchmarks typically use reconstructed indoor scans with explicit point clouds or meshes.Examples include ScanRefer and ReferIt3D, followed by multi-target and expanded task variants.
- Image-based benchmarks: Image-based grounding addresses posed-camera partial observability, but earlier datasets focused mainly on single-view RGB-D or outdoor driving scenarios.EmbodiedScan and MMScan later introduced ego-centric multi-view streams.
- Image-based benchmarks: Embodied multi-view benchmarks do not necessarily verify language against exact evaluation observations and often treat supporting frames as order-agnostic sets.SeqAlign3DVG instead conditions expressions on the exact single-view or temporally ordered input.
- Method landscape: Geometry-based methods commonly use instance-level contextual modeling, relation reasoning, multi-view projections, detection-transformer formulations, or foundation models.These approaches generally assume explicit 3D geometry or target closed-set/open-world variants.
- Image-based benchmarks: RGB-only grounding avoids depth input at inference but must infer 3D cues from visual observations, with many prior monocular methods tailored to outdoor driving.BIP3D supports indoor posed-image grounding with optional depth-map input.
- Positioning: SeqAlign3DVG’s depth-free framework uses shared voxel lifting, ROVM, and PLVF without target phrase-token supervision or a pretrained detector prior.The comparison specifically contrasts these choices with BIP3D’s depth-free variant.
3 The SeqAlign3DVG Benchmark
SeqAlign3DVG constructs single-view and temporally ordered sequence samples whose language is tied to the exact evaluated observations. Human verification enforces observability and unique identifiability, while diagnostic subsets expose distractor and anchor-related difficulty.
- Benchmark design: SeqAlign3DVG combines strict observation alignment, ordered sequences, and unified single-view and sequence grounding.Its expressions are written and verified against the exact observation unit paired with each sample.
- Construction: Single-view samples use one sufficiently visible RGB frame, whereas sequence samples use temporally contiguous clips preserved in natural order.Language is attached to the selected observation unit rather than a scene-level annotation or post-hoc support-view retrieval.
- Quality control: Human verification accepts samples only when targets are uniquely identifiable and all appearance cues, anchors, and spatial relations are visually supported.Evidence outside the evaluation input is disallowed, and violating samples are rewritten.
- Construction: Figure 2a pairs candidate views or sequences with draft descriptions before human verification against strict observability criteria.Figure 2b summarizes difficulty levels and anchor categories.
- Quality control: Approximately 59% of sequence drafts and 41% of single-view drafts are rewritten during independent blind verification.At least two annotators check each sample.
- Dataset scale: The benchmark contains 9,622 single-view and 14,493 observation-sequence samples, with many queries combining appearance, anchors, relations, and same-class distractors.It is designed to be diagnostic rather than merely large.
- Diagnostic subsets: Difficulty subsets use same-class distractor counts, while anchor subsets use the number of distinct anchor categories in each expression.Easy, Medium, and Hard correspond to 0, 1–2, and ≥3 distractors.
4 Methodology
The method converts RGB observations into 3D voxel representations, aggregates sequences with query-aware relevance ordering, and progressively fuses language for final 3D grounding. Its design preserves reliable evidence while focusing detailed reasoning on promising spatial candidates.
- 2D-to-3D Voxel Feature Lifting: The pipeline lifts each RGB observation into a per-view 3D voxel volume using camera projection, retaining separate volumes for sequence inputs.For single-view grounding, the single lifted volume is used directly; sequences are passed to ROVM for aggregation.
- Relevance-Ordered Voxel Memory: ROVM ranks valid views by query relevance using target-aware descriptors and reliability cues before aggregation.The cues include projection coverage, target-focus concentration, and sequence position.
- Relevance-Ordered Voxel Memory: ROVM uses a compact target slot derived from sentence-level text and pooled voxel context to estimate voxel-wise target priors for view comparison.Voxel features are projected into text space and matched against the target slot to produce sigmoid target-support scores.
- Relevance-Ordered Voxel Memory: Relevance-ordered conservative memory refinement lets confident views establish the target state while later views selectively fill missing details without overwriting reliable evidence.The refined memory is blended with a robust all-view baseline so uncertain regions remain close to the average representation.
- Progressive Language–Voxel Fusion: PLVF performs coarse-to-fine language–voxel reasoning by forming a coarse target hypothesis before revisiting full text tokens on the most promising voxels.This progressive design keeps expensive token-level reasoning off the full voxel grid and focuses it on a cleaner candidate set.
5 Experiments
Experiments evaluate SeqAlign3DVG on single-view and observation-sequence grounding under protocol-aware depth conditions, including component ablations. The method achieves the best depth-free performance in both settings, while PLVF improves accuracy and efficiency and ROVM strengthens sequence aggregation.
- Evaluation Protocol: Acc@0.25 and Acc@0.50 measure the percentages of predicted boxes whose IoU with ground truth exceeds 0.25 and 0.50, respectively.
- Evaluation Protocol: The experiments report single-view and observation-sequence results, with explicit labels distinguishing depth input, depth supervision, phrase supervision, and detector priors.Depth-free performance excludes ground-truth depth input and depth supervision.
- Main Results: The method attains the best depth-free performance in both single-view and observation-sequence settings.Methods using ground-truth depth input or depth supervision are reported separately for transparency.
- Main Results: 51.30%/22.39% Acc@0.25/0.50 on observation sequences exceeds BIP3D with detector initialization by 1.25/2.27 percentage points without target phrase-token supervision or a pretrained 3D detector prior.The BIP3D comparison is 50.05%/20.12% under the stated alternative protocol.
- Main Results: 44.54% Acc@0.25 and 16.86% Acc@0.50 are achieved on single-view grounding without ground-truth depth input or supervision.Grounding DINO + Back-project reaches 38.25%/8.52% while using ground-truth depth maps at inference.
- Ablation Study: PLVF improves Acc@0.25 by 6.29 points on single-view and 4.63 points on sequence grounding.It also reduces single-view computation by 35.7%, latency from 63 to 59 ms, and peak memory from 3026 to 2810 MB versus dense full-grid cross-attention.
- Ablation Study: ROVM raises sequence performance from 46.09%/20.40% to 51.30%/22.39%, while relevance-ordered writing exceeds reverse writing by 0.96/0.52 points.
6 Conclusion
The paper introduces SeqAlign3DVG as a strictly observation-aligned indoor benchmark for grounding from single views and temporally ordered sequences, together with a unified voxel-based framework. ROVM supports query-aware sequence aggregation, PLVF supports efficient coarse-to-fine language–voxel reasoning, and experiments show state-of-the-art depth-free performance.
- SeqAlign3DVG provides a strictly observation-aligned benchmark for single-view and temporally ordered observation-sequence 3D grounding.
- The unified voxel-based framework uses ROVM for query-aware sequence aggregation and PLVF for efficient coarse-to-fine language–voxel reasoning.
- Extensive experiments demonstrate state-of-the-art performance under the depth-free protocol.