PaperlayerView exact revision on arXiv

Source-linked AI summary

VLMs Can Describe, But Not Measure: Object-Centric Scene Understanding for Robotic Manipulation

Enrico Saccon, Tommaso Faraci, Iñigo De La Ossa Zarzuelo, Luigi Palopoli, Marco Roveri, Matteo Saveriano

arXiv:2609.28184v1cs.ROcs.CV

TL;DR

Robotic manipulation in unseen environments needs both semantic understanding and reliable metric information, but VLMs are less reliable at geometric estimates. This paper combines VLM annotations with depth-grounded object regions from a single RGB-D observation; the framework preserves comparable semantic performance while reducing geometric error relative to direct VLM inference.

  • Problem

    VLMs provide strong semantic understanding but are less suited to precise metric information needed for physical interaction in previously unseen environments.

    Source: §I · p. 1

  • Method

    The framework separates semantic interpretation from metric geometry, combining VLM descriptions of segmented entities with depth measurements in a task-independent object-centric representation.

    Source: IRGB, IDEPTH · p. 3, §VI · p. 8

  • Results

    With SAM3, average per-configuration median depth error falls from 192.0 mm with direct VLM inference to 71.3 mm with DA3 and mask-median aggregation, a 62.8% reduction.

    Source: D. Depth Techniques Ablation · p. 6

  • Takeaways & Limitations

    The representation can serve as an interface to symbolic and language-based planning frameworks, and the paper demonstrates its integration in a real-robot sorting task.

    Source: §I · p. 1

    Future work will investigate richer geometric representations and evaluate perception-error propagation to task-level performance in more complex, less structured environments.

    Source: §VI · p. 8

Abstract

from arXiv · show

Robotic operation in previously unseen environments requires both semantic understanding and reliable metric information. While vision--language models (VLMs) provide strong semantic capabilities, their geometric estimates remain less reliable. In this paper, we propose a VLM-driven, modular perception framework for scene understanding using off-the-shelf approaches. Starting from a single RGB-D observation, the scene is segmented into object-level regions, annotated by a VLM, and grounded with depth information to construct a task-independent object-centric representation. Experiments on 151 tabletop scenes show that the proposed decomposition preserves strong semantic performance while substantially improving localization and depth estimation over direct VLM inference. The resulting representation is also integrated with a task-planning framework for robotic execution.

I. INTRODUCTION

Robotic manipulation in unseen environments needs semantic understanding and metric geometry, capabilities that VLMs and geometric methods provide unevenly. The paper combines them in a task-independent object-centric representation for downstream planning.

I. INTRODUCTION

  • VLMs provide strong open-vocabulary semantics but less reliable metric information, while geometric methods recover 3D structure with weaker semantics.The introduction frames this mismatch as a barrier to robotic manipulation in previously unseen environments.Source: §I · p. 1
  • The framework separates VLM-driven semantic interpretation from geometric estimation by grounding object segments with depth from a single RGB-D observation.It uses off-the-shelf segmentation and multimodal models without scene-specific training or fine-tuning.Source: §I · p. 1
  • The resulting representation combines semantic and geometric information in a task-independent form that can interface with symbolic and language-based planning frameworks –[9].The paper demonstrates integration with a real-robot sorting task.Source: §I · p. 1

A. Learning and Foundation Models for Robotic Manipulation

  • BridgeData provides large-scale manipulation trajectories across diverse environments; these approaches improve behavioral generalization but primarily learn observation-to-action mappings from previously collected robot data, rather than explicit reusable representations of unseen scenes.Language-conditioned methods combine semantic reasoning with geometric information, but their representations are task-conditioned.Source: A. Learning and Foundation Models for… · p. 2

B. Geometric and Open-Vocabulary Scene Understanding

The paper situates its approach among open-vocabulary and geometric scene-understanding methods, emphasizing a compact representation from one RGB-D observation rather than persistent mapping or task-specific perception.

B. Geometric and Open-Vocabulary Scene Understanding

  • Single-view depth prediction can be ambiguous and need not be metric, while RGB-D sensing provides direct metric measurements useful for manipulation.Multi-view observations can alleviate single-view limitations but do not fit settings with only one observation.Source: B. Geometric and Open-Vocabulary Scene… · p. 2
  • Transformer-family methods, provide powerful tools for point-cloud semantic segmentation and perform well on target data distributions, but their vocabulary and generalization remain limited; this work targets open-vocabulary understanding without scene-specific training.Source: B. Geometric and Open-Vocabulary Scene… · p. 2
  • Foundation models can separate class-agnostic entity localization from semantic labeling, and this work couples region-level semantics with metric RGB-D measurements for physical interaction.The approach processes depth independently while retaining RGB foundation models for semantic reasoning.Source: B. Geometric and Open-Vocabulary Scene… · p. 2

C. Open-Vocabulary 3D Representations

  • Unlike persistent 3D mapping and open-vocabulary segmentation approaches, this work extracts a compact object-centric representation from one RGB-D observation with natural-language semantics and directly measured metric information.The cited approaches include ConceptFusion [23], ConceptGraphs, OpenMask3D, and Open3DIS.Source: C. Open-Vocabulary 3D Representations · pp. 2–3

D. Structured Representations for Task Planning

The paper separates perception from task reasoning: a single RGB-D observation yields a generic object-centric representation that can be interpreted for different tasks and used by planning approaches.

D. Structured Representations for Task Planning

  • Unlike task-specific assembly work, the proposed perception is separated from task reasoning and constructs a generic representation without object-specific perception models or scene-specific fine-tuning.The resulting knowledge base can be interpreted for the task at hand and interfaced with higher-level planning approaches [8].Source: D. Structured Representations for Task… · p. 3

A. Scene Understanding Pipeline

  • The target is a task-independent, object-centric representation of a previously unseen workspace constructed from one RGB-D observation..Source: A. Scene Understanding Pipeline · p. 3
  • The pipeline decouples semantics and metric geometry: a VLM describes retained scene entities while their image-space masks associate them with depth measurements.The branches are fused into a unified task-independent object-centric representation for downstream reasoning and robotic tasks.Source: IRGB, IDEPTH · p. 3

B. Scene Decomposition

Scene decomposition produces candidate binary masks, filters unsuitable regions, and uses retained masks as the shared image-space reference for semantic and geometric grounding.

B. Scene Decomposition

  • A class-agnostic segmentation model produces an initial set of binary masks from the RGB image.The initial proposals are filtered to remove small or redundant segments, background regions, and dominant planar structures such as floors, walls, or supporting surfaces.Source: B. Scene Decomposition · p. 3
  • Each retained segment is converted into an object-focused observation by applying its binary mask to the RGB image..Source: B. Scene Decomposition · p. 3
  • The segmentation mask serves as the common image-space reference for associating semantic and geometric information with the same physical entity.The method prefers the mask over the object-focused crop for that association.Source: B. Scene Decomposition · p. 3

C. Context-Aware Semantic Annotation

The VLM receives both the full scene and a localized object region so it can use scene context while describing a specific entity. The framework separately derives object geometry from valid depth measurements within the same mask.

C. Context-Aware Semantic Annotation

  • The annotation combines the complete scene for context with a localized object region that identifies the entity being described.Descriptions may include identity, visual attributes, functional properties, affordances, and qualitative relationships; metric quantities are recovered separately from depth.Source: C. Context-Aware Semantic Annotation · p. 3

D. Geometric-Semantic Association

  • The representative object depth is the median of valid depth measurements inside the object mask.Using the median reduces the influence of isolated erroneous measurements and pixels incorrectly assigned near segmentation boundaries.Source: D. Geometric-Semantic Association · pp. 3–4
  • The object mask supplies image-space localization, which is combined with representative metric depth to define the object's geometric information.The same formulation can accommodate metric monocular depth estimation when measured depth is unavailable.Source: D. Geometric-Semantic Association · p. 4

E. Object-Centric Scene Representation

The framework combines each object's segmentation mask, semantic description, and geometric information into an object-level representation. The task-independent representation can be queried for downstream reasoning and planning.

E. Object-Centric Scene Representation

  • Each object-level representation combines its segmentation mask, semantic description, and geometric information.The mask is the common image-space reference linking the semantic and geometric branches; the masked observation is used only as visual input to annotation.Source: E. Object-Centric Scene Representation · p. 4
  • The resulting scene representation can be queried to identify relevant entities, reason over their properties and relationships, and retrieve metric information for planning or manipulation.Because it is constructed independently of the task, the same representation can be reused across different instructions.Source: E. Object-Centric Scene Representation · p. 4
  • The framework is modular: segmentation, annotation, and depth-estimation backends can be replaced independently while preserving the object-level interface.Source: E. Object-Centric Scene Representation · p. 4

IV. DATASET

The evaluation dataset contains 151 tabletop scene configurations captured with an RGB-D camera. It varies object identity, number, and arrangement, and uses marker-augmented images only to create reference annotations.

IV. DATASET

  • 151 tabletop scene configurations were collected with an Orbbec Femto Mega camera, with 1920 × 1080 RGB images and corresponding 640 × 576 16-bit depth images.The depth observations are registered to the RGB images because the sensors have different formats and are not directly pixel-aligned.Source: §IV · p. 4
  • The dataset includes 26 distinct objects and scenes containing one to five task objects, with variation in object identity, appearance, size, shape, and spatial arrangement.Repeated instances of the same objects at different locations support consistency evaluation across scene configurations.Source: §IV · p. 4
  • ArUco-augmented images provide reference identities and geometry but are never input to the perception framework; models are neither trained nor fine-tuned on the evaluation images.This establishes a zero-shot, task-independent evaluation setting.Source: §IV · p. 4

V. EXPERIMENTAL EVALUATION

The evaluation compares direct VLM predictions of object semantics and geometry with a framework that separates VLM-based semantics from specialized geometric estimation. Direct VLMs show strong semantic recognition but fall short on depth estimation.

A. Evaluation Protocol and Metrics

B. VLM-Only Baseline

  • The evaluation dataset visual includes clean RGB observations, corresponding ArUco-augmented images, and associated depth observations.The baseline prompts each model with a single RGB image to jointly predict object classes, image-space bounding boxes, and metric distance, without fine-tuning or in-context demonstrations.Source: Figure 3 · p. 5, B. VLM-Only Baseline · p. 5

A. Evaluation Protocol and Metrics

  • Geometric evaluation measures image-space localization error in pixels and depth error in millimeters against ArUco references, using median error and RMSE for localization and median absolute error and RMSE for depth.Depth metrics include only matched objects with a valid depth estimate.Source: A. Evaluation Protocol and Metrics · pp. 4–5

B. VLM-Only Baseline

  • GPT 5.4 reaches 95.9% recall, while the tested VLMs fall short when estimating object depth.GPT 5.4, GPT 5.4-mini, and Sonnet 4.6 also provide 2D position estimates with median errors comparable to the full pipeline.Source: B. VLM-Only Baseline · pp. 5–6

C. Pipeline experiments

The experiments show that adding specialized perception tools greatly improves geometric estimates while preserving semantic performance comparable to direct VLM inference. The pipeline uses segmentation to ground VLM annotations and depth measurements on the same objects.

C. Pipeline experiments

  • SAM3 used noun-phrase preconditioning from the same VLM used for annotation, while SAM1, SAM2, and FastSAM generated automatic class-agnostic masks before VLM annotation.The VLM annotation used both the full scene and each object crop as context, without task-specific demonstrations.Source: C. Pipeline experiments · p. 6
  • Position error was measured relative to each mask’s bounding box, and object depth was the median of depth values within the mask.The experiments used depth metadata from RGB-D images.Source: C. Pipeline experiments · p. 6
  • Across six SAM3 VLM backends, median depth error fell from 192.0 mm with direct VLM inference to 12.1 mm with the pipeline, a 179.9 mm (93.7%) reduction.SAM3 with GPT 5.4 achieved 94.1% recall, compared with 95.9% for its VLM-only baseline.Source: C. Pipeline experiments · p. 6
  • The experiments support the framework’s central claim: VLMs annotate semantics effectively, while specialized tools provide more reliable localization and depth without sacrificing semantic accuracy.The comparison is between direct VLM inference of annotations and depth and the pipeline’s separate semantic and geometric processing.Source: C. Pipeline experiments · p. 6

D. Depth Techniques Ablation

The depth ablation finds that mask-median aggregation is more reliable than sampling the bounding-box center, both with measured RGB-D and predicted depth. Predicted depth with mask aggregation also improves on direct VLM estimates.

D. Depth Techniques Ablation

  • The ablation compared measured RGB-D and DA3 depth using mask-median extraction against sampling the pixel at each mask’s bounding-box center.The study also evaluated whether the complete object region improves on a single representative image location.Source: D. Depth Techniques Ablation · p. 6
  • Mask-median extraction reduced median depth error from 15.8 mm to 13.2 mm with RGB-D data and from 100.1 mm to 69.9 mm with DA3.The reductions were approximately 16.7% and 30.1%, respectively; the comparison is consistent across evaluated segmentation and annotation combinations.Source: D. Depth Techniques Ablation · p. 6, Table III · p. 7
  • With SAM3 across six VLM backends, DA3 plus mask-median aggregation reduced median depth error from 192.0 mm to 71.3 mm (120.7 mm, or 62.8%) and RMSE from 272.0 mm to 131.5 mm (140.6 mm, or 51.7%).Measured RGB-D yielded the most accurate geometric estimates; DA3 remains useful when an RGB-D sensor is unavailable.Source: D. Depth Techniques Ablation · pp. 6–7

E. Robot Demonstration

The robot demonstration applies the perception pipeline to sorting tasks, while the conclusion presents the resulting representation as task-independent and suitable for downstream robotic execution. The authors identify richer geometry and task-level error propagation in more complex environments as future work.

E. Robot Demonstration

  • The robot demonstration used SAM3 with GPT 5.4, measured RGB-D depth, and mask-median extraction; its annotations and geometric data generated a behavior tree through framework [8], executed via ROS2.The workflow began from a natural-language task description and an RGB-D image captured with an Orbbec Femto Mega camera.Source: E. Robot Demonstration · p. 7
  • Three tests used different object arrangements, including sorting blocks into colored trays and a sequence that placed one block on another.The first test is depicted in Figure 4.Source: E. Robot Demonstration · pp. 7–8

VI. CONCLUSION

  • The framework constructs a task-independent, object-centric representation of previously unseen workspaces from a single RGB-D observation.Segmentation identifies entities, VLMs annotate them, and specialized geometric tools associate metric spatial information.Source: §VI · p. 8
  • The conclusion reports that the framework preserves comparable semantic performance while substantially reducing geometric error, and that mask aggregation is more robust than a bounding-box-center measurement.It also identifies monocular depth estimation as an alternative when RGB-D sensing is unavailable.Source: §VI · p. 8
  • Future work will examine object pose and complete shape estimation, and how perception errors propagate to task performance in more complex, less structured environments.Source: §VI · p. 8

Related papers

Loading 2609.28184v1…