M3SunAgent: Monocular 3D Spatial Understanding Agent for Metric Depth Estimation and 3D Visual Grounding

Motivation

• Instance-level depth and 3D grounding provide complementary spatial information: physical distance on one hand, and object position, dimensions, and orientation on the other.Both require linking a language description to an image and estimating quantitative spatial properties.Source: §I · p. 1

Research gap

• Scene-level depth maps do not directly provide the metric distance of a language-referred object, while specialized grounding models may lack flexibility for other spatial queries.Spatial VLMs may also require substantial 3D training data and computation, and can struggle to estimate metric distance.Source: §I · p. 2

Approach

• M3SunAgent converts natural-language queries into executable programs that coordinate visual tools and geometric operations for both spatial tasks.For depth, it localizes the target and aggregates selected-point estimates; for grounding, it combines VLM attributes with back-projection and 2D-to-3D lifting.Source: §I · p. 2

Evaluation and contributions

• The paper introduces M3SI, a 2,910-sample benchmark for instance-level metric depth, and reports that M3SunAgent outperforms spatial VLM baselines across evaluations on M3SI and Mono3DRefer.The introduction describes its performance as competitive with task-specific vision methods.Source: §I · p. 2

A. Monocular metric depth estimation

• Scene-level depth maps do not directly provide the distance to a language-identified object, and a single point estimate may be inadequate for extended or partially occluded targets.A target region can also contain background, motivating reliable target measurements and methods that reduce unreliable samples’ influence.Source: A. Monocular metric depth estimation · p. 2

B. Monocular 3D visual grounding

• Monocular 3D grounding recovers a referred object’s 3D location and extent from one image and language, but existing specialized methods may lack flexibility for other spatial queries.The cited methods include Mono3DVG, cross-modal interaction, dimension-decoupled text encoding, and VLM-based approaches.Source: B. Monocular 3D visual grounding · p. 3

C. LLM Agents for Spatial Understanding

• Prior LLM agents plan subtasks, compose visual operations, or coordinate spatial tools, while their application to monocular metric depth and 3D grounding remains underexplored.M3SunAgent addresses this gap with a unified framework coordinating visual models and geometric tools.Source: C. LLM Agents for Spatial Understanding · p. 3

A. Overview of M3SunAgent

• A single RGB image and natural-language query feed a planner that selects depth estimation or 3D grounding, then passes the resulting spatial estimate to the LLM for response formulation.Intermediate results remain associated with the same referred object across operations.Source: A. Overview of M3SunAgent · p. 3

B. Spatial Visual Programming

• The LLM generates a DSL program from the query, and an interpreter executes its operators through registered visual tools and geometric computations.The LLM is called again to combine the computed distance or 3D box with the query; separating operators from implementations allows tools to be added or replaced.Source: B. Spatial Visual Programming · pp. 3–5
• The two workflows coordinate different operators: depth uses LOCATE, DEPTH, and AGGREGATE, while grounding uses GROUND, BACKPROJECT, LIFT, and BUILD.The depth workflow aggregates point predictions into an instance-level distance; the grounding workflow combines a projected center, depth, size, and yaw into a 3D box.Source: Figure 1 · p. 4, B. Spatial Visual Programming · p. 4, Figure 2 · p. 5

LOCATE

• LOCATE finds the referred instance with an open-vocabulary detector and returns its 2D box to focus subsequent depth sampling on the target.DEPTH then invokes a registered model using the image, target box, and camera intrinsics.Source: C. Instance-Level Metric Depth… · p. 5

DEPTH

• The DEPTH operator normalizes effective focal length and samples nine target-box points for metric depth prediction, avoiding reliance on a single location.The pattern includes a center, four axial points, and four points near the corners.Source: C. Instance-Level Metric Depth… · p. 5

AGGREGATE

• AGGREGATE removes non-finite and negative values, skips outlier removal when fewer than four valid depths remain, and returns the median of retained estimates.The filtering addresses background pixels and unreliable samples near boundaries or occluded regions; retained values are constrained by quartile-based filtering.Source: C. Instance-Level Metric Depth… · pp. 5–6

D. Monocular 3D Visual Grounding Pipeline

• The grounding program runs GROUND, BACKPROJECT, LIFT, and BUILD to produce a target 3D bounding box in camera coordinates.The interpreter executes the operations and passes intermediate results between them.Source: D. Monocular 3D Visual Grounding… · p. 6
• GROUND uses a VLM to predict the target’s 2D box, projected 3D center, and depth, using semantic cues in the description.The description may convey relative position, distance, or relationships with other objects.Source: D. Monocular 3D Visual Grounding… · p. 6
• BACKPROJECT uses the projected center, depth, and camera intrinsics to recover a 3D center that adapts to different camera configurations.The projected center is the image-plane projection of the target’s 3D center, not necessarily the center of its 2D box.Source: D. Monocular 3D Visual Grounding… · p. 6
• LIFT estimates dimensions and 3D rotation, while BUILD combines the center, dimensions, and yaw into the final box.The lifting tool uses a specialized 2D-to-3D model; the paper notes it reduces the 3D geometric information that the VLM must predict directly.Source: D. Monocular 3D Visual Grounding… · pp. 6–7

1) Datasets:

• M3SI adapts ETH3D into a 2,910-sample benchmark, using median Euclidean distances converted from sampled axial depths as instance-level ground truth.Its annotation pipeline identifies and describes candidate objects, obtains their boxes, samples depth values within them, and converts axial depth using camera intrinsics.Source: 1) Datasets · p. 7
• Mono3DRefer has 29,990 training, 5,735 validation, and 5,415 test samples; the training split fine-tunes the target-localization VLM.Source: 1) Datasets · p. 7
• The M3SI examples show target objects in blue boxes, with descriptions and ground-truth camera distances in meters beneath each image.Source: Figure 4 · p. 7, 1) Datasets · p. 7
• Mono3DRefer test subsets cover target uniqueness, depth bands of Near, Medium, and Far, and KITTI-defined Easy, Moderate, or Hard localization difficulty.The depth bands are 0–15 m, 15–35 m, and greater than 35 m; difficulty reflects occlusion and truncation.Source: 1) Datasets · p. 7

2) SOTA models for comparison:

• Metric depth baselines include general-purpose VLMs, spatially trained VLMs, and DepthVLM, with all models evaluated on the same test dataset.The listed baselines include GPT-5, Pixtral-12B, InternVL, Qwen, Molmo2-8B, LLaVA-1.5-7B, SpaceLLaVA, SpaceQwen, and DepthVLM.Source: 2) SOTA models for comparison · p. 7
• Grounding baselines span VLM-based methods, including MonoVLM, and computer-vision methods, with results drawn from their original papers.The computer-vision baselines include Mono3DVG, Mono3DVG-TGE, and two-stage methods.Source: 2) SOTA models for comparison · p. 7

3) Evaluation metrics:

• Depth evaluation reports absolute and relative errors over valid numerical predictions, while δ<0.25 counts samples below 25% relative error and treats invalid predictions as incorrect.Reported errors include MAE, RMSE, MedAE, MRE, and MedRE.Source: 3) Evaluation metrics · p. 8
• The depth error distribution separates outputs into five mutually exclusive categories, including valid predictions with relative error of at least 50% and invalid outputs.The first four categories cover valid predictions across relative-error ranges.Source: 3) Evaluation metrics · p. 8
• Grounding evaluation reports mIoU and Acc@0.25 and Acc@0.5 to measure average 3D-box overlap and the share meeting specified IoU thresholds.Higher mIoU indicates greater overall overlap and reflects predicted position, dimensions, and orientation.Source: 3) Evaluation metrics · p. 8

• At the 10% threshold, M3SunAgent reaches 23.71% versus Molmo2-8B's 13.23%; at 50%, it reaches 85.09% versus second-best 59.48%.Source: Figure 5 · p. 9, 1) Quantitative Evaluation · p. 10
• M3SunAgent achieves the best performance across all reported error metrics and valid predictions for all 2,910 test samples; 52.61% have relative error below 25%.Source: 1) Quantitative Evaluation · p. 9
• The grounding tables report accuracy on Mono3DRefer and selected subsets, including mIoU across occurrence, distance, and difficulty.Source: Table II · p. 10, Table III · p. 10, Table IV · p. 11
• M3SunAgent combines open-vocabulary target localization with specialized-VLM metric depth estimation; DepthVLM's dense-depth head does not improve this benchmark.Source: 1) Quantitative Evaluation · p. 10

• The real-world examples use mobile-phone images, EXIF-estimated camera intrinsics, and handheld laser rangefinder distances as ground truth.Source: 2) Qualitative Evaluation · p. 11
• Green boxes mark detected targets, while red points indicate M3SunAgent's selected depth-sampling locations.Source: 2) Qualitative Evaluation · p. 11
• M3SunAgent predicts physical distances close to laser measurements; some general-purpose VLM outputs have large errors, vague descriptions, or no valid distance.Source: 2) Qualitative Evaluation · p. 11

1) Quantitative Evaluation:

• On Unique and Multiple, Acc@0.25 is 67.94% and 71.58%, respectively; Multiple mIoU is 3.24 percentage points above MonoVLM-MiMo.Source: 1) Quantitative Evaluation · p. 12
• Far and Hard mIoU is 30.53% and 34.83%, respectively, 0.93 and 2.85 percentage points below MonoVLM-MiMo; distant and occluded or truncated targets remain challenging.Source: 1) Quantitative Evaluation · p. 12

2) Qualitative Evaluation:

• Qualitative examples show general-purpose VLMs make large yaw-angle and 3D-center errors, while M3SunAgent's 3D boxes overlap the ground truth more closely.Source: 2) Qualitative Evaluation · p. 12

1) Quantitative Evaluation:

• On Mono3DRefer, M3SunAgent reaches 70.90% Acc@0.25 and 42.66% Acc@0.5, while matching or exceeding MonoVLM variants on most major metrics.Source: §V · p. 12, 1) Quantitative Evaluation · p. 12

V. CONCLUSIONS

• The conclusion frames the shared framework as connecting language-guided target identification with quantitative spatial estimation across both tasks.Source: §V · p. 12