PaperlayerView exact revision on arXiv

Source-linked AI summary

HM3D-M3: A Benchmark for Multi-Agent, Multi-Modal, Multi-Object Goal Navigation

Hongyu Ding, Kanyu Xu, Mingjia Chen, Zixuan Chen, Jieqi Shi, Jing Huo, Yifan Zhang, Yang Gao

arXiv:2610.20121v1cs.RO

TL;DR

A way to measure how well a team handles everyday instructions with several goals specified in different forms and tracked jointly is missing. HM3D-M3 benchmarks teams searching photorealistic scenes for unordered category, instance-image, and instance-description goals, with trained HiGA and training-free LaViRA-M3 baselines. Adapted methods reach 16–27% SR, versus 37.9% for HiGA and 41.0% for LaViRA-M3, with complementary baseline strengths.

  • Problem

    A way to measure how well a team handles everyday instructions with several goals specified in different forms and tracked jointly is missing.

    Source: §I · p. 1

  • Method

    HM3D-M3 benchmarks teams searching photorealistic scenes for unordered category, instance-image, and instance-description goals, with trained HiGA and training-free LaViRA-M3 baselines.

    Source: §I · p. 2

  • Results

    Adapted methods reach 16–27% SR, versus 37.9% for HiGA and 41.0% for LaViRA-M3, with complementary baseline strengths.

    Source: §I · p. 2

  • Takeaways & Limitations

    The task remains far from solved, and difficulty grows with the number of goals and is only partly relieved by adding agents.

    Source: §VII · p. 7

    The evaluation assumes a shared coordinate frame and lockstep execution, omits communication cost and localization drift, and reports a single run with qualitative real-world deployment.

    Source: §VII · p. 7

Abstract

from arXiv · show

Rapid progress in AI agents and robotics makes it a matter of the near future that humans are served by a team of robots. Such a team will receive several goals at once, each specified in whatever form is convenient for the human, such as a category name, a photo, or a spoken description. Existing benchmarks study multi-agent, multi-modal, and multi-object goal navigation in isolation, and a benchmark that places all three in one episode is missing. To fill this gap, we introduce HM3D-M3. A team of K agents starts at distributed locations in a photorealistic indoor scene and searches for an unordered set of M object goals, each given as an open-vocabulary category, an instance image, or an instance description. HM3D-M3 provides 1,000 validation episodes over 3,926 goals from 379 open-vocabulary categories, 100K observation-assignment training samples, and two team-level metrics, goal-level SR and makespan-normalized SPL. We further contribute two baselines. HiGA is a trained hierarchical goal assigner on top of a frozen open-vocabulary navigation policy, and LaViRA-M3 is a training-free team of VLM agents with shared mapping and progress analysis. Adapted multi-robot and zero-shot methods reach 16--27% SR, while our baselines reach 38--41% with complementary strengths and distinct failure modes. Furthermore, we deploy LaViRA-M3 on a heterogeneous three-robot team. Project page: https://robo-lavira.github.io/hm3d-m3/

I. INTRODUCTION

HM3D-M3 targets everyday team requests for multiple goals in different forms, a joint task existing benchmarks do not define. It introduces the benchmark and two complementary baselines, which outperform adapted methods in goal-level SR.

I. INTRODUCTION

  • Existing benchmarks study the three task dimensions separately or score combined methods on borrowed single-agent episodes, leaving no benchmark that defines and measures the joint team task.The task also involves partial-observability allocation, instance-level verification, and joint completion cost.Source: §I · pp. 1–2
  • HM3D-M3 defines team navigation for an unordered set of goals specified as categories, instance images, or instance descriptions in photorealistic scans.A goal counts as found when an agent declares it within 1 m of the instance.Source: §I · p. 2
  • The paper contributes trained HiGA, training-free LaViRA-M3, and adaptations of four existing multi-robot and zero-shot methods.HiGA assigns goals using a shared map, agent poses, and per-goal CLIP heat maps; LaViRA-M3 coordinates through shared maps, a goal board, and progress analysis.Source: §I · p. 2
  • Adapted methods reach 16–27% SR, versus 37.9% for HiGA and 41.0% for LaViRA-M3, whose strengths are complementary.HiGA is strongest on category goals, while LaViRA-M3 is strongest on image and text goals.Source: Figure 1 · p. 1, §I · p. 2
  • The benchmark includes 1,000 validation episodes, 3,926 goals from 379 categories, 100K observation–assignment training samples, and team-level metrics.The validation set covers 36 held-out scenes.Source: Figure 1 · p. 1, §I · p. 2
  • LaViRA-M3 is also deployed on a heterogeneous team of three robots completing mixed-modality goal sets in an office.The robots use local VLMs and the same coordination channels as in simulation.Source: §I · p. 2

II. RELATED WORK

Prior navigation benchmarks add open-vocabulary, multi-object, or multimodal capabilities, while team-navigation work evaluates narrower tasks or uses borrowed episodes. HM3D-M3 combines the benchmark dimensions, and its baselines draw on trained and training-free navigation approaches.

II. RELATED WORK

  • Earlier benchmarks progress from closed-set to open-vocabulary and instance-specific goals, while multi-object and multimodal variants often use fixed-order or per-subtask scoring.Habitat-MAS evaluates heterogeneous teams, but limits multi-object navigation to a closed vocabulary and a single modality.Source: §II · p. 2
  • Recent multi-robot systems combine some task dimensions, but evaluate on single-agent episodes with per-subtask scoring rather than defining a team episode distribution, budget, and metrics.DM3-Nav and AnyGoal are the closest cited systems; DM3-Nav introduces makespan-normalized SPL.Source: §II · p. 2
  • The baselines build on two policy families: trained open-vocabulary navigation and training-free methods that use maps or foundation models.LaViRA and Uni-LaViRA decompose decisions into language, vision, and robot actions; LaViRA-M3 extends them to a team.Source: §II · p. 3

A. Task Definition

An episode combines a scene, distributed agent start poses, and an unordered set of object goals, each specified in one modality. The team is evaluated on whether it finds those goals within the episode budget.

A. Task Definition

  • Each episode contains a scene, K agent start poses, and M goals, with each goal specifying an object instance and one modality’s content.Category-name goals accept any instance of that category; image and description goals refer to a specific instance.Source: A. Task Definition · p. 3
  • A goal succeeds when an agent declares it within 1 m of a navigable viewpoint of an acceptable instance; the evaluator checks declarations offline.The episode ends when all goals are found or every agent exhausts its 500-action or 3d∗ path-length budget.Source: A. Task Definition · p. 3

B. Episode Generation

Episodes are generated on HM3DSem floors with distinct-category goals, balanced modalities, and distributed starts that jointly cover the goals. The paper reports validation-set composition and defines goal-level success and makespan-normalized efficiency metrics.

B. Episode Generation

  • Goals are sampled as M distinct-category instances on a floor, with balanced modalities and K separated starts from which every goal is reachable.Image goals follow the InstanceImageNav protocol; descriptions distinguish instances by attributes and neighboring objects, with spot-checking.Source: B. Episode Generation · p. 3
  • Validation contains 1,000 episodes across 36 scenes and 3,926 goals; training scenes are disjoint and yield 100K observation–assignment samples.Validation goals comprise 1,299 ON, 1,313 TN, and 1,314 IIN goals; episodes sample K∈{2,3,4} and M∈{3,4,5}.Source: B. Episode Generation · p. 4
  • The oracle makespan uses exact assignment enumeration when KM≤105 and a load-balanced greedy assignment otherwise; it is an upper bound on the true optimum.It uses each agent’s open nearest-neighbor tour and scales the episode budget and efficiency normalization.Source: B. Episode Generation · pp. 3–4

C. Metrics

  • Goal-level SR measures the fraction of goals found, while makespan-normalized SPL rewards division of labor and weights efficiency by that fraction.The authors use goal-level rather than episode-level success because all-goals-found success is rarely achieved by current methods.Source: C. Metrics · p. 4

IV. BASELINES

All methods receive the same goal specifications, posed RGB-D observations, and shared occupancy map, with found goals removed from agents’ lists. Adapted methods share a recognition module so comparisons focus on coordination strategies.

IV. BASELINES

  • All methods use the same multimodal goal set, every agent’s posed RGB-D observations, and a shared occupancy map; declared-found goals are removed from all lists.The evaluator’s verdict on declarations is never revealed to the team.Source: §IV · p. 4

A. Adapted Methods

  • Adapted methods and HiGA share an OWLv2-based recognition module, using detector confidence for ON, CLIP similarity for TN, and SuperGlue matching for IIN goals.A point-goal policy approaches accepted candidates before declaring them found.Source: A. Adapted Methods · p. 4
  • The adapted methods compare frontier coordination: VLFM scores frontiers, EGO-Swarm uses geometric repulsion, Co-NavGPT2 assigns frontiers centrally, and DM3-Nav coordinates decentralized agents.Qwen3.5-27B replaces GPT-4o for Co-NavGPT2 so VLM-based methods use the same model.Source: A. Adapted Methods · p. 4

B. HiGA, a Hierarchical Goal Assigner

HiGA separates goal assignment from navigation and trains its assigner using oracle-derived assignments. LaViRA-M3 coordinates VLM agents through shared maps, a goal board, and progress summaries.

B. HiGA, a Hierarchical Goal Assigner

  • HiGA pairs a learned upper-level goal assigner with a frozen DAgRL+OD navigation policy, recomputing assignments when goals are found, agents idle, or candidates appear.A transformer scores agent–goal pairs from map, pose, and goal heat-map inputs; Hungarian decoding produces assignments, trained by behavior cloning and three DAgger rounds.Source: B. HiGA, a Hierarchical Goal Assigner · p. 4

C. LaViRA-M3, a Team of VLM Agents

  • LaViRA agents use a VLM for high-level actions, a VLM to ground directions in bounding boxes, and fast-marching map planning for robot actions.The shared map supports planning and avoids re-exploring covered regions; the shared goal board records unresolved or found status, while progress summaries inform teammates’ prompts.Source: C. LaViRA-M3, a Team of VLM Agents · pp. 4–5
  • LaViRA-M3 declares a goal only after a confident identity judgment and a depth check within 2 m, a looser gate than the benchmark’s 1 m success radius.Division of labor is not enforced; it emerges as the VLM reads the goal board and teammates’ progress summaries.Source: C. LaViRA-M3, a Team of VLM Agents · p. 5

V. EXPERIMENTS

Experiments compare adapted methods, HiGA, and LaViRA-M3 on HM3D-M3’s 1,000 validation episodes. LaViRA-M3 leads overall success and efficiency, while the baselines have complementary modality strengths.

A. Setup

  • Experiments use one evaluation run on 1,000 validation episodes, with agents limited to 500 actions and a path length of 3 d∗.LaViRA-M3 uses a 79° field of view, while HiGA and adapted methods use 42°; a sensor-matched LaViRA-M3 control is also evaluated.Source: A. Setup · p. 5

B. Main Results

  • On the HM3D-M3 validation split with 1,000 episodes, LaViRA-M3 finds 41.0% of goals—the best result—and has the best SPL, 20.3%.Adapted methods find 16–27% of goals; HiGA gains 11 points over the best adapted method.Source: Table II · p. 6, B. Main Results · p. 6
  • HiGA leads category goals at 53.0% ON, while LaViRA-M3 leads instance goals at 41.4% IIN and 40.9% TN.HiGA has 18.8% IIN and 7.1% SPL; LaViRA-M3’s wider camera adds 14 SR points, and its sensor-matched camera adds 3.Source: B. Main Results · p. 6

C. Ablations

Ablations compare HiGA goal-assignment strategies and LaViRA-M3 coordination channels. Learned assignment improves HiGA over a hand-designed rule, while removing either LaViRA-M3 channel reduces success.

C. Ablations

  • 16 points separate learned HiGA assignment from the hand-designed rule, and DAgger adds 3.8 points over behavior cloning.The learned assigner uses heat maps before detection; DAgger’s gains concentrate on ON and IIN.Source: C. Ablations · p. 6, Table III · p. 6
  • Removing LaViRA-M3’s shared goal board drops SR by 7.8 points, while withholding progress analysis drops SR by 3.7 points and SPL by 3.1 points.Removing both channels costs 11.7 points; the fixed assignment raises SPL from 20.3 to 21.6 while reducing instance-goal success.Source: C. Ablations · p. 6, Table III · p. 6
  • With the 42° Stretch camera, LaViRA-M3’s SR falls by 11.4 points across modalities while SPL rises to 23.0.The narrower view finds fewer goals but reaches those it finds with less detouring.Source: C. Ablations · p. 6, Table III · p. 6

D. Scaling with K and M

LaViRA-M3’s success rate rises with more agents but falls as goals increase; failure analysis identifies different dominant miss patterns for the two baselines.

D. Scaling with K and M

E. Failure Analysis

  • LaViRA-M3 misses 59.0% of goals, mostly after acting; 41.8% are pose misses, while only 8.3% are never claimed.Pose misses include stopping at the wrong place or misidentifying an object or room, not only the 2 m depth gate.Source: E. Failure Analysis · p. 7
  • HiGA misses 62.1% of goals, with perception limiting its paradigm; the two baselines’ failure profiles are complementary.The paper suggests pairing VLM instance recognition with a learned policy for approach, stop selection, and close-range identity confirmation.Source: E. Failure Analysis · p. 7

VI. REAL-WORLD DEPLOYMENT

LaViRA-M3 is deployed on a heterogeneous three-robot team in an office, using the same coordination channels as in simulation. The conclusion reports benchmark headroom, complementary baseline strengths, and limitations in the evaluation setting.

VI. REAL-WORLD DEPLOYMENT

  • The office deployment uses an Agilex wheeled platform and two Unitree quadrupeds, each running a local LaViRA agent and exchanging maps, goal boards, and progress analysis.Source: §VI · p. 7

VII. CONCLUSION

  • 41% SR leaves ample headroom; HiGA and LaViRA-M3 have complementary strengths, and difficulty grows with goals, only partly eased by more agents.Source: §VII · p. 7
  • The study assumes shared coordinates and lockstep execution, omits communication cost and localization drift, spotchecks VLM-written text goals, reports one run, and presents qualitative real-world results.Source: §VII · p. 7

Related papers

Loading 2610.20121v1…