MuSTMM: Learning Sound-Triggered Mobile Manipulation across Multiple Scenes

• The benchmark addresses the gap between audio-visual navigation datasets without physical interaction and manipulation datasets with limited structural variation or no sound rendering.Existing STMM evaluation varies object layouts within limited scene structures, leaving performance in unseen structures under-explored.Source: §1 · p. 2
• MuSTMM combines sound rendering and physical interaction across 31 scenes, varying both architecture and object layouts for single-event and multi-event evaluation in unseen scenes.Its 23 training and 8 held-out test scenes are disjoint, and the configurable pipeline supports extensions to scenes, assets, and task configurations.Source: §1 · p. 2
• The navigation policy accumulates depth observations in a shared coordinate frame and projects them into a BEV occupancy map, fusing map features with acoustic and temporal context into a goal descriptor.The descriptor carries target-related and scene-structure priors for navigation to the region needed for subsequent interaction.Source: §1 · pp. 2–3
• 75.28% navigation success is achieved by the explicit-spatial-memory policy, while navigation remains a major execution bottleneck in held-out scenes, even with ground-truth skill chains.Memory ablations show that explicit spatial memory combined with compressed spatio-temporal memory outperforms either component alone.Source: §1 · pp. 2–3

Embodied Navigation and Manipulation Benchmark

• STMM combines sound rendering and physical interaction, but its evaluations primarily vary object configurations within limited architectural structures, leaving unseen-scene performance under-explored.MuSTMM introduces architecturally distinct scenes while preserving object-layout variation and both capabilities.Source: §2 · p. 3
• Audio-visual navigation benchmarks support spatial sound or sound-source navigation without subsequent physical interaction, while manipulation benchmarks provide interaction but may lack sound rendering or architectural diversity.HomeRobot evaluates mobile manipulation across diverse scenes with scene-disjoint splits, but does not incorporate spatial sound rendering.Source: §2 · p. 3

Embodied Navigation

• Prior navigation methods use spatial maps or multimodal memories; MuSTMM complements memory-based sound-goal estimation with geometry accumulated from pose-aligned depth observations.Its goal descriptor provides target-related and structural priors for navigating to the region required for interaction.Source: §2 · p. 3

• STMM tasks require robots to respond to sound events by navigating to source objects and interacting with them, either in single-event tasks or sequentially with two prioritized objects in multi-event tasks.Interactions include relocating sounding rigid objects or changing articulated-object states.Source: §3 · p. 3
• MuSTMM supports sound rendering and physical interaction across 31 scenes, with variation in architectural structure and object layout.Figure 2 contrasts dataset capabilities, room-count distribution, and furniture assets.Source: §3 · p. 4, Figure 2 · p. 4
• Structure variation spans room number, shape, size, connectivity, and architectural geometry across 31 HSSD scenes, including 11 room counts from four to 17 rooms.The dataset precomputes 14.36 million binaural RIRs using sampled source-receiver positions and four receiver orientations.Source: §3 · p. 4
• Layout variation combines episode-wise placements of sounding targets and silent source-capable distractors with scene-specific arrangements of 855 non-source household furniture assets.The furniture provides diverse visual backgrounds and spatial contexts distinct from the task-related objects.Source: §3 · p. 4, Figure 2 · p. 4
• The dataset contains 24.91k training episodes across 23 scenes and 5.20k testing episodes across eight held-out scenes, covering single-event and multi-event tasks with disjoint scene sets.Source: §3 · p. 4
• The scene-disjoint split provides a testbed for assessing sound grounding and physical interaction in unseen scenes, while the configurable generator supports extensions to scenes, assets, and task configurations.Source: §3 · p. 4

4.1 TASK PLANNING AND POLICY EXECUTION

• The planner predicts an ordered skill chain once from initial observations, then activates the corresponding policies sequentially to execute the skills.For articulated-object interactions, the policy outputs base velocities, seven arm-joint actions, and a gripper action at each control step.Source: §4.1 · p. 5

4.2 NAVIGATION POLICY MODEL

• The navigation policy predicts a sound-source location from spatial and spatio-temporal context, then uses fused map features as the policy branch’s decoder query for navigation actions.The policy branch processes observation history with a transformer encoder-decoder and uses actor and critic heads for action prediction and value estimation.Source: §4.2 · pp. 5–6
• The spatially augmented goal descriptor combines accumulated scene geometry with sound and temporal features to provide goal-location and scene-structure priors for navigation.Depth points are transformed into world coordinates, accumulated, and projected into an egocentric BEV occupancy map; the map feature is fused with the spatio-temporal embedding for the policy branch.Source: §4.2 · pp. 5–6

4.3 LEARNING OBJECTIVE

• Navigation training uses dense progress rewards with time and collision penalties, while a PPO loss trains the policy branch.The reward reflects progress toward a collision-safe goal and adds a success reward when the policy stops within the goal region; PPO combines policy, critic, and entropy terms.Source: §4.3 · pp. 6–7
• The goal predictor has a separate location loss that supervises target direction, normalized distance, and spatial-map location predictions.Its spatial-memory terms align a predicted occupancy-map distribution and soft-argmax location with the ground-truth target; the location predictor is optimized separately from PPO.Source: §4.3 · p. 7

5 EXPERIMENT

• The evaluation tests single-event relocation and state transition plus sequential two-source tasks, using Planning Success Rate and stricter Completion Success Rate.Baselines include an Oracle, zero-shot Omni-MLLMs, and a supervised setting-specific planner; policies use PPO training for 5 × 10^6 environment steps.Source: §5 · p. 7

5.1 RESULT

• 62.51% CSR for object relocation and 39.51% CSR for state transition are the best non-oracle results, with navigation losses during execution.The progressive curves show execution-stage losses for both single-event tasks.Source: §5.1 · pp. 7–8, Figure 4 · p. 8
• 39.50% Oracle and 31.60% Qwen2.5-Omni-7B completion success rates show the difficulty of multi-event execution.Completion declines as execution proceeds through two navigation skills.Source: §5.1 · p. 8, Figure 4 · p. 8
• Qualitative executions show object relocation, door opening, and priority-ordered interaction with two simultaneous sound sources.In the multi-event example, the robot handles the primary source before the secondary one despite interference.Source: §5.1 · p. 8, Figure 5 · p. 9

5.2 ABLATION STUDY AND FURTHER DISCUSSION

• 75.28% navigation success outperforms SAVi (15.68%) and MAGNet (71.47%) on object relocation.Table 1a compares the proposed navigation policy with existing policies.Source: §5.2 · p. 8, Table 1 · p. 9
• Combining Explicit Spatial Memory and Compressed Spatio-Temporal Memory reaches 75.28% success, versus 36.00% and 44.08% when used alone.The results support complementary roles for explicit and compressed memory.Source: §5.2 · p. 8, Table 1 · p. 9
• Head depth achieves 94.55% open-refrigerator success, arm depth 92.91%, and both views together 95.36%.The head view guides base movement, while the arm view focuses on refrigerator interaction regions.Source: §5.2 · p. 9, Table 1 · p. 9
• For state-transition planning, sound alone reaches 59.50% success, head RGB alone 11.61%, and their combination 61.49%.The results identify sound as the primary skill-chain cue, with visual context contributing complementary information.Source: §5.2 · p. 9, Table 1 · p. 9

6 CONCLUSION

• MuSTMM supports sound rendering and physical interaction across scenes, with scene-disjoint single-event and multi-event evaluation in unseen scenes.The Navigation Policy Model accumulates depth observations into a BEV occupancy map and fuses map features with sound context to provide target-location and scene-structure priors.Source: §6 · p. 9
• Explicit spatial memory complements compressed spatio-temporal memory to improve sound-source navigation, but navigation remains a major execution bottleneck in unseen scenes.The conclusion identifies multi-event tasks as a particularly challenging setting.Source: §6 · p. 9

7 AI USE STATEMENT

• Generative AI tools were used for readability edits, specifically grammar and spelling checks, and not for scientific figures or plagiarism.The authors state that they reviewed the AI-assisted revisions and take responsibility for the final content.Source: §7 · p. 10

8.1 PROMPT FOR TASK PLANNING

• Task-planning prompts share a system prompt for rules and logic and a user prompt containing visual and acoustic environmental context.The standardized structure is used across task planners for comparison.Source: §8.1 · p. 12

8.1.1 PROMPT FOR TASK PLANNING IN THE SINGLE-EVENT SETTING.

• The single-event system prompt defines the skill library, sound-to-skill-chain mapping, planning constraints and JSON format, and response template.The user prompt supplies visual and acoustic observations to begin planning.Source: §8.1.1 · p. 12

1. Category A: Relocation Objects

• Mechanical alarms, ringing phones, and Furby toys map to [“nav”, “pick”, “place”].Source: §1 · p. 12
• Talkies, switches, radios, music boxes, Bluetooth speakers, earbuds, watches, and tablets use the same navigation, pickup, and placement plan.Source: §1 · p. 12
• The prescribed plan for each relocation object is navigation followed by picking and placing it.Source: §1 · p. 12

2. Category B: State Transition

• A doorbell maps to navigation followed by opening the door, while running water maps to navigation followed by closing the sink.Source: §2 · p. 12
• Air-fryer beeps and a running dishwasher map to navigation followed by opening the corresponding appliance.Source: §2 · p. 12
• Microwave beeps, a refrigerator alarm, and a running washing machine map to navigation followed by opening the corresponding appliance.Source: §2 · pp. 12–13

8.1.2 PROMPT FOR TASK PLANNING IN THE MULTI-EVENT SETTING.

• The multi-event planner prompt combines visual frames and sound data, and its five segments include event priority and constrained output formatting.Source: §8.1.2 · p. 13

1. Category A: Relocation Objects; 2. Category B: State Transition

• The skill mapping assigns relocation sounds to navigation, picking, and placing, while state-transition sounds map to navigation and an object-specific action.Source: §8.1.2 · pp. 13–14

8.1.2 PROMPT FOR TASK PLANNING IN THE MULTI-EVENT SETTING.

• For two sounds, the planner acts on the designated first source before identifying and planning for the second, excluding other potential objects.Source: §8.1.2 · p. 14

5. Expected JSON Structure; 2. Audio Input

• The multi-event output is a JSON plan with exactly “first sound” and “second sound” keys, while the prompt also supplies the observation instruction.Source: §8.1.2 · p. 14, §5 · p. 14, §2 · p. 14

8.2 ADDITIONAL ABLATION STUDY

• 75.40% for sound alone exceeds 71.70% with head RGB included in the multi-event SSP modality ablation.Source: §8.2 · p. 14, Table 2 · p. 14

8.3 DETAILED NUMERICAL RESULTS ON MUSTMM

• Table 3 reports planning, individual-skill, conditional-skill, and full-task completion rates; PSCSR tracks completion through stage k, and CSR covers the entire task.Source: §8.3 · pp. 14–15, Table 3 · p. 15

8.4 ADDITIONAL QUALITATIVE RESULT

• Navigation failures include stopping by a visually distracting refrigerator and being diverted by acoustic echoes before the step limit.Source: §8.4 · p. 15, Figure 6 · p. 16

8.5 EPISODE STATISTIC

• Episodes with two articulated sound sources are excluded because structural variation makes safe interaction regions difficult to maintain.Source: §8.5 · p. 16, Figure 7 · p. 16