Source-linked AI summary
ViSMoE: Visual-Aware Sparse Mixture-of-Experts for Embodied Referring Expression Grounding
Shuo Feng, Piji Li
TL;DR
Embodied Referring Expression Grounding requires an agent to navigate and localize a remote object, while prior methods process views and objects indistinguishably. ViSMoE uses sparse Mixture-of-Experts with visual-aware routing, and experiments on REVERIE and SOON demonstrate its effectiveness and superiority over previous state-of-the-art methods.
Problem
Embodied Referring Expression Grounding must model both navigational views and destination objects, but prior approaches process these heterogeneous inputs indistinguishably.
Method
ViSMoE combines sparse Mixture-of-Experts with a visual-aware routing policy to specialize processing for diverse visual inputs.
Results
ViSMoE outperforms previous methods on REVERIE and SOON, improving nearly all reported REVERIE metrics and all SOON metrics over DUET.
Takeaways & Limitations
ViSMoE demonstrates the effectiveness of using specialized experts and visual-aware routing to process visual features in embodied AI.
Abstract
from arXiv · showhide
Embodied Referring Expression Grounding is the task of enabling an agent to navigate in real environments and to localize a remote object based on natural language instructions. In this scenario, the agent needs to select one view for navigation at each step and identify a specific object among all candidate objects at the destination. However, most of the previous approaches fail to distinguish between views and objects, instead processing them using the vanilla vision encoder, which results in ambiguous representations of both views and objects. To address the above issues, we propose ViSMoE, which equips sparse Mixture-of-Experts with a visual-aware routing policy for the embodied agent. This framework processes different types of visual information specifically, resulting in discriminative visual representations for both views and objects. Experimental results on REVERIE and SOON datasets demonstrate that ViSMoE outperforms the previous state-of-the-art methods, showing the superiority of our proposed method.
1 Introduction
Embodied Referring Expression Grounding requires agents to navigate by selecting views and localize target objects, but prior methods process these heterogeneous inputs indistinguishably. ViSMoE addresses this with task-specialized experts and visual-aware routing, with extensive results on REVERIE and SOON demonstrating effectiveness.
- Task Motivation: Embodied Referring Expression Grounding requires selecting a discrete view for each navigation step and identifying the target object at the destination.The task combines language-guided navigation with referring expression grounding.
- Heterogeneous Visual Inputs: Discrete views capture room layout and spatial context, whereas candidate objects provide detailed information about specific items.The two visual inputs differ in both content and task relevance.
- Limitations of Prior Work: Prior approaches concatenate view and object embeddings and feed them into shared encoders, despite their distinct distributions and associated tasks.This unified processing introduces challenges for modeling the two input types simultaneously.
- Proposed Framework: ViSMoE uses sparse Mixture-of-Experts to disentangle task-specific visual features for heterogeneous inputs.Its experts specialize in particular tasks or feature subspaces.
- Proposed Framework: The visual-aware routing policy enhances expert specialization for diverse visual inputs within the ViSMoE framework.The contribution targets more specialized processing of views and objects.
- Experimental Evidence: Extensive results on REVERIE and SOON demonstrate ViSMoE's effectiveness and the benefits of MoE for processing visual features in embodied AI.The introduction identifies these experiments as evidence for the framework's effectiveness.
2 Related Work
Related work addresses embodied grounding through enhanced view representations and applies sparse Mixture-of-Experts to task-specific and multi-input processing. ViSMoE builds on these directions for the unified navigation-and-object-grounding setting.
- Embodied Grounding: Prior embodied grounding methods enhance view representations with external knowledge or corresponding object features to improve vision-language alignment.These approaches focus on enriching the semantics of view representations.
- Sparse Mixture-of-Experts: Recent sparse Mixture-of-Experts studies use task-specific routers to activate relevant model components across NLU and visual dense prediction tasks.The cited studies report strong results across multiple tasks.
- Sparse Mixture-of-Experts: Sparse Mixture-of-Experts has also been used to address distributional differences in multi-input scenarios, providing context for ViSMoE's unified navigation-and-object-grounding task.The related work connects multi-input processing with the paper's combined task setting.
3 Method
ViSMoE models views and candidate objects through specialized visual processing, combining dual-scale cross-modal reasoning with visual-aware sparse expert routing. It replaces dense FFNs, routes tokens using visual type information, and trains the resulting model with multi-task objectives and auxiliary load balancing.
- Task Formulation: The task combines sequential viewpoint selection with destination object identification from natural-language instructions.The agent navigates an indoor environment and selects among candidate objects after stopping.
- Base Model: The base model encodes language, views, and candidate objects, then uses local and global cross-modal reasoning for navigation and grounding.Local visual features and global node representations attend to language; global and local view features support navigation, while object features support grounding.
- Visual-Aware Sparse Mixture-of-Experts: ViSMoE replaces the vision encoder’s dense FFN with an SMoE layer that processes visual tokens through selected experts.The router selects the top-k experts, whose outputs are combined as a weighted sum.
- Visual-Aware Sparse Mixture-of-Experts: Visual-aware routing adds type-specific embeddings so view and object tokens are directed toward different experts rather than distributed indiscriminately.The routing policy uses view and object type embeddings, while the vanilla SMoE lacks explicit constraints for token-type specialization.
- Visual-Aware Sparse Mixture-of-Experts: An auxiliary load-balancing loss promotes even token distribution across experts while preserving type-aware routing.The loss uses the average routing weight and token-allocation fraction for each expert, with α set to 0.001.
- Training and Inference: Pre-training combines masked language modeling, masked region classification, single-step action prediction, object grounding, and the auxiliary loss before DAgger-based fine-tuning.Fine-tuning adds navigation-policy training with SAP and DAgger demonstrations.
4 Experiments
Experiments evaluate ViSMoE on REVERIE and SOON, comparing benchmark performance and ablations of its routing configuration and placement. ViSMoE improves reported results over DUET, while routing choices affect computation and performance.
- Experimental Setup: REVERIE averages 18-word instructions and 4–7-step paths, while SOON averages 47-word instructions and 2–21-step paths.REVERIE panoramas include predefined object bounding boxes; SOON provides detailed descriptions of target scenes and objects.
- Main Results: ViSMoE improves REVERIE SR and RGS over DUET by 3.20% and 1.56% on validation, and by 1.65% and 1.50% on test.These results are reported for the REVERIE benchmark.
- Main Results: On SOON, ViSMoE improves SPL by 3.55% and RGSPL by 0.56% on validation, and SPL by 0.11% and RGSPL by 1.03% on test.The reported SOON gains are measured against DUET across the listed metrics.
- Main Results: SOON contains more complex instructions and candidate sets than REVERIE, averaging 11.3 versus 5.31 nouns and 45.8 versus 3.42 candidate objects per navigation step.The paper links the larger RGSPL improvement on SOON to tasks with greater visual and linguistic complexity.
- Ablation Studies: Ablations report improvements from sparse MoE, auxiliary load-balancing loss, and visual-aware routing over the vanilla DUET baseline.The visual-aware routing ablation substantially improves over the sparse-MoE configuration without that routing policy.
- Ablation Studies: Increasing top-k activates more experts and raises computational overhead, while performance remains consistent for top-k values of 2 or 4.Performance also gradually increases as ViSMoE layers are added across panorama and local-scale cross-modal transformer blocks.
5 Conclusion
ViSMoE applies sparse Mixture-of-Experts with visual-aware routing to process diverse visual inputs in embodied AI. The framework is positioned to support more robust and precise navigation systems.
- ViSMoE uses sparse Mixture-of-Experts with a visual-aware routing policy for diverse visual inputs.The framework is designed for processing visual features in embodied AI.
- The framework highlights Mixture-of-Experts as an approach for processing visual features in embodied AI.
- ViSMoE aims to advance robust and precise navigation systems for real-world scenarios.