Source-linked AI summary
MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World
Yining Hong, Zishuo Zheng, Peihao Chen, Yian Wang, Junyan Li, Chuang Gan
TL;DR
Current multimodal LLMs mainly focus on 2D understanding and lack the ability to interact with 3D environments while capturing multisensory information. MultiPLY pairs an interaction dataset with object-centric scene representations and action/state tokens for an embodied LLM. It outperforms baselines on object retrieval, tool use, multimodal captioning, and task decomposition.
Problem
Existing multimodal LLMs mainly focus on 2D understanding and lack the ability to interact with 3D environments and capture multisensory information beyond vision and language.
Source: §1 · p. 2Method
MultiPLY combines the 500k-example Multisensory Universe dataset with object-centric scene representations and action/state tokens to instruction-tune an embodied LLM.
Source: §1 · p. 2, §4.1 · p. 4Results
MultiPLY outperforms baseline models by a large margin on object retrieval, tool use, multimodal captioning, and task decomposition.
Source: §1 · p. 2Takeaways & Limitations
The paper concludes that an LLM can incorporate multisensory interactive data collected by an agent actively exploring and interacting with an environment.
Source: §6 · p. 9MultiPLY lacks detailed navigation and control policies and instead uses pre-defined policies to carry out actions.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Human beings possess the capability to multiply a melange of multisensory cues while actively exploring and interacting with the 3D world. Current multi-modal large language models, however, passively absorb sensory data as inputs, lacking the capacity to actively interact with the objects in the 3D environment and dynamically collect their multisensory information. To usher in the study of this area, we propose MultiPLY, a multisensory embodied large language model that could incorporate multisensory interactive data, including visual, audio, tactile, and thermal information into large language models, thereby establishing the correlation among words, actions, and percepts. To this end, we first collect Multisensory Universe, a large-scale multisensory interaction dataset comprising 500k data by deploying an LLM-powered embodied agent to engage with the 3D environment. To perform instruction tuning with pre-trained LLM on such generated data, we first encode the 3D scene as abstracted object-centric representations and then introduce action tokens denoting that the embodied agent takes certain actions within the environment, as well as state tokens that represent the multisensory state observations of the agent at each time step. In the inference time, MultiPLY could generate action tokens, instructing the agent to take the action in the environment and obtain the next multisensory state observation. The observation is then appended back to the LLM via state tokens to generate subsequent text or action tokens. We demonstrate that MultiPLY outperforms baselines by a large margin through a diverse set of embodied tasks involving object retrieval, tool use, multisensory captioning, and task decomposition.
1. Introduction
The introduction frames embodied multisensory reasoning as a gap in current largely passive models and presents MultiPLY and interaction data as a response.
- Existing multimodal LLMs excel at vision-language tasks but struggle with 3D interaction and nonvisual sensing; 3D-LLM adds point-cloud reasoning but is costly and object-inefficient.Source: §1 · p. 2
- Related modality-binding methods often target one object or 2D images, leaving large-scale 3D interaction and touch or temperature sensing beyond scope.Source: §1 · p. 2
- The paper identifies scarce interaction data, inadequate scene and multisensory representations, and passive single-step LLM tuning as barriers to linking words, actions, and percepts.It argues models should shift between an object-centric scene overview and detailed object observations during interaction.Source: §1 · p. 2
- Multisensory Universe contains 500k agent-collected 3D interaction data spanning captioning, question answering, dialogue, manipulation, and task decomposition.The environments use HM3D [46], enriched with interactive objects from ObjectFolder [20] and Objaverse.Source: §1 · p. 2
- For instruction tuning, MultiPLY represents scenes with abstract object-centric representations and uses action tokens (NAVIGATE, OBSERVE, TOUCH, HIT) to elicit sensory state-token observations.Different sensor encoders and sensor-to-image adapters connect the observations to the LLM.Source: §1 · p. 2
- MultiPLY outperforms baselines by a large margin on object retrieval, tool use, multisensory captioning, and task decomposition.Source: §1 · p. 2
- At inference, MultiPLY emits actions, receives resulting multisensory observations, appends them as state tokens, and generates subsequent steps.Source: §1 · p. 2
2. Related Works
The section contrasts established sensor-specific multisensory learning and passive multimodal LLMs with MultiPLY’s interactive, multisensory 3D embodied focus.
Multisensory Learning
- Prior multisensory work spans visual-audio localization, editing, and generation [27] [14] [16] [19] [7] [18] [25] [9] [15] and visual-tactile retrieval, manipulation, and reconstruction [3] [21] [4] [5] [36] [48] [49].MultiPLY combines visual, audio, tactile, and thermal information in an interactive 3D environment for diverse embodied tasks.Source: §2 · p. 3
Multi-modal Large Language Models
3. The Multisensory-Universe Dataset
The Multisensory-Universe dataset is built by adding interactive objects to 3D scenes, collecting their sensory data, and using embodied agents to generate paired interaction-language training data.
3.1. Inputting Interactive Objects into 3D Scenes
- HM3D scenes are augmented with 1–10 interactive objects selected by ChatGPT from ObjectFolder and Objaverse, with boxes and sensory properties chosen to support ambiguity-resolving and compatible interactions.HM3D contains 216 spaces and 3,100 rooms; ObjectFolder provides 1k meshes with impact sounds and material annotations, while Objaverse provides 800K objects, from which indoor-compatible objects are selected.Source: §3.1 · pp. 3–4
3.2. Object Sensor Data Acquisition
- Object sensor data include simulated tactile deformation, ChatGPT-matched AudioSet ambient sounds, ObjectFolder impact sounds, and temperatures derived from object labels.DiffTactile reads marker positions before and after contact; impact sounds are queried from implicit sound fields using hitting position and force.Source: §3.2 · p. 4
3.3. Embodied Agents for Data Collection
- ChatGPT proposes interaction tasks, pseudo-ground-truth action sequences, and language reasoning from object labels; an embodied agent explores, executes actions, records sensory feedback, and yields stepwise input-output data.Tasks span multisensory captioning, question answering, dialogue, navigation, manipulation, tool use, rearrangement, and task decomposition.Source: §3.3 · p. 4
4. MultiPLY
MultiPLY represents a 3D scene with object-centric features, then alternates between generating action tokens and receiving multisensory observations through state tokens. Its design covers scene representation, interaction, sensor-state encoding, training, and inference.
4. MultiPLY
- The framework turns generated action tokens into environment interactions and appends the resulting observations as state tokens for subsequent generation.The overview depicts multisensory object details as available through interaction rather than solely from the initial scene representation.Source: §4 · p. 4, Figure 3 · p. 5
4.1. Object-Centric Scene Representations
- MultiPLY builds an object-centric scene representation from ConceptGraphs and CLIP features, fusing image views into 3D and optionally encoding ambient sound with CLAP.The representation contains O × 1024 object features with position embeddings; ambient sound is encoded as a 1024-dimensional feature, and both are provided to the LLM as initial inputs.Source: §4.1 · p. 4
4.2. Action Tokens
- Action tokens select objects by language-to-visual attention and support navigation, observation, touch, impact-sound capture, pick-up, put-down, and looking around.Navigation can use a predefined pathfinder module and is not the paper’s research focus.Source: §4.2 · p. 5
4.3. State Tokens
- State tokens encode observed object points, impact sound, tactile readings, and temperature, using modality-specific feature extraction and projectors where needed.Object point clouds and sensory features use 1024-dimensional representations; sound, tactile, and temperature features are projected into the LLM feature space.Source: §4.3 · p. 5
4.4. Training & Inference
- MultiPLY uses LLaVA, aligns sensor features with lightweight adapters, then instruction-tunes on Multisensory Universe with language-model and object-selection losses.At inference, the LLM generates an action, an agent executes it in Habitat-sim, and the resulting state-token observations feed the next generation step.Source: §4.4 · pp. 5–6
5. Experiments
MultiPLY is evaluated in a simulator across object retrieval, tool use, multisensory captioning, and task decomposition, with evaluation scenes and objects excluded from training. The reported results include gains over baselines in object retrieval and task decomposition, and over PointBind-LLM in multisensory captioning.
5.1. Object Retrieval
- MultiPLY outperforms retrieval baselines; multisensory models beat single-sensor models, 3D models beat 2D models, and LLMs beat similarity-based retrieval models.MultiPLY resolves ambiguity by interacting with and reasoning about different sensor data individually.Source: §5.1 · pp. 6–7, Table 1 · p. 7
5.2. Tool Use
- Binding-based methods perform poorly in tool use, where selecting an appropriate tool requires reasoning about individual sensory properties and object functionality.The task asks the agent to retrieve a tool based on a described situation and goal, with LLM-based methods also providing reasons for their choices.Source: §5.2 · p. 7, Table 2 · p. 7
5.3. Multisensory Captioning
- 3D-based LLMs overall outperform 2D VLMs in multisensory captioning, and MultiPLY outperforms PointBind-LLM.Unlike holistic visual baselines, the task requires navigating to and interacting with an object to describe it across senses.Source: Table 3 · p. 7, §5.3 · pp. 7–8
5.4. Task Decomposition
- Models without interaction perform poorly in task decomposition, while MultiPLY outperforms the baselines by a large margin.The authors attribute this partly to hallucinated objects and to baselines' whole-scene inputs, which may miss nuanced objects.Source: Table 4 · p. 8, §5.4 · p. 8
5.5. Qualitative Examples
- Qualitative examples demonstrate that MultiPLY interacts with objects in embodied environments to gather multisensory information.The examples are shown in Figure 4; further examples are in the supplementary materials.Source: §5.5 · p. 8, Figure 4 · p. 8
6. Conclusion
MultiPLY incorporates multisensory interactive data into language models, supported by a 500k-example dataset collected through active exploration and interaction; it currently relies on predefined action policies.
6. Conclusion
- MultiPLY incorporates multisensory interactive data into large language models.The paper presents it as a multisensory LLM.Source: §6 · p. 9
- Multisensory Universe contains 500k multisensory data collected by an agent actively exploring and interacting with an environment.Source: §6 · p. 9
- MultiPLY lacks detailed navigation and control policies, instead using predefined policies for actions; the authors describe these aspects as orthogonal and integrable in future work.Source: §6 · p. 9
A. Dataset · A.1. More details on Scene Construction
For scene construction, MultiPLY adds objects to HM3D scenes using ChatGPT-generated object bounding boxes, material properties, and temperatures.
A.1. More details on Scene Construction
- ChatGPT is prompted to generate object bounding boxes, object materials and their properties, and temperatures for objects added to HM3D scenes.Figure 5 shows the prompts used for adding objects to the scene.Source: §A.1 · p. 13, Figure 5 · p. 13
A.2. More details on Sensor Data Acquisition … A.2.4 Temperature
The appendix details how tactile, impact-sound, ambient-sound, and temperature data are obtained for objects, using simulations, sampled observations, and prompted category or temperature assignments.
A.2. More details on Sensor Data Acquisition
- This section elaborates on how sensor data are obtained from the objects, giving detailed information about the process by which those object-related sensor data are acquired.Source: §A.2 · p. 13
A.2.1 Tactile
- Tactile data use ChatGPT-generated DiffTactile material parameters and gripper-marker displacement images, with 16 touch positions sampled per object and one image returned randomly.ChatGPT maps object type and softness descriptions to simulation parameters; tactile readings become images by drawing arrows between markers’ initial and final positions.Source: §A.2.1 · p. 13, Figure 6 · p. 14, Figure 7 · p. 14
A.2.2 Impact Sound
- Impact sounds come from ObjectFolder’s implicit sound fields, with 10 mesh locations sampled per object and one sound returned randomly.A striking location is input to the object’s sound field to obtain the corresponding impact sound.Source: §A.2.2 · pp. 13–14
A.2.3 Ambient Sound
- For ambient sound, ChatGPT matches AudioSet ontology entries to object categories, using AudioSet’s hierarchical labels and descriptions.Audio labels trace from child ontology nodes toward the root, with granularity becoming coarser at higher levels.Source: §A.2.3 · p. 14, Figure 8 · p. 15
A.2.4 Temperature
- The temperature subsection provides a prompt for generating a temperature value for each object.The prompt is shown in Figure 9.Source: §A.2.4 · p. 14, Figure 9 · p. 15
A.3. More details on Task Construction
Task-construction prompts generate Multisensory-Universe language tasks by representing expected observations in state-token placeholders, then appending the observations obtained after the agent acts.
A.3. More details on Task Construction
- Prompts specify language tasks with expected observations in state-token placeholders; after executing actions, the agent’s actual observations are appended to those tokens.An expected observation may itself be expressed in language, such as a tactile map when an object is touched.Source: §A.3 · p. 15, Figure 10 · p. 16
B. Experiments … B.3. More Qualitative Examples
The experiments describe MultiPLY’s training and sensory-encoding setup, explore single-, double-, and triple-modality variants, and include additional qualitative examples.
B.1. Experimental Details
- MultiPLY is fine-tuned from LLaVA-v1.5-7B with Adam at a 1e-6 learning rate and batch size 2112 on 4×132 V100s.Adapter training takes 2 hours; full fine-tuning takes less than one day.Source: §B.1 · p. 16
- The original LLaVA mm projector adapts scene representations and object point clouds, while sound, tactile, and temperature adapters are single 1024-to-1024 linear layers.These adapters map inputs of size 1024 to outputs of size 1024.Source: §B.1 · p. 16
- The default LLaVA CLIP vision encoder encodes objects, point clouds, tactile images, and temperature images.For objects, Segment Anything isolates and crops 2D objects, which are merged into 3D objects following ConceptGraph; object pixels are projected into 3D to obtain point clouds.Source: §B.1 · p. 16
- For scene construction, each object has one CLIP feature, encoded with LLaVA’s default CLIP vision encoder from a cropped, masked 2D object before objects are merged from 2D to 3D.Source: §B.1 · p. 16
B.2. Ablative Studies
- Table 5 explores MultiPLY with single, double, or triple modalities.The supplied passage identifies the modality configurations but does not report their results.Source: §B.2 · p. 16
B.3. More Qualitative Examples
- Figure 11 presents more qualitative examples of MultiPLY.The supplied caption identifies the examples but provides no further description of their content.Source: Figure 11 · p. 17