PaperlayerView exact revision on arXiv

Source-linked AI summary

Magma: A Foundation Model for Multimodal AI Agents

Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, Yuquan Deng, Lars Liden, Jianfeng Gao

arXiv:2502.13130v1cs.CVcs.AIcs.HCcs.LGcs.RO

TL;DR

Magma addresses the challenge of building one multimodal agent foundation model that supports understanding, planning, and action across digital and physical environments. It pretrains on heterogeneous data using SoM for action grounding and ToM for action planning, and achieves strong results across UI navigation, robotic manipulation, and vision-language tasks. Its recommended scope remains the controlled settings for which it was designed, with human oversight required for UI actions.

  • Problem

    Existing multimodal understanding and action-taking tasks have domain gaps between verbal outputs, 2D UI coordinates, and 3D robotic actions, motivating a shared surrogate-task interface.

  • Method

    Magma jointly pretrains on heterogeneous vision-language, UI, robotics, image, and video data using SoM for actionable objects and ToM for object movements.

  • Results

    Magma achieves new state-of-the-art results on UI navigation and robotic manipulation while maintaining vision-language performance comparable to state-of-the-art large multimodal models.

  • Takeaways & Limitations

    SoM and ToM help Magma learn action grounding and planning, supporting strong spatial-temporal reasoning across downstream agentic tasks.

  • Takeaways & Limitations

    Magma is designed for controlled Web UI, Android-simulator, and enclosed robotic-manipulation settings, with a human required to control every UI action.

Abstract

from arXiv · show

We present Magma, a foundation model that serves multimodal AI agentic tasks in both the digital and physical worlds. Magma is a significant extension of vision-language (VL) models in that it not only retains the VL understanding ability (verbal intelligence) of the latter, but is also equipped with the ability to plan and act in the visual-spatial world (spatial-temporal intelligence) and complete agentic tasks ranging from UI navigation to robot manipulation. To endow the agentic capabilities, Magma is pretrained on large amounts of heterogeneous datasets spanning from images, videos to robotics data, where the actionable visual objects (e.g., clickable buttons in GUI) in images are labeled by Set-of-Mark (SoM) for action grounding, and the object movements (e.g., the trace of human hands or robotic arms) in videos are labeled by Trace-of-Mark (ToM) for action planning. Extensive experiments show that SoM and ToM reach great synergy and facilitate the acquisition of spatial-temporal intelligence for our Magma model, which is fundamental to a wide range of tasks as shown in Fig.1. In particular, Magma creates new state-of-the-art results on UI navigation and robotic manipulation tasks, outperforming previous models that are specifically tailored to these tasks. On image and video-related multimodal tasks, Magma also compares favorably to popular large multimodal models that are trained on much larger datasets. We make our model and code public for reproducibility at https://microsoft.github.io/Magma.

1. Introduction

Magma is presented as a unified foundation model for multimodal agents that understand inputs and predict actions across digital and physical environments. It uses SoM and ToM to bridge verbal multimodal understanding with spatial-temporal action capabilities, achieving strong results across agentic and multimodal tasks.

  • Motivation: Magma targets multimodal understanding and action prediction for human-goal-driven tasks in both digital and physical environments.The model is intended to understand inputs semantically, spatially, and temporally, then produce executable action sequences.
  • Approach: A single foundation model is pretrained on heterogeneous UI, robotics, instructional-video, and image-text datasets for zero-shot transfer across downstream settings.The training strategy avoids sequentially training one domain and adapting to another.
  • Approach: SoM labels actionable objects in images, while ToM labels object movements in videos to transform non-action-labeled data into vision-language-action data.Examples include clickable UI buttons for SoM and human-hand or robotic-arm traces for ToM.
  • Evaluation: Magma evaluates UI navigation, vision-language understanding, and robotic manipulation, spanning evolving digital environments, visual events, and 3D physical interaction.The evaluation covers Mind2Web and AITW, GQA and VideoMME, and Bridge and LIBERO.
  • Results: Magma achieves new state-of-the-art results on UI navigation and robotic manipulation while remaining comparable to state-of-the-art large multimodal models on vision-language tasks.The contribution summary also reports strong performance on BLINK and video question-answering benchmarks without instruction fine-tuning or with fewer pretrained frames, respectively.

2. Related Work

Related work includes large multimodal models and vision-language-action systems that extend language models with visual inputs, action trajectories, visual traces, or video-token prediction. These approaches motivate Magma’s unified treatment of multimodal understanding and action.

  • Large Multimodal Models: Large multimodal models integrate visual data with open-source or proprietary language-model backbones and have achieved substantial progress.Examples discussed include GPT-4V, LLaVA, and related multimodal models.
  • Vision-Language-Action Models: OpenVLA is described as an open-source vision-language-action foundation model fine-tuned from an open-source Prismatic VLM backbone.This places OpenVLA among systems that connect visual-language representations with actions.
  • Action and Video Modeling: Prior work uses visual traces for robot-arm actions, trace prompting for spatial-temporal awareness, or latent video-token prediction for learning from videos.The cited examples include LLARVA, TraceVLA, and video-learning approaches based on VQVAE tokens.

3. Multimodal Agentic Modeling

Magma unifies multimodal understanding with spatial-temporal action grounding and planning across digital and physical environments. Its modeling framework converts heterogeneous visual and action outputs into a common token-based interface, using Set-of-Mark and Trace-of-Mark supervision to support joint training.

  • 3.1. Problem Definition: Magma represents a generalist agent as a policy that maps past visual observations, a textual task, and context to verbal or spatial output tokens.The interface covers language responses, UI actions, and robotic end-effector displacements.
  • 3.1. Problem Definition: The unified interface expresses UI actions through semantic action tokens plus coordinates or boxes, and robotic actions through 6-DoF end-effector displacements.Both modalities are converted into textual tokens for model learning.
  • 3.2.1. Set-of-Mark for Action Grounding: SoM addresses action grounding by marking actionable image regions or points, allowing the model to select candidate marks and recover their original coordinates for atomic actions.Candidate regions can come from segmentation, detection, or domain-specific proposal networks.
  • 3.2.2. Trace-of-Mark for Action Planning: ToM extends mark supervision across time by predicting future traces of valid marks, encouraging temporal understanding and look-ahead planning while using fewer tokens than next-frame prediction.The traces represent object movements in future video frames and focus on action-related dynamics rather than ambient contents.
  • 3.3. Modeling: The model uses a vision encoder and decoder-only language model to autoregressively process variable-resolution images or videos alongside task descriptions, applying SoM broadly but excluding ToM from discrete-screenshot UI data.Table 1 specifies the data-type application pattern for the two supervision schemes.

4. Multimodal Agentic Pretraining

Magma’s agentic pretraining combines heterogeneous data with SoM and ToM supervision, bridging multimodal understanding and action grounding or planning across digital and physical environments.

  • 4.1. Datasets: Magma curates data from robotics, UI navigation, instructional videos, and multimodal understanding sources for joint pretraining.The suite includes UI screenshots, robotic trajectories, instructional-video samples, and image-text pairs.
  • 4.2. SoM and ToM: SoM assigns textual marks to actionable objects in images, while ToM represents object movements in videos for action planning.SoM is applied across data types, whereas ToM is unsuitable for UI screenshots because they are discrete image sequences.
  • 4.2.2. SoM and ToM for Videos and Robotic Data: CoTracker extracts fine-grained traces for human hands, robotic arms, and objects without requiring object recognition.The method uses point tracking to obtain temporal trajectories from video segments.
  • 4.2.2. SoM and ToM for Videos and Robotic Data: Camera-motion handling applies homography transformation before classifying traces into foreground and background groups and clustering them separately.Foreground traces are selected by motion magnitude, then clustered with K-Means before SoM is applied to the first frame.
  • 4.1. Datasets: The resulting pretraining suite contains close to 2.7M UI screenshots, 970K robotic trajectories, over 25M video samples, and 1.2M image-text pairs.The data covers digital and physical environments together with verbal, spatial, multimodal-understanding, and agentic tasks.
  • 4.3. Pretraining: A shared vision encoder and language model generate verbal, spatial, and action outputs, with SoM and ToM bridging multimodal understanding and action supervision.The full model is pretrained with the whole dataset and the proposed mark and trace annotations.

5. Experiment

Magma is evaluated as a unified multimodal agent foundation model across UI navigation, robotic manipulation, spatial reasoning, and multimodal understanding. Across these settings, the results support heterogeneous pretraining with SoM and ToM as a way to improve both agentic and general multimodal capabilities.

  • Zero-Shot Evaluation: Magma’s zero-shot evaluation spans UI action grounding and navigation, robotic manipulation, and generic multimodal understanding tasks.The evaluation uses ScreenSpot, VisualWebBench, SimplerEnv, VQA benchmarks, and POPE.
  • Zero-Shot Evaluation: 19.6%: Magma surpasses OpenVLA on SimplerEnv, nearly doubling the average success rate across Google Robot and Bridge tasks.Magma also performs strongly on challenging tasks where most baselines fail, and pretrained Magma outperforms the robotics-only variant.
  • Ablation Studies: SoM and ToM are necessary for effective heterogeneous pretraining: simply combining UI and robotics data hurts both tasks, while unified marking enables gains in verbal and spatial intelligence.Adding video data alone slightly improves performance but does not close the domain and action-space gap.
  • Efficient Finetuning: Magma transfers effectively with moderate finetuning, outperforming general-domain and specialized models on Mind2Web and AITW UI navigation tasks.The authors attribute the performance to SoM and ToM modeling, which facilitate action grounding for UI navigation.
  • Efficient Finetuning: Magma achieves higher average success rates across all LIBERO task suites using only 10 finetuning trajectories per suite, and removing SoM and ToM reduces performance.The evaluation performs 100 trials per task suite.
  • Multimodal Understanding: Magma performs competitively on spatial and multimodal benchmarks, including parity with CogVLM despite using approximately 29M versus 1.5B pretraining images.It also gains approximately 28% over IG-VLM and SF-LLaVA, while removing videos causes an approximately 8% BLINK drop.

6. Conclusion

Magma is presented as a foundation model that understands and acts on multimodal inputs across different environments. The authors report that SoM and ToM support action grounding and planning, while noting important data, scope, and safety limitations.

  • Magma is designed to understand and act on multimodal inputs to complete agentic tasks in different environments.
  • SoM and ToM prediction tasks help Magma learn to ground and plan actions, respectively.
  • Magma significantly outperforms baselines on downstream UI navigation and robotic manipulation tasks.The experiments also report strong spatial-temporal reasoning ability.
  • Limitations: The instructional-video data are not representative of global human identities, activities, or societal diversity, creating potential training biases.The authors plan to include disclaimers when publishing the models.
  • Limitations: Magma is intended for controlled Web UI, Android-simulator, and robotic-manipulation settings rather than broad application to other tasks.For UI navigation, researchers should keep a human in control of every generated action.
  • Limitations: The model is not usable as-is for exploitation scenarios, although specific malicious training data could enable automated UI navigation.The authors characterize this as a generic risk associated with agentic models.

Supplementary Material

The supplementary material specifies shared experimental settings for Magma pretraining and finetuning. It uses multi-crop processing to accommodate differing image resolutions across data types.

  • Table 9 summarizes the experimental settings for Magma pretraining and finetuning, using up to 32 Nvidia H100s or 64 AMD MI300 GPUs.
  • Pretraining: Pretraining uses 512 as the base image size and resizes inputs to at most four crops for UI and image data.Video and robotics data use one crop.
  • Finetuning: Downstream finetuning follows the recipe in Table 9, while the vision encoder adapts to task-specific image resolutions.

B.1. Pretraining Data

The supplementary material describes heterogeneous pretraining data for UI interaction, instructional videos, robotics, and multimodal image understanding. UI examples use marked spatial supervision, while videos and robotics provide action-oriented trajectories or triplets.

  • UI Data: UI pretraining data come from SeeClick and Vision2UI, with additional bounding-box annotations and OCR-derived text boxes used to provide grounded supervision.Vision2UI includes boxes whose content property is not null.
  • UI Data: Magma-PT-UI contains tasks that map bounding boxes, points, natural-language queries, and content to one another for action grounding and UI understanding.The task directions include predicting descriptions, content, bounding-box coordinates, and point coordinates.
  • Instructional Videos: Instructional-video data include third-person and egocentric sources such as Epic-Kitchens, Something-Something v2, and Ego4D.Something-Something v2 contributes around 160K training and validation video clips, while the full dataset contains 220,847 seconds-long clips.
  • Instructional Videos: Epic-Kitchens annotations support first-frame description prediction and Trace-of-Mark prediction, while Something-Something v2 supplies labeled everyday human-object actions.
  • Robotics Data: Robotics pretraining follows OpenVLA’s mixture, containing 9.4M image-language-action triplets from 326K trajectories across 23 datasets.
  • Image Data: Magma-PT-Image combines 1.2M synthetic ShareGPT4V image-text pairs with 665K LLaVA-1.5 image-instruction examples.The two sources support global visual understanding and familiarity with varied human instructions, respectively.

B.1.5. Data Statistics

The data-statistics analysis compares action vocabulary across UI navigation, robotic manipulation, and instructional-video pretraining data. The datasets contribute complementary verbs related to navigation, object manipulation, and everyday interactions.

  • Figure 13 analyzes verb distributions in UI navigation, robotic manipulation, and instructional-video datasets.
  • UI annotations contain task-guiding verbs such as “locate” and “turn,” while robotics annotations include “pick,” “push,” and “slide.”The authors describe these vocabularies as complementary across the two action-oriented domains.
  • The robotics annotations are described as valuable for learning to reason about interactions with everyday objects.

B.2.1. UI Agent Navigation

This section describes the datasets, evaluation settings, and instruction-tuning data used to assess Magma for UI navigation and related multimodal capabilities.

  • UI navigation benchmarks: UI navigation evaluation covers ScreenSpot, Mind2Web, and AITW, spanning screenshots, web interactions, and Android navigation.ScreenSpot contains 600 screenshots paired with 1.2K instructions across iOS, Android, macOS, Windows, and web pages.
  • UI navigation benchmarks: ScreenSpot zero-shot grounding uses OmniParser to propose actionable regions, while Mind2Web uses cropped webpage screenshots centered on ground-truth action boxes.The evaluation protocols are designed for vision-based UI agents and action grounding.
  • UI navigation benchmarks: AITW experiments use a substantially smaller training subset to examine efficient fine-tuning across General, Install, GoogleApps, Single, and WebShopping splits.The original dataset contains up to 715K trajectories and 5.7M screenshots, whereas the experiments use listed instruction counts for each split.
  • Training data: The pretraining and tuning materials combine robotics, UI, instructional-video, image, and video-language datasets rather than relying on a single domain.The image instruction-tuning collection contains 820K samples, while the video tuning set contains approximately 1.6M samples from 178K videos.

B.2.5. Details about SoM for training and evaluation

Magma constructs Set-of-Mark prompts by extracting candidate actionable regions from structured interfaces and vision-model proposals.

  • Candidate region extraction: The SoM prompt uses candidate bounding boxes extracted in three ways.The supplied passage introduces the three-way extraction scheme but does not enumerate all three methods.
  • Candidate region extraction: DoM Tree annotations derive mobile-screenshot boxes from HTML code and Android view hierarchies for model pretraining.These annotations supplement boxes extracted from HTML code in the SeeClick and Vision2UI data.
  • Candidate region extraction: For zero-shot ScreenSpot evaluation, OmniParser supplies bounding boxes without local semantics to support fair comparison with state-of-the-art methods.The passage distinguishes these evaluation-time boxes from the pretraining annotations.

C. Qualitative Analysis

Qualitative examples show Magma handling mobile UI tasks and a difficult real-robot manipulation task, including sequential actions and precise object placement.

  • Mobile UI navigation: Magma completes mobile tasks such as checking Tokyo weather and installing Instagram from the home page through sequential interaction.The examples use new observations and action history, although the model was never trained on the full trajectory.
  • Robotics manipulation: In the sausage-and-hotdog rollout, Magma precisely picks up the sausage and moves it smoothly above the hotdog.The comparison uses the same complicated real-robot task for OpenVLA and Magma.
  • Robotics manipulation: The real-robot comparison qualitatively presents OpenVLA and Magma on identical manipulation tasks, with OpenVLA failing and Magma succeeding.The task requires precise spatial understanding and planning.
Loading 2502.13130v1…