PaperlayerView exact revision on arXiv

Source-linked AI summary

DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents

Kai Shi, Jun Yang, Ni Yang, Binqiang Pan, Qingsong Xie, Chao Zhang, Zhenyu Yang, Tianhuang Su, Haonan Lu

arXiv:2510.19336v1cs.CV

TL;DR

Multitask MLLM fine-tuning still faces the challenge of selecting data mixtures that maximize downstream performance. DaMo predicts downstream task performance to select mixtures, and the paper reports gains on PhoneAgentBench and general benchmarks over DML.

  • Problem

    Choosing data mixtures to maximize downstream performance remains challenging for multitask MLLM fine-tuning, while prior mixture work focuses on pretraining validation loss.

    Source: §1 · p. 1

  • Method

    DaMo fits a trainable neural network to predict downstream task performance from data mixtures and extrapolates to select promising mixtures for MLLM training.

    Source: §1 · p. 2, §4.3 · p. 6

  • Results

    DaMo gains 3.38% average performance on PhoneAgentBench and 2.57% average score over DML on general benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DaMo’s reported results extend across PhoneAgentBench and general benchmarks, and the method can be extended to other models and tasks.

    Source: §6 · p. 9

    The study assumes sample order within datasets can be disregarded and the data-mixture ratio remains fixed throughout training; dynamic mixtures remain exploratory.

    Source: D LIMITATIONS AND FUTURE WORK · p. 21

Abstract

from arXiv · show

Mobile Phone Agents (MPAs) have emerged as a promising research direction due to their broad applicability across diverse scenarios. While Multimodal Large Language Models (MLLMs) serve as the foundation for MPAs, their effectiveness in handling multiple mobile phone tasks simultaneously remains limited. Although multitask supervised fine-tuning (SFT) is widely adopted for multitask learning, existing approaches struggle to determine optimal training data compositions for peak performance. To address this challenge, we propose DaMo (Data Mixture Optimizer) - a novel solution employing a trainable network that predicts optimal data mixtures by forecasting downstream task performance for any given dataset ratio. To support comprehensive evaluation, we introduce PhoneAgentBench, the first specialized benchmark to evaluate MLLMs on multimodal mobile phone tasks, comprising 1235 QA pairs spanning diverse real-world industrial mobile application scenarios. Demonstrating strong predictive capability (R^2=0.81) in small-scale pilot experiments, DaMo efficiently extrapolates optimal data mixing configurations. Our results show DaMo achieves a 3.38% performance improvement on PhoneAgentBench compared to alternative methods. Furthermore, extensive experiments across established benchmarks including BFCL-v3, MME-Reasoning, MME-Perception, and OCRBench reveal DaMo's superior generalization, outperforming other approaches by 2.57% in terms of average score. When used solely for MLLM optimization on the BFCL-v3 task, DaMo improves the metrics by 12.47% than other methods. Notably, DaMo maintains robust scalability, preserving its effectiveness when applied to other model architectures. The code and dataset are available at https://github.com/OPPO-Mente-Lab/DaMo.git

1 INTRODUCTION

DaMo addresses the difficulty of choosing data mixtures for multitask MLLM fine-tuning by predicting downstream task performance. The introduction also presents PhoneAgentBench, a benchmark for evaluating mobile-agent capabilities, and reports gains across phone-agent and general benchmarks.

1 INTRODUCTION

  • Existing MLLMs struggle to integrate the diverse capabilities needed for agents that handle multiple mobile tasks.The introduction identifies multitask SFT as the predominant approach, while optimal data blending across datasets and downstream tasks remains challenging.Source: §1 · p. 1
  • DaMo uses a trainable neural network to predict downstream task performance for a given data mixture and extrapolate an optimal mixture for training.The method addresses prior mixture approaches that predict pretraining validation loss rather than downstream performance.Source: §1 · p. 2
  • PhoneAgentBench covers complex task planning, device-native tool use, multimodal memory, and screen-context understanding in real-world phone interactions.The passage reports 2,350 test cases, with 55% requiring simultaneous use of more than three tools.Source: §1 · p. 2
  • 3.38% average performance gain on PhoneAgentBench over DML Ye et al. (2024), alongside a 2.57% average-score gain over DML on general benchmarks.The general benchmarks are BFCL-v3, MME-perception, MME-reasoning, and OCRBench.Source: §1 · p. 2
  • DaMo reports Pearson correlations of 0.75–0.95 between predicted and actual scores across other models, with downstream-task gains over other methods.The method is presented as scalable across models.Source: §1 · pp. 2–3

2 RELATED WORK

Related work distinguishes pretraining mixture optimization from the less-developed problem of choosing multi-source data mixtures for fine-tuning. Existing agent benchmarks also leave gaps in holistic, multimodal assessment of real-world mobile-phone scenarios.

Data Mixing

  • Pretraining mixture methods optimize perplexity, using approaches such as Group DRO, bandit selection, and joint optimization of proportions and data scaling.The passage contrasts these PPL-focused objectives with fine-tuning objectives.Source: §2 · p. 3
  • Fine-tuning mixture work remains limited: industrial solutions use costly manual iteration, SFTMix handles intradataset ratios, and MoE methods lack interpretable criteria.The passage identifies general multi-source mixing schemes and theoretical guidance as open gaps.Source: §2 · p. 3

Agent Benchmark

  • Existing benchmarks are often single-dimensional, unimodal, or GUI-focused, leaving no comprehensive assessment of multimodal phone-agent planning, tool use, and memory.General agent benchmarks lack granularity or multimodal interaction, while mobile benchmarks primarily target GUI tasks.Source: §2 · p. 3

3 PHONEAGENTBENCH

PhoneAgentBench is designed to assess mobile agents across diverse capabilities in real-world application scenarios. Its MT-Plan task evaluates multimodal phone-task planning from images and queries, with plans represented as directed acyclic graphs.

3 PHONEAGENTBENCH

  • Existing benchmarks lack systematic multimodal evaluation of mobile agents across planning, tool use, and memory, while mobile-specific benchmarks primarily focus on GUI tasks.The section frames this mismatch with real-world phone-agent scenarios as a challenge for practical evaluation.Source: §3 · p. 3
  • PhoneAgentBench comprises six curated datasets for holistic assessment of phone-agent performance across capabilities relevant to real-world mobile applications.Table 1 presents details of the tasks in the suite.Source: §3 · p. 3, Table 1 · p. 4
  • MT-Plan evaluates multimodal task planning in phone-agent scenarios using an image and query as input and a plan as output.Its tasks use real photos or mobile screenshots, and plans must invoke at least two tools.Source: §3 · pp. 3–4
  • MT-Plan queries and plans are constructed by annotators, and three annotators cross-validate the data for accuracy.Queries are designed to be concise, colloquial, and aligned with daily user needs.Source: §3 · p. 4
  • The MT-Plan evaluator scores predicted plans against gold plans using the length of the longest ordered action sequence from similarity-matched pairs.Source: §3 · p. 4

4 METHODOLOGY

DaMo formulates multitask fine-tuning as selecting a data mixture that maximizes average downstream-task performance. It fits a predictor from sampled training configurations, then uses that predictor to rank mixtures for training.

4.1 PROBLEM FORMULATION

  • The objective is to choose mixture p* that maximizes the overall average downstream-task score after fine-tuning from θ0 for t steps.Source: §4.1 · pp. 4–5

4.2 PERFORMANCE PREDICTION OF DOWNSTREAM TASKS

  • Single-dataset MMU training improves ACU, degrades APP-Rec, leaves MM-NER uncorrelated, and gives MT-Plan initial gains followed by declines.Source: §4.2 · pp. 5–6
  • In APP-Rec+MMU mixtures, the APP-Rec score surface is non-convex and fluctuates across training steps and dataset ratios, challenging exponential and power-law fits.Source: §4.2 · p. 6
  • DaMo fits an MLP to map data mixture and training step to downstream-task performance.Source: §4.2 · p. 6

4.3 OPTIMAL DATA MIXTURE EXTRAPOLATION

  • DaMo predicts downstream-task scores across Pfix, ranks the mixtures, and trains the top-k candidates, avoiding exhaustive empirical testing.The method first samples mixing ratios, trains checkpoints, and evaluates them to fit its predictor.Source: §4.2 · p. 5, §4.3 · p. 6

5 EXPERIMENTS

The experiments test DaMo on multitask fine-tuning, downstream benchmark performance, and transfer to other models. They report strong mixture-fitting accuracy, gains over baselines, and improved prediction correlation after calibration.

5.1 EXPERIMENTS SETTINGS

  • The experiments use 12 open-source and self-built datasets totaling 220K instructions, evaluate ten tasks across four open-source benchmarks, and compare uniform, natural, and DML mixtures.The benchmarks are BFCL-V3, MME-perception, MME-reasoning, and OCRBench.Source: §5.1 · p. 6

5.2 FITTING SCORE OF NEURAL NETWORK

  • 250 samples from a 1.3 · 10^7-point mixture space yield R2 = 0.81 in 10-fold cross-validation.The MLP fitting score gradually converged at 250 samples.Source: §5.2 · p. 7, Table 2 · p. 7

5.3 DOWNSTREAM TASK PERFORMANCE OF UNSEEN DATA MIXTURES

  • DaMo’s top 50 predicted mixtures achieve significantly higher overall average scores than baseline methods, while random-mixture scores are approximately normally distributed without a discernible baseline advantage.The authors characterize excellent mixtures as extremely sparse.Source: §5.3 · p. 7, Figure 4 · p. 7
  • 13.73% overall average score improvement is reported when DaMo’s top-1 mixture is used, with PhoneAgentBench rising from 44.83% to 68.18% over the native model.The authors also report stable gains over DML across almost all tasks.Source: §5.3 · p. 8, Table 3 · p. 8
  • 29.32% to 47.43% on BFCL-V3 is reported for DaMo (BFCL-V3), which searches mixtures using that benchmark alone.DaMo is reported to outperform baselines across the open-source benchmarks, with this enhancement sustained without task-curated training data.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 EXTENSION TO OTHER MODELS

  • Across target models, DaMo’s predicted and actual overall average scores generally correlate above 0.75; fitting a linear layer on 20 calibration samples reduces discrepancies and raises correlation above 0.9.The mapping is used as a model-agnostic predictor to compensate for model discrepancies.Source: §5.4 · p. 9, Figure 5 · p. 9

6 CONCLUSION

DaMo predicts downstream performance to optimize data mixtures for multitask fine-tuning of multimodal LLMs. The paper also introduces PhoneAgentBench and reports efficacy on phone-agent and general benchmarks, with extension to other models and tasks.

6 CONCLUSION

  • DaMo optimizes data mixtures for multitask fine-tuning of multimodal LLMs by predicting downstream task performance.The prediction uses neural-network-based modeling.Source: §6 · p. 9
  • PhoneAgentBench is introduced to evaluate multimodal LLMs on phone-agent tasks.Source: §6 · p. 9
  • Experiments report DaMo’s efficacy on PhoneAgentBench and general benchmarks, outperforming state-of-the-art methods, and state that it extends to other models and tasks.Source: §6 · p. 9

A DETAILS OF EXPERIMENTS SETTING

The experiments build training and evaluation samples from randomly selected mixtures, fit an MLP, and use its predictions to select mixtures for further training. They also test transfer to three other model architectures.

A.1 IMPLEMENT DETAILS

  • 250 random mixtures from Pfix were trained on InternVL2.5-4B, with checkpoints saved at four steps to produce 1,000 evaluated samples.Training used eight NVIDIA H20 GPUs.Source: §A.1 · p. 14
  • A two-layer MLP with 100 neurons in each hidden layer was fitted on the 1,000 samples and assessed by 10-fold cross-validation using R2.The MLP was built with sklearn.MLPRegressor.Source: §A.1 · p. 14
  • The MLP predicted performance across Pfix, after which the 50 mixtures with the highest predicted scores were selected for model training and validation.Source: §A.1 · p. 14
  • Transfer tests applied InternVL2.5-4B DaMo to Qwen2.5VL-3B-Instruct, Qwen2.5VL-7B-Instruct, and InternVL3-14B using a small number of random mixtures.The tests examined prediction correlation and competitiveness of selected mixtures against baselines.Source: §A.1 · p. 14

A.2 EVALUATION DATASETS

PhoneAgentBench’s evaluation data preparation uses filtering, synthetic data generation, and manual verification.

A.2 EVALUATION DATASETS

  • PhoneAgentBench’s evaluation-data workflow combines data filtering, synthetic data generation, and manual verification.Source: §A.2 · p. 14

A.2.1 MULTIMODAL TASK PLANNING

This appendix describes PhoneAgentBench task and training-data designs, evaluation measures, and the DaMo optimization procedure. The tasks cover planning, multimodal reference and context understanding, entity extraction, mobile functions, app recognition, and image-text relevance.

A.2.1 MULTIMODAL TASK PLANNING

  • Task-planning complexity is represented by DAG edges per node, while diversity uses pairwise Rough-L query similarity; the appendix compares MT-Plan with T-Eval on both measures.The supplied passages describe the comparison but do not report its outcome.Source: §A.2.1 · p. 15

A.2.2 MULTIMODAL REFERENCE RESOLUTION

  • MM-RR is a binary task: determine whether a question refers to image content, with examples of both a non-reference and an image-grounded reference.Figure 6 supplies the examples used to illustrate the two labels.Source: §A.2.2 · p. 15, Figure 6 · p. 15

A.2.3 MULTIMODAL NER; A.2.4 MOBILE FUNCTION CALL; A.2.5 AGENT CONTEXT UNDERSTANDING

  • MM-NER evaluates entity extraction with F1 on 376 curated image-only samples, while Mobile-FC tests mobile API selection and parameter filling, and ACU tests anaphora resolution in multimodal, multi-turn dialogue.MM-NER labels seven entity categories; Mobile-FC defines 50 interfaces and scores exact function-and-parameter matches, whereas ACU uses BLEU.Source: §A.2.3 · p. 15, §A.2.4 · p. 16, §A.2.5 · p. 16

A.2.6 APP RECOGNITION; A.3 TRAINING DATASETS

  • APP recognition evaluates naming an app from its interface image; the training-data overview lists open-source and self-built datasets, with the latter drawn mainly from synthesis and industrial scenarios.The self-built collection includes MMIE, APP-Rec, RR, MMU, FC, TP, and ITR.Source: §A.2.6 · p. 17, §A.3 · p. 17

A.3.1 MULTIMODAL INSTRUCTION EVOLUTION

  • MMIE contains 1.8K multimodal QA items generated through six phases, from intent analysis and scenario expansion to task decomposition, question generation, validation, and naturalization.The phases aim to produce more sophisticated and diversified questions from an initial query, image, and available tools.Source: §A.3.1 · pp. 17–18

A.3.2 APP RECOGNITION; A.3.3 REFERENCE RESOLUTION; A.3.4 MULTIMODAL UNDERSTANDING

  • APP-Rec uses screenshots from 100 installed apps, RR builds image-grounded positive and image-mismatched negative examples, and MMU uses 1–4-round dialogues to rewrite final inputs without omitted or referential content.APP-Rec contains 22.8K items, RR 10.5K, and MMU 21.1K samples.Source: §A.3.2 · p. 18, §A.3.3 · p. 18, §A.3.4 · p. 18

A.3.5 FUNCTION CALLING; A.3.6 TASK PLANNING; A.3.7 IMAGE-TEXT RELEVANCE

  • FC covers tool selection and parameters, TP adds dependent multistep operations, and ITR labels whether an image can answer a question; their datasets include 500 tool types, 26.8K TP items, and 9.7K ITR items.Most FC instructions require one tool; TP uses #E-prefixed outputs to pass parameters to later steps, while ITR uses positive and image-mismatched negative examples.Source: §A.3.5 · pp. 18–19, §A.3.6 · pp. 19–20, §A.3.7 · p. 20

B ALGORITHM

  • Algorithm 1 searches fixed data mixtures and training steps with the fitted MLP, trains the MLLM using the selected configuration, then evaluates the resulting model on the test dataset.Its inputs include training and test datasets, initial MLLM parameters, the mixture space, and fitting data for the MLP.Source: B ALGORITHM · p. 20

C EXTENSIBLE VALIDATION

DaMo transfers mixture predictions across model families and sizes, with correlations above 0.75. A 20-sample linear calibration improves prediction correlation to 0.90 and outperforms direct extrapolation.

C EXTENSIBLE VALIDATION

  • Pearson correlations stay above 0.75 across target models, and a 20-sample linear mapping raises correlation to 0.90 while outperforming direct extrapolation.The evaluation spans Qwen2.5VL-3B, Qwen2.5VL-7B, and InternVL3-14B; mapped performance is reported as superior in Table 5.Source: C EXTENSIBLE VALIDATION · p. 21

D LIMITATIONS AND FUTURE WORK

The study assumes sample order within datasets is irrelevant and mixture ratios remain fixed during training. Dynamic-mixture optimization remains exploratory, without a systematic extrapolation method or quantified cost and performance trade-offs.

D LIMITATIONS AND FUTURE WORK

  • DaMo assumes dataset sample order is irrelevant and mixture ratios stay fixed; dynamic-mixture extrapolation and its computational costs and performance gains remain unestablished.Future work proposes MCTS with reinforcement learning to determine stage-specific mixtures, alongside sample-quality inputs and benchmark updates.Source: D LIMITATIONS AND FUTURE WORK · p. 21
Loading 2510.19336v1…