PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Improvement in Multimodal Large Language Models: A Survey

Shijian Deng, Kai Wang, Tianyu Yang, Harsh Singh, Yapeng Tian

arXiv:2510.02665v1cs.CL

TL;DR

MLLM self-improvement faces multimodal challenges, and before this work no comprehensive survey focused on methods for the area. This paper structures the literature around data collection, organization, and model optimization, and synthesizes evaluations, applications, challenges, and future directions.

  • Problem

    No comprehensive survey focused on MLLM self-improvement, despite multimodal challenges such as alignment-related hallucination and limits on models generating their own image data.

    Source: §1 · p. 1

  • Method

    The survey organizes representative MLLM self-improvement methods by data collection, data organization, and model optimization, while reviewing evaluations and downstream applications.

    Source: §1 · p. 2, §8 · p. 9

  • Results

    The survey synthesizes representative methods, commonly used evaluations and applications, and current challenges and opportunities for future research.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The authors present the survey as a guide for researchers exploring and developing MLLM self-improvement methods.

    Source: §8 · p. 9

    The survey is macro-level, may omit recent developments, and excludes work focused only on LLMs or agents.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Recent advancements in self-improvement for Large Language Models (LLMs) have efficiently enhanced model capabilities without significantly increasing costs, particularly in terms of human effort. While this area is still relatively young, its extension to the multimodal domain holds immense potential for leveraging diverse data sources and developing more general self-improving models. This survey is the first to provide a comprehensive overview of self-improvement in Multimodal LLMs (MLLMs). We provide a structured overview of the current literature and discuss methods from three perspectives: 1) data collection, 2) data organization, and 3) model optimization, to facilitate the further development of self-improvement in MLLMs. We also include commonly used evaluations and downstream applications. Finally, we conclude by outlining open challenges and future research directions.

1 Introduction

Self-improvement in MLLMs could reduce reliance on costly human data collection, but multimodal challenges and the lack of a comprehensive survey leave the field without a focused synthesis. This survey reviews prior work, organizes methods and evaluations, and identifies future directions.

1 Introduction

  • Self-improvement uses MLLMs to obtain their own training data and improve models, offering a route beyond costly scaling and potential performance ceilings of static training.Prior work cited in the passage reports reduced hallucinations and improved general-task performance at relatively low cost.Source: §1 · p. 1
  • Figure 1 depicts a cycle in which a seed MLLM generates data, the data are organized into a dataset, and training produces an improved model that can seed another round.Source: Figure 1 · p. 1
  • No comprehensive survey had focused on MLLM self-improvement, so this paper reviews prior work and identifies challenges in the area.Earlier surveys covered text-only LLM self-improvement, MLLMs broadly, or narrower topics such as hallucinations.Source: §1 · p. 1
  • Multimodal alignment can contribute to hallucination, and most current MLLMs cannot generate images directly, limiting their ability to produce all their own training data.Source: §1 · p. 1
  • The survey organizes methods around data collection, data organization, and model optimization, and also covers evaluations, downstream applications, and future challenges.Source: §1 · p. 2

2 Overview

The survey frames MLLM self-improvement as a spectrum of automated methods, distinguishing model training from response refinement and covering related approaches across image, video, and agent settings.

2 Overview

  • The survey reviews representative methods for reducing human involvement in model improvement, spanning self-refinement, peer improvement, image MLLMs, video MLLMs, and multimodal agents.Source: §2 · p. 2, §2.2 · p. 3

2.1 Definition

  • Self-improvement updates model parameters through training, whereas self-refinement updates a response in context; the training loop can iterate with the improved model as its next seed.The survey includes assisted approaches using external models on the less-independent end of its self-improvement spectrum.Source: §2.1 · pp. 2–3

2.2.1 Self-Refinement and Peer-Improvement

  • Early self-refinement and peer-improvement work uses training-free hallucination interventions or GPT-4 assistance to build or refine multimodal capabilities without starting from human supervision.Source: §2.2.1 · p. 3

2.2.2 Self-Improvement in Image Large Language Models

  • Image-MLLM methods include instruction-data enhancement, self-questioning, object description-and-location cycles, trainable hallucination interventions, preference data, and internal checks or rewards.Examples named in the passage include VIGC, SQ-LLaVA, SC-Tune, M3ID, SeVa, SIMA, FiSAO, and CSR.Source: §2.2.2 · p. 3

2.2.3 Extensions to Video

  • Video self-improvement includes preference generation grounded in video content and adapting STaR to use labeled video datasets for instruction tuning.Source: §2.2.3 · p. 3

2.2.4 Multimodal Agents

  • For multimodal agents, self-improvement covers self-play for image identification and better game decision-making in Blackjack and ALFWorld.Source: §2.2.4 · p. 3

3 Seed Models

Seed MLLMs need not be exceptionally strong, but they require certain capability floors for self-improvement loops to function. The survey describes common seed choices and alternative training from scratch.

3 Seed Models

  • A seed model needs capability floors for the self-improvement loop; without them, it may generate low-quality data and the loop may stall or collapse.Source: §3 · p. 3
  • Some capabilities are costly to retrofit through self-improvement alone, including robust text-in-the-wild handling, video temporal aggregation, and coherent reasoning traces.Source: §3 · p. 3
  • Common seed choices include LLaVA, Qwen-VL, InstructBLIP, MiniGPT4, and Video-LLaVA, while some studies train seed models from scratch for greater control or specific needs.Source: §3 · p. 4

4 Data Collection

Autonomous data collection can reduce human dependence and support continuous, scalable learning, with methods ranging from unguided sampling to guided generation and negative-example construction.

4 Data Collection

  • Autonomous collection is intended to reduce human intervention and enable more efficient, continuous, and scalable learning compared with labor-intensive conventional collection.Source: §4 · p. 4

4.1 Random Sampling

  • Random sampling draws from a model’s existing knowledge without guidance and can yield diverse data, but may waste resources on redundant or irrelevant samples.Source: §4.1 · p. 4

4.2 Guided Data Generation

  • Guided generation uses designed prompts and pipelines to steer outputs toward desired responses; Chain-of-Thought and search methods such as beam search or MCTS can support sample efficiency.Source: §4.2 · p. 4

4.3 Negative Samples

  • Negative samples can be created with misleading prompts, distorted images, attention masking, or decoding paths that make responses less grounded in visual context.These examples target the model’s ability to distinguish correct from incorrect responses.Source: §4.3 · pp. 4–5

5 Data Organization

Data organization verifies and transforms generated data before reuse, since its quality shapes the robustness and reliability of self-improvement. The survey covers verification, dataset arrangement, and iterative collection-organization cycles.

5 Data Organization

  • Organizing generated data through verification and processing is essential because its quality determines self-improvement robustness and reliability.The section groups this work into verification methods, dataset arrangement, and collection-organization loops.Source: §5 · p. 5

5.1 Verification Methods

  • The survey compares rule-based and model-based verification methods, while emphasizing that verification is one stage in a broader data organization process.The cited table is identified as a comparison of data verification methods.Source: §5.1 · p. 5, Table 3 · p. 6

5.1.1 Rule-Based Verification

  • Rule-based verification applies predefined criteria, offering computational efficiency but limited flexibility across diverse scenarios.Methods include majority voting and ground-truth alignment, such as accepting bounding boxes above an IoU threshold.Source: §5.1.1 · p. 5

5.1.2 Model-Based Verification

  • Model-based verification uses peer models or self-critique for nuanced assessments, at the cost of additional computation.Self-critique can assess correctness and relevance at token, sentence, or output level.Source: §5.1.2 · p. 5

5.1.3 Verification from the Environment

  • Verification can also draw on feedback from real-world or simulated environments, including games.This is a distinct verification source alongside rules and models.Source: §5.1.3 · p. 6

5.2 Dataset Arrangement

  • Dataset arrangement filters unwanted data, edits or refines outputs, and can schedule hard examples for later curriculum learning.The resulting data may be formatted for SFT or DPO, or used to train reward or judge models and for evaluation.Source: §5.2 · p. 6

5.3 Collection-Organization Loop

  • Collection and organization can form an iterative loop in which curated data shapes later data generation and model updates.Data-centric iterations can improve data quality and reduce noise; model-centric iterations support co-evolution of data and model capability.Source: §5.3 · p. 6

6 Model Optimization

The survey describes supervised fine-tuning, reinforcement learning, and direct preference optimization as approaches for updating a seed model from organized data. It also reviews enhanced variants, negative-sample use, and multi-stage training.

6 Model Optimization

  • After data organization, MLLMs can be optimized with supervised fine-tuning, reinforcement learning, or direct preference optimization, whose advantages and disadvantages the survey compares.The survey notes that these training approaches are connected.Source: §6 · p. 6

6.1 Supervised Fine-tuning

  • SFT trains predictions toward ground-truth responses, supporting instruction following and a broader range of general tasks.Its objective is to minimize the discrepancy between predictions and dataset targets.Source: §6.1 · pp. 6–7

6.2 Reinforcement Learning

  • RL seeks outputs that receive high rewards and has been used without human demonstrations, particularly for preference alignment and reasoning.GRPO is described as an efficient alternative to PPO because it does not require a value model.Source: §6.2 · p. 7

6.3 Direct Preference Optimization

  • DPO is an offline, reinforcement-learning-free preference method that can use negative as well as positive examples.It increases the likelihood of preferred outputs relative to dispreferred ones.Source: §6.3 · p. 7

6.4 Other Enhanced Variants

  • Enhanced preference-optimization variants add components such as regularization to support stability and prevent overfitting to preference data.The survey describes these as adjustments to the classic method.Source: §6.4 · p. 7

6.5 Alternative Ways of Using Negative Samples

  • Negative samples can also be paired with self-reflection and chain-of-thought correction to identify and correct model errors.The described approach generates detailed reasoning steps during correction.Source: §6.5 · p. 7

6.6 Curriculum

  • Multi-stage optimization is common, but stages may hurt performance, combined stages may yield gains, and RL can itself be divided into substages.The cited studies report differing findings about the effects of training stages.Source: §6.6 · p. 7

7 Dataset and Evaluation

The survey reviews MLLM datasets and evaluation practices, then reports patterns across benchmarks: method-task fit, seed strength, and cross-benchmark inconsistency matter. It also identifies recurring capability bottlenecks, judge leakage, and efficiency trade-offs.

7 Dataset and Evaluation

  • No benchmark has been specifically designed for MLLM self-improvement, so studies generally use existing benchmarks and compare results with seed and state-of-the-art models.LLM-Evolve is an attempted new benchmark, but it uses a non-parametric setting.Source: §7 · p. 7

7.1 Dataset

  • MLLM-created datasets support self-improvement through varied constructions, including feedback instructions, knowledge-aligned grounding samples, and preference pairs.Examples include VLFeedback, DeepPerception, and OmniAlign-V-DPO; the survey compiles dataset comparisons.Source: §7.1 · pp. 7–8, Table 5 · p. 9

7.2 Benchmarks

  • Evaluation uses benchmarks for general multimodal knowledge, visual reasoning, hallucination, medical imaging, video question answering, and judging ability.Examples include MMMU, Mathvista, POPE, VQA-RAD, MSVD-QA, and VL-RewardBench.Source: §7.2.1 · p. 8, §7.2.2 · p. 8, §7.2.3 · p. 8

Method-Task Match.

  • Rule- or verification-based RL yields the largest absolute gains on verifiable tasks, while preference or AI-feedback data most reliably lowers hallucination metrics and improves helpfulness or faithfulness.The cited hallucination metrics include POPE and AMBER.Source: Method-Task Match · p. 8

Seed Strength Matters.

  • Relative improvement over the seed typically shrinks for stronger seeds, although strong seeds show more stable gains and better final performance under identical pipelines.The survey gives STIC-style pipelines as an example.Source: Seed Strength Matters · p. 9

Cross-Benchmark Inconsistency.

  • Gains are inconsistent across benchmarks: methods that help compositional reasoning can regress on perception-heavy tasks, and pairwise rank correlations are often modest.Improvement on one benchmark suite does not guarantee improvement on others.Source: Cross-Benchmark Inconsistency · p. 9

Persistent Bottlenecks.

  • Recurring bottlenecks include fine-grained spatial reasoning, multi-image consistency, long-horizon video grounding, diagram understanding, noisy OCR, and open-world hallucination without tightly verified visual evidence.The survey also warns that closely related judges used for curation and evaluation can inflate scores.Source: Persistent Bottlenecks · p. 9, Judge/Reward Leakage · p. 9

Efficiency Analysis.

  • Verification cost varies: rule-based checks are generally cheapest, model-based verification costs more, and real-environment feedback can cost most while yielding high-quality feedback.Prompt guidance can reduce the search space, while negative samples can make use of low-scoring generations.Source: Efficiency Analysis · p. 9

8 Conclusion

The conclusion presents a structured survey of MLLM self-improvement, comparing representative work across data collection, data organization, and model optimization. It also summarizes evaluations and applications and identifies challenges and future opportunities.

8 Conclusion

  • The survey organizes representative MLLM self-improvement work across data collection, data organization, and model optimization, and summarizes evaluations, applications, challenges, and future opportunities.It aims to guide researchers exploring and developing self-improvement methods for MLLMs.Source: §8 · p. 9

Limitations

The survey gives macro-level coverage of current MLLM self-improvement, provides a complete three-part taxonomy, and describes applications where task data is scarce, including math and science reasoning.

Limitations

  • The survey is limited to macro-level analysis of current MLLM self-improvement; rapid developments may be missed, and LLM-only or agent work is outside its scope.The authors note that some methods excluded from the review may be adaptable to MLLMs.Source: Limitations · p. 10

A Full Taxonomy

  • The appendix gives the complete taxonomy hierarchy across Data Collection, Data Organization, and Model Optimization, with representative works on each branch.The main paper provides only an overview of the taxonomy.Source: A Full Taxonomy · p. 16

B Applications

  • Self-improvement can address applications lacking related instruction data by generating data autonomously and acquiring skills for downstream tasks.Source: B Applications · p. 16

B.1 Math & Science

B.2 Control

A preference-data synthesis framework improves object segmentation and trajectory generation in visuo-motor control tasks; Figure 3 places self-improvement methods in a three-step taxonomy.

B.2 Control

  • 15.50% improvement was achieved in four visuo-motor control tasks by an automatic preference-data synthesis framework using an MLLM and image segmentation tool, with GPT-4o judging.Source: §B.2 · p. 16
  • Figure 3 presents a three-step taxonomy of self-improvement in MLLMs, with methods varying according to requirements.Source: Figure 3 · p. 17

B.3 Healthcare

The survey describes self-improvement applications in healthcare, personalization, 3D, and embodied intelligence, then identifies multimodal challenges and future directions involving verification, generalization, scalability, and autonomy.

B.3 Healthcare

  • STLLaVA-Med achieves zero-shot performance on VQA-RAD, SLAKE, and PathVQA using only 9% of available medical data; LLaVA-ASD supports explainable screening.STLLaVA-Med uses a larger model as a biomedical expert to guide DPO fine-tuning on auto-generated medical visual instruction data.Source: §B.3 · p. 18

B.4 Personalization

  • Automated dataset construction and training pipelines let users personalize MLLMs for their own use with minimal additional effort.Source: §B.4 · p. 18

B.5 3D and Embodied Intelligence; C.2 Better Seed Models and Emerging Modalities

  • In 3D and embodied intelligence, MLLMs transfer 2D understanding via projected masks, while SELU and MART improve agents; expansion to audio, 3D, and actions increases action-space complexity.MLLMs equipped with 3D encoders enhance spatial-relation understanding, fundamental to application in 3D environments.Source: §B.5 · p. 18, §C.2 · p. 19

C.1 Uniqueness of Multi-Modality

  • Multimodal tasks require spatial and temporal objectives beyond text-sequence prediction, and cross-modal alignment or distillation without high-quality data might introduce hallucinations.Source: §C.1 · pp. 18–19

C.3 Omni I/O

  • Restricted input/output pipelines often require external datasets or separate models to generate non-text inputs, limiting self-verification and correction; forcing these functions may compound hallucinations.Omni I/O would let models handle arbitrary modality combinations as both input and output and generate their own multimodal training data.Source: §C.3 · p. 19

C.4 Biases and Robust Verification

  • Generated data requires verification and organization, but these steps do not guarantee bias or errors are eliminated; reliable, efficient verification remains essential for iterative improvement.Environmental feedback and cross-modal consistency checks could provide verification signals.Source: §C.4 · pp. 19–20

C.5 Generalization

  • Task-specific self-improvement may plateau, motivating a general framework that recursively improves across universal tasks and potentially adapts architecture, learning algorithms, and knowledge representation.Source: §C.5 · p. 20

C.6 Scalability; C.7 Higher Autonomy

  • Current frameworks are often small-scale, and despite reducing data-generation and verification workload, still require human ideation, coding, experiments, and outcome evaluation.The survey identifies satisfactory scalability and greater autonomy, such as R&D automation, as directions for addressing these constraints.Source: §C.6 · p. 20, §C.7 · p. 20

D Related Surveys

Existing surveys address MLLMs or self-improvement in text-only LLMs, while this survey focuses on self-improvement in MLLMs. It organizes methods into data collection, data organization, and model optimization to guide future development.

D Related Surveys

  • Existing surveys cover MLLMs (Yin et al., 2024) or LLM self-improvement (Tao et al., 2024), but not specifically self-improvement in MLLMs.Source: D Related Surveys · p. 20
  • Adjacent surveys cover MLLM reinforcement learning as a specific self-improvement domain and self-evolving agents rather than MLLMs (Sun et al., 2025b; Wu et al., 2025b; Gao et al., 2025; Fang et al., 2025).Source: D Related Surveys · p. 20
  • The survey structures MLLM improvement research into data collection, data organization, and model optimization to analyze techniques across modules.Source: D Related Surveys · p. 20
  • The survey aims to guide future development toward more effective self-improvement methodologies for MLLMs.Source: D Related Surveys · p. 20

Related papers

Loading 2510.02665v1…