PaperlayerView exact revision on arXiv

Source-linked AI summary

Counteracting Matthew Effect in Self-Improvement of LVLMs through Head-Tail Re-balancing

Xin Guo, Zhiheng Xi, Yiwen Ding, Yitao Zhai, Xiaowei Shi, Xunliang Cai, Tao Gui, Qi Zhang, Xuanjing Huang

arXiv:2510.26474v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Iterative self-improvement can develop a Matthew effect in which successful trajectories increasingly favor simple queries over difficult ones, contributing to performance bottlenecks. The paper introduces four head-tail rebalancing strategies and reports improved visual reasoning across two LVLMs. Its conclusion identifies larger models and broader datasets as directions for future work.

  • Problem

    During iterative LVLM self-improvement, simple-query trajectories increasingly dominate while difficult queries are squeezed out, contributing to performance bottlenecks.

    Source: §1 · p. 2

  • Method

    The paper combines threshold clipping and repeat-based padding with adaptive-weighted and guided trajectory resampling to rebalance head and tail data.

    Source: §1 · p. 2

  • Results

    The rebalancing strategies reduce head dominance and increase tail proportion, improving performance ceilings and visual reasoning performance.

    Source: §1 · p. 2, §7 · p. 11

  • Takeaways & Limitations

    Head-tail rebalancing is reported to mitigate the Matthew effect in visual self-improvement across the evaluated models and tasks.

    Source: §1 · p. 2, §7 · p. 11

    The conclusion identifies evaluation on larger models and broader datasets as future work.

    Source: §7 · p. 11

Abstract

from arXiv · show

Self-improvement has emerged as a mainstream paradigm for advancing the reasoning capabilities of large vision-language models (LVLMs), where models explore and learn from successful trajectories iteratively. However, we identify a critical issue during this process: the model excels at generating high-quality trajectories for simple queries (i.e., head data) but struggles with more complex ones (i.e., tail data). This leads to an imbalanced optimization that drives the model to prioritize simple reasoning skills, while hindering its ability to tackle more complex reasoning tasks. Over iterations, this imbalance becomes increasingly pronounced--a dynamic we term the "Matthew effect"--which ultimately hinders further model improvement and leads to performance bottlenecks. To counteract this challenge, we introduce four efficient strategies from two perspectives: distribution-reshaping and trajectory-resampling, to achieve head-tail re-balancing during the exploration-and-learning self-improvement process. Extensive experiments on Qwen2-VL-7B-Instruct and InternVL2.5-4B models across visual reasoning tasks demonstrate that our methods consistently improve visual reasoning capabilities, outperforming vanilla self-improvement by 3.86 points on average.

1. Introduction

The introduction frames iterative self-improvement as a response to scarce high-quality SFT annotations, but identifies an accumulating head-tail imbalance as a source of visual-reasoning bottlenecks. It proposes four rebalancing strategies and reports gains across two LVLMs.

1. Introduction

  • Self-improvement lets LVLMs iteratively explore and learn from successful trajectories, while high-quality annotated datasets for SFT remain insufficient.The introduction reports that self-improvement with varying sampling numbers reaches bottlenecks and can decline in visual reasoning.Source: §1 · pp. 1–2
  • Across iterations, simple high-accuracy queries increasingly dominate while difficult tail queries are squeezed out, a pattern linked to performance degradation.The introduction also describes a trend toward shorter average response lengths.Source: Figure 1 · p. 1, §1 · p. 2
  • The paper proposes four head-tail rebalancing strategies: threshold clipping, repeat-based padding, adaptive-weighted resampling, and guided resampling.They span distribution reshaping and trajectory resampling; guided resampling starts reasoning from varying intermediate steps.Source: §1 · p. 2
  • The proposed methods significantly enhance visual reasoning across Qwen2-VL-7B-Instruct and InternVL2.5-4B.Experiments use sampling numbers K=8 and K=16, respectively.Source: §1 · p. 2

2. Related Work

Prior work studies self-critique, correction, and data-efficient self-improvement, but leaves iterative behavior less thoroughly explored. Work on distribution bias has focused mainly on text reasoning, while this paper investigates visual scenarios and proposes rebalancing strategies.

2.1. Self-improvement in visual reasoning.

  • Prior visual self-improvement work includes self-critique, selective correction, and preference optimization, but does not thoroughly explore the iterative process.Some approaches use separate critic models, while others unify response generation and self-critique or selectively revise erroneous reasoning segments.Source: §2.1 · p. 3
  • The paper focuses on vanilla self-improvement and proposes targeted strategies to improve its performance.Related work describes self-critic and self-correction methods, often with separate critic models that require additional resources.Source: §2.1 · pp. 2–3

2.2. Distribution bias in self-generated data.

  • Prior distribution-bias studies mainly address text reasoning; SynthRL rebalances visual queries by rewriting them but does not investigate the iterative process.This paper reports an in-depth investigation of LVLM behavior during visual self-improvement and introduces four rebalancing strategies.Source: §2.2 · p. 3

3. “Matthew Effect” in Self-improvement

The section formulates iterative self-improvement as exploration, filtering, and learning, then examines how its data and responses become imbalanced over iterations. It reports performance bottlenecks and a growing head-tail imbalance in visual reasoning.

3.1. Formulating Self-improvement

  • Vanilla self-improvement initializes from Mbase and repeats exploration, filtering, and learning for T iterations.The section frames this process as underexplored in visual reasoning scenarios.Source: §3.1 · p. 3, §3 · p. 3

Exploration.

  • During exploration, the prior iteration’s model generates K responses for each base query.These generated responses are sampled at each iteration.Source: Exploration · p. 3

3.2. Plateaus in Varying Sampling Numbers

  • At iteration 1, K=16 outperforms K=4 by 3.89 points; by iteration 5, K=16 performs 0.39 points worse than K=4.Across sampling numbers, performance improves early, converges to bottlenecks, and can decline in later iterations.Source: Figure 2 · p. 4, §3.2 · p. 4

Differences between original and self-generated data

  • Self-generated data has 51.1% level-1 samples and nearly no level-5 samples, while its responses average 277 tokens versus 395 for original data.Some self-generated responses are under 10 tokens and omit chain-of-thought reasoning, even under CoT prompting.Source: Figure 2 · p. 4, Differences between original and… · p. 5

Filtering.

  • The next model is trained by fine-tuning the base model on the filtered dataset.Source: Learning · p. 4

Learning.

  • Learning minimizes negative log-likelihood on filtered data, and repeated cycles are described as producing higher-quality responses that enhance subsequent performance.Source: Learning · p. 4

Matthew effect over iterations

  • Across iterations, 100%-accuracy data takes a growing share while 25%-accuracy data is squeezed out of Dtrain, intensifying the difficult-tail shortfall.The authors call this distribution shift the Matthew effect and say it limits self-improvement performance ceilings.Source: Figure 3 · p. 5, Matthew effect over iterations · p. 5
  • Average response length declines across iterations; level-5 responses shrink by 56.5%, reaching 136 tokens by iteration 5 as the difficulty-length pattern reverses.Level-1 response length remains relatively stable, while higher sampling numbers mitigate the overall length decline.Source: Figure 3 · p. 5, Matthew effect over iterations · p. 5

4. Methodology

The methodology rebalances head and tail data through distribution reshaping and trajectory resampling. Its four strategies reduce head frequency, repeat underrepresented queries, weight resampling by failure rate, or guide exploration from intermediate reasoning steps.

4. Methodology

  • The framework combines distribution reshaping and trajectory resampling through four strategies: TC, RP, AR, and GR.The first pair changes sample frequencies; the second pair resamples trajectories.Source: §4 · p. 6

Threshold clipping

  • Threshold Clipping (TC) randomly truncates responses so each query contributes at most L correct responses, increasing the tail’s proportion by reducing the head.The training set uses filtered responses with indices up to L.Source: Threshold clipping · p. 6

Repeat-based padding

  • Repeated-based Padding (RP) repeats queries with too few correct samples until all queries appear with equal frequency K in the next training set.Source: Repeat-based padding · p. 6

Adaptive-weighted resampling

  • Adaptive-weighted Resampling (AR) adjusts query resampling weights by fail rate and performs K−k_i additional resampling operations for a query with k_i successful trajectories out of K.The authors note that duplication can reduce diversity and trigger overfitting, motivating resampling strategies.Source: Adaptive-weighted resampling · p. 6

Guided resampling

  • Guided Resampling (GR) starts exploration from different intermediate reasoning steps to navigate toward promising trajectories and progressively learn complex reasoning.The paper presents GR as an alternative to AR’s brute-force sampling, which has limited efficiency improvements.Source: Guided resampling · p. 7

5. Experiments

The experiments compare head-tail re-balancing with vanilla self-improvement across LVLMs and visual reasoning datasets. Re-balancing improves performance, mitigates head dominance, and helps models reach their peak performance in the final iteration.

5.1. Experimental Setups

Head-tail re-balancing improves the performance of self-improvement across various models

5.2. Main Results

  • 10.13 points: vanilla self-improvement raises Qwen2-VL-7B-Instruct's average performance at K=8, but doubling K yields only a 0.05-point optimal average gain at twice the computational cost.The reported 0.05-point difference compares K=16 with K=8.Source: §5.2 · p. 8

Head-tail re-balancing improves the performance of self-improvement across various models

  • Re-balancing strategies, especially GR, more often reach optimal performance in the final iteration, unlike vanilla self-improvement's frequent final-iteration shortfall.The paper describes this as evidence of greater stability and potential for further improvement.Source: Head-tail re-balancing improves the… · p. 8
  • 51.1% to 24.8%: RP reduces head data, while tail data rises from 1.5% to 6.6% on Qwen2-VL-7B-Instruct at K=16.AR shows limited tail augmentation; GR provides step-by-step guidance and raises performance from 41.36 to 43.94.Source: Head-tail re-balancing improves the… · p. 9, Figure 4 · p. 9

6. Discussion

The discussion examines re-balancing variants, sampling efficiency, self-correction, and visual inputs. It reports benefits from iterative sampling and re-balancing, including stronger tail-data reasoning when images are available.

6.1. Ablation Study

Variants of resampling strategies.

  • AR and GR at K=8 achieve comparable or better performance than vanilla self-improvement at K=16, with lower resampling requirements.AR uses K−k_i operations, about 50% fewer; GR resamples only once on minimal tail data.Source: Variants of resampling strategies · pp. 9–10

6.2. Self-improvement as Efficient Sampling

  • 3.34 and 1.79 points: RP outperforms batch sampling and vanilla iterative sampling, respectively, using 40 samples across five iterations.Iterative sampling also outperforms batch sampling, and distribution reshaping remains effective with iterative sampling.Source: §6.2 · p. 10, Table 3 · p. 10

6.3. Self-correction Benefits Self-improvement

  • Figure 5 compares average performance for vanilla and self-correction in visual self-improvement.The supplied passage identifies the comparison but gives no outcome.Source: Figure 5 · p. 10

6.4. The Power of “Seeing”

  • 18.8 points: RP's advantage when using images on tail data, as re-balanced strategies further amplify the reasoning gain from vanilla self-improvement.The comparison evaluates tail-data performance with and without images.Source: §6.4 · p. 11, Figure 6 · p. 11

7. Conclusion

The conclusion identifies an iterative head-tail imbalance as a performance bottleneck in visual self-improvement and presents four strategies to rebalance successful trajectories. Experiments report improved performance ceilings; future work is directed toward larger models, broader datasets, and more efficient re-balancing.

7. Conclusion

  • The Matthew effect describes simple head samples progressively dominating successful trajectories while difficult tail data becomes increasingly narrow.The conclusion identifies this imbalance as a challenge behind performance bottlenecks in visual self-improvement.Source: §7 · p. 11
  • The four strategies are threshold clipping, repeat-based padding, adaptive-weighted resampling, and guided resampling, spanning distribution reshaping and trajectory resampling.They are intended to counteract the Matthew effect during self-improvement.Source: §7 · p. 11
  • The strategies reduce head dominance and increase tail proportion, improving performance ceilings; future work will study larger models, broader datasets, and more efficient re-balancing.Source: §7 · p. 11

A.1. Dataset Details

The appendix describes MMPR-mini and its out-of-domain evaluation datasets, then explains difficulty categorization and the shared prompt setup. MMPR-mini contains mathematical visual reasoning queries with ground truth but no chain-of-thought trajectories.

A.1. Dataset Details

  • MMPR-mini contains 7,980 randomly extracted MMPR samples, with 7,183 for training and 797 for in-domain testing; MathVerse and We-Math provide out-of-domain evaluation.The subset focuses on mathematical visual reasoning queries in multiple formats, with ground truth but no CoT trajectories.Source: §A.1 · p. 17

A.2. Difficulty Level Categorization

  • Queries are assigned five difficulty levels using pass@64 performance, with levels ordered from easiest to most difficult.The categorization prioritizes a balanced distribution across difficulty levels.Source: §A.2 · p. 17

A.3. Prompt Details

  • A unified prompt template is used for training, sampling, and testing across datasets.Figure 7 presents the prompt for those phases.Source: §A.3 · p. 17, Figure 7 · p. 17

B. Error Type Analysis

The analysis classifies incorrect responses into five error types, reports how self-improvement and rebalanced strategies affect them, and illustrates repeat-based padding and guided resampling.

B. Error Type Analysis

  • Incorrect responses are classified as No Reasoning Process, Comprehension Error, Knowledge Error, Logic Error, or Format Error.GPT-4.1 performs the classification with temperature 0.1 and up to three outputs.Source: B. Error Type Analysis · pp. 17–18
  • Self-improvement reduces comprehension and logic errors, while rebalanced strategies further improve question and image comprehension and reasoning logic.Error patterns vary by model; Qwen2-VL-7B-Instruct shows more No Reasoning Process errors under threshold clipping, while InternVL2.5-4B shows few.Source: B. Error Type Analysis · pp. 18–19
  • Rare correct responses were misclassified as incorrect under exact-match rules when π appeared in different formats.The authors report that these cases were manually examined.Source: B. Error Type Analysis · p. 19

C. Case Study

  • Repeat-based padding and guided resampling correct vanilla self-improvement errors involving geometric relationships and confusion between height and radius.The examples show improved visual comprehension and geometric element understanding.Source: C. Case Study · p. 19

D. Matthew Effect Mitigation

Across additional model and sampling configurations, the proposed strategies mitigate the Matthew effect; repeat-based padding performs best overall, and guided resampling generally exceeds adaptive-weighted resampling.

D. Matthew Effect Mitigation

  • The additional configurations examine Qwen2-VL-7B-Instruct at K=8 and InternVL2.5-4B at K=8 and K=16.The difficulty-level distributions are shown in Figures 12–14.Source: D. Matthew Effect Mitigation · p. 19
  • Repeat-based padding shows the best Matthew effect mitigation overall, while guided resampling generally outperforms adaptive-weighted resampling.The authors report that this ordering aligns with the strategies’ test-set performance patterns.Source: D. Matthew Effect Mitigation · p. 19
  • A Figure 9 example was misclassified under exact-match rules because π appeared in different formats.Source: Figure 9 · p. 20

E. Batch Sampling and Iterative Sampling

The section compares batch and iterative sampling, then explores using self-correction to refine outputs for later self-improvement training.

E. Batch Sampling and Iterative Sampling

  • Iterative sampling improves the in-domain test set by 5.65 points and average performance by 3.17 points on Qwen2-VL-7B-Instruct.The comparison combines 16 samples across five iterations for iterative sampling and contrasts them with 80 direct samples from the base model.Source: E. Batch Sampling and Iterative Sampling · p. 20, Table 6 · p. 21
  • Distribution-reshaping strategies do not consistently perform well in iterative sampling; threshold clipping reduces data quantity and repeat-based padding can cause excessive repetition.The authors suggest these factors may hinder the strategies in efficient sampling settings.Source: E. Batch Sampling and Iterative Sampling · p. 20

F. Applying Self-correction to Self-improvement

  • Self-correction refines model outputs and incorporates them into subsequent training iterations.The resulting training data excludes failed corrections and samples with insufficient chain-of-thought reasoning length.Source: F. Applying Self-correction to… · pp. 21–22
  • Applying self-correction to self-improvement yields substantial performance gains, with iterative correction enabling longer reasoning chains that can correct vanilla self-improvement errors.The example shows correction of identification errors during tangent calculations on MMPR-mini.Source: F. Applying Self-correction to… · p. 22, Figure 17 · p. 27

Related papers

Loading 2510.26474v1…