PaperlayerView exact revision on arXiv

Source-linked AI summary

Injecting Distributional Awareness into MLLMs via Reinforcement Learning for Deep Imbalanced Regression

Yao Du, Shanshan Song, Xiaomeng Li

arXiv:2605.01402v2cs.CLcs.CVcs.LG

TL;DR

MLLMs struggle with long-tailed numerical regression because token-level and pointwise supervision favors dense target regions and leaves cross-sample structure underused. The paper introduces CCC-GRPO, a batch-level distribution-aware reinforcement-learning framework, and reports consistent benchmark improvements, especially in medium- and few-shot regimes. The approach also addresses a scope gap in prior DIR methods, which target non-generative real-valued regressors rather than token-based MLLMs.

  • Problem

    Token-level SFT and pointwise rewards bias MLLM regression toward dense regions, causing regression-to-the-mean behavior and unreliable predictions for under-represented targets.

  • Method

    The paper uses GRPO with a CCC-based batch-level relational reward that aligns predicted and ground-truth distributions in correlation, scale, and mean.

  • Results

    CCC-GRPO consistently improves over SFT and existing MLLM regression methods across unified long-tailed benchmarks, with particularly strong gains in medium- and few-shot regimes.

  • Takeaways & Limitations

    Distribution-aware cross-sample supervision provides a practical way to improve tail-region robustness without architectural modification or task-specific heuristics.

  • Takeaways & Limitations

    Practical deployment requires subgroup-level evaluation, fairness auditing, and calibration because distribution-level objectives do not guarantee equitable performance across demographic subpopulations.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) struggle with numerical regression under long-tailed target distributions. Token-level supervised fine-tuning (SFT) and point-wise regression rewards bias learning toward high-density regions, leading to regression-to-the-mean behavior and poor tail performance. We identify the lack of cross-sample relational supervision as a key limitation of existing MLLM training paradigms. To address it, we propose a distribution-aware reinforcement learning framework based on Group Relative Policy Optimization, which introduces batch-level comparison-based supervision via the Concordance Correlation Coefficient-based reward to align predicted and ground-truth distributions in terms of correlation, scale, and mean. The framework is plug-and-play, requiring no architectural modification. Experiments on a unified suite of long-tailed regression benchmarks show consistent improvements over SFT and existing MLLM regression methods, with particularly strong gains in medium- and few-shot regimes.

1. Introduction

Long-tailed continuous targets expose a mismatch between token-level MLLM supervision and numerical regression, producing regression-to-the-mean behavior. The paper proposes batch-level, distribution-aware reinforcement learning and a unified benchmark to address this limitation.

  • Motivation: Long-tailed targets concentrate training data in dense regions while sparsely representing many valid values, making numerical regression difficult for MLLMs.MLLMs generate continuous quantities as discrete token sequences optimized with next-token prediction.
  • Motivation: Token-level SFT ignores numerical proximity, ordering, and global magnitude, amplifying dominant patterns and weakening corrective signals for rare targets.This misalignment leads to pronounced regression-to-the-mean behavior.
  • Approach: Standard GRPO improves sequence-level supervision but remains pointwise, whereas CCC-GRPO compares predictions within minibatches to provide distribution-aware relational supervision.The proposed objective jointly considers each output and the mean predictions of other samples.
  • Evaluation: The unified benchmark reformulates four long-tailed numeric datasets into a multimodal dialogue setting with over 129k samples and shot-aware balanced test splits.This protocol preserves natural long-tailed training distributions for systematic and fair evaluation.
  • Approach: CCC-GRPO uses batch-level correlation-guided rewards to align predicted and ground-truth distributions and mitigate collapse without modifying model architecture.The design targets robustness in sparse and tail regions.
  • Evaluation: Experiments and ablations show that batch-level distribution-aware supervision improves stability and accuracy in under-represented regions.The empirical analysis establishes evidence for regression-oriented alignment of MLLMs.

2. Related Work

Prior deep imbalanced regression methods target non-generative models with explicit continuous prediction heads, while MLLM approaches must handle tokenized autoregressive outputs. Existing reinforcement-learning approaches similarly provide context for sequence-level optimization but do not resolve this specific mismatch.

  • Deep Imbalanced Regression: Deep imbalanced regression methods address skewed continuous targets through distribution smoothing, structural consistency, ranking regularization, or uncertainty modeling.These methods were developed primarily for feature-based regression models.
  • Deep Imbalanced Regression: Existing DIR methods assume direct optimization over real-valued outputs and do not account for tokenized supervision, autoregressive decoding, or sequence-level optimization in MLLMs.Their formulation therefore does not address how these mechanisms interact with long-tailed continuous targets.
  • Numerical Regression in MLLMs: MLLM numerical-regression approaches include architectural modifications, explicit reasoning procedures, and methods designed to bridge discrete and continuous representations.Evaluations also report numerical-perception deficiencies despite scaling or chain-of-thought prompting.
  • Reinforcement Learning for Post-training: Reinforcement learning enables sequence-level objectives beyond next-token prediction, extending MLLM post-training beyond token-level supervision.Early approaches commonly rely on point-wise scalar rewards, while later work explores reward scaling, distributional robustness, and group-wise comparisons.

3. Method

The method uses GRPO to provide value-level, batch-aware supervision for MLLM regression. It compares sampled predictions with within-minibatch context and rewards distributional agreement using CCC.

  • Multi-Generation Regression Outputs: GRPO samples multiple generation trajectories for each input and summarizes their numeric outputs with an empirical mean.The mean provides a stable, low-variance estimate during reward computation.
  • Batch-Level Relational Comparison: Each sampled prediction is evaluated against a relational comparison set containing mean predictions from other minibatch samples.This replaces isolated point-wise evaluation with contextual batch-level comparison.
  • Batch-Level Relational Comparison: The comparison construction uses other samples’ empirical means as stable anchors, reducing reward noise and avoiding entanglement among their stochastic generations.Ground-truth and prediction vectors are ordered by fixed minibatch indices for deterministic comparison.
  • Concordance-Based Distributional Reward: CCC serves as the distributional reward, measuring agreement between predicted values and ground-truth targets.CCC captures correlation, scale consistency, and mean alignment, while penalizing variance collapse and mean shift.
  • Optimization: A lightweight format-validity reward supports stable computation, while GRPO normalizes sampled rewards within each group to calculate relative advantages.The framework is presented as robust numeric supervision without architectural modification.

4. Experiments

Experiments use a unified, shot-aware benchmark covering four long-tailed multimodal regression tasks. CCC-GRPO consistently improves under-represented-region performance while retaining competitive dense-region accuracy, with ablations attributing gains to batch-level distribution-aware rewards.

  • Benchmark and evaluation: The benchmark contains over 129k samples across four multimodal regression tasks, preserving naturally imbalanced training distributions and using approximately balanced test ranges.The tasks cover age estimation, movie-rating prediction, and skeletal-maturity estimation.
  • Benchmark and evaluation: Shot-aware evaluation partitions targets into many-shot, medium-shot, and few-shot regions and reports MAE alongside GM to assess accuracy and error uniformity.MAE measures average error, while GM complements it by penalizing concentrated or frequent errors.
  • Main results: CCC-GRPO consistently improves under-represented regions across four tasks while maintaining competitive performance in dense regions.The main comparisons use matched MLLM backbones, prompting formats, and decoding protocols.
  • Main results: 7.67 to 5.62 medium-shot MAE and 8.36 to 6.40 few-shot MAE are achieved over SFT on AgeDB-DIR with Qwen2.5-VL-3B.Many-shot performance remains stable.
  • Main results: 8.12 medium-shot MAE and 16.35 few-shot MAE replace SFT values of 11.21 and 21.51 on IMDB-Movie-DIR under Qwen2.5-VL-3B.CCC-GRPO also improves over Regression Reward, whose corresponding values are 10.51 and 21.14.
  • Main results: CCC-GRPO attains the lowest MAE in nearly all shot regimes on IMDB-WIKI-DIR and the strongest GM in many- and medium-shot regions.On BoneAge-DIR, it suppresses extreme errors particularly in sparse regions, although GM can increase slightly in some many-shot settings.
  • Main results: +23.55% overall MAE improvement over SFT is reported for BoneAge-DIR in the larger-backbone evaluation, where the task is substantially more challenging.The method remains effective despite BoneAge-DIR’s multi-peaked training-label distribution.
  • Ablation study: Batch-level CCC rewards provide the best overall trade-off in ablations by jointly enforcing correlation, scale, and mean consistency.Performance differences across GRPO variants are minor, and results remain stable across sampled-generation counts and batch sizes.

5. Conclusions

The paper identifies point-wise, token-level supervision as fundamentally limited for long-tailed continuous regression in MLLMs. It addresses this with batch-level relational reinforcement learning and reports improved robustness in medium- and few-shot regions without architectural modification.

  • Conclusion: Point-wise, token-level objectives bias optimization toward dense regions, encourage regression-to-the-mean behavior, and fail to preserve global numeric structure.These limitations produce unreliable predictions in under-represented regimes.
  • Conclusion: The proposed GRPO framework replaces isolated per-sample errors with batch-level relational comparison and uses a CCC reward to align correlation, scale, and mean.The approach requires no architectural modification or task-specific heuristics.
  • Conclusion: The method substantially improves robustness in medium- and few-shot regions, with experiments and ablations identifying distribution-aware batch-level supervision as the primary performance driver.The conclusion attributes gains to supervision design rather than architectural changes.

Impact Statement

The paper notes that distribution-level optimization may interact with demographic bias and safety requirements, so deployment requires subgroup evaluation and application-specific safeguards.

  • Distribution-level objectives do not guarantee equitable performance across demographic subgroups and may shift error distributions.The paper recommends subgroup evaluation, fairness auditing, and calibration analysis.
  • In safety-critical applications such as BoneAge estimation, distribution-level objectives may trade off reliability on common cases without appropriate constraints.The method is presented as a training-time strategy rather than a deployment prescription.

A. Extended Experimental Results

Extended experiments show that CCC-GRPO improves errors primarily for medium-, few-shot, and sparse target regions while maintaining comparable head-region accuracy, with a documented head–tail trade-off under extreme imbalance.

  • Sorted Error Distribution Analysis: CCC-GRPO’s sorted error curves lie below SFT’s in medium- and few-shot regions across datasets, indicating lower errors on under-represented targets.The methods are comparable on many-shot samples, while CCC-GRPO suppresses extreme high-error cases.
  • MAE Gain Across Density Regions: MAE improvements over SFT concentrate in low- and medium-density regions, while dense-region performance remains comparable.This pattern reflects mitigation of regression collapse in sparse regimes without explicitly sacrificing head-region accuracy.
  • Complementary Error Metrics: CCC-GRPO shows strong performance across MSE, MAE, and GM, with particularly pronounced MSE gains in sparse regions.Because MSE amplifies large errors, these gains highlight reduced extreme deviations.
  • Head–Tail Trade-off under Extreme Imbalance: On IMDB-Movie-DIR, many-shot performance mildly degrades, reflecting a trade-off between tail reliability and mean-optimal accuracy under extreme imbalance.The paper proposes hybrid objectives balancing head and tail performance as future work.
  • Interpreting GM Degradation on BoneAge-DIR: On BoneAge-DIR, CCC-GRPO improves MAE and shot-region uniformity but increases GM error in many-shot regions.The paper attributes this to GM’s multiplicative sensitivity, label ambiguity, and a small fraction of extreme-tail samples rather than uniform degradation.
  • Scaling to Larger MLLM Backbones: Scaling from Qwen2.5-VL-3B to 7B preserves the qualitative pattern of improved medium- and few-shot robustness and suppressed extreme errors.

B. Dataset Construction and Imbalance Characteristics

The benchmark preserves naturally imbalanced training distributions while using approximately balanced test sets over the supported target range.

  • All benchmarks use naturally imbalanced training distributions and approximately balanced test sets for evaluation.Figures 13 and 14 visualize the contrasting training and testing target distributions.

B.1. AgeDB-DIR

The paper constructs a unified MLLM regression benchmark from four naturally imbalanced tasks, evaluates on balanced test distributions, and standardizes numeric-only outputs and reward parsing.

  • Imbalance Characteristics: Training distributions remain naturally imbalanced while evaluation uses approximately balanced test sets over the supported target range.
  • AgeDB-DIR: AgeDB-DIR contains 12,208 training images and 2,140 balanced-test images spanning ages 0–100, with bin densities ranging from 1 to 353.
  • Evaluation Protocol: Models are evaluated in a pure numeric-prediction setting without chain-of-thought, explanatory text, or intermediate steps.Each task requires a single numeric output.
  • Output and Reward Handling: Outputs use <answer> tags, and numeric parsing extracts the first enclosed value while invalid or out-of-range generations receive no format reward.Valid formatted outputs receive the constant reward c = 0.5.

C. Experimental Details

The experiments compare MLLM-specific baselines, conventional deep imbalanced regression methods, and several point-wise, ranking-based, and correlation-based rewards. The proposed CCC reward extends batch-level relational supervision by aligning correlation, scale, and mean.

  • Baselines and setup: The benchmark compares Qwen2.5-VL backbones using zero-shot generation, token-level SFT, classical DIR methods, and reward-based or reinforcement-learning baselines.Classical DIR methods were originally designed for non-generative regression pipelines with explicit continuous prediction heads.
  • Implementation: The experiments use GRPO post-training with Qwen2.5-VL-3B/7B, LoRA adapters, AdamW, four sampled generations per input, and a KL coefficient of 0.04.The main-paper implementation uses rank-64 LoRA adapters with scaling factor 128 and an initial learning rate of 1 × 10^-5.
  • Reward baselines: Standard regression rewards optimize point-wise absolute error but remain dominated by high-frequency regions and prone to regression-to-the-mean behavior.DISCO MAE adds frequency-aware reweighting so rarer bins exert stronger influence during optimization.
  • Correlation and ranking rewards: Spearman rewards encourage batch-level monotonic consistency while remaining agnostic to absolute scale.They can improve tail performance but may sacrifice calibration in dense regions.
  • CCC reward: CCC-based rewards jointly align predicted and ground-truth correlation, scale, and mean across samples.This provides continuous numeric supervision beyond ordering consistency and point-wise error.

D. Extended Discussion and Analysis

The discussion frames long-tailed MLLM regression as a batch-level distributional problem and explains how CCC rewards counter collapse through covariance, variance, and mean alignment. It also identifies batch composition, computational cost, dimensional scope, and empirical coverage as boundaries.

  • Batch-level supervision: Batch-relative supervision evaluates each prediction against other minibatch samples, making the learning signal sensitive to distributional structure rather than marginal accuracy alone.The formulation can expose tail samples to non-vanishing supervision without explicit reweighting or resampling.
  • Batch statistics: Performance improvements saturate with moderate batch sizes and a small number of sampled generations, but extremely small or non-representative batches may make rewards noisy.The method does not require label-aware or stratified batching and remains effective under standard random sampling.
  • Scope: The framework is demonstrated on scalar regression, while broader empirical validation beyond representative MLLM benchmarks remains future work.Its claimed generality concerns the learning principle rather than the breadth of empirical coverage.
  • Context construction: Mean predictions from other samples provide a stable, low-variance contextual reference for estimating batch-level distributional structure.This avoids unstable coupling among all stochastic generations, whose possible relational orderings scale as K^(N−1).
  • Computational considerations: GRPO training increases computational cost: AgeDB-DIR takes approximately 3 hours versus around 30 minutes for SFT under the stated configuration.The added cost comes from reinforcement learning with multiple sampled trajectories.
  • Why distribution-aware supervision matters: Point-wise objectives are dominated by many-shot regions, allowing head-centered predictions to achieve low average error while performing poorly on tail values.This produces the classic regression-to-the-mean failure mode under increasing imbalance.
  • CCC mechanism: CCC penalizes degenerate constant predictions through covariance and variance while also penalizing mean shifts and scale mismatch.Unlike pure rank objectives, it therefore preserves absolute numeric structure as well as ordering.
  • CCC mechanism: For tail samples, collapse toward the head region lowers covariance with batch targets and increases mean mismatch, producing lower CCC rewards.This creates a tail-sensitive signal without explicit reweighting or resampling.
Loading 2605.01402v2…