PaperlayerView exact revision on arXiv

Source-linked AI summary

ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Models

Thomas De Min, Subhankar Roy, Stéphane Lathuilière, Elisa Ricci, Massimiliano Mancini

arXiv:2603.19466v2cs.CV

TL;DR

MLLMs have not been adequately evaluated on whether they seek help when visual information is ambiguous or insufficient. The paper introduces ProactiveBench, a seven-dataset benchmark spanning intervention-based multimodal tasks, and evaluates current models and reinforcement-learning fine-tuning. Existing models generally lack proactiveness, while fine-tuning improves the behavior even in unseen scenarios, though substantial accuracy gaps remain.

  • Problem

    Existing studies focus on reactive MLLM settings, leaving whether models can request help for ambiguous or unanswerable visual queries unknown.

  • Method

    The paper introduces ProactiveBench by repurposing seven datasets to evaluate proactive visual interventions across diverse tasks and interaction settings.

  • Results

    Current MLLMs generally lack proactiveness, with 8.2% accuracy on ROD versus 98.3% in the reference setting, while hinting provides only marginal accuracy gains.

  • Takeaways & Limitations

    Reinforcement-learning fine-tuning improves proactive behavior and generalizes to unseen scenarios, supporting proactiveness as a learnable capability.

  • Takeaways & Limitations

    The average accuracy gap remains large, with 40.7% in the proactive setting versus 75.1% with reference images.

Abstract

from arXiv · show

Effective collaboration begins with knowing when to ask for help. For example, when trying to identify an occluded object, a human would ask someone to remove the obstruction. Can MLLMs exhibit a similar "proactive" behavior by requesting simple user interventions? To investigate this, we introduce ProactiveBench, a benchmark built from seven repurposed datasets that tests proactiveness across different tasks such as recognizing occluded objects, enhancing image quality, and interpreting coarse sketches. We evaluate 22 MLLMs on ProactiveBench, showing that (i) they generally lack proactiveness; (ii) proactiveness does not correlate with model capacity; (iii) "hinting" at proactiveness yields only marginal gains. Surprisingly, we found that conversation histories and in-context learning introduce negative biases, hindering performance. Finally, we explore a simple fine-tuning strategy based on reinforcement learning: its results suggest that proactiveness can be learned, even generalizing to unseen scenarios. We publicly release ProactiveBench as a first step toward building proactive multimodal models.

1 Introduction

ProactiveBench addresses the unexplored question of whether MLLMs can request visual help when queries are ambiguous or unanswerable. It evaluates this behavior across diverse intervention scenarios and finds limited proactiveness in current models, while fine-tuning improves it even on unseen scenarios.

  • Motivation: Current MLLM studies largely focus on reactive settings, leaving their ability to seek help under ambiguity unknown.Reactive behavior includes hallucinating or abstaining when information is insufficient.
  • Benchmark: ProactiveBench evaluates whether MLLMs can request user interventions that provide additional visual cues for answerable queries.The benchmark repurposes seven existing datasets for tasks including sketch recognition and product identification.
  • Benchmark: The benchmark covers occlusion removal, camera and object movement, image-quality enhancement, and requests for additional visual details.Each sample includes an ambiguous starting frame, a complete reference frame, and intervening frames.
  • Findings: Evaluation of 22 MLLMs reveals limited proactiveness, even when models are explicitly hinted to behave proactively.The authors report that models often abstain or hallucinate instead of requesting useful visual cues.
  • Findings: Fine-tuning for proactiveness improves the behavior in unseen scenarios, suggesting that proactive responses can be learned.The paper presents this result as a promising direction for building proactive multimodal models.

2 Related work

Prior MLLM benchmarks assess visual question answering, reasoning, world knowledge, and multimodal inputs, while embodied-AI work studies agents and active sensing. However, these lines of research do not evaluate proactiveness for ambiguous or unanswerable queries.

  • MLLM benchmarking: MLLM benchmarks have expanded from visual question answering to reasoning, world knowledge, and multiple-image or video inputs.These benchmarks target increasingly complex multimodal capabilities.
  • Research gap: Existing benchmarks and related agent research do not evaluate proactiveness when queries are ambiguous or unanswerable.This distinction motivates a dedicated benchmark for requesting information or intervention.
  • Related paradigms: Embodied-AI research evaluates language models integrated with agents, while active vision studies dynamic control of sensing strategies.Active-vision applications include viewpoint planning, object recognition, reconstruction, and manipulation.

3 ProactiveBench

ProactiveBench evaluates whether MLLMs can answer or request help through structured and open-ended interactions across seven intervention scenarios. Its construction filters out cases solvable without intervention, leaving a benchmark focused on proactive behavior.

  • Evaluation: Proactiveness is defined as either providing a correct answer or requesting help that makes an otherwise unanswerable query answerable.The benchmark evaluates this definition in multiple-choice and open-ended settings.
  • Evaluation: In MCQA, the MLLM acts as a policy over finite states and actions, with proactive suggestions transitioning the environment to new visual states.Abstaining or selecting a wrong category terminates the evaluation with an incorrect prediction.
  • Evaluation: OEG evaluation is restricted to single-turn scenarios because free-form answers and proposed actions are difficult to interpret and may be inapplicable.An LLM judge scores valid category predictions and proactive suggestions, including equivalent action descriptions.
  • Scenarios: ProactiveBench spans seven scenarios involving occlusion, temporal ambiguity, uninformative views, image quality, extra sketch details, and camera movement.Interventions include moving objects, inspecting other frames, changing viewpoints, improving quality, and requesting additional strokes.
  • Construction: The benchmark uses sequences with ambiguous initial frames and more informative frames, pairing datasets with multi-turn MCQA interactions and broader OEG suggestions.The construction exploits ordered frames where available and uses existing test or validation data.
  • Filtering: Filtering removes samples that models can solve at the first turn, reducing one dataset’s eligible proportion from 32.5% to 6.4% and leaving 7,557 of 17,909 samples.This procedure aims to ensure that proactive suggestions are needed for strong scores.

4 Are MLLMs proactive?

ProactiveBench evaluates whether MLLMs request useful interventions rather than abstaining or guessing when visual information is insufficient. Across MCQA and OEG, current models generally perform poorly, show no simple scale relationship, and can be misled by hints, conversation histories, and in-context examples.

  • Evaluation protocol: ProactiveBench evaluates 22 open- and closed-weight MLLMs using accuracy and proactive suggestion rate across its splits.The benchmark protocol feeds models the question, current image, and, for MCQA, valid action options; OEG reports correct categories and valid proactive suggestions separately.
  • MCQA results: Proactiveness does not increase monotonically with model size: InternVL3-1B reaches 27.1% accuracy versus 12.7% for InternVL3-8B, while LLaVA-1.5-7B reaches 24.8% versus 13.0% for LLaVA-OV-72B.The reported comparisons also show proactive suggestion rates of 0.7 versus 0.3 and 0.9 versus 0.3, respectively.
  • MCQA results: 79.8% reference accuracy falls to more than a 60% underperformance when models must navigate ambiguous scenarios through proactive suggestions.On ROD, average accuracy is 8.2% with proactive navigation versus 98.3% with reference frames.
  • OEG results: In OEG, models still tend to abstain or hallucinate, and multi-turn performance remains extremely low: Qwen2.5-VL-7B reaches 3.9% accuracy with 0.1 psr.The paper reports that OEG performance is worse than MCQA and finds no simple monotonic relationship between model size and proactiveness.
  • Eliciting proactiveness: Hints increase MCQA psr by 1.9 on average, but accuracy averages only 25.8% (+8.3%) and models blindly select proactive suggestions in 16.0% of cases.Hinted actions can be preferred over correct predictions, while conversation history lowers average accuracy by 7% and raises psr from 0.5 to 1.8.

5 Can MLLMs learn proactiveness from data?

The paper tests whether proactiveness can be learned through reinforcement-learning post-training, using QuickDraw and COCO data to assess generalization to unseen scenarios. RL improves proactive behavior, but substantial accuracy gaps remain compared with reference images.

  • Training setup: Training on QuickDraw and COCO enables evaluation of generalization to unseen proactive scenarios.The datasets cover abstract and natural images and provide sufficient training data.
  • Training setup: Reinforcement learning jointly rewards correct predictions and valid proactive suggestions while favoring response efficiency.Correct predictions receive rc = 1, valid proactive suggestions receive rp ∈{0.5, 0.75, 1.0}, and wrong answers receive rw = 0.
  • Results: Lower proactive-suggestion rewards generally balance effectiveness and efficiency better than rewards equal to correct predictions.The paper reports this balance for rp < rc, while larger rp values increase proactive-suggestion rates and can cause excessive proactive behavior.
  • Results: 37.4% versus 34.0% of o4-mini is achieved by the evaluated RL-tuned models, except Qwen2.5-VL-3B with rp = 1.0.The comparison refers to accuracy and the previously evaluated models in Table 1.
  • Limitations: 40.7% versus 75.1% is the average accuracy gap between the learned models and the reference setting.The remaining gap indicates open challenges in correctly eliciting proactiveness, despite encouraging generalization.

6 Conclusion

ProactiveBench evaluates whether MLLMs can request human intervention when visual inputs are ambiguous or unanswerable. The findings show limited current proactiveness, marginal gains from hinting, harmful conditioning biases, and promising but incomplete learning through data.

  • Benchmark and findings: ProactiveBench evaluates proactiveness when human intervention is required to make visual queries answerable.It repurposes seven datasets into seven scenarios spanning single- and multi-turn MCQA and OEG settings.
  • Benchmark and findings: Existing MLLMs generally prefer abstaining or hallucinating instead of requesting useful visual interventions.This conclusion is reported across the benchmark’s evaluated settings.
  • Benchmark and findings: Hinting increases proactive behavior but produces only marginal accuracy gains.The conclusion summarizes the benchmark’s elicitation analysis.
  • Benchmark and findings: Conversation histories and few-shot examples bias action distributions toward lower accuracy.The paper identifies these conditioning strategies as hindering performance.
  • Benchmark and findings: Learning proactiveness from data is possible, although eliciting it remains challenging.The paper releases ProactiveBench to support future work on proactive MLLMs.

A Dataset details and environment implementation

The appendix details environments that begin with ambiguous or uninformative visual states and allow predefined interventions before answering. It also describes filtering designed to retain samples that require proactiveness.

  • ROD environment: ROD tests moving a completely occluding object before selecting the hidden object’s category.The initial frame is unanswerable because the target object is invisible, and models may move the occluder or abstain.
  • VSOD environment: VSOD tests waiting or rewinding an occluded video before predicting a speaker, people count, or event type.The first frame contains an occlusion that may prevent the question from being answered.
  • MVP-N environment: MVP-N tests object suggestions and camera rotations from an uninformative viewpoint until discriminative features become visible.Models can rotate the object or change the camera angle, typically for about three actions per sample.
  • ImageNet-C environment: ImageNet-C tests requesting image-quality improvements before classifying strongly corrupted images.The environment uses ImageNet validation samples with one of eight corruptions and includes valid and invalid proactive options.
  • Filtering: IN-C decreases from 4,856 to 1,095 samples after filtering.Datasets that are easier to classify on the first turn undergo larger reductions, while Figure 10 reports original and post-filtering sizes.

C Training details

The RL post-training implementation uses approximately 27k examples and the GRPO trainer, with model-specific GPU resources and tabulated hyperparameters.

  • Training resources: Approximately 27k training examples combine 17k QuickDraw samples with the remainder from MS-COCO.The appendix reports the dataset composition for the RL post-training experiment.
  • Training resources: Qwen2.5-VL-3B training uses 4 NVIDIA A100 GPUs, while LLaVA-NeXT-Mistral-8B uses 16.The larger model required more GPUs according to the implementation details.
  • Hyperparameters: GRPO uses batch size 512, AdamW, and a learning rate of 2 × 10^-5.These hyperparameters are listed in the training-details table.

D Dataset examples

Figures 12–18 provide dataset examples returned by the environment in the first state. These examples document the initial inputs used across ProactiveBench environments.

  • Figures 12–18 show examples from the ProactiveBench datasets.
  • Each example depicts an input returned by the environment in its first state.
  • The examples collectively cover the dataset environments used to construct the benchmark.

E Extended results

The extended results examine model actions, oracle performance, and hinted prompting across all evaluated models. They also document computational requirements and show variation in proactive suggestion rates, including 4% for InternVL3 78B.

  • Action distributions: Extended results compare action distributions with random proactive options and with prompt hints across all evaluated models.Figures 20 and 21, together with Tables 7–9, report these comparisons using accuracy, proactive suggestion rate, aggregate accuracy, category accuracy, and correct proactive suggestions.
  • Computational details: Most experiments used one A100 GPU, 32GB of RAM, and 8 CPU cores, while history and few-shot settings used two A100 GPUs.Typical experiments lasted about one hour; history- and few-shot-conditioned experiments averaged two hours and took at most eight hours, depending on dataset and model.

F Broader impacts statement

ProactiveBench evaluates whether MLLMs request additional input under ambiguous visual information, a capability linked to collaborative and user-aligned AI. The broader-impact discussion also highlights risks from excessive questioning and over-reliance on user feedback.

  • Motivation: ProactiveBench targets MLLMs’ ability to request additional input when visual information is ambiguous or insufficient.The benchmark is motivated by interactive and safety-critical applications such as assistive tools and autonomous systems.
  • Risks: Promoting proactiveness must be balanced against over-questioning or inefficient behavior.
  • Risks: Over-reliance on user feedback may create misuse risks in adversarial settings.The statement contrasts this risk with the benchmark’s emphasis on interpretability and abstention over hallucination.

G Licenses

The supplementary material lists licenses for the reused datasets and evaluated models. It also reports tables for oracle, hinted, multi-turn, and open-ended evaluation results, while noting that LLMs assisted with manuscript polishing.

  • Dataset and model licenses: The reused datasets carry varied licenses, including MIT, Apache License 2.0, CC-BY-4.0, and one dataset released without a license.
  • Disclosure: The paper states that LLMs were used for writing polish and manuscript proofreading.
  • Supplementary results: The supplementary tables cover oracle accuracy, hinted MCQA accuracy and proactive suggestion rate, multi-turn OEG results, and OEG aggregate metrics.The OEG tables report aggregate accuracy, correctly predicted categories, and correct proactive suggestions.
Loading 2603.19466v2…