Source-linked AI summary
MM-IFEval-Pro: A Multilingual and Attack-Resistant Benchmark for Instruction-Following in Vision-Language Models
Changming Xiao, Zhenliang Ni, Jinhui He, Han Shu, Jie Hu
TL;DR
Existing multimodal instruction-following benchmarks have limited language coverage and insufficient adversarial safety scenarios for multilingual, safety-sensitive evaluation. MM-IFEval-Pro introduces a bilingual benchmark with fine-grained constraints and instruction-hijacking cases, alongside a Chinese- and adversarially enriched reinforcement-learning training set. The training approach improves performance on MM-IFEval-Pro and transfers to other multimodal benchmarks.
Problem
Existing multimodal instruction-following benchmarks generally lack Chinese samples and sufficient coverage of adversarial scenarios such as instruction hijacking.
Method
MM-IFEval-Pro combines Chinese and English multimodal tasks with 4 major task categories, 24 subcategories, 8 constraint categories, 52 subcategories, and an average of 3.0 constraints per sample, supported by an enriched reinforcement-learning training set.
Results
The Chinese- and adversarially enriched training set improves performance on MM-IFEval-Pro and transfers effectively to other mainstream multimodal benchmarks.
Takeaways & Limitations
MM-IFEval-Pro provides a broader assessment of multilingual and adversarial multimodal instruction following, while its training set supports cross-task and cross-language generalization.
Abstract
from arXiv · showhide
As vision-language models (VLMs) rapidly advance in image understanding, cross-modal reasoning, and complex instruction execution, instruction-following capability has become a key indicator of their reliability and practicality. However, existing multimodal instruction-following benchmarks still suffer from limited language coverage and insufficient adversarial safety scenarios, making them inadequate for evaluating real-world multilingual and safety-sensitive settings. To address these gaps, we present MM-IFEval-Pro, a multimodal instruction-following benchmark covering Chinese and English tasks as well as diverse instruction hijacking cases. MM-IFEval-Pro includes 4 major task categories and 24 subcategories and 8 instruction categories with 52 subcategories, with each sample containing an average of 3.0 constraints to realistically simulate complex instruction scenarios. We further construct a reinforcement-learning training set enriched with Chinese and adversarial instructions, which significantly improves model performance on MM-IFEval-Pro and transfers effectively to other mainstream multimodal benchmarks, demonstrating strong cross-task and cross-language generalization.
1 Introduction
MM-IFEval-Pro addresses gaps in multilingual coverage, adversarial safety evaluation, and reliable assessment of complex multimodal instruction following. It combines bilingual, fine-grained tasks and constraints with adversarial scenarios and reinforcement-learning training data.
- Motivation: Existing multimodal benchmarks generally lack Chinese samples and sufficiently diverse safety-adversarial scenarios such as instruction hijacking.These gaps limit evaluation in multilingual and safety-sensitive settings.
- Benchmark design: MM-IFEval-Pro covers 4 major task categories, 24 subcategories, 8 constraint categories, and 52 constraint subcategories in Chinese and English.Each sample contains an average of 3.0 instruction constraints to represent complex interactions.
- Benchmark design: The benchmark models instruction hijacking as an independent category and targets multilingual, safety-sensitive, multi-constraint interactions.Its design is intended to better reflect real-world multimodal scenarios.
- Verification and training: The benchmark and training corpus use rule-based verification functions to check constraints deterministically and support reward computation.Figure 1 presents the training corpus, deterministic constraint checking, and multi-constraint evaluation workflow.
- Training and results: The reinforcement-learning training set contains Chinese and adversarial instructions and improves MM-IFEval-Pro performance while transferring to other multimodal benchmarks.The reported transfer demonstrates cross-task and cross-language generalization.
2 Related Work
Prior work established verifiable instruction-following evaluation and multimodal benchmarks, but important gaps remain in Chinese-specific verification and robustness to visual instruction conflicts. MM-IFEval-Pro builds on this landscape by combining multilingual, multimodal, and adversarial evaluation concerns.
- Instruction following: IFEval introduced executable verification for constraints such as length, keywords, and format, while later benchmarks increased constraint diversity and compositional complexity.Preference optimization and reinforcement learning with verifiable rewards have also been explored for precise instruction following.
- Instruction following: Most verifiable instruction-following benchmarks remain English-centric, while Chinese-specific verifiable constraints remain underexplored.CFBench covers Chinese scenarios but mainly relies on checklist-based LLM judging rather than programmatically verifiable rules.
- Multimodal instruction following: Multimodal benchmarks such as MIA-Bench, MM-IFEval, and VC-IFEval assess layered constraints, visual constraints, or vision-dependent grounding.These efforts broaden multimodal instruction-following evaluation beyond text-only settings.
- Visual instruction hijacking: Visual instruction hijacking can arise from adversarial perturbations or competing text embedded in images, including low-contrast overlays on natural images.These attacks can steer models away from intended behavior or disrupt intended instructions.
3 MM-IFEval-Pro
MM-IFEval-Pro builds bilingual, constraint-aware multimodal instruction-following data with dedicated visual instruction-hijacking tests and capability-benchmark augmentation. Its pipeline uses language-aware verifiers, explicit competing instructions, and rule-based checks to produce executable, assessable samples across diverse task and constraint categories.
- Bilingual construction: MM-IFEval-Pro makes Chinese and English first-class languages through language-aware schemas, generation protocols, and paired verifiers.Constraints are annotated by language support, language-specific types are gated during construction, and verification follows Chinese or English conventions.
- Visual Instruction Hijacking: The visual instruction-hijacking subset pairs a readable image-embedded fake task with a genuine OCR query, testing whether models transcribe rather than execute the visible instruction.Instances enforce language consistency between the OCR query and rendered text, and the same protocol generates evaluation and RL-training samples.
- Visual Instruction Hijacking: A binary rule-based verifier scores hijacking resistance by comparing model output with the rendered fake text using normalized Levenshtein distance.A score of 1 denotes adherence to the OCR query, whereas 0 typically indicates execution of the visually embedded task; the verifier also supplies the RLVR reward.
- Capability Benchmark Augmentation: Capability-benchmark augmentation converts implicit output-format requirements into explicit constraints for grounding and numerical VQA tasks.Grounding samples jointly verify serialization and box overlap, while numerical samples verify wrapper format and precision against ground truth.
- Pipeline and Statistics: The resulting data support programmatic checking of both format and content, enabling RLVR and reducing failures caused by extraction-incompatible responses.The benchmark spans 4 major task categories, 24 task subcategories, 8 instruction-constraint categories, and 52 constraint subcategories; the training set has 13,547 samples averaging 3.0 constraints, and the test set has 1,279 samples.
4 Experiment
Experiments evaluate MM-IFEval-Pro across instruction-following, general multimodal, and instruction-hijacking settings. GRPO training with its data improves benchmark performance and transfers across tasks and languages, while exposing robustness differences among models.
- Main Results: MM-IFEval-Pro training raises Qwen3-VL-8B-Instruct’s average score from 78.73% to 89.31% and InternVL3.5-8B’s from 64.64% to 87.55% on MM-IFEval-Pro.The reported gains use instruction-hijacking data and accompany improvements on other instruction-following benchmarks.
- Main Results: MM-IFEval-Pro finetuning improves STEM performance while retaining strong VQA and OCR performance across broader multimodal benchmarks.For example, Qwen3-VL-8B-Instruct rises from 62.55% to 64.00% on MMMU and from 40.72% to 42.13% on MathVision.
- Main Results: Instruction-hijacking robustness varies substantially across models: MiMo-VL-7B-RL-2508 and InternVL-3.5-8B degrade considerably, whereas MiniCPM-V4.5 and Qwen3.5-35B-A3B remain relatively stable.Qwen3-VL-8B-Instruct gains in both without-IH and with-IH settings after MM-IFEval-Pro finetuning.
- Ablation Study: Before finetuning, Qwen3-VL-8B-Instruct and InternVL3.5-8B follow misleading visual pseudo-tasks; afterward, both resist these distractions and follow the user’s intended instructions.The failure cases illustrate vulnerability to instruction conflicts introduced through image content.
5 Conclusion
MM-IFEval-Pro addresses multilingual coverage and adversarial robustness gaps through a bilingual, constraint-rich benchmark with instruction-hijacking scenarios. Its multilingual and adversarial training data yields consistent gains across instruction-following and broader multimodal benchmarks.
- Conclusion: MM-IFEval-Pro combines Chinese and English visual tasks, complex constraint-rich instructions, and realistic instruction-hijacking scenarios to assess multimodal instruction following.The benchmark is designed to address shortcomings in multilingual coverage and adversarial robustness.
- Conclusion: GRPO training on multilingual and adversarial instructions produces consistent gains across instruction-following and general multimodal benchmarks.The conclusion presents the benchmark and accompanying training approach as a foundation for more robust and broadly applicable multimodal systems.