PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity Recognition

Jinlong Ma, Yu Zhang, Xuefeng Bai, Kehai Chen, Yuwei Wang, Zeming Liu, Jun Yu, Min Zhang

arXiv:2602.04486v1cs.CL

TL;DR

MLLMs for end-to-end GMNER can rely on unimodal shortcuts instead of verifying information across modalities. This paper proposes MCR, whose structured reasoning improves performance across GMNER, MNER, and visual grounding benchmarks while mitigating modality bias.

  • Problem

    End-to-end MLLM-based GMNER is limited by modality bias, while cascaded uses introduce cumulative error propagation and additional computational costs.

  • Method

    MCR enforces structured cross-modal reasoning through schema-injected reasoning chains and constraint-guided optimization aligned with GRPO.

  • Results

    MCR achieves superior performance across GMNER, MNER, and visual grounding benchmarks, including an 11.87% F1 improvement over MQSPN on GMNER.

  • Takeaways & Limitations

    The results demonstrate that structured reasoning is essential for precise cross-modal grounding and can mitigate modality biases.

  • Takeaways & Limitations

    MCR may struggle to generalize to unseen entities absent from the underlying MLLMs’ pre-training corpus.

Abstract

from arXiv · show

Grounded Multimodal Named Entity Recognition (GMNER) aims to extract text-based entities, assign them semantic categories, and ground them to corresponding visual regions. In this work, we explore the potential of Multimodal Large Language Models (MLLMs) to perform GMNER in an end-to-end manner, moving beyond their typical role as auxiliary tools within cascaded pipelines. Crucially, our investigation reveals a fundamental challenge: MLLMs exhibit $\textbf{modality bias}$, including visual bias and textual bias, which stems from their tendency to take unimodal shortcuts rather than rigorous cross-modal verification. To address this, we propose Modality-aware Consistency Reasoning ($\textbf{MCR}$), which enforces structured cross-modal reasoning through Multi-style Reasoning Schema Injection (MRSI) and Constraint-guided Verifiable Optimization (CVO). MRSI transforms abstract constraints into executable reasoning chains, while CVO empowers the model to dynamically align its reasoning trajectories with Group Relative Policy Optimization (GRPO). Experiments on GMNER and visual grounding tasks demonstrate that MCR effectively mitigates modality bias and achieves superior performance compared to existing baselines.

1 Introduction

The introduction frames GMNER as a task that jointly identifies textual entities and grounds them in visual regions, then identifies modality bias as a central obstacle to end-to-end MLLM-based reasoning. It presents MCR, combining schema injection and verifiable optimization to enforce structured cross-modal reasoning and improve grounding performance.

  • Task Background: GMNER jointly extracts named entities from text and grounds them to corresponding visual bounding boxes for structured multimodal representations.The task supports downstream applications including recommendation systems and knowledge-based question answering.
  • Motivation: Existing MLLM approaches usually use models as auxiliary components in cascaded GMNER pipelines, causing cumulative error propagation and additional computational costs.
  • Modality Bias: Direct end-to-end MLLM application exposes modality bias, as textual bias can disregard visual evidence while visual bias can override textual semantics.Examples include grounding “Iggy” to “Kevin Durant” based on textual context and hallucinating “Manchester United” from visual cues despite its absence from the text.
  • Proposed Method: MCR mitigates modality bias by enforcing structured cross-modal reasoning through Multi-style Reasoning Schema Injection and Constraint-guided Verifiable Optimization.MRSI converts abstract constraints into executable reasoning chains using diverse injected templates, while CVO supports autonomous exploration and verification of reasoning trajectories.
  • Contributions: MCR achieves superior performance on multiple benchmarks, supporting the importance of structured reasoning for precise cross-modal grounding.

2 Related Work

GMNER extends multimodal named entity recognition by requiring both entity recognition and visual grounding, while prior work emphasizes cross-modal alignment and generalization. This work instead targets MLLMs’ modality bias by using explicit cross-modal reasoning for consistency verification.

  • GMNER: GMNER requires models to recognize named entities and localize visually present entities with bounding boxes.It extends MNER from image-text entity extraction and classification to joint recognition and visual grounding.
  • GMNER: Existing GMNER studies refine cross-modal alignment to suppress visual noise and improve generalization to unseen entities.The cited approaches focus on alignment refinement and broader entity generalization.
  • MLLM Reasoning: Recent MLLM research emphasizes integrating heterogeneous data and introducing explicit reasoning processes through language-level Chains of Thought.These developments motivate using MLLMs for complex multimodal reasoning rather than only feature processing.
  • Modality Bias: Prior work identifies modality bias in MLLMs, with reinforcement learning from human feedback used to distinguish hallucinated from grounded content.These strategies rely on extensive preference datasets to mitigate modality-specific inclinations.
  • Proposed Direction: MCR addresses GMNER modality bias as cognitive shortcuts by explicitly modeling modality interplay to verify entity existence and spatial alignment.The method enforces rigorous cross-modal consistency instead of relying on unimodal heuristics.

3 Task Formulation

GMNER takes a sentence and its associated image and decomposes the task into multimodal named entity recognition and entity extraction and grounding. MNER identifies and types textual entities, while EEG determines visual presence and outputs either a bounding box or None.

  • Task Decomposition: GMNER decomposes into Multimodal Named Entity Recognition (MNER) and Entity Extraction & Grounding (EEG).MNER recognizes entities in the sentence and assigns each a predefined type; EEG parallels generalized Visual Grounding.
  • Multimodal Named Entity Recognition: MNER produces pairs (e_i, t_i), with e_i as an entity span and t_i as its corresponding predefined type.The task operates on entities in the sentence and assigns each entity a semantic category.
  • Entity Extraction & Grounding: For each textual entity e_i, EEG decides whether it is visually present in v and outputs its bounding box b_i or None.When present, the bounding box uses top-left and bottom-right corner coordinates.

4 Methodology

MCR addresses modality bias by enforcing explicit, constraint-centered cross-modal reasoning through MRSI and CVO. MRSI injects diverse reasoning schemas, while CVO uses verifiable rewards and GRPO-style optimization to strengthen and stabilize reasoning.

  • MCR Framework: MCR combines Multi-style Reasoning Schema Injection (MRSI) and Constraint-guided Verifiable Optimization (CVO) to enforce cross-modal consistency reasoning.MRSI organizes modality-specific reasoning schemas, while CVO uses those schemas with GRPO to strengthen reasoning.
  • Multi-style Reasoning Schema Injection: MRSI injects diverse, constraint-centered reasoning schemas guided by entity recognition, type classification, visual entailment, and visual grounding constraints.Schemas are generated in multiple styles from templates, LLMs, and MLLMs using image–text inputs and labels.
  • Multi-style Reasoning Schema Injection: Diverse schemas prevent trajectory collapse and are injected into MLLMs through supervised fine-tuning, retaining explicit reasoning paths for cross-modal consistency checking.MRSI explicitly introduces constraints and reasoning paths so predictions can be checked across modalities.
  • Constraint-guided Verifiable Optimization: CVO defines rule-based verifiable rewards for entity count, span, type, entailment, and localization, including Hungarian-matched token-level F1 for entity spans and IoU-based grounding.Grounding rewards threshold IoU values and linearly map qualifying overlaps into [0, 1].
  • Constraint-guided Verifiable Optimization: CVO samples diverse responses, scores them with verifiable rewards, filters training data using reward statistics, and updates the policy with a clipped, length-normalized GRPO-style objective.Clipping limits overly aggressive updates, while length normalization improves comparability across responses without requiring a critic.

5 Experiments

The experiments evaluate MCR for grounded multimodal named entity recognition across multiple datasets and compare explicit-reasoning MLLM approaches with pipeline and unified baselines. MCR consistently outperforms existing baselines, including by 11.87% F1 scores over MQSPN on GMNER.

  • Datasets: Experiments use Twitter-GMNER for GMNER, MNER-MI for multi-image MNER, and GREC for visual grounding.Twitter-GMNER and its MNER and EEG subtasks are evaluated in the main experiments, with additional evaluations on MNER-MI and GREC.
  • Baselines: Existing approaches are categorized as pipeline or unified methods according to whether they use separate or unified processing.The passage states that this categorization follows prior work, though the supplied text truncates the detailed criterion.
  • Results: Explicit Chain-of-Thought or Few-shot prompting improves training-free performance over direct MLLM application.The passage reports notable performance gains from introducing explicit reasoning through CoT or Few-shot prompting.
  • Results: 11.87% F1 scores is the improvement of MCR over MQSPN, the previous best unified method, on GMNER.MCR integrates Multi-style Reasoning Schema Injection and Constraint-guided Verifiable Optimization, and the passage reports that all MLLMs consistently outperform existing baselines.

Type Methods GMNER MNER EEG

MCR improves GMNER, MNER, and EEG performance while mitigating visual and textual modality bias. Ablations and training analyses attribute these gains to structured cross-modal verification, verifiable optimization, and diverse reasoning schemas.

  • Overall Performance: MCR surpasses existing methods across GMNER, MNER, and EEG, including a 10.97% F1-score improvement over MQSPN on EEG.It also exceeds SCANNER by 2.11% F1 and direct SFT by 8.05% and 7.57% F1 on Qwen2.5VL-7B and MimoVL-7B, respectively.
  • Performance on MNER and VG: MCR outperforms SFT on MNER and improves GREC N-acc across models, indicating reduced visual and textual bias.On MNER, the second-stage CVO generally surpasses first-stage MRSI; N-acc measures entailment between text and image for zero-region cases.
  • Ablation Study: Skipping MRSI causes an 18.95% F1-score drop, showing that its stepwise cross-modal verification path is a necessary prerequisite for CVO.Removing reasoning instructions also produces a 3.90% F1-score drop under the same training conditions.
  • Ablation Study: Strengthening MRSI with CVO yields a 2.04% gain, whereas continuing MRSI with CVO data alone produces only a marginal 0.18% improvement.This contrast supports CVO’s role in further enhancing cross-modal verification.
  • Reasoning Dynamics: Multi-style reasoning ultimately achieves higher rewards and F1 scores with more stable optimization, while producing controlled exploration and increasingly concise reasoning patterns.Single-style reasoning performs better early because of focused supervision, but later exhibits larger fluctuations.
  • Modality Bias Mitigation: MCR reduces visual bias to a near-negligible level and improves all three GMNER textual-bias metrics by nearly +14% over SFT on Qwen2.5VL-7B.N-Count and N-Rate quantify recalled entities absent from the sentence, while N-Pre and N-Rec evaluate text-only grounding predictions.

6 Conclusion

The work reformulates GMNER as an end-to-end generative reasoning task and identifies modality bias from unimodal shortcuts. It proposes MCR for structured cross-modal verification, with evaluations and ablations supporting its effectiveness and design choices.

  • Contribution: GMNER is reformulated as an end-to-end generative reasoning task, exposing modality bias caused by unimodal cognitive shortcuts.MLLMs often fail to perform rigorous cross-modal verification.
  • Contribution: MCR enforces structured cross-modal reasoning to mitigate modality bias.The method is designed to address MLLMs’ reliance on unimodal shortcuts.
  • Results: Comprehensive evaluations on GMNER, MNER, and Visual Grounding benchmarks show that MCR mitigates modality biases and achieves superior performance compared to existing baselines.The evaluations indicate that MCR enables rigorous cross-modal verification.
  • Results: Ablation experiments validate the necessity of MCR’s design choices and the stability of its optimization mechanism.The conclusion reports a comprehensive suite of ablation experiments.

7 Limitations · A Ethical Considerations · A.1 Potential Risks

MCR reduces modality bias across GMNER, MNER, and VG tasks but remains limited by the underlying MLLMs’ parametric knowledge and potential real-world misuse. The paper addresses these concerns through dataset and content safeguards while urging privacy-conscious deployment.

  • 7 Limitations: MCR remains constrained by the underlying MLLMs’ parametric knowledge limits.Its entity recognition relies on the model’s internal knowledge base.
  • 7 Limitations: MCR may struggle to generalize to entities absent from the pre-training corpus.This limitation specifically affects recognition of unseen entities.
  • 7 Limitations: The framework’s reported modality-bias mitigation spans GMNER, MNER, and VG tasks.The passage characterizes this performance as promising despite the framework’s knowledge constraints.
  • A Ethical Considerations: Entity grounding capabilities could be misused for surveillance purposes.This is identified as a potential risk associated with the work.
  • A.1 Potential Risks: Model hallucinations could lead to misinformation in real-world applications.The paper flags hallucination-driven misinformation as another potential risk.
  • A.1 Potential Risks: The authors mitigate potential risks by using publicly available datasets and strictly filtering harmful content.They also urge practitioners to exercise caution and respect user privacy when deploying these models.

A.3 Code and Data … D.2 Token-level F1 score

The paper specifies implementation safeguards, modality-aware GMNER constraints, diversified reasoning instructions, and reward mechanisms for entity counting and token-level span matching. These components require textual entity extraction, semantic typing, visual visibility and localization, while optimizing structured multimodal outputs.

  • A.3 Code and Data: MCR’s images and generated contexts exclude harmful, unethical, and offensive content, with curated annotations planned for open-source release upon acceptance.The data does not compare harmful, ethical, or offensive content between image pairs and follows original licensing and usage policies.
  • B Modality-specific Constraints: GMNER extracts entities from text, assigns categories primarily from textual evidence, and grounds visible entities with bounding boxes while marking absent entities as invisible.Visual cues supplement textual information when entity typing is uncertain.
  • C Multiple Styles Reasoning Schema: MRSI constructs diverse reasoning styles and paths using templates, LLMs, and MLLMs, with corresponding prompts for GMNER.The instructions require modality-relevant execution and outputs consistent with labels.
  • C.1 Instruction: The formal GMNER reasoning route sequentially extracts text entities, assigns person, organization, location, or miscellaneous types, checks image visibility, and outputs bboxes or None in fixed tags.The route prioritizes text, forbids image-only entities, and requires a final synthesized answer.
  • C.2 Reasoning Styles and Paths: Different reasoning paths can vary output formats and the ordering of entity triples, entity attributes, and reasoning presentation within the same sample.Examples demonstrate step-by-step analysis of text and image evidence, including visibility and localization decisions.
  • D More Details about Rewards: The reward details combine entity-count control with token-level span matching to optimize structured predictions.Entity-count rewards address excessive or insufficient recall, while span rewards average optimally matched token-level F1 scores.
  • D.1 Entity Count Rewards: Entity-count rewards compare predicted and gold counts, scaling penalties by the true count so small-count errors receive harsher penalties than large-count errors.The reward uses separate penalty weights for excessive and insufficient recall.
  • D.2 Token-level F1 score: Token-level F1 uses longest contiguous token overlap to compute precision and recall, then applies Hungarian matching for optimal one-to-one predicted–gold span alignment.The entity span reward is the average token-level F1 across matched pairs.

D.3 Data Preparation · E More Experiment Details

The method constructs multi-style reasoning schemas for MRSI and partitions them between schema injection and CVO. Sampling-based filtering is further applied to improve efficiency and reduce training-collapse risk.

  • D.3 Data Preparation: D.3 Data Preparation: The method constructs a multi-style reasoning schema set DR during MRSI.This construction aims to prevent over-reliance on fixed templates and mitigate training collapse in CVO.
  • D.3 Data Preparation: D.3 Data Preparation: Only subset D1 of DR is used for MRSI.The passage defines D1 as the portion allocated to MRSI.
  • D.3 Data Preparation: D.3 Data Preparation: The remainder D2 = DR\D1 is allocated to CVO.CVO uses D2 to calibrate and optimize cross-modal verification on core constraints.
  • D.3 Data Preparation: D.3 Data Preparation: CVO uses D2 to calibrate cross-modal verification on core constraints.The allocation is intended to support verification calibration within CVO.
  • D.3 Data Preparation: D.3 Data Preparation: CVO uses D2 to optimize cross-modal verification on core constraints.The passage describes optimization as a second purpose of allocating D2 to CVO.
  • D.3 Data Preparation: D.3 Data Preparation: Sampling-based filtering is applied to D2 to improve training efficiency.The same filtering step is also used to reduce collapse risk.
  • D.3 Data Preparation: D.3 Data Preparation: The filtering procedure is designed to reduce training-collapse risk.The passage links this objective to improving training efficiency during CVO preparation.

E.1 Datasets · E.2 Baselines

The experiments use Twitter-GMNER, MNER-MI, and GREC, with incompatible multi-region GREC samples excluded from the GMNER setting. Baselines span unified, pipeline, and end-to-end methods, including several MLLMs evaluated under varied prompting and fine-tuning conditions.

  • E.1 Datasets: The training data come from Twitter-GMNER for GMNER and NER, MNER-MI for multi-image multimodal NER, and GREC for generalized visual grounding.Twitter-GMNER is used in the main experiments, while MNER-MI and GREC are evaluated additionally in Section 5.3.
  • E.1 Datasets: GREC samples are filtered when one textual description corresponds to multiple image regions, because this conflicts with the GMNER setting.The paper reports dataset statistics in Table 5 alongside this dataset setup.
  • E.2 Baselines: GMNER baselines are categorized as unified methods, which predict entity–type–location triples in one pass, or pipeline methods, which decompose prediction into multiple stages.Unified approaches are described as reducing error propagation and improving over early pipelines, while recent LLM-based pipelines are also considered.
  • E.2 Baselines: Pipeline baselines include ITA-VinVL-EVG, BARTMNER-VinVL-EVG, Scanner, and UnCo, combining alignment, staged extraction and grounding, or uncertainty-aware collaboration.These methods respectively use image–text alignment, BART with EEG, NER and visual grounding with external knowledge, and collaboration between small models and multimodal language models.
  • E.2 Baselines: Unified baselines include MNER-QG, H-index, and TIGER, which formulate multimodal recognition and grounding through machine reading comprehension or sequence generation.MNER-QG grounds entity queries in textual and visual evidence, while H-index and TIGER generate structured outputs.
  • E.2 Baselines: End-to-end baselines comprise GLM4.5VL, Qwen2.5VL, and MimoVL, selected for multimodal understanding, reasoning, and visual grounding capabilities.The models are evaluated under direct instruction prompting, Chain-of-Thought prompting, and CoT with 3-shot demonstrations.
  • E.2 Baselines: The end-to-end evaluation also includes supervised fine-tuning, with models fine-tuned on GMNER training data.This provides an SFT baseline alongside the prompting-based settings.

E.3 Evaluation Metrics · E.4 Implementation Details · E.5 Case Study

The paper defines metrics for GMNER, visual grounding, and modality bias, details MRSI and CVO training settings, and uses case studies to show MCR mitigating textual and visual bias while exposing knowledge-related failures.

  • E.3 Evaluation Metrics: GMNER evaluation uses Precision, Recall, and F1, counting a triple as correct only when its entity, type, and location are all correct.This evaluation covers GMNER and its MNER and EEG subtasks.
  • E.3 Evaluation Metrics: Visual grounding uses no-target accuracy for absent entities, Precision for present targets, and IoU ≥0.5 for correct one-target localization.A no-target prediction is correct when the predicted bounding box is None.
  • E.3 Evaluation Metrics: N-Pre, N-Rec, and N-F1 quantify textual bias, while N-Count and N-Rate measure image-only entity recall; lower N-Count and N-Rate indicate weaker visual bias.These metrics target entities absent from the image or sentence, respectively.
  • E.4 Implementation Details: MRSI training uses diverse reasoning schemas, two or five epochs depending on the model, learning rate 0.0001, batch size 16, and eight L20 GPUs.Qwen2.5VL is trained for 2 epochs, MimoVL for 5 epochs, and training lasts 4 hours.
  • E.4 Implementation Details: CVO training runs for 2 epochs with learning rate 0.000005, batch size 64, eight generations per input, clipping thresholds 0.15 and 0.25, and β = 0.005.It uses warmup ratio 0.05, temperature 1.5, top-k 200, top-p 0.95, and 20 hours on eight L20 GPUs.
  • E.5 Case Study: Case studies show MCR prevents textual grounding errors by verifying cross-modal consistency, rejecting NBA-for-NFL and elderly-man-for-Donald-Trump mismatches.MCR explicitly checks image semantics against textual entities before grounding.
  • E.5 Case Study: MCR also reduces visual bias by rejecting image-only NBA recall and avoiding Rory Calhoun misclassification caused by a distracting cat.The method surfaces multimodal evidence, identifies each entity’s modality source, and relies on sentence semantics and internal knowledge for typing.
  • E.5 Case Study: Failure cases remain when GMNER requires broad cross-domain knowledge beyond the model’s coverage or when the model has acquired incorrect knowledge.The paper illustrates these limitations through MCR failure cases involving knowledge and entity spans.
Loading 2602.04486v1…