Source-linked AI summary
SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenes
Chuhan Wang, Xintong Li, Jennifer Yuntong Zhang, Junda Wu, Chengkai Huang, Lina Yao, Julian McAuley, Jingbo Shang
TL;DR
MLLMs often struggle to keep reasoning grounded in complex visual scenes, while text- or answer-based preference methods can bypass visual evidence. SceneAlign uses scene-graph perturbations to create plausible but structurally incorrect rationales for DPO alignment, and experiments across seven benchmarks report consistent gains in reasoning-intensive tasks.
Problem
MLLM rationales can hallucinate entities, mis-ground relations, skip steps, or over-specify reasoning, while existing preference methods leave visual grounding untouched.
Method
SceneAlign constructs graph-grounded positive reasoning, applies controlled perturbations to create hard negatives, filters them, and performs DPO alignment.
Results
Across seven benchmarks, SceneAlign reports consistent 3–5% gains on reasoning-intensive tasks and consistently improves grounding consistency, reasoning coherence, and answer accuracy.
Takeaways & Limitations
Structure-aware supervision yields more faithful multimodal reasoning than traditional text-based perturbations within the evaluated visual reasoning setting.
Takeaways & Limitations
The study focuses on single-image reasoning and relies on GPT-based chain-of-thought generation and automatic evaluation, which may not fully capture fine-grained human judgments.
Abstract
from arXiv · showhide
Multimodal large language models often struggle with faithful reasoning in complex visual scenes, where intricate entities and relations require precise visual grounding at each step. This reasoning unfaithfulness frequently manifests as hallucinated entities, mis-grounded relations, skipped steps, and over-specified reasoning. Existing preference-based approaches, typically relying on textual perturbations or answer-conditioned rationales, fail to address this challenge as they allow models to exploit language priors to bypass visual grounding. To address this, we propose SceneAlign, a framework that leverages scene graphs as structured visual information to perform controllable structural interventions. By identifying reasoning-critical nodes and perturbing them through four targeted strategies that mimic typical grounding failures, SceneAlign constructs hard negative rationales that remain linguistically plausible but are grounded in inaccurate visual facts. These contrastive pairs are used in Direct Preference Optimization to steer models toward fine-grained, structure-faithful reasoning. Across seven visual reasoning benchmarks, SceneAlign consistently improves answer accuracy and reasoning faithfulness, highlighting the effectiveness of grounding-aware alignment for multimodal reasoning.
1 Introduction
SceneAlign addresses unfaithful visual reasoning by using scene graphs to create controllable, structurally grounded contrastive supervision. It targets grounding failures that textual or answer-level perturbations leave indistinguishable and trains models toward faithful reasoning.
- Motivation: MLLMs often produce rationales decoupled from visual evidence, especially in complex scenes with numerous objects and intricate relations.Reported failures include object hallucination, relation mis-grounding, incomplete grounding, and over-specification.
- Motivation: Token- and answer-level rationale perturbations can encourage coherent language while leaving visual grounding untouched and grounding failures unlocalizable.Such negatives allow reliance on language priors rather than specific scene elements.
- Approach: SceneAlign uses scene graphs to represent objects, attributes, and relations, providing explicit structure for localizing reasoning-critical entities.The framework constructs graph-grounded positive reasoning before intervention.
- Approach: Controlled graph interventions swap, remove, or replace nodes and edges to simulate realistic grounding failures and regenerate corresponding reasoning chains.Examples include removing “upside down” or replacing “on” with “hold.”
- Training: SceneAlign filters perturbed samples for plausibility and diversity, then applies DPO to contrast visually grounded positives with semantically coherent but visually inconsistent negatives.The resulting preference signal is designed to expose diverse reasoning failure modes.
2 Related Works
Prior multimodal reasoning methods often treat reasoning as flat text, missing graph-level dependencies among visual entities and relations. SceneAlign instead perturbs scene graphs to construct structurally inconsistent rationales and uses preference optimization to align reasoning with visual structure.
- Prior work: SceneAlign builds on negative CoT sampling, preference optimization, and scene-graph-based multimodal reasoning.Its distinctive combination targets structural consistency in the reasoning process.
- Prior work: Answer-oriented preference methods can miss structural reasoning errors because flawed reasoning may still produce a correct answer.This limits supervision based only on final correctness.
- Gap: Previous studies often represent reasoning as a flat textual sequence, neglecting graph-level dependencies among entities and relations.This leaves the reasoning process insufficiently aligned with visual scene structure.
- SceneAlign: SceneAlign introduces structure-aware perturbations guided by scene graphs, allowing models to learn from both grounded and structurally inconsistent rationales.The framework uses these contrastive rationales for preference optimization.
3 Methodology
SceneAlign constructs scene-graph-grounded positive reasoning, generates structured counterfactual negatives through targeted graph perturbations, filters them for informativeness and diversity, and applies DPO to favor scene-faithful reasoning.
- Positive Reasoning Generation: SceneAlign constructs scene-graph–grounded positive reasoning traces that explicitly reference visual entities, attributes, and relations.The framework uses scene graphs as visual anchors so reasoning remains tied to the underlying scene structure.
- Construction of Negative Scene Graphs: It identifies the subgraph referenced by a positive chain of thought and perturbs its elements while leaving the remaining scene graph unchanged.This localizes structural changes to reasoning-critical content while preserving semantically relevant context.
- Construction of Negative Scene Graphs: Four transformations—swap, replace, shorten, and overthink—model relation mis-grounding, hallucination-like substitution, incomplete grounding, and over-specification.The operators exchange relation roles, substitute residual elements, remove key evidence, or add redundant content.
- Construction of Negative Scene Graphs: The resulting counterfactual scene graphs cover major complex-scene reasoning errors and provide structured negatives for preference optimization.A mix mode and consistency constraints expose recurring hallucination, mis-grounding, skipped-inference, and over-specification patterns.
- Selection of Negative Reasoning Chains: SceneAlign filters negative graphs by Jaccard overlap and selects diverse candidates, excluding near-duplicates and irrelevant outliers.These steps produce semantically consistent but precisely grounded hard negatives for contrastive supervision.
- Scene-Graph Preference Optimization: DPO trains the MLLM to assign higher likelihood to scene-consistent reasoning chains than to structurally inconsistent alternatives.The final preference data pairs positive chains with selected negative chains to produce aligned parameters.
4 Experiments
SceneAlign is evaluated across multiple models, benchmarks, and ablations, consistently improving visual reasoning performance through scene-graph-guided preference alignment. Its structured perturbations target distinct grounding failures, while moderate overlap thresholds and carefully selected negatives provide balanced supervision.
- Experimental Settings: SceneAlign is evaluated across five MLLMs and benchmarks assessing visual grounding and complex scene reasoning.The evaluation includes pretrained and SFT baselines, prior CoT preference methods, and multiple reasoning-oriented and vision-centric benchmarks.
- Main Results: SceneAlign consistently outperforms pretrained and SFT baselines, with average gains of 3%–5% on reasoning-intensive benchmarks.The strongest improvements occur on EMMA, HallusionBench, and MMMU-Reasoning, which require compositional reasoning and multientity grounding.
- Ablation Studies: Removing any of the swap, replace, shorten, or overthink operators reduces performance on benchmarks sensitive to the corresponding failure mode.The full method achieves the greatest and most balanced improvements by integrating all four operators.
- Ablation Studies: Relation-level grounding contributes the largest gains, while entity grounding remains important and attribute grounding provides complementary effects.The ablation identifies subject–object binding as especially critical for grounded reasoning.
- Ablation Studies: SceneAlign’s full negative-sampling method combines scene-graph filtering with diversity sampling to avoid trivial, irrelevant, or excessively distant negatives.The method uses three negatives per instance as a reported trade-off between efficiency and effectiveness.
- Case Study: A case study shows that scene-graph-grounded reasoning links the man, motorcycle, surrounding people, and paper to support inspecting or diagnosing the motorcycle.SceneAlign’s structured perturbations instead create natural but structurally inconsistent chains exposing role misalignment, hallucination, missing evidence, and over-specification.
5 Conclusion
SceneAlign aligns multimodal chain-of-thought reasoning with visual scene structure by generating coherent but structurally inconsistent rationales through controlled graph perturbations. Across seven benchmarks, it delivers consistent 3–5% gains on reasoning-intensive tasks.
- 5 Conclusion: SceneAlign improves grounding-faithful multimodal reasoning by aligning chain-of-thought rationales with scene-level structure.Its contrastive supervision uses semantically coherent but structurally inconsistent rationales generated through controlled scene-graph perturbations.
- 5 Conclusion: 3–5% gains across seven benchmarks on reasoning-intensive tasks support structure-aware supervision over traditional text-based perturbations.The conclusion attributes these gains to more faithful reasoning supervision.
Limitations
The study is limited to single-image reasoning and does not cover multi-image or video inputs, where temporal and cross-view consistency create additional grounding challenges.
- The study evaluates single-image reasoning and excludes multi-image or video-based inputs.These settings introduce temporal and cross-view consistency challenges beyond the study’s scope.
A.1 Analysis of Data Reliability
A human spot check on 150 randomly sampled A-OKVQA instances assessed the reliability of scene graphs and positive rationales.
- 150 randomly sampled A-OKVQA instances were used for a human spot check of scene graphs and positive rationales.The evaluation examined whether question-relevant entities and relations were captured correctly.
- The gpt-4o-generated scene graphs captured all question-relevant entities and relations in every checked case.Occasional redundant background-object nodes appeared but did not degrade positive CoT quality.
A.2 Prompt Design
SceneAlign uses structured prompts to separate scene grounding from reasoning and generate positive and negative chains with controlled structural contrast.
- The scene-graph prompt extracts entities, attributes, and relations in strict JSON format to anchor reasoning to visual structure.This prompt design explicitly structures the scene information used during reasoning.
- Positive CoTs use both the image and scene graph, whereas negative CoTs use only the graph to create controlled contrast in structure consistency.The differing inputs separate scene grounding from reasoning during contrastive generation.
A.3 Case Study Explanation
The case study shows that SceneAlign produces structure-consistent positive chains of thought, while graph perturbations reveal distinct grounding failure modes.
- SceneAlign yields structure-consistent positive CoTs in the case study.
- Perturbations expose distinct failure modes in the case study.
A.4 Training Curves
The paper uses structured prompts and scene-graph-based reasoning examples to support training and analysis of multimodal models. Training curves compare five models across four reward-related metrics over one epoch.
- Structured Scene Graph Generation: The structured scene-graph prompt represents relevant entities, attributes, and spatial or semantic relationships in strict JSON format.This representation targets objects and relations relevant to answering each image-question pair.
- Positive CoT Generation: Positive reasoning uses the image and scene graph for step-by-step answers while treating scene-graph elements as the visual scene itself.The prompt instructs the model not to mention the data source.
- Negative CoT Generation: Negative reasoning is generated through a separate step-by-step prompt based on the structured scene graph.The negative CoT prompt is presented alongside the positive CoT generation prompt in the framework materials.
- Training Curves: Figure 8 tracks accuracies, chosen, margins, and rejected for five multimodal models over one training epoch.The models are Qwen2.5-VL-3B-Instruct, Qwen3-VL-4B-Instruct, Qwen2.5-VL-7B-Instruct, InternVL3-8B, and Llama3-LLaVA-Next-8B-hf.
- Case Study: Scene-Graph–Grounded Reasoning: The case study examines a man, motorcycle, and paper in a scene requiring structured reasoning beyond surface-level visual cues.The associated question asks what activity the man on the floor is engaging in with respect to the motorcycle.