Source-linked AI summary
HANIA: Planner-Guided Multimodal Graph Evidence Selection for Grounded Question Answering
Zafar Ali, Asad Khan, Nimbeshaho Thierry, Nabila Amir, Adam A. Q. Mohammed, Pavlos Kefalas
TL;DR
Multimodal question answering needs better control of noisy, incomplete, and weakly grounded evidence than long contexts or flat retrieval provide. HANIA constructs provenance-linked multimodal graph evidence, plans descriptive and relational requirements, and prunes it before frozen-decoder prediction; on ScienceQA, it reports 87.80% accuracy with competitive evidence filtering without target-dataset fine-tuning or iterative retrieval.
Problem
Multimodal QA lacks compact, well-grounded evidence selection when visual, textual, and relational information must support multi-step reasoning.
Method
HANIA extracts question-relevant visual evidence, constructs a provenance-preserving multimodal graph, plans evidence needs with finite-state control, and prunes for coverage.
Results
87.80% accuracy, 68.57% evidence precision, 62.31% recall, and 65.29% F1 were achieved on ScienceQA.
Takeaways & Limitations
Structured planning and compact graph-guided retrieval support evidence-grounded QA without target-dataset fine-tuning, decoder modification, external knowledge retrieval, or iterative retrieval.
Takeaways & Limitations
All pretrained components remain frozen, and configuration choices are selected on the validation split before testing.
Abstract
from arXiv · showhide
Multimodal question answering remains sensitive to noisy, incomplete, and weakly grounded evidence. Long unstructured contexts can introduce redundancy and encourage unsupported generation, while flat retrieval may overlook relations needed for multi-step reasoning. We present HANIA, a planner-guided multimodal graph framework for evidence-grounded question answering. HANIA processes the supplied image and text using a frozen vision-language model to extract concise question-relevant visual evidence with explicit abstention. It then constructs an input-grounded multimodal graph and applies a two-group finite-state planner to coordinate descriptive and relational evidence. Coverage-aware pruning retains a compact evidence set based on relevance, graph confidence, concept coverage, and modality diversity. The selected passages, visual statements, and graph triples are provided to a frozen instruction-tuned decoder. We evaluate HANIA on ScienceQA using answer accuracy, evidence-filtering quality, evidence-budget sensitivity, and efficiency. The results show that structured evidence planning and compact graph-guided retrieval can support competitive multimodal question answering without target-dataset fine-tuning or iterative retrieval. The code is available at https://github.com/Zafar-southeast/HANIA.
1 Introduction
HANIA addresses the challenge of grounding multimodal question answering by constructing compact, question-aware graph evidence from supplied images and text, then controlling its selection before decoding.
- 1 Introduction: Existing multimodal KG-assisted QA must connect heterogeneous visual, textual, and relational evidence under limited computational budgets.Extracted triples may be weakly aligned with visual content, while vision–language models can produce unsupported answers.
- 1 Introduction: Many graph-aware methods remain text-centric, assume pre-existing symbolic graphs, or depend on complex training and retrieval pipelines.They provide limited support for constructing compact, question-aware graph evidence directly from supplied multimodal inputs.
- 1 Introduction: HANIA extracts concise question-relevant visual evidence with abstention and builds a provenance-preserving graph from visual and textual evidence.Controlled rules and GLiREL extract question-aware relations while retaining links to original sources.
- 1 Introduction: A two-group finite-state planner coordinates descriptive and relational requirements, while coverage-aware pruning selects evidence using relevance, confidence, coverage, and modality diversity.The selected evidence is serialized for a frozen decoder without target-dataset fine-tuning, decoder modification, external knowledge retrieval, or iterative retrieval.
- 1 Introduction: HANIA contributes a planner-guided multimodal graph framework, an input-grounded graph with explicit provenance, and an evaluation on ScienceQA.The evaluation measures answer accuracy, evidence-filtering quality, evidence-budget sensitivity, and efficiency.
2 Related Work
Prior work combines graph reasoning, multimodal QA, retrieval augmentation, and structured LLM reasoning, while HANIA targets compact graph evidence construction from supplied multimodal context.
- 2 Related Work: KG-QA methods use structured triples for interpretable reasoning but can struggle with unseen entities, missing relations, and scalability.Recent LLM-based approaches add graph-aware prompting and traversal for multi-hop reasoning.
- 2 Related Work: Multimodal QA coordinates reasoning over images, text, and relational information, with prior methods exploring multimodal graphs, message passing, and vision-to-graph construction.These approaches address heterogeneous signals and alignment noise in grounded reasoning.
- 2 Related Work: HANIA presents a planner-guided multimodal graph framework that combines explicit provenance, finite-state coordination, and coverage-aware pruning before frozen-decoder prediction.Its pretrained components remain frozen and it uses no external knowledge retrieval.
- 2 Related Work: Structured retrieval methods support visually rich inputs, but many systems still rely on coarse aggregation or long multimodal contexts.HANIA instead converts supplied context into compact question-aware graph evidence and organizes it through constrained planning.
3 Methodology: HANIA
HANIA constructs compact, question-aware multimodal graph evidence from supplied inputs, then plans and prunes that evidence before constrained answer selection. Its frozen pipeline preserves provenance, coordinates descriptive and relational requirements, and inserts insufficient-evidence markers when support is unavailable.
- 3.1 Input-Grounded Evidence and Graph Construction: HANIA extracts question-relevant visual evidence, retains original text, and constructs a provenance-preserving graph linking retained elements to supporting visual statements or text spans.The vision-language model abstains when requested visual evidence cannot be determined; the graph includes entities, properties, numerical values, and retained relations.
- 3.1 Input-Grounded Evidence and Graph Construction: Question-aware relation vocabularies cover attributes, counts, comparisons, and spatial relations, while deterministic templates and GLiREL produce candidate visual and textual relations.Relations are retained only when entities are valid, the relation belongs to the selected vocabulary, confidence meets the threshold, and the relation is relevant to the question.
- 3.2 FSM-Constrained Two-Group Planning: A finite-state planner organizes requirements into descriptive and relational groups and proposes bounded action sequences under valid-transition constraints.The descriptive group covers attributes, colors, materials, counts, and shapes; the relational group covers spatial, comparative, and object-level relations.
- 3.3 Planner-Guided Retrieval and Coverage-Aware Pruning: HANIA greedily prunes candidate passages, visual statements, and graph triples using relevance, graph support, planner alignment, concept coverage, modality diversity, and redundancy penalties.The candidate pool is linked to original sources, and coverage-aware selection avoids retaining redundant evidence while missing required concepts.
- 3.3 Planner-Guided Retrieval and Coverage-Aware Pruning: Selection stops at the evidence budget or a score threshold, inserting an insufficient-evidence marker when a required planner group remains unsupported.Retained evidence is serialized by descriptive and relational group with provenance, associated graph triples, and supporting modality.
- 3.4 Structured Evidence Prompting and Answer Generation: A frozen instruction-tuned decoder selects among the provided answer options using only serialized evidence, while all pretrained components remain frozen and validation-selected settings are fixed before testing.The constrained formulation avoids unrestricted generation and does not fine-tune models on ScienceQA.
4 Experimental Evaluation
HANIA is evaluated on ScienceQA under matched protocols using accuracy, evidence-filtering quality, evidence-budget sensitivity, and latency. Controlled comparisons, ablations, and budget sweeps assess its evidence-planning components and operating trade-offs.
- 4.1 Dataset and Evaluation Protocol: ScienceQA evaluation uses the official test split with supplied context only, frozen baseline backbones, matched metrics and hardware, five-run averages, confidence intervals, and corrected significance tests.Answer accuracy is primary; evidence precision, recall, and F1 use reference relevance annotations.
- 4.2 Overall Results and Component Ablation: 87.80% accuracy makes HANIA the top result, exceeding Qwen3-VL Direct by 2.40 points and Flat-RAG by 0.90 points under controlled comparisons.The Qwen3-VL Direct difference has a 95% CI of [1.95, 2.85] with Holm-adjusted p < 0.003; the Flat-RAG difference has a 95% CI of [0.52, 1.28] with p = 0.008.
- 4.2 Overall Results and Component Ablation: HANIA improves evidence quality over Flat-RAG by 2.86 F1 points and exceeds SelF-Reasoner by 0.56 accuracy points.The Flat-RAG F1 difference has a 95% CI of [2.10, 3.50], while the SelF-Reasoner accuracy difference has a 95% CI of [0.18, 0.94] with p = 0.021.
- 4.2 Overall Results and Component Ablation: Removing any component reduces performance; fixed top-k ranking causes the largest precision drop, insufficient-evidence markers the largest accuracy reduction, and graph removal lowers accuracy and evidence F1.These ablations support the joint contribution of provenance-linked graph construction, constrained planning, coverage-aware pruning, and unsupported-evidence handling.
- 4.3 Filtering Quality and Evidence-Budget Selection: Increasing k improves recall but gradually reduces precision; accuracy peaks at 87.80% with k = 5, while k = 7 yields the highest filtering F1.At k = 9, recall gains accompany lower precision and accuracy, and mean latency increases from 1.72 to 3.51 seconds per question; k = 5 was selected on validation.
- 4.4 Implementation Details: The pipeline extracts visual evidence, constructs provenance-linked relations, plans and prunes evidence once, then predicts with a frozen decoder.It uses no external knowledge retrieval, iterative retrieval, decoder modification, or target-dataset fine-tuning; parameters are fixed before testing.
5 Conclusion and Future Work
HANIA constructs question-aware graph evidence from supplied visual and textual context, coordinates descriptive and relational requirements with finite-state planning, and prunes evidence before frozen-decoder prediction. On ScienceQA, it achieves 87.80% accuracy and the study identifies k = 5 as the best answer-accuracy setting, while noting important evaluation boundaries.
- 5 Conclusion and Future Work: HANIA combines supplied-context graph construction, finite-state planning, and coverage-aware pruning before prediction with a frozen decoder.The framework targets question-aware evidence selection for multimodal question answering.
- 5 Conclusion and Future Work: 87.80% accuracy, 68.57% evidence precision, 62.31% recall, and 65.29% F1 are reported on ScienceQA.The evidence-budget analysis identifies k = 5 as best for answer accuracy with moderate latency.
- 5 Conclusion and Future Work: Controlled comparisons and ablations report joint benefits from provenance-linked graphs, constrained planning, coverage-aware pruning, and explicit unsupported-evidence handling without fine-tuning or decoder modification.The conclusion frames these components as improving answer accuracy and evidence quality over direct prompting and flat relevance ranking.
- 5 Conclusion and Future Work: The study is limited to one benchmark and does not yet include detailed graph-quality, faithfulness, or question-type analyses.Future work extends evaluation to additional multimodal benchmarks and improves visual relation extraction and planner constraints.