PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphLoom: Reliability-Calibrated Graph Evidence Routing for Multimodal KG-RAG

Zafar Ali, Asad Khan, Aalia Malik, Pavlos Kefalas

arXiv:2608.15056v1cs.AI

TL;DR

Multimodal QA systems struggle to select reliable evidence from heterogeneous, noisy sources while keeping multi-hop reasoning grounded and efficient. GraphLoom routes compact, reliability-calibrated graph evidence through bounded retrieval and selective decoder injection, achieving consistent gains in answer quality and faithfulness across ScienceQA, MultiModalQA, and OK-VQA, with further retrieval improvements on MultiModalQA.

  • Problem

    Reliable multimodal KG-assisted QA remains challenging because relevant evidence is distributed across visual, textual, tabular, external, and inferred relational sources, while large contexts complicate selection and decoding.

  • Method

    GraphLoom constructs instance-level multimodal knowledge graphs, retrieves compact subgraphs with bounded expansion, and selectively routes reliability-calibrated graph memories through a frozen decoder with bounded corrective retrieval.

  • Results

    GraphLoom shows consistent gains in answer quality and evidence grounding across ScienceQA, MultiModalQA, and OK-VQA, with additional retrieval gains on MultiModalQA and stable performance under noisy evidence pools.

  • Takeaways & Limitations

    Reliability-aware evidence routing provides an effective alternative to larger models or longer multimodal contexts within the evaluated QA settings.

  • Takeaways & Limitations

    GraphLoom’s filtering thresholds are selected on validation data, fixed during testing, and evaluated for robustness only under moderate ±20% threshold perturbations.

Abstract

from arXiv · show

Multimodal retrieval-augmented generation (RAG) systems often rely on long unstructured contexts or aggressively expanded evidence graphs, which can introduce noisy evidence, weaken multi-hop reasoning, and increase unsupported generation. We present GraphLoom, a reliability-calibrated multimodal knowledge-graph RAG framework for compact and faithful evidence routing. Given a question and its associated multimodal input, GraphLoom constructs an instance-level multimodal knowledge graph from grounded scene descriptions, extracted relational triples, and external commonsense knowledge. Instead of injecting all retrieved evidence into the generator, GraphLoom performs reliability-aware subgraph retrieval with bounded expansion and selectively routes high-utility evidence through hierarchical graph memory slots and joint graph-sequence attention in a frozen language model. To improve robustness in complex reasoning settings, GraphLoom further combines interleaved retrieval with budgeted corrective retrieval, enabling adaptive multi-hop evidence refinement under noisy retrieval conditions. We evaluate GraphLoom on ScienceQA, MultiModalQA, and OK-VQA, including large distractor evidence pools that approximate noisy external knowledge retrieval. Experimental results show consistent gains in answer quality and evidence faithfulness over strong multimodal RAG, graph-retrieval, and open-source vision-language baselines, with improved retrieval quality on MultiModalQA and stable performance under noisy evidence pools. Additional analyses using MiniCheck-based verification, human evaluation, and latency profiling show that reliability-calibrated graph evidence routing provides an effective alternative to long-context multimodal evidence injection.

1 Introduction

GraphLoom addresses noisy, incomplete multimodal evidence and context-budget constraints with reliability-calibrated graph retrieval, bounded corrective refinement, and controlled decoder-time evidence injection. Evaluated on ScienceQA, MultiModalQA, and OK-VQA, it improves answer quality, evidence grounding, and MultiModalQA retrieval over strong baselines while analyzing robustness, scalability, and latency.

  • Method: GraphLoom constructs an instance-level multimodal knowledge graph from grounded scene descriptions, extracted relational triples, visual relation priors, and commonsense knowledge.The described components use Qwen3-VL-Instruct, REBEL, Visual Genome, and ConceptNet.
  • Method: For complex multi-hop questions, interleaved retrieval triggers bounded refinement, while an adequacy evaluator permits at most one budgeted corrective action.Corrective actions include targeted expansion, query reformulation, or evidence re-scoring.
  • Method: A reliability-calibrated router activates only a small subset of hierarchical graph slots at each decoding step, injecting compact evidence into a frozen Llama-3.1-8B-Instruct decoder.The interface combines HieraSlot memories, multimodal prefix memories, and KG-JSA++ joint graph–sequence attention.
  • Contribution: GraphLoom treats multimodal evidence retrieval, graph organization, and decoder-time evidence use as a controlled knowledge-management problem.It is introduced as a reliability-calibrated multimodal KG-RAG framework for controlled evidence retrieval, organization, and injection.
  • Evaluation: Across ScienceQA, MultiModalQA, and OK-VQA, GraphLoom improves answer quality and evidence grounding over strong multimodal RAG, graph-retrieval, and vision–language baselines, with further retrieval gains on MultiModalQA.The evaluation also covers retrieval quality, faithfulness, scalability, latency, and robustness under distractor evidence.

2 Related Work

Prior work advances multimodal QA through vision–language models, multimodal knowledge graphs, structured retrieval, efficiency methods, and faithfulness evaluation. GraphLoom positions itself as controlled, reliability-calibrated evidence management that routes graph evidence within a frozen decoder and evaluates both answer quality and retrieval behavior.

  • Multimodal QA: Vision–language models have improved image–text reasoning, but often rely on dense visual tokens, long prompts, or implicit parametric knowledge.Examples include BLIP-2, LLaVA, InstructBLIP, MiniGPT-4, Flamingo, Qwen3-VL, GPT-5, and Gemini 2.5.
  • Multimodal knowledge graphs: Multimodal knowledge-graph research supports structured multi-hop reasoning, cross-modal graph construction, and query-conditioned subgraph retrieval.Examples include MMKGR, MMKG–LLM integration, Pythia-RAG, and EvoGraph-R1.
  • Structured retrieval: Recent RAG systems emphasize structured, iterative, utility-aware, and corrective retrieval across graph, multimodal, visual-textual, and weighted-knowledge-graph settings.Examples include G-Retriever, SKURG, IRCoT, CRAG, VisDoM, and Talk2Doc.
  • Efficiency: Efficiency methods reduce long-context, repeated-retrieval, KV-cache, latency, and memory costs, while GraphLoom adds bounded subgraph retrieval, sparse self-context, decoder-time slot selection, and latency reporting.Attention-sink retention and sparse attention are identified as related cost-reduction approaches not designed for graph-evidence routing.
  • Faithfulness and positioning: GraphLoom distinguishes prior work by treating multimodal KG-RAG as controlled evidence management, routing reliability-calibrated graph slots inside a frozen decoder, and jointly evaluating quality, retrieval, robustness, latency, and faithfulness.Faithfulness-related work includes FActScore, MiniCheck, and RARR; GraphLoom uses independent automatic verification and human evaluation.

3 Methodology: GraphLoom · 3.1 Augmented Multimodal KG Construction · 3.2 Compact Evidence Subgraph Retrieval

GraphLoom builds a question-independent, instance-level multimodal knowledge graph and retrieves compact, reliability-filtered evidence subgraphs from local and optionally noisy external pools. Its bounded, task-conditioned pipeline uses multimodal scoring, fixed validation-selected budgets, and staged evidence refinement to control retrieval noise and decoder input.

  • 3 Methodology: GraphLoom: GraphLoom constructs an instance-level multimodal knowledge graph from grounded scene descriptions, relational triples, and external visual or commonsense knowledge.The graph supports controlled evidence retrieval, organization, and decoder-time injection for each question and associated multimodal input.
  • 3 Methodology: GraphLoom: The framework separates offline evidence construction from online query-conditioned retrieval, slot routing, decoder injection, and conditional correction.This separation enables reporting both full end-to-end and cached online latency.
  • 3 Methodology: GraphLoom: For multi-hop questions, GraphLoom permits at most one hypothesis-guided retrieval round and one budgeted corrective action when evidence is insufficient.A post-generation verification–revision stage edits unsupported spans, while independent evaluators and human validation assess final faithfulness.
  • 3.1 Augmented Multimodal KG Construction: Each instance graph GMMKG = (V, E) represents multimodal evidence under a shared (s, r, o) schema, independently of the question used later for retrieval.Its local triples share a retrieval space with external candidate evidence for compact and noisy-pool evaluation.
  • 3.1 Augmented Multimodal KG Construction: When images are available, Qwen3-VL-Instruct generates grounded scene descriptions, REBEL extracts triples, and entity mentions undergo canonicalization and alias linking.Each extracted triple receives a normalized extraction-confidence score confext(t) ∈ [0, 1] derived from generation likelihood.
  • 3.1 Augmented Multimodal KG Construction: Confidence-aware filtering uses a default edge threshold of τedge = 0.5, with thresholds selected on validation data and fixed during testing.The study also reports robustness under ±20% perturbations of the main evidence-selection and verification thresholds.
  • 3.1 Augmented Multimodal KG Construction: External enrichment adds relevant VG150 visual-relation and ConceptNet commonsense neighbors within at most two hops, retaining relations above τKB = 0.7.Subsequent retrieval applies explicit degree and total-edge budgets to prevent uncontrolled evidence expansion.
  • 3.2 Compact Evidence Subgraph Retrieval: Compact retrieval scores verbalized candidate triples with multimodal similarity and supports standard instance-level retrieval or larger noisy pools containing optional external distractor triples.Triple embeddings can be precomputed and indexed for large pools, while validation-selected budgets K, dmax, and Nmax remain fixed during testing.

3.3 Interleaved Multi-hop Retrieval · 3.4 Evidence Memories: Fusion and HieraSlot

GraphLoom uses bounded interleaved retrieval to recover missing multi-hop connectors while controlling retrieval and generation overhead. It routes retrieved multimodal and structured graph evidence through a fixed global prefix and independently competing entity and relation memory slots.

  • 3.3 Interleaved Multi-hop Retrieval: GraphLoom performs at most two bounded retrieval rounds to recover missing connectors for multi-hop reasoning.The second round is triggered after selecting bridge nodes from the initial retrieved subgraph.
  • 3.3 Interleaved Multi-hop Retrieval: Candidate bridge nodes satisfy deg(v) ≥2 and are ranked by betweenness centrality within the retrieved subgraph.
  • 3.3 Interleaved Multi-hop Retrieval: A frozen Llama-3.1-8B-Instruct decoder generates an intermediate query capped at 15 tokens, allowing one additional retrieval round.Retrieval is rerun with the original multimodal input and intermediate query.
  • 3.3 Interleaved Multi-hop Retrieval: The interleaved step improves multi-hop evidence coverage while keeping retrieval and generation calls bounded, with its cost reported separately.
  • 3.4 Evidence Memories: Fusion and HieraSlot: Retrieved evidence conditions the frozen decoder through a global multimodal prefix and structured graph-slot memories.
  • 3.4 Evidence Memories: Fusion and HieraSlot: The global prefix concatenates the question with facts from the retrieved subgraph, fuses text and visual encodings, and uses a fixed prefix length L_p=32.The fused representation is converted into prefix key/value pairs for KG-JSA++.
  • 3.4 Evidence Memories: Fusion and HieraSlot: Each retrieved triple is represented by complementary entity and relation slots whose keys and values encode different combinations of entities, relations, and objects.Entity slots capture entity-specific attributes, while relation slots encode relational patterns.
  • 3.4 Evidence Memories: Fusion and HieraSlot: Entity and relation slots compete independently during routing, enabling fine-grained selection of evidence before mapping selected representations into KG-JSA++ key/value space.

3.5 Reliability-Calibrated Slot Routing

GraphLoom routes a compact active set of graph slots at each decoding step using semantic utility and reliability estimates. Only the selected slots reach KG-JSA++, limiting decoder-time exposure to noisy triples under a validated budget.

  • Slot selection: At each decoding step, GraphLoom selects a small active set of graph slots from S_HieraSlot.Candidate-slot routing uses slot content, retrieval score, extraction confidence, source type, graph centrality, and the current decoder state.
  • Reliability calibration: The router combines semantic utility with reliability calibration into a step-dependent routing score.The reliability-calibration strength is controlled by β, while decoder-state features make the scores vary across decoding steps.
  • Evidence routing: Only slots in A_t are passed to KG-JSA++, keeping decoder-time evidence use compact and reducing exposure to noisy triples.The active-slot budget k is fixed after validation, and its sensitivity is reported experimentally.

3.6 KG-JSA++: Joint Attention with Sparse Self-KV

KG-JSA++ jointly attends to routed graph memories and decoder context within the frozen Llama-3.1-8B-Instruct decoder, enabling reliability-aware evidence use. Sparse self-KV retention reduces dependence on the full generated history while preserving selected contextual tokens.

  • Joint graph-sequence attention: KG-JSA++ uses one joint attention operation so routed graph memories and internal decoder context compete directly for attention.This replaces separate cross-attention with fine-grained, reliability-aware evidence integration inside the frozen decoder.
  • Sparse self-KV: Sparse self-KV retains recent tokens, beginning-of-sequence sink tokens, and prior tokens selected by cosine similarity to the current query.The retained cache is the union of recent, sink, and similarity-selected token indices.
  • Reliability-aware attention: Slot positions receive a reliability-aware logit bias before a single softmax combines external memories with fused-prefix and sparse self-context scores.The bias encourages reliable evidence while preserving semantic relevance, with γ fixed after validation.
  • Efficiency: Per-step attention-score computation depends on fixed prefix length, active slot budget, and retained sparse self-context rather than directly on full generated history.Under fixed head dimension and retained sparse context, this contrasts with O(t) for full causal self-attention.

3.7 Corrective Retrieval and Evidence-Bound Revision · 3.8 Answer Prediction

GraphLoom addresses imperfect retrieval through budgeted corrective actions and evidence-bound answer revision, then predicts answers from graph memory with reproducible decoding and an evidence-safety fallback. Its revision process verifies atomic claims against multimodal textualized evidence and restricts edits to unsupported spans.

  • 3.7 Corrective Retrieval and Evidence-Bound Revision: GraphLoom triggers at most one corrective action when its evidence-adequacy score falls below a threshold, using graph, query, and retrieval-history signals.The evaluator combines subgraph statistics, question difficulty, and prior retrieval actions into E_eval; corrective retrieval occurs if E_eval < τ_eval.
  • 3.7 Corrective Retrieval and Evidence-Bound Revision: The action selector chooses among Expansion Boost, Query Reformulation, and Verbalization Repair under a remaining evidence budget.Expansion Boost reruns bounded expansion, Query Reformulation clarifies the query, and Verbalization Repair rescoring targets low-confidence triples.
  • 3.7 Corrective Retrieval and Evidence-Bound Revision: After generation, GraphLoom decomposes the draft into atomic claims and checks each against retrieved graph, grounded scene, and available textual evidence.The verification context can include serialized table context when applicable, and claim-level NLI scores determine whether additional retrieval is needed.
  • 3.7 Corrective Retrieval and Evidence-Bound Revision: Claims below the entailment threshold trigger retrieval of relevant MMKG triples and constrained revision that preserves supported spans while editing unsupported ones.The constrained prompt is sent to the frozen decoder and limits revisions to unsupported spans whenever possible.
  • 3.7 Corrective Retrieval and Evidence-Bound Revision: Final faithfulness is evaluated independently through verification and human validation rather than relying solely on the verifier used during revision.This separation prevents reported H-Rate results from being tied only to the internal DeBERTa-v3 NLI model.
  • 3.8 Answer Prediction: For answer prediction, each option is scored by length-normalized log-likelihood under the frozen decoder conditioned on memory context M.Open-ended questions instead use autoregressive decoding conditioned on M.
  • 3.8 Answer Prediction: GraphLoom fixes decoding settings across controlled comparisons and can emit [NO_EVIDENCE] when retrieved evidence fails adequacy requirements, stopping decoding.When sampling is used, the random seed and number of runs are fixed consistently for reproducibility.

3.9 Computation and Latency Accounting · 3.10 Training Objective

GraphLoom separates cacheable graph construction from query-time evidence routing and generation when accounting for computation and latency. Its training objective updates lightweight routing and fusion modules with losses for answer generation, calibration, slot utility, and router–attention alignment while keeping backbone models frozen.

  • 3.9 Computation and Latency Accounting: GraphLoom reports full-pipeline and online latency separately to distinguish cacheable preprocessing from query-time routing and generation costs.Full-pipeline latency includes graph construction, retrieval, decoding, verification, and revision; online latency assumes cached instance-level graphs.
  • 3.10 Training Objective: The backbone models remain fixed, while training optimizes the multimodal fusion projection, HieraSlot projections, and reliability router.The frozen components include Qwen3-VL encoders and the Llama-3.1-8B-Instruct decoder.
  • 3.10 Training Objective: The end-to-end objective combines answer generation, router calibration, contrastive slot utility, and router–attention alignment.These correspond to generation, calibration, contrastive-futility, and alignment loss terms.
  • 3.10 Training Objective: The validation-selected loss weights are {𝜆gen, 𝜆cal, 𝜆cf, 𝜆align} = {1.0, 0.5, 0.3, 0.2} and remain fixed for test evaluation.All loss weights, thresholds, and routing/retrieval budgets are selected on validation data before testing.
  • 3.10 Training Objective: Lgen applies label-smoothed negative log-likelihood to the gold option in multiple-choice QA or the reference answer in open-ended QA.The loss conditions on memory context M produced by the multimodal prefix and routed graph slots.
  • 3.10 Training Objective: Lcal calibrates router confidence against binary slot-utility targets, while Lcf applies a margin loss to useful–misleading slot pairs.The utility target indicates whether a slot provides supporting evidence for the target answer, and the margin is validation-selected.
  • 3.10 Training Objective: Lalign aligns router probabilities with decoder attention over selected evidence slots, encouraging routing to match evidence used during generation.Attention is averaged across decoder heads, layers, and answer-token steps, then renormalized over the same slot set; the final faithfulness evaluator is not used for training.
  • 3.10 Training Objective: Trainable modules use AdamW, linear warmup, cosine decay, and gradient clipping at 1.0, with early stopping based on 𝑆val = 0.7𝑆ans + 0.3𝑆faith.Validation answer and grounding scores determine early stopping, while thresholds and routing/retrieval budgets are fixed before test evaluation.

4 Experimental Evaluation

GraphLoom is evaluated across answer quality, retrieval, faithfulness, robustness, scalability, and efficiency on ScienceQA, MMQA, and OK-VQA. It achieves strong multimodal QA performance while maintaining evidence faithfulness under noisy retrieval and bounded inference costs.

  • Answer quality: GraphLoom achieves 92.5% accuracy on ScienceQA, 55.2 EM / 66.5 F1 on MMQA, and 70.5% VQA accuracy on OK-VQA.Gains are larger on MMQA and OK-VQA, which require external knowledge, multimodal alignment, or multi-hop reasoning.
  • Comparative analysis: GraphLoom outperforms IRCoT, CRAG, Flat-RAG, and CLIP-RAG on MMQA while remaining competitive with GPT-5 and Gemini 2.5 Pro.The results associate gains with reliability-calibrated routing, KG-JSA++, structured slots, and selective graph-memory routing.
  • Faithfulness and efficiency: GraphLoom achieves the lowest unsupported-claim rate among evaluated systems, with independent automatic verification and human validation reducing circularity risk.At inference, it retrieves 38.2 triples per query, activates at most |A𝑡| ≤4 graph slots per decoding step, and reaches 119 ms online latency with cached graphs.
  • Ablations: Removing the router increases H-Rate from 10.5 to 17.3, while removing KG-JSA++ causes the largest degradation in answer quality and faithfulness.Interleaved retrieval mainly improves EM/F1, corrective retrieval more strongly affects H-Rate, and post-hoc revision substantially reduces unsupported claims.
  • Scalability: With one million distractor triples, Recall@5 decreases by 5.2 points and MRR by 0.058, while MMQA F1 drops by only 1.4 points.Bounded expansion, reliability-aware slot selection, and fixed limits on Nmax and k keep generation cost independent of candidate-pool size.
  • Robustness: MMQA EM and F1 remain within 0.8 points of the default setting across threshold perturbations, while the default provides the best accuracy–faithfulness balance.Threshold changes trade evidence coverage and noise, with stricter evaluation and revision thresholds generally lowering H-Rate while potentially reducing answer quality.

5 Conclusion and Future Work

GraphLoom retrieves compact evidence subgraphs and injects selected graph memories into a frozen decoder through HieraSlot and KG-JSA++. Across ScienceQA, MultiModalQA, and OK-VQA, it improves answer quality and faithfulness, including MMQA retrieval, while remaining stable under noisy evidence pools and moderate threshold variation.

  • Conclusion: GraphLoom retrieves compact evidence subgraphs and routes selected graph memories through HieraSlot and KG-JSA++ in a frozen decoder.The framework combines reliability-calibrated retrieval with hierarchical graph memory injection and joint graph-sequence processing.
  • Conclusion: Experiments on ScienceQA, MultiModalQA, and OK-VQA show consistent gains in answer quality and faithfulness, with additional improvements in MMQA retrieval.The reported evaluation spans three multimodal question-answering benchmarks.
  • Conclusion: Scalability, robustness, and latency analyses demonstrate stable performance under noisy evidence pools and moderate threshold variation.These analyses support reliability-aware evidence routing as an alternative to large evidence injection.

GenAI Usage Disclosure

OpenAI ChatGPT assisted with implementation, presentation, and language-related tasks, while the authors reviewed and validated all generated code and reported outputs and retained responsibility for the final work.

  • GenAI Usage Disclosure: OpenAI ChatGPT assisted with code generation, debugging, experimental output preparation, LaTeX, language refinement, and consistency checking.The authors reviewed and validated all generated code and reported outputs, made the methodological and technical decisions, and take full responsibility for the final work.
Loading 2608.15056v1…