Source-linked AI summary
GUIDE: Guiding Internal Evidence with Language Instructions
Soyeon Caren Han, Hyunsuk Chung, Jinwoo Kim, Seungyeon Ji, Kyungreem Han
TL;DR
Large multimodal models may follow output instructions without controlling the evidence they rely on, leaving shortcut-associated cues influential. GUIDE addresses this by combining grouped adaptation pathways with instruction-conditioned gating and evaluates whether reliance shifts while task behavior is preserved. Across diverse reasoning, classification, and generation benchmarks, it reports structured instruction-aligned redistribution and improved robustness under targeted evidence perturbations.
Problem
Multimodal models can follow instructions about outputs without systematically controlling the evidence sources contributing to their predictions.
Method
GUIDE combines taxonomy-aligned grouped parameter-efficient adaptation with instruction-conditioned gating to modulate evidence pathways during reasoning and generation.
Results
Across multimodal reasoning, classification, and generation benchmarks, GUIDE produces instruction-aligned pathway and functional-reliance shifts while largely preserving task behavior and improving robustness to targeted perturbations.
Takeaways & Limitations
GUIDE extends multimodal instruction following toward regulating how different evidence sources contribute to predictions, beyond controlling outputs alone.
Takeaways & Limitations
GUIDE does not guarantee isolated evidence pathways, and its perturbations are approximate interventions that may affect multiple evidence categories simultaneously.
Abstract
from arXiv · showhide
Large multimodal models follow instructions about what to generate, but not necessarily about what evidence to rely on. Hence, models may continue to depend on shortcut-associated cues even when instructions suggest otherwise. We introduce GUIDE, a framework for controlling internal evidence usage through language instructions. GUIDE combines grouped parameter-efficient adaptation with instruction-conditioned gating to modulate multimodal evidence pathways during reasoning and generation. We further introduce a pathway-level evaluation framework that characterizes instruction-conditioned evidence modulation through reliance sensitivity, controlled perturbation analysis, pathway modulation, and autoregressive decoding dynamics. Across multimodal reasoning, classification, and generation, GUIDE induces structured and instruction-aligned redistribution of evidence reliance while largely preserving task behavior. Experiments on GQA, TextVQA, MM-IMDb, CREMA-D, RAVDESS, and Flickr30K show that GUIDE improves robustness under targeted evidence perturbations and enables controllable modulation across diverse multimodal settings. This suggests that multimodal instruction following can extend beyond output control toward regulating how different evidence sources contribute to model predictions.
1 Introduction
GUIDE addresses the gap between following instructions about outputs and controlling the evidence used to produce them. It modulates internal evidence reliance so similar answers can arise from different pathways while largely preserving task behavior.
- Multimodal models may follow instructions about what to generate without systematically changing which evidence sources support their predictions.The same answer can be supported by object relationships, OCR, visual attributes, or contextual background cues.
- GUIDE combines grouped parameter-efficient adaptation with instruction-conditioned gating to regulate multimodal evidence pathways during reasoning and generation.The framework operates through grouped internal pathways rather than changing task objectives or input representations.
- Semantic-focused instructions increase reliance on action and interaction pathways, whereas appearance-focused instructions increase reliance on visual attribute pathways.
- GUIDE evaluates instruction-conditioned evidence use through pathway modulation, controlled perturbations, reliance sensitivity, and autoregressive decoding dynamics.
- Across reasoning, classification, and generation, GUIDE produces instruction-aligned evidence redistribution, improved robustness under targeted perturbations, and largely preserved task behavior.The evaluation spans GQA, TextVQA, MM-IMDb, CREMA-D, RAVDESS, and Flickr30K.
2 Related Work
Prior multimodal instruction-following and representation-control methods mainly target outputs or latent behavior, while offering limited explicit control over internally selected multimodal evidence. GUIDE instead treats instructions as interventions on evidence reliance itself.
- Existing multimodal instruction-following methods primarily align outputs with instructions rather than controlling the internal evidence used to produce them.Prompt engineering, controllable decoding, attention steering, and visual grounding can influence generated behavior without explicitly regulating evidence selection.
- Feature steering and representation-editing methods manipulate latent representations or activations but primarily focus on text-only settings.
- Bias-mitigation methods generally address dataset-level spurious correlations during training rather than supporting user-specified evidence control at inference.
- GUIDE translates language instructions into explicit control over internal multimodal evidence usage, separating output controllability from evidence controllability.Similar predictions can arise from different internal evidence-reliance patterns.
3 GUIDE
GUIDE organizes multimodal evidence into task-relevant grouped adaptation pathways and dynamically gates them using the instruction and evolving model state. It evaluates both pathway activation changes and functional reliance changes under controlled perturbations.
- Problem formulation: GUIDE formulates evidence control as changing pathway activation and functional reliance while keeping predictions approximately similar across instructions.
- Evidence organization: The framework groups semantic, appearance, contextual, motion, acoustic, demographic, and OCR/textual evidence according to each benchmark’s dominant structure.GQA further decomposes semantic evidence into object, relation, and action groups, with appearance represented by an attribute group.
- Grouped LoRA Adaptation: Grouped LoRA decomposes one low-rank adaptation into taxonomy-aligned branches whose contributions are dynamically modulated by instruction-conditioned gates.The gates can selectively amplify or suppress pathways during inference without requiring explicit representation disentanglement.
- Instruction-Conditioned Gating: Pathway-specific gates depend on both instruction representations and current hidden states, allowing multiple pathways to remain active and change throughout generation.For autoregressive tasks, gates are computed at each decoding step.
- Evaluation Measures: Gate Modulation Response measures instruction-driven pathway activation changes, while Reliance Sensitivity measures functional dependence on targeted evidence.GMR and RS therefore distinguish activation modulation from corresponding changes in functional reliance.
- Evaluation Measures: Reliance is measured by masking, corruption, replacement, or suppression perturbations that target pathway-associated evidence while preserving overall task solvability.
4 Evaluation Setup
The evaluation tests controllable evidence modulation across multimodal reasoning, classification, and generation benchmarks. It combines instruction families, targeted evidence perturbations, task metrics, pathway metrics, and GQA gate visualizations.
- GUIDE is evaluated across multimodal reasoning, classification, and generation settings to assess evidence controllability, generalization, and task-semantic preservation.
- Benchmarks: The benchmark suite includes GQA and TextVQA for reasoning, diverse multimodal datasets for classification, and Flickr30K for image captioning.Caption generation tests whether evidence modulation persists throughout autoregressive decoding.
- Instructions: Instruction families encode semantic, appearance, background-suppression, motion, and OCR-focused evidence preferences depending on the benchmark.Paraphrased and compositional variants test generalization beyond fixed lexical templates.
- Perturbations: Taxonomy-aligned perturbations target semantic, appearance, contextual, OCR/textual, motion, and acoustic evidence through masking, corruption, replacement, and suppression.The interventions are designed to degrade targeted evidence while preserving overall task solvability.
- Metrics: Evaluation reports standard task-performance metrics alongside Gate Modulation Response and Reliance Sensitivity.GMR measures pathway activation changes, while RS measures functional dependence on targeted evidence.
- Visualization: The GQA gate-activation heatmap compares Object, Relation, Action, Attribute, Background, and Demographic pathways under different instruction conditions.
5 Results
GUIDE consistently aligns pathway activations and functional reliance with instructions across multimodal tasks, while improving robustness and often preserving predictions during substantial internal evidence shifts.
- Pathway Modulation: Across datasets, semantic-focused instructions increase semantic and relational pathway activation, while appearance-focused and suppression-oriented instructions selectively alter targeted pathways.Neutral instructions produce comparatively balanced activation patterns.
- Functional Evidence Reliance: Across tasks, Reliance Sensitivity patterns align with instructed evidence preferences, increasing sensitivity to semantic, relational, interaction-related, or appearance-related evidence.GQA and additional benchmarks show corresponding changes in functional evidence reliance.
- Counterfactual Robustness: Semantic-focused and background-suppression instructions remain highly stable under color and background perturbations, whereas object removal reduces stability across all instruction conditions.Object-level evidence remains task-critical across routing preferences.
- Perturbation Robustness: Appearance-focused instructions degrade most rapidly under increasing color-shift strength, while semantic-focused and background-suppression instructions retain comparatively higher accuracy.Robustness curves are averaged across the evaluation set over multiple perturbation strengths.
- Decoding-Time Routing: Different instructions produce persistent pathway activation changes throughout Flickr30K autoregressive decoding, yielding captions that emphasize actions and interactions or visual attributes.Appearance suppression reduces appearance-specific details while preserving semantic and contextual pathways.
- Same Answer, Different Evidence: GQA preserves the same answer in 93.8% of cases despite substantial shifts in pathway modulation and functional reliance.Table 1 reports same-prediction rate for datasets other than Flickr30K, which uses caption similarity.
- Ablations: Removing grouped pathways or replacing dynamic instruction-conditioned gates reduces pathway modulation, functional reliance shifts, and perturbation robustness.Prompt-only control exhibits the weakest evidence-modulation behavior across the representative ablations.
6 Conclusion
GUIDE extends multimodal instruction following from controlling outputs toward regulating the evidence sources contributing to predictions. Its effects remain instruction-aligned across reasoning, classification, and generation while largely preserving task behavior.
- Conclusion: GUIDE combines taxonomy-aligned grouped adaptation pathways with instruction-conditioned gating to modulate multimodal evidence during reasoning and generation.The framework supports different evidence-reliance patterns under natural-language instructions.
- Conclusion: Across multimodal benchmarks, GUIDE produces pathway modulation, corresponding functional-reliance shifts, and improved robustness under targeted evidence perturbations.These effects are reported across reasoning, classification, and generation tasks.
- Qualitative Evidence Routing: On GQA and TextVQA, semantic-focused instructions emphasize action, interaction, OCR, and text-relevant evidence, whereas appearance-focused instructions prioritize visual attributes and contextual cues.The same input can therefore receive different evidence emphasis and sometimes different prediction behavior.
Limitations
GUIDE provides an operational intervention space rather than perfectly disentangled or isolated evidence pathways.
- Scope Boundary: GUIDE does not explicitly disentangle semantic factors or guarantee perfectly isolated evidence pathways.Pathway activations should not be interpreted as uniquely corresponding to fully disentangled semantic concepts.
- Scope Boundary: The perturbation protocols are approximate operational interventions, and evaluation is focused primarily on controlled multimodal benchmarks rather than open-ended agents or deployment settings.Future work may pursue more principled causal intervention and localization methods.
Potential Risks
Controllable evidence routing may obscure reasoning or introduce artifacts, so GUIDE should not be treated as guaranteeing fairness, bias removal, or full interpretability.
- Misuse Risk: Selective evidence amplification or suppression may obscure model reasoning behavior while preserving similar output behavior.Evidence suppression could unintentionally hide undesirable biases.
- Misuse Risk: Imperfect localization and perturbation protocols may introduce unintended artifacts or incompletely suppress sensitive cues.The paper therefore frames GUIDE as a research tool for operational intervention settings.
Ethics Statement
The human evaluation used voluntary annotators who consented to the procedure and assessed outputs from publicly available benchmarks without collecting sensitive personal data or personally identifiable information.
- Voluntary annotators were recruited for the human evaluation and provided consent before participating.
- The annotation tasks evaluated multimodal model outputs and excluded sensitive personal data, medical information, and high-risk decision making.
- Evaluation data came from publicly available research benchmarks under their original licenses, and no personally identifiable information was collected or stored.
D Training Objective
GUIDE trains grouped adaptation pathways and instruction-conditioned gates with the original task objective while keeping the pretrained backbone frozen. Bernoulli entropy regularization encourages selective pathway activation without requiring pathway labels.
- GUIDE updates grouped low-rank adaptation parameters and gating networks while keeping the pretrained multimodal backbone frozen.This preserves pretrained capabilities while enabling lightweight instruction-conditioned modulation.
- The gating network learns instruction-sensitive pathway modulation jointly with the task objective without explicit pathway supervision or manually annotated evidence labels.The approach also does not require explicitly disentangled internal representations.
- Independent sigmoid gates assign each pathway activation gc ∈ [0, 1] without normalization across pathways.Each pathway is treated independently rather than competing through a normalized gate distribution.
- Bernoulli entropy regularization encourages confident and selective pathway activation, with λgate controlling regularization strength.The final objective combines task loss and gate regularization: L = Ltask + Lgate.
E Extended Conflict Routing Results
Across contrasting evidence-routing instructions, GUIDE preserves task behavior while substantially changing pathway activations and functional reliance. The modulation is continuous across instruction mixtures, structured across decoding depth, and stable across paraphrased and compositional variants.
- Extended conflict routing: GUIDE preserves high output consistency across datasets despite substantial changes in pathway activation and functional reliance.Semantic–appearance contrasts generally produce the largest modulation, while neutral or partially aligned pairs produce smaller changes.
- Extended conflict routing: Strongly contrasting evidence preferences maintain relatively stable task behavior across visual, textual, acoustic, and generation settings.These results support continuous and compositional evidence modulation rather than isolated prompt templates or single-pathway switching.
- Compositional conflict routing: As α shifts from appearance-oriented to semantic-oriented reasoning, semantic and appearance pathway activations change smoothly rather than switching abruptly.Balanced conditions produce intermediate activation patterns while task performance remains comparatively stable.
- Layer-wise modulation: Deeper Flickr30K layers increasingly emphasize semantic pathways while gate entropy decreases, indicating progressively more selective activation during generation.Earlier layers show stronger appearance-related activation, with intermediate layers displaying more balanced patterns.
- Instruction generalization: Semantic-intent and appearance-intent instructions form separated Reliance Sensitivity clusters across template-based, paraphrased, and compositional variants.The stable separation suggests evidence modulation generalizes beyond fixed lexical formulations.
H Full Ablation Results
Full ablations and additional analyses show that GUIDE's grouped routing and dynamic gating support structured evidence modulation, functional reliance shifts, perturbation robustness, and task-consistent outputs across multimodal benchmarks.
- Ablations: Removing taxonomy-aligned grouped pathways substantially reduces pathway modulation and changes in functional reliance across benchmarks.The full ablation study reports consistent reductions when grouped pathways are removed.
- Ablations: Replacing dynamic instruction-conditioned gates with static routing decreases perturbation robustness and evidence-modulation behavior.Prompt-only control exhibits the weakest evidence-modulation behavior.
- Gate activation: Instruction-conditioned gating increases semantic and interaction-related routing groups for semantic-focused instructions and appearance-related groups for appearance-focused instructions.Suppression-oriented instructions reduce targeted routing-group activation while preserving other task-relevant pathways across benchmarks.
- Functional reliance: Reliance Sensitivity shifts systematically with instruction conditions, increasing for targeted semantic, interaction, or appearance pathways while preserving sensitivity to other task-relevant pathways.Higher RS represents greater functional dependence on the corresponding evidence pathway.
- Perturbation stability: Semantic-focused and suppression-oriented instructions remain comparatively stable under appearance- and context-related perturbations, whereas appearance-focused instructions degrade more under appearance perturbations.Across evaluated datasets, perturbations to core semantic evidence produce the largest performance degradation.
- Efficiency: GUIDE keeps computational overhead substantially below full-model fine-tuning by freezing pretrained parameters.The appendix contains detailed hyperparameters, backbone configurations, and implementation details.
- Human evaluation: Human evaluation on 50 randomly sampled Flickr30K examples rated instruction alignment and semantic preservation across instruction conditions.Annotators used a 4-point scale for both criteria.
- Qualitative analysis: Qualitative examples align gate-activation changes with corresponding changes in functional reliance during reasoning and generation.Semantic-focused instructions emphasize task-relevant semantic structure, while appearance-focused instructions emphasize visual attributes and related cues.