From Routing Signals to Selective Review: Visual regrounding in MoE VLMs

1 INTRODUCTION

• Target-absence failures occur when a model accepts a false visual premise and supplies an attribute for an object absent from the image.Because the question requests an attribute rather than an existence judgment, a fluent answer can obscure the missing visual evidence.Source: §1 · p. 2
• Existing detectors primarily use generated responses, hidden states, or uncertainty, while whether MoE routing reveals target absence before generation remains unclear.Reviewing every input can add inference cost; the paper investigates routing as a signal for selective correction.Source: §1 · p. 2
• The framework extracts routing probabilities at target-noun tokens, trains a model-specific L2-regularized linear detector, and uses its prediction to gate target-aware review.Predicted-present inputs retain the standard response; predicted-absent inputs receive a review request, without changing model weights or router parameters.Source: §1 · p. 2
• Routing-gated target-aware review improves end-to-end accuracy across both models and datasets, with Qwen gains of 22.25 points on GQA-Inpaint and 12.17 points on OBER.The framework also reports improvements for Gemma on both datasets.Source: §1 · p. 2
• On GQA-Inpaint, Qwen and Gemma routing probes reach test ROC-AUCs of 0.9988 and 0.9956, respectively.On external OBER, their ROC-AUCs are 0.8095 and 0.7781, while a GQA-calibrated threshold transfers less reliably.Source: §1 · p. 2
• Routing encodes target presence and absence with identifiable layer- and expert-level structure, and selective review concentrates additional inference on likely failures.False-positive review causes limited harm overall, but cross-dataset threshold shifts mean ranking transfers more reliably than a fixed decision threshold.Source: §1 · p. 2

2.1 VISUAL GROUNDING FAILURES AND FALSE-PREMISE EVALUATION

• False-premise evaluation asks whether a model recognizes that requested information presupposes an absent object, rather than merely judging explicit object presence.Removed-object benchmarks preserve contextual cues that can make a deleted object seem plausible; this work studies target-absence failures rather than all visual hallucinations.Source: §2.1 · p. 3

2.2 INTERNAL SIGNALS FOR GROUNDING DETECTION

• Grounding detectors use response confidence or consistency, hidden-state uncertainty, entity-level monitoring, and pre-generation query or visual representations.Prior pre-generation work leaves open which other internal representations expose visual support for a particular referent; this paper examines target-token MoE routing.Source: §2.2 · p. 3

2.3 MOE ROUTING ANALYSIS AND BEHAVIORAL CONTROL

• MoE routing organization varies across models and layers, motivating analysis under specific inputs rather than universal functional labels for experts.Prior MoE interventions steer computation or fine-tune experts, whereas this paper uses unchanged routing features to select a review prompt.Source: §2.3 · p. 3

2.4 VISUAL VERIFICATION AND SELECTIVE REVIEW

• Visual correction methods modify decoding, verify claims, or ground answers with image descriptions, but their availability does not establish that every input needs correction.Prompted reconsideration can also introduce errors; prior selective review uses uncertainty signals, while this paper uses a supervised target-absence routing probe.Source: §2.4 · pp. 3–4

3 METHODOLOGY

• Each example pairs an image and target-referencing question with a binary label, and the pipeline extracts routing, estimates target absence, then reviews only above a threshold.Target-present and target-absent images may share a source scene, but each image–question pair is processed independently.Source: §3 · p. 4

3.1 TARGET-TOKEN ROUTING REPRESENTATION

• The method maps the target noun phrase to tokenizer sub-token positions and averages the router’s complete pre-selection probabilities over that span across layers.The resulting vector retains probabilities for all experts, including experts outside the Top-k executed set; Qwen’s vector has 6,144 dimensions.Source: §3.1 · p. 4

3.2 ROUTING-BASED TARGET-ABSENCE DETECTION

• A model-specific L2-regularized linear probe maps standardized layer–expert routing features to an absence score, with training statistics frozen for validation, test, and external data.Regularization is selected by validation ROC-AUC and the decision threshold by validation balanced accuracy; test labels do not influence these choices.Source: §3.2 · p. 4
• The linear probe supports layer–expert interpretation, but coefficient magnitude is a conditional predictive contribution, not a standalone activation effect or causal importance.Coefficient direction associates greater routing probability with target absence or presence, depending on its sign.Source: §3.2 · p. 5

3.3 ROUTING-GATED SELECTIVE REVIEW

• The thresholded score gates the unchanged model: predicted-present inputs keep the standard response, while predicted-absent inputs receive a target-aware review prompt.Review asks the model to verify whether the target is present and state explicitly when it is absent; only triggered inputs incur an additional forward pass.Source: §3.3 · p. 5

4.2.1 ROUTING GATE RELIABILITY

• On GQA-Inpaint, Qwen and Gemma achieve 0.9988 and 0.9956 ROC-AUC; on OBER, the ranking signal remains useful, but GQA-selected thresholds transfer less reliably.The manually audited GQA set used to test the review policy includes validation images.Source: §4.2.1 · p. 6, Figure 2 · p. 6
• Routing reaches 0.9988 ROC-AUC and 98.54% accuracy versus 0.9979 and 98.06% for mean hidden states; 500 features retain 0.9984 ROC-AUC versus 0.9988 with all 6,144.Routing retains explicit layer–expert coordinates; the feature-count comparison does not measure VLM inference cost.Source: Table 1 · p. 6, Figure 3 · p. 6, §4.2.1 · p. 7

4.2.2 ROUTING-GATED VISUAL REGROUNDING

• Target-aware review raises Qwen accuracy from 66.75% to 89.00% on GQA and 83.39% to 95.57% on OBER; Gemma rises from 80.00% to 93.42% and 93.91% to 95.30%.Target-aware review yields positive gains for both models and datasets, while generic review gives smaller gains in all four settings.Source: Figure 4 · p. 7, §4.2.2 · p. 7
• Selective review beats always review for Qwen overall: 89.00% versus 86.25% on GQA and 95.57% versus 95.30% on OBER, while avoiding 49.58% and 16.09% of review calls.Always reviewing present images lowers their accuracy; selective review retains most correct present-image answers.Source: §4.2.2 · p. 7, Table 2 · p. 7

4.2.3 CORRECTION ON TRIGGERED ABSENT IMAGES

• On triggered target-absent images, Qwen OBER accuracy rises from 66.49% to 95.79%, while Gemma rises from 93.10% to 100%.Gemma's smaller gain reflects limited room for improvement on this subset.Source: §4.2.3 · pp. 7–8

4.2.4 COST OF FALSE TRIGGERS

• After false triggers, target-aware review turns 7.90% of initially correct Qwen answers and 7.98% of Gemma answers wrong, exceeding generic review's 0.99% and 5.32%.The GQA harm rates are based on only two images per model, with five sampled answers per image, limiting evidence about dataset-level risk.Source: Figure 5 · p. 8, §4.2.4 · p. 8

4.3.1 EXPERT WEIGHT DISTRIBUTION

• Four of the eight highest-ranked experts are in Layer 2, while weights spread across layers and the largest coefficient is only 0.164% of their absolute-weight sum.Weights indicate how the probe uses routing features, not the causal role of individual experts.Source: §4.3.1 · pp. 8–9, Table 3 · p. 9, Figure 6 · p. 9

4.3.2 LAYER-WISE DETECTION

• Layer 2 alone reaches 0.9836 test ROC-AUC and 94.33% accuracy with 128 features, while the full probe reaches 0.9988 ROC-AUC.Combining layers improves detection beyond the early-layer probe.Source: §4.3.2 · p. 9, Figure 6 · p. 9

• Target-token routing reveals target absence even when the model’s final answer accepts the false premise.Source: §5 · p. 9
• Selective target-aware review improves accuracy by up to 22.25 percentage points for Qwen and 13.42 for Gemma across GQA-Inpaint and OBER.The method does not modify model weights or router parameters.Source: §5 · p. 9
• Qwen analyses find target-absence information in early MoE layers and distributed across experts, revealing a gap between routing evidence and final answers.Source: §5 · p. 9

• Large language models assisted with experimental scripts, including linear-probing training scripts.Source: AI USE STATEMENT · p. 10
• They also helped identify relevant prior work, check manuscript logic and grammar, and improve writing clarity and fluency.Source: AI USE STATEMENT · p. 10
• The authors tested AI-generated code, independently verified suggested references, and reviewed and took responsibility for the final content.They also used AI to draft parts of the experimental results section.Source: AI USE STATEMENT · p. 10

A.1 DATA AND EVALUATION PROTOCOL

• The 31,222-pair GQA-Inpaint probe cohort was split into 20,000 training, 5,446 validation, and 5,776 held-out test pairs, each contributing present and absent examples.Source: §A.1 · p. 12
• Source-scene pairs stayed within one split; policy evaluation used 120 GQA pairs and 115 retained OBER pairs, separate from the held-out detector test.The GQA review cohort included validation examples, while detector performance was measured on the separate 5,776-pair test split.Source: §A.1 · p. 13, Table 4 · p. 13
• A researcher audited image presence and absence, and answers on absent-target images were marked wrong when they invented an attribute, relation, or state.The GQA review cohort covered three difficulty groups and four question types; sampled answers from one image were treated as repeated observations, not independent images.Source: §A.1 · p. 13

A.2 MODELS, ROUTING FEATURES, AND PROBE TRAINING

• The probes use 6,144 Qwen or 3,840 Gemma layer–expert probabilities, averaged across target sub-tokens when needed, without changing model or router weights.Routing is logged at the target-object span during prompt prefill; extraction was exact on audited templates but is not established for unrestricted questions.Source: §A.2 · p. 13
• Each model’s probe is standardized and trained separately, with weight decay selected by validation ROC-AUC and its threshold set by validation balanced accuracy.Held-out GQA test and OBER labels are not used for fitting or selecting frozen detectors.Source: §A.2 · p. 13, Table 5 · p. 14
• Review prompts prepend generic or target-aware instructions to the unchanged question and short-answer request, with model-specific separators for target-aware sentences.Policy answers were sampled at temperature 0.7, top-p 0.8, top-k 20, and a 64-token limit.Source: §A.2 · p. 13, Table 6 · p. 14

A.3 ADDITIONAL DETECTOR RESULTS

• On OBER, the frozen GQA thresholds produce different operating points, especially for Qwen, whose absence recall is high but specificity is low.AUC also decreases for both models; Table 7 reports detector performance with the GQA-validation-selected threshold frozen across datasets.Source: §A.3 · p. 14, Table 7 · p. 14
• 0.9988 ROC-AUC and 98.54% accuracy for all-layer routing compare with 0.9979 ROC-AUC and 98.06% accuracy for mean hidden states on Qwen/GQA.Text-only input embeddings reach 0.5000 test ROC-AUC and 50.00% accuracy; Table 8 covers target-token hidden-state probes.Source: §A.3 · p. 14, Table 8 · p. 15
• The comparison does not show routing outperforming every hidden-state representation; its scope is Qwen on GQA, and several hidden layers are highly predictive.Routing coordinates identify a layer and expert, unlike an undifferentiated representation.Source: §A.3 · p. 15
• 65.22% to 77.75% is Qwen's mean OBER balanced-accuracy change with local threshold calibration; Gemma changes from 70.87% to 73.19%.These held-out diagnostic means use labeled OBER calibration pairs and are not zero-shot policy results; threshold changes do not improve ROC-AUC on a fixed set.Source: §A.3 · p. 15
• 0.5151 ± 0.0618 is the mean validation ROC-AUC across 20 independently shuffled-label Gemma control runs, consistent with chance.Each run selected its epoch using shuffled validation labels and was evaluated against true validation labels.Source: §A.3 · p. 15

A.4 ADDITIONAL REVIEW-POLICY RESULTS

• End-to-end accuracy combines target-present and target-absent images, and a reviewed answer replaces the standard answer only when the frozen GQA threshold is exceeded.Table 10 reports standard generation and two routing-gated review prompts; gains are absolute percentage-point differences from standard answering.Source: §A.4 · p. 15, Table 10 · p. 16
• 22.25±0.23 points on GQA and 12.17±0.81 on OBER are Qwen's mean target-aware gains across five sampled repetitions per image.The gain is positive in all five repetitions on both datasets, but the repetitions use the same images rather than independent dataset replications.Source: §A.4 · p. 16, Figure 8 · p. 17
• 89.00% versus 86.25% on GQA and 95.57% versus 95.30% on OBER are Qwen selective versus always-review accuracy with target-aware review.The gate sends 50.42% of GQA and 83.91% of OBER images to review, avoiding 49.58% and 16.09% of potential review calls, respectively.Source: §A.4 · p. 16, Table 11 · p. 17
• Target-aware review yields larger gains on absent images but can reduce accuracy on present images, especially when applied to every input.Table 12 separates outcomes by true target presence; selective review retains standard answers when the gate does not trigger.Source: §A.4 · p. 17, Table 12 · p. 17
• Figure 9 compares standard, generic, and target-aware answers only on absent images sent to review, excluding absent cases missed by the gate.The triggered subset may differ between models and its denominator differs from the full absent-image rows in Table 12.Source: §A.4 · pp. 17–18, Figure 9 · p. 18
• 2.75 points is Qwen's selective-review advantage over always review on GQA (95% CI [1.17, 4.58]); on OBER it is 0.26 points (95% CI [0, 0.70]).The OBER gain is small and its interval includes zero.Source: §A.4 · p. 18
• Among 405 initially correct Qwen answers on false-triggered present images, generic review makes four wrong while target-aware review makes 32 wrong.A changed answer is not necessarily harmed; the GQA per-dataset estimate is unstable because only two images contribute.Source: §A.4 · p. 18, Table 13 · p. 18

A.5 LAYER-WISE ROUTING

• 0.9836 test ROC-AUC and 94.33% accuracy are reached by Qwen Layer 2 alone, versus 0.9988 ROC-AUC for all layers.Layer 0 reaches 0.9006 test ROC-AUC, and Layer 1 is the first above 0.90 on validation; Table 14 lists all 48 classifiers.Source: §A.5 · pp. 18–19, Table 14 · p. 19

A.6 TOP-K ROUTING PROBES

• The top-K experiment ranks Qwen's 6,144 routing dimensions by absolute standardized weight, then refits probes at prespecified K values using the same split and validation selection.It does not measure VLM latency or FLOPs because the router still computes its full probability distribution.Source: §A.6 · p. 19

A.7 EXPERT WEIGHT RANKING DETAILS

• The expert-weight ranking uses mean standardized coefficients across three seeds; positive weights raise the predicted absence score and negative weights lower it.The ranking is within the joint linear probe, not evidence of standalone expert accuracy or causal importance.Source: §A.7 · p. 19