Looks the Same, Answers Differently: Flip-Direction Steering for Robust Vision-Language Reasoning

1 INTRODUCTION

• Subtle visual variations can change final predictions even in frontier models, while early token changes may remain locally plausible and make intervention timing difficult.Source: §1 · pp. 1–2
• Across 18 settings, FLIPDIR improves the recovery-preservation harmonic mean by 2.8 points on average and up to 6.3 points over the strongest baseline in each setting.Source: §1 · p. 2
• FLIPDIR estimates a low-rank flip-inducing activation subspace from contrastive original–flipped inputs and uses a margin-based gate to steer hidden states at uncertain decoding steps.Source: §1 · pp. 2–3
• The paper analyzes answer flips through associated activation shifts and decoding divergence.Source: §1 · p. 3
• VISFLIP groups variants by whether they change a model’s original prediction, separately evaluating recovery and preservation across scientific reasoning, robotics, and medical VQA.Source: §1 · pp. 2–3

2 RELATED WORK

• Prior corruption benchmarks measure accuracy or caption consistency, but accuracy can reward correcting an original error and fixed-set consistency can reflect barely impactful variations [Zhang et al., 2024a].Source: §2 · p. 3
• Safety steering targets identifiable behaviors: refusal can follow a single activation direction [Arditi et al., 2024], while multimodal defenses suppress responses linked to adversarial visual features [Wang et al., 2025a].Source: §2 · p. 3
• Hallucination mitigation includes decoding methods based on visual contrast, token dependencies, or uncertainty [Leng et al., 2024; Huang et al., 2024; Xu et al., 2026], and activation steering toward grounded behavior [Liu et al., 2025].Source: §2 · p. 4

3.1 PROBLEM SETUP

• The task is to recover the original answer when a mild visual perturbation changes it while preserving answers on unaffected inputs.Calibration is model-, dataset-, and perturbation-family-specific; the intervention is reused on new inputs without a clean reference or advance flip-status knowledge.Source: §3.1 · p. 4

3.2 ESTIMATING THE FLIP-INDUCING SUBSPACE

• FLIPDIR estimates a fixed low-rank subspace from paired activation differences between original inputs and answer-flipping variants, using final prompt-token states after multimodal prefill.It uses uncentered SVD, retaining k singular directions; the estimated subspace may also include other perturbation-related variation.Source: §3.2 · pp. 4–5
• The leading direction captures 15.6% to 21.2% of activation-difference energy versus 0.6% for a Gaussian null in Qwen3-VL-8B/Robo2VLM lens-smudge pairs.The result uses 255 pairs with d = 4096; the observed concentration suggests a compact subspace can capture substantial activation change, though retaining more directions need not improve steering on new inputs.Source: §3.2 · p. 5

3.3 MARGIN-GATED STEERING

• At decoding steps, FLIPDIR attenuates the hidden-state projection onto the estimated subspace and leaves the orthogonal component unchanged.A stateful gate opens below a lower margin threshold, closes above an upper threshold, and otherwise retains its prior state; prompt prefill is unchanged.Source: §3.3 · pp. 5–6
• Divergence tokens have lower median margins in every one of 15 Gemma and Qwen settings; pooled, 93.0% of divergence tokens versus 11.7% of non-divergence tokens have margin below 0.3.The association motivates using low margin to trigger steering; the gate does not require the original input at inference.Source: §3.3 · p. 5, Figure 5 · p. 6

4 VISFLIP

• VISFLIP groups variants by whether they change a chosen model’s original prediction, separately measuring recovery of flipped predictions and preservation of stable ones.The framework is instantiated across 18 settings spanning scientific reasoning, robot-scene understanding, and medical VQA.Source: §4 · p. 6
• The science-reasoning settings use M3CoT physics questions with exposure shift, white-balance shift, and double JPEG compression.These variations cover illumination, camera color processing, and image recompression.Source: §4 · p. 6
• The robot-scene settings use Robo2VLM with exposure shift, lens contamination, and motion blur.The benchmark covers spatial relations, robot state, and action outcomes in manipulation scenes.Source: §4 · p. 6
• The medical VQA settings use GMAI-MMBench with window-level variation, modality-specific defocus blur, and small detector rotation.The dataset spans clinical tasks, perceptual granularities, and imaging modalities; its splits are stratified by task and granularity while preserving modality proportions.Source: §4 · p. 6
• Table 2 summarizes parent-image pools across the three domains, including parent-image counts by split and representative tasks or modalities.The framework supports reproducible variant generation, model-specific flip labeling, and recovery–preservation evaluation.Source: Table 2 · p. 6, §4 · p. 6

5.1 EXPERIMENTAL SETUP

• Evaluation covers two open-weight VLMs, Base and three inference-time baselines, with clean and non-flip preservation and flip recovery combined by a harmonic mean.The test split is reserved for final evaluation; subspaces are estimated from training data and layer and strength are selected on validation data.Source: §5.1 · pp. 7–8

5.2 MAIN RESULTS

• FLIPDIR achieves the highest H(C, N, F) in all 18 settings, exceeding the strongest baseline by 2.8 points on average.The gains reflect a better balance between recovering original predictions and preserving stable ones, rather than higher task accuracy.Source: §5.2 · p. 8

5.3 TRANSFER ACROSS DATASETS AND PERTURBATION COMBINATIONS

• Transferring only U across twelve off-diagonal settings reduces H by 0.1–6.8 points versus target-matched subspaces, while retaining target-specific layers and strengths.The largest decrease is 71.5 versus 78.3 for Robo2VLM lens smudge → M3CoT exposure on Qwen; this tests transfer with target-specific calibration.Source: §5.3 · p. 8, Figure 7 · p. 9

5.4 ABLATION STUDIES

• Flip-conditioned representation differences achieve the highest H in both ablation evaluations, despite similar mean shift norms for flip-inducing and non-flipping perturbations.The norms are 15.2 versus 15.6; results suggest answer flips are associated with shift direction rather than magnitude.Source: §5.4 · pp. 8–9, Table 5 · p. 9
• Margin-gated steering improves H in all six comparisons, while partial attenuation at α = 0.5 beats both α = 0.1 and α = 1 in one setting.Across five rank-sensitivity settings, increasing k beyond 8 yields no consistent gain, though the best rank varies by setting.Source: §5.4 · p. 9, Table 6 · p. 9, Figure 9 · p. 9
• With lower-layer activations reused, estimated decoding FLOPs increase by 8% on average at matched sequence lengths.Steering requires an additional forward pass only when the margin gate opens.Source: §5.4 · p. 9, Table 7 · p. 9

6 CONCLUSION

• FLIPDIR selectively attenuates a learned flip subspace during decoding without weight updates or a clean reference, and VISFLIP separates recovery from preservation across three domains.The conclusion covers scientific reasoning, robot-scene understanding, and medical VQA.Source: §6 · pp. 9–10
• Across 18 settings, FLIPDIR improves the harmonic mean of preservation and recovery by 2.8 points on average over the strongest baseline.Source: §6 · p. 10
• A single subspace that generalizes across perturbation types and supports steering throughout decoding without disrupting stable predictions remains an open goal.The authors identify setting-specific calibration and margin gating as aspects that such a shared subspace might remove.Source: §6 · p. 10

AI USE STATEMENT

• The authors report using AI tools for writing, ideation, research, and code implementation, with authors guiding and reviewing the work and taking responsibility for the manuscript and artifacts.Source: AI USE STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

• The planned release includes the complete VISFLIP dataset, construction and experimental code, evaluation scripts, configurations, seeds, provenance metadata, and extracted flip subspaces.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 BENCHMARK CONSTRUCTION

• Algorithm 1 builds VISFLIP for a specified dataset, model, and variation family by obtaining clean answers, sampling perturbations, and labeling child answers relative to each parent.Parents with unnormalizable clean answers are excluded; full-budget runs evaluate all draws, while pair-search runs can stop after finding both flip and non-flip children.Source: §A.1 · p. 14, Table 8 · p. 14

A.2 PERTURBATION DEFINITIONS AND RANGES

• Perturbations include exposure, white balance, double JPEG, lens smudge, and motion blur; reported ΔE is the mean per-pixel CIE76 difference in CIELAB space.The lens-smudge operator combines wipe smears, droplets, dust specks, blur, and haze; medical operators are specified separately.Source: §A.2 · pp. 14–15, Table 9 · p. 15

A.3 DATASET COMPOSITION AND TASK COVERAGE

• M3CoT covers eight physics topics, Robo2VLM eight robot-scene question types, and GMAI-MMBench clinical tasks, perceptual granularities, and imaging modalities.Disjoint parent pools support offline subspace estimation, validation selection, and final testing; validation pool size depends on settings.Source: §A.3 · p. 15

A.4 SPLIT CONSTRUCTION AND FILTERING

• M3CoT and Robo2VLM exclude parents whose clean answers cannot be normalized, while GMAI-MMBench uses fixed-seed stratified allocation across clinical-task and perceptual-granularity groups.GMAI-MMBench train/test task, granularity, and modality marginals differ by about two percentage points or less.Source: §A.4 · p. 15

A.5 MEDICAL DATA AND PREPROCESSING

• Medical preprocessing excludes MRI from the isotropic-blur protocol and uses modality-specific defocus kernels, window-level shifts, and small detector rotations.Images are downscaled when needed to model-specific maximum long sides before perturbations are applied.Source: §A.5 · p. 15

B.1 EVALUATION POOL SIZES

• Table 10 lists calibration-pair counts and the clean, non-flip, and flip denominators used for evaluation.Source: §B.1 · p. 16, Table 10 · p. 16

B.2 BUDGET-MATCHED CROSS-MODEL EVALUATION

• Under a common budget of 15 perturbation draws per parent, all four evaluated models exhibit answer flips; on flipped inputs, the two open-weight models achieve only about 30% accuracy.Flip rate decreases with clean accuracy in this comparison, but not with parameter count; instability also overlaps across models.Source: §B.2 · p. 16

• M3CoT flip rates were Qwen3-VL-8B/Gemma-3-12B: exposure 42.7%/61.9%, white balance 44.3%/60.6%, and JPEG 43.6%/66.4%.Source: §B.3 · p. 16
• Robo2VLM flip rates were Qwen3-VL-8B/Gemma-3-12B: exposure 19.1%/26.3%, lens smudge 15.4%/27.0%, and motion blur 16.8%/33.0%.Source: §B.3 · p. 16
• GMAI flip rates were Qwen3-VL-8B/Gemma-3-12B: window level 58.3%/51.0%, defocus blur 44.3%/38.5%, and detector rotation 57.3%/63.0%.Source: §B.3 · p. 16

C.1 FLIPDIR CONFIGURATION

• All settings use rank k = 8 and gate thresholds (τo, τc) = (0.3, 0.7), with layer and strength selected separately for each model and perturbation setting.Table 12 lists selected configurations and corresponding main-table results.Source: §C.1 · p. 16, Table 12 · p. 17

C.2 STEERING IMPLEMENTATION

• The estimated subspace is applied at the selected decoding layer using two matrix-vector products, costing O(dk) arithmetic without forming a d×d projection.The unedited distribution updates the margin gate; only an open gate triggers a second, steered forward.Source: §C.2 · p. 17

C.3 HYPERPARAMETER SELECTION

• Six layer–strength candidates are evaluated on validation triplets, with validation parents disjoint from calibration and test parents.The cited setting has 47 available Qwen3-VL defocus-blur validation flip pairs and approximately 24 selection triplets.Source: §C.3 · p. 17

C.4 BASELINE ADAPTATIONS / C.5 DETERMINISTIC INFERENCE AND RELEASED ARTIFACTS

• Baseline adaptations address model-specific embedding and token conventions, while open-weight runs use deterministic decoding and saved flip subspaces for exact reproduction.One complete setting reproduced all 1,040 generations; GPU re-extraction can introduce numerical differences. The planned release includes benchmark instances, perturbation and evaluation tools, and extracted subspaces.Source: §C.4 · p. 17, §C.5 · p. 17

D.1 ACTIVATION ENERGY CONCENTRATION

• The top 8 activation-difference directions capture 48.6–53.7% of energy, compared with 4.7% for the same-size Gaussian null.At rank 16, the corresponding values are 59.4–65.7% and 9.2%; Figure 4 and Figure 10a plot cumulative energy.Source: §D.1 · p. 18

D.2 TOKEN MARGINS AT DECODING DIVERGENCE

• For Gemma-3-12B, divergence and non-divergence token margins are analyzed from stored unedited traces, complementing the Qwen analysis.The two runs share a decoding prefix at each step; after a mismatch, they are realigned with the same token, and each distribution is normalized separately.Source: §D.2 · p. 18, Figure 10 · p. 18

D.3 LIMITS OF STATIC DIAGNOSTICS

• Subspace alignment, steering coordinates, and clean-run margins are weak standalone predictors of recovery or flips, so FLIPDIR uses the subspace for intervention rather than flip detection.The diagnostics do not establish the benefit of dynamic gating; that benefit is evaluated by a controlled gate ablation.Source: §D.3 · p. 18

E.1 RECOVERY–PRESERVATION TRADE-OFFS

• FLIPDIR is Pareto-nondominated in all 18 settings for preservation P = (C + N)/2 and flip recovery F, though other methods can also lie on the frontier.These comparisons show trade-offs, not dominance at every preservation level.Source: §E.1 · pp. 18–19

E.2 TRANSFER ACROSS DATASETS AND PERTURBATIONS

• Across the cross-domain source-to-target combinations shown in Figure 7, transferred subspaces retain at least 91% of target-matched H.Table 13 expands the transfer results to include all measured combinations, including within-domain sources.Source: §E.2 · p. 20, Table 13 · p. 21

F ABLATIONS AND SENSITIVITY

• The appendix ablations and sensitivity analyses supplement the rank, gate, and strength analyses with tabulated rank results, full gate comparisons, and calibration-pair sensitivity.The cited overview points to Tables 14–16.Source: F ABLATIONS AND SENSITIVITY · p. 21

F.1 FLIP-SUBSPACE RANK

• The rank sweep covers six settings, with other components fixed at the selected configuration; rank k = 8 is marked in the table.The settings cover two non-medical perturbation families and one medical perturbation family per model; row maxima are bolded.Source: §F.1 · p. 21, Table 14 · p. 21

F.2 MARGIN-GATE ABLATION

• The margin gate improves H in all six tested settings compared with applying steering at every decoding step.Table 15 reports C/N/F/H with and without the gate; Figure 13 visualizes the changes in percentage points for each metric.Source: §F.2 · p. 22, Table 15 · p. 22, Figure 13 · p. 22

F.3 CALIBRATION SAMPLE SIZE

• The calibration sample-size analysis varies the number of pairs used to estimate U while keeping other components fixed.Each subset uses the first N parents in sorted identifier order, with at most one flip pair per parent; the basis uses uncentered SVD.Source: §F.3 · p. 22, Table 16 · p. 22

G.1 COMPUTE ACCOUNTING

• Conditional FLIPDIR has an estimated mean decoding cost of 1.08× standard decoding and uses about 41–50% fewer decoding FLOPs than VCD.The mean uses 15 settings with available traces; replayed gate-open fractions are proxies for conditional execution, not measurements from a deployed conditional implementation. The FLOP comparison concerns decoding arithmetic, not end-to-end latency.Source: §G.1 · p. 23

G.2 GATE ACTIVATION STATISTICS

• Gate activation statistics replay thresholds (0.3, 0.7) with hysteresis on stored unedited traces, including clean, flip, and non-flip generations.The replay uses twelve parents per setting across shards; unavailable traces are marked N/A.Source: §G.2 · p. 24, Table 17 · p. 24

G.3 RUNTIME

• Across six medical settings, VCD runtime is 1.8–2.8 times FLIPDIR runtime, with higher H for FLIPDIR in each setting.Measurements use a shared serving regime with four to five concurrent processes per GPU and do not isolate per-query latency.Source: §G.3 · p. 24

H.1 ANSWER-FLIP RECOVERY

• A Gemma-3-12B Robo2VLM exposure-shift example shows FLIPDIR advancing reasoning to the next phase and returning the answer to the clean answer and ground truth.The figure marks where the unedited and steered trajectories diverge; the example uses layer 12 and strength 0.5.Source: Figure 14 · p. 25, §H.1 · p. 25

H.2 PERTURBATION INTENSITY EXAMPLES

• The examples show fixed-band exposure and white-balance shifts, double-JPEG draws, robot-scene smudges, and medical image variations.The medical examples include window-level shifts, modality-gated defocus blur, and detector rotations.Source: §H.2 · p. 26, Figure 15 · p. 26, Figure 16 · p. 26