FearCaut-Qwen: Affective Steering in a Vision-Language Model Shifts the Decision Criterion for Hazard Assessment

1. Introduction

• VLMs demand substantial evidence before declaring a hazard, so high accuracy can coexist with low hazard recall.In the SeisMLLM reconstruction, only 27.0% of genuinely unsafe buildings were flagged, with no false Red predictions.Source: §1 · p. 2
• Signal detection theory separates a shift in the safety decision criterion from a change in how well the VLM distinguishes damaged from undamaged structures.The paper uses this decomposition to assess whether a change in recall reflects the decision mechanism that produced it.Source: §1 · p. 3
• The study asks whether manipulating an emotion-relevant circuit can change decisions on an engineering hazard task, and whether that reflects criterion movement or improved discrimination.Fear is the proposed affective direction, motivated by findings that it can produce pessimistic risk estimates and risk-averse choice.Source: §1 · pp. 2–3
• FearCaut-Qwen localizes an affective direction in Qwen2.5-VL-7B using emotion-rich natural scenes, then tests steering on a two-stage SeisMLLM-1K building assessment pipeline.The direction is causally validated on held-out emotion data using sparse-neuron knockout and distributed steering before transfer to the building task.Source: §1 · p. 3

2.1 The influences of affect on safety decision making

• Fear has been associated with increased risk estimation and decreased risky decision-making, but a high-powered experiment found no significant shift in risk preferences from incidental emotion.This mixed evidence motivates empirical validation of emotion-based manipulation in a model.Source: §2.1 · p. 4
• Human research distinguishes changes in evidence processing from changes in decision caution, paralleling this paper’s separation of discrimination and criterion.Other work describes fear-related changes in decision tendency without changes in stimulus discriminability.Source: §2.1 · p. 4

2.2 Mechanistic interpretability and activation steering in vision–language models

• Probing can locate where a variable is decodable, but causal methods such as knockout and activation steering test whether a component matters or can change behavior.A probe may read information that the model does not use; knockout tests necessity, while adding a direction tests sufficiency.Source: §2.2 · p. 5
• Prior work reports localizable and manipulable affective representations, while leaving open whether they can change a decision criterion in a subsequent engineering safety task.This transfer question motivates examining both sparse components and a distributed residual-stream direction.Source: §2.2 · pp. 5–6

3.1 Overview

• The method treats a circuit as causally implicated components spanning a layer band, sparse SwiGLU coordinates, and a distributed residual-stream direction.Knockout and steering test source-task effects before the identified fear/threat circuit is tested for bidirectional changes in building decisions.Source: §3.1 · p. 6

3.2 Building damage assessment dataset

• SeisMLLM-1K is a case-level, multi-view benchmark for post-earthquake building safety evaluation, and this analysis retains its published 65-case test split alongside a 1,241-case training set.The public release contains 1,306 building cases and 3,421 images.Source: §3.2 · pp. 7–8

3.3 Signal detection formalization in building damage assessment

• Signal detection theory treats Red/Unsafe as signal and Green or Yellow as noise, separating evidence discrimination, d′, from the response criterion, c.A lower c denotes greater readiness to issue Red, whereas a higher c is more conservative; the analysis also uses a symmetric rate correction to avoid biasing c.Source: §3.3 · pp. 8–9
• The paper interprets criterion shifts and discrimination changes as distinct mechanisms because classification metrics alone can improve when a model simply becomes readier to flag Red.Its stated classification rule distinguishes a criterion shift without discrimination change from cases where discrimination also changes.Source: §3.3 · pp. 9–10

3.4 VLM model and the two-stage assessment pipeline

• No parameters are fine-tuned; FearCaut-Qwen intervenes only on forward-pass activations, with primary runs using deterministic greedy decoding.The backbone is Qwen/Qwen2.5-VL-7B-Instruct.Source: §3.4 · p. 10
• Stage 1 processes all case images into a structured perceptual profile, and Stage 2 uses that profile to determine a safety posting.The model scores the three complete posting strings by summed token log-probability and selects the highest normalized candidate probability.Source: §3.4 · p. 11
• Stage 2 retrieves the three highest-scoring train-fit cases using lexical, semantic, and categorical similarity, alongside ATC-20-1-grounded rules.The query case is excluded from retrieval, and no test-case ground truth enters the Stage 2 context.Source: §3.4 · p. 12
• The deterministic reference rule agrees with published labels on 81.5% of test cases, but is used for retrieval and circuit analysis rather than reported as the model prediction.Its agreement is 86.9% on train-fit cases and 85.1% on validation cases.Source: §3.4 · p. 12

3.5 Forced candidate scoring and the decision rule

• The primary decision rule scores the three complete posting strings by summed conditional log-probability, normalizes them, and selects the highest-probability posting.Forced candidate scoring avoids relying on free-text parsing when steering may change phrasing or produce an unparseable response.Source: §3.5 · pp. 12–13

3.6 Source emotion dataset and decontamination

• FindingEmo whole-scene images were mapped to happiness, sadness, fear/threat, and neutral classes, with neutral serving as the reference contrast.The source data were chosen to evoke affect without sharing structural-damage content with the target task; three independent screens were used for decontamination.Source: §3.6 · pp. 13–15
• Of 2,457 downloaded images, 20 duplicates and 671 semantic-screen failures were removed, leaving 1,766; none matched a target image.Exclusion was class-dependent: 45% for fear/threat and 41% for sadness, versus 7% for happiness and 17% for neutral, reflecting disaster imagery in negatively valenced web scenes.Source: §3.6 · p. 15

3.6.1 Realised valence–arousal separation of the source classes

• Sadness was more aroused than fear/threat (3.75 vs. 3.41) and more negative in valence (−2.16 vs. −1.82), contrary to the intended separation.Fear/threat was slightly less aroused (Welch t = −2.74, p = 0.006, d = −0.20) and less negative in valence (t = 5.95, d = 0.44); the planned dissociation was not realized.Source: §3.6.1 · pp. 15–16
• The retained images were divided into three pairwise-disjoint, class-balanced splits for discovery, selection validation, and held-out emotion testing.The splits contained 844, 280, and 280 images, respectively; neuron rankings and steering directions were computed only on discovery data, with choices made on validation data.Source: §3.6.1 · p. 16

3.7 Activation capture: neuron definition, token policy and streaming statistics

• A sparse neuron is one coordinate of the post-gating SwiGLU intermediate representation.The captured vector is the one read by the down-projection.Source: §3.7 · p. 16
• The primary capture policy pools the last prompt token, which precedes any emitted class token and is identical in form across source stimuli.Full activation tensors were not retained; per-neuron sufficient statistics were streamed, including mean absolute activation, top-activation frequency, and class-conditioned response histograms.Source: §3.7 · pp. 16–17
• Balanced accuracy was 0.661 at layer 0 under the first-answer-token policy but chance-level 0.250 under the last-prompt-token policy.The first-answer token encodes the answer the model is about to give, creating a spurious appearance of early-layer affect coding; primary analyses therefore use the last prompt token.Source: §3.7 · pp. 16–17

3.8 Layer-wise affect probing and selection of the candidate band

• Class-balanced multinomial logistic probes were fit to standardized residual vectors, with discovery-split transforms applied unchanged to other splits.Regularization was selected on the validation split by balanced accuracy, and held-out performance was evaluated with bootstrap confidence intervals.Source: §3.8 · p. 17
• The longest contiguous run among the eight highest validation-scoring layers selected layers 22–24 as the candidate band.This band bounds the later circuit comparison and the layers at which steering directions are defined.Source: §3.8 · p. 17
• A successful linear probe shows that affect information is available at a layer, but does not establish that the layer is necessary or that any coordinate is causal.The paper therefore proceeds from probing to intervention before transferring anything.Source: §3.8 · p. 17

3.9 Sparse neuron selectors and Borda consensus

• The sparse selector set used K = 256 neurons for fear/threat, K = 128 for happiness, and K = 256 for sadness.K was selected by validation balanced accuracy using the selected neurons as features, then frozen.Source: §3.9 · p. 19
• Selected neurons concentrate in upper decoder blocks and extend above the probe band.The four selectors combine contrast effect, mean deviation, activation frequency, and response-distribution entropy; Borda consensus combines their normalized rankings.Source: §3.9 · pp. 18–19

3.10 Distributed steering directions

• Distributed directions are class-mean differences in residual-stream states at layers 22–24, whereas sparse fear/threat neurons span layers 17–27.Only 51 of the 256 fear/threat neurons fall inside layers 22–24, so the sparse units and directions are not two parameterizations of one object.Source: §3.10 · p. 20
• Scaling steering by residual-stream norm would make interventions roughly seven times larger than the empirical contrast and produce degenerate generation.The paper instead expresses steering strength in units of one source class-mean difference.Source: §3.10 · p. 20

• The sparse operator scales selected pre-projection activation coordinates, while the distributed operator adds a scaled unit direction at a decoder block's output.The sparse scale distinguishes knockout, attenuation, and amplification; the distributed direction is installed as a block-output hook.Source: §3.11 · p. 20
• Both operators are strict no-ops at their baseline settings, and interventions can target prompt processing, generation, or both.Primary conditions apply interventions during both phases; the no-op hooks return the original tensor object, preserving bit-identical logits.Source: §3.11 · p. 21
• Matched sparse-neuron controls vary neuron count, layer distribution, activation magnitude, or whether labels were shuffled during circuit selection.The shuffled-label control reruns the selection pipeline, testing the procedure rather than only its selected output.Source: §3.11 · p. 21
• The activation-magnitude control preserves layer distribution and perturbed activation energy, while distributed controls use random directions with matched perturbation magnitude.Twenty independent draws are used per control family in the specificity audit.Source: §3.11 · pp. 21–22

3.12 Source-domain causal validation and operating-point selection

• The source-domain specificity gate requires TSI>1, a target-recall drop of at least 0.05, and an effect larger than the best matched control.The target-specificity index compares target-class recall drop with mean absolute recall change across the other three classes.Source: §3.12 · p. 22
• Fear/threat passed the source usability rule at α=±0.5, happiness at α=+0.5, and sadness at no tested distributed strength.The non-degeneracy rule limits accuracy loss and target-class prediction frequency to reject saturation.Source: §3.12 · p. 22

3.13 Target intervention, stage resolution and circuit analysis

• The target design applies frozen fear/threat directions to building cases and compares perception-stage with reasoning-stage intervention while holding baseline context fixed for reasoning.It also compares positive and negative steering, random directions, neuron amplification or knockout, and layer-matched random neurons.Source: §3.13 · p. 23

3.14 Statistical analysis

• Eight independently generated baselines produced identical predictions on all 65 cases, confirming determinism.Target comparisons use paired case-level bootstrap intervals, with earthquake-event cluster bootstrap as a robustness analysis.Source: §3.13 · p. 23, §3.14 · p. 23

4.1 Evidence of a badly placed criterion

• Red recall was 0.270 despite Red precision 1.000, with d' = 1.521 and c = +1.354, indicating usable discrimination alongside a conservative hazard criterion.Sweeping the threshold alone raised Red recall from 0.27 to 1.00 while macro-F1 peaked at 0.453 for 32 Red postings, showing that reported performance changes with policy despite unchanged evidence.Source: §4.1 · pp. 24–25
• Oracle Stage-1 raised accuracy to 0.692 and d' to 2.416 while c remained +0.907; at 38 Red postings, steering achieved macro-F1 0.556 versus 0.376 for thresholding.At that count, d' was 1.027 under steering versus 0.529 under thresholding, showing that improved perception alone did not relocate the criterion.Source: §4.1 · pp. 24–25

4.2 Affect is linearly decodable in late layers and localizable to sparse neurons

• Held-out emotion decoding reached balanced accuracy 0.714 at layer 22, where valence peaked at R² = 0.729; sparse selection identified 256 fear/threat neurons across late layers.Valence was more linearly accessible than arousal, which peaked at R² = 0.297 at layer 26.Source: §4.2 · p. 26

4.3 The source circuit is causally necessary and behaviourally sufficient

• Fear-circuit knockout reduced fear/threat recall by 0.129 while other-class mean recall changed −0.005; fear steering at α = +0.5 raised predicted fear fraction by 0.457 versus 0.086 for norm-matched random steering.On validation, fear predictions rose from 0.054 at baseline to 0.129 at α = +0.5, 0.179 at +1.0, and 0.604 at +2.0, where balanced accuracy collapsed.Source: §4.3 · pp. 27–28
• The identified affective circuit was causally manipulable through selective knockout, null controls, and usable bidirectional steering on held-out emotion data.Large fear strengths and positive sadness strengths were discarded as degenerate; the modest fear strength was retained as the operating point.Source: §4.3 · p. 29

4.4 The frozen circuit moves the building decision bidirectionally

• Distributed fear/threat steering raised Red recall from 0.270 to 0.757 (+0.486; 95% CI [+0.306, +0.667], p < 0.001); subtracting the same direction reduced recall to 0.000 (−0.270; 95% CI [−0.419, −0.133], p < 0.001).Source: §4.4 · p. 29

4.5 The effect is criterion movement, and discrimination is not improved

• Fear/threat steering shifted criterion c from +1.354 to −0.161 (Δc = −1.515), while discrimination d' fell from 1.521 to 1.028 (Δd' = −0.493).The reported criterion-to-discrimination ratio was 3.1, indicating that the behavioral change was dominated by threshold movement, not improved discrimination.Source: Figure 16 · p. 32, §4.5 · p. 32

4.6 Criterion control originates during perception, not during rule-based reasoning

• Perception-only steering increased Red recall by 0.459 (95% CI [+0.278, +0.634], p < 0.001), recovering 94.4% of the whole-pipeline effect; reasoning-only steering increased Red recall by 0.027.Perception-only changes were Δd' = −0.480 and Δc = −1.428, closely matching both-stage steering.Source: §4.6 · p. 33
• Severity ranks rose under both steering directions, but decisions moved oppositely; severity-change and decision-change correlation was only +0.308, consistent with an effect during evidence construction.Parse success was similar across conditions, and the intervention was not a simple monotonic change in perceived damage.Source: §4.6 · p. 34

4.7 The effect is threat-specific

• +0.486 Red recall and +0.174 macro-F1 for fear/threat steering contrasted with happiness’s +0.027 Red recall (p=0.84) and +0.006 macro-F1 (p=0.94) at α=+0.5.The happiness direction therefore did not reproduce the fear/threat effect at the same usable strength.Source: §4.7 · pp. 34–35, Figure 18 · p. 35
• No valid distributed sadness comparison was available, so fear/threat could not be fully separated from generic negative valence.Some happiness and sadness steering strengths were excluded as degeneration failures because 75% to 89% of Stage-1 outputs hit the token limit.Source: §4.7 · pp. 35–36
• The targeted fear-responsive-unit knockout effect was +0.129, beyond the pooled 80-draw null interval [−0.0143, +0.0429] (p=0.012).The four control families included same-K, layer-matched, activation-matched, and shuffled-label nulls.Source: §4.7 · p. 35, Figure 19 · p. 36

4.8 Sparse neurons and the distributed direction dissociate

• Amplifying or ablating the 256-neuron set changed Red recall by only +0.027, versus +0.486 for the distributed direction; these are not a common effect size.The sparse metric is a recall deficit, whereas the distributed metric is a prediction-tendency change.Source: §4.8 · p. 36
• Fear overlapped with unsafe-decision neurons by 1 neuron versus 1.15 expected (p=0.686), while the results point to a distributed perceptual shift propagating through unmodified reasoning.The fear-versus-damage-severity overlap was 0 versus 1.15 expected (p=1.000); the two structural sets overlapped by 4 neurons, 3.47-fold above chance (p=0.029).Source: §4.8 · pp. 36–37

5.1 Criterion control as the engineering contribution

• Baseline performance showed adequate discrimination (d' = 1.521) but a conservative criterion (c = +1.354); supplying ground-truth perception raised d' to 2.416 while c remained +0.907.The discussion argues that improving recognition alone would not fix the conservative operating policy.Source: §5.1 · p. 37
• Steering raised Red recall by 0.486 and macro-F1 by 0.174, while Δc = −1.515 versus Δd' = −0.493 (3.1 ratio), indicating criterion shift without improved evidence.The authors interpret the model as readier to act on evidence it already had.Source: §5.1 · p. 38
• At the same 38 Red postings, internal steering yielded macro-F1 0.556 and d' = 1.027, versus 0.376 and 0.529 for external thresholding.Thresholding alone moved macro-F1 from 0.21 to 0.45 and Red recall from 0.27 to 1.00 without changing evidence; the authors caution that the test split is Red-majority and the macro-F1 gain is partly mechanical.Source: §5.1 · p. 38

5.2 Mechanism: a distributed perceptual shift

• Reasoning-only steering on Stage-1 text recovered 5.6% of the effect versus 94.4% for perception-only steering; complete Stage-2 contexts were byte-identical in all 65 cases, locating vulnerability in image-to-profile transformation.Source: §5.2 · p. 39
• Positive and negative steering both raised mean hazard severity while moving postings in opposite directions; case-level severity and decision changes correlated only +0.308.The authors interpret this as a change to the character of the perceptual report as a whole, not a simple severity shift.Source: §5.2 · p. 39
• The 256 source-task neurons were selectively causal but transferred nothing; unsafe and severity circuits overlapped 3.5-fold above chance, supporting distributed steering over a neuron list.The residual-stream direction pools weakly aligned features, including features outside the selector’s top ranks.Source: §5.2 · p. 39

5.3 Limitations and future work

• Experiments cover one backbone, affect corpus, and building benchmark, so they establish controllability in this system, not generality; the authors call for replication across models, corpora, and hazards.They also propose applying the same framework to latent factors such as urgency, authority, optimism, fatigue, or demographic stereotype.Source: §5.3 · pp. 39–40
• The 65-case Red-heavy test split limits power: only one of 24 macro-F1 comparisons survived multiplicity correction, and the structural null used five draws at α = 1.0 rather than α = 0.5.Pairing and event-cluster bootstrap reduce but cannot eliminate uncertainty; the authors prioritize larger, event-disjoint benchmarks.Source: §5.3 · p. 40
• Current interventions constrain the mechanistic explanation, but set overlap is associational and none of the analyses traces the causal path.The authors identify activation patching as the natural next step.Source: §5.3 · p. 40
• Patching matched baseline–steered cases could test emotion-sensitive MLP activations, candidate attention-head outputs, and residual-stream states to localize causal transmission, following prior methods.The circuit analysis has identified matched case pairs whose postings differ between baseline and steered runs.Source: §5.3 · p. 40
• Head-level attribution, matched-layer comparison of sparse and distributed families, and sparse-dictionary decomposition could test whether the steering direction is one feature or a bundle.These analyses are proposed as further tests of the mechanistic account.Source: §5.3 · p. 40

6. Conclusion

• 27.0% of genuinely unsafe buildings receive an Unsafe posting, and the pipeline never issues a false Red.The paper attributes this conservative behavior to a decision criterion of c = +1.354 despite d′ = 1.521.Source: §6 · pp. 40–41
• Fear-direction injection raises Red recall from 0.270 to 0.757, while subtracting the direction reduces Red recall to zero.A norm-matched random direction and a matched happiness direction show no significant effect.Source: §6 · p. 41
• The intervention shifts the criterion by −1.515 while discrimination is not improved (Δd′ = −0.493).The conclusion identifies criterion movement, rather than improved discrimination, as the mechanism.Source: §6 · p. 41
• At the same operating point, thresholding output probabilities yields d′ = 0.529 against 1.027, costing more discrimination than internal steering.The paper argues that the internal route is not merely a re-description of output calibration.Source: §6 · p. 41
• Perception-only intervention reproduces 94.4% of the effect, whereas reasoning-only intervention over byte-identical evidence reproduces 5.6%.The conclusion places the mechanism in construction of the perceptual report rather than rule-based reasoning.Source: §6 · p. 41
• The authors argue that unsafe-decision tendency should be measured separately from discrimination and can be set at inference without retraining.They recommend reporting d′ and c alongside conventional metrics because accuracy can mistake a criterion shift for competence.Source: §6 · p. 41

• SeisMLLM-1K is publicly available from its original release, while FindingEmo distributes annotations and source URLs rather than image files.Source: Data availability · p. 41
• The study reports the annotation index, image identifier lists for all three splits, SHA-256 manifests, and decontamination decisions.Source: Data availability · p. 41
• Configuration files, run metadata, prediction records, frozen circuit records with content hashes, and analysis code are available from the authors on request.Source: Data availability · p. 41