PaperlayerView exact revision on arXiv

Source-linked AI summary

It Is Not Seeing the Hazard: A Frozen Vision-Language Safety Score Measures Its Caption Bank

Samuel Tetteh, Cody Fleming

arXiv:2610.09517v1cs.CVcs.LG

TL;DR

Policy outcomes cannot establish whether a vision-language safety score detects hazards or tracks correlated scene features. The paper audits a frozen CLIP score using passive trajectories and controlled caption, hazard-geometry, and viewpoint tests; the results support a scene-template explanation, not established hazard perception.

  • Problem

    Policy return and collision rate cannot show whether a safety score measures hazards or instead correlates with failure for another reason.

    Source: §1 · p. 1

  • Method

    The authors audit a frozen CLIP prompt-margin score on trajectories from policies that never observe it, matching hazard geometry and varying captions, encoder, and camera view.

    Source: §1 · p. 1, §3 · p. 4

  • Results

    73% of the pre-contact deviation remains in a score along the captions’ shared direction, supporting a scene-template explanation rather than established hazard perception.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    VLM safety scores should be validated as measurements rather than treated as semantic ground truth; caption geometry, viewpoint sensitivity, and constant-confidence baselines provide direct tests.

    Source: §7 · p. 8

    The evidence supports claims about tested configurations, not every VLM safety signal or trained policy; the closed-loop experiment covers one environment, algorithm, and placebo family.

    Source: §7 · p. 8

Abstract

from arXiv · show

Frozen vision-language models increasingly provide safety signals for reinforcement learning. Their use assumes that similarity to language describing danger indicates the hazard itself. Yet policy return and collision rate cannot reveal whether a score detects hazards or responds to correlated features of the scene. VLM-based methods have reported gains in driving and safe-RL benchmarks by converting image-text similarity into rewards, costs, or confidence weights. Such signals promise to reduce reliance on manually designed feedback. They may also reflect prompt structure, embedding geometry, or camera viewpoint, leaving their safety meaning unverified. To address this gap, we present a controlled evaluation of a frozen CLIP prompt-margin safety score. We apply the score to trajectories generated by policies that never receive it, match pre-contact observations to contact-free observations with comparable hazard geometry, and vary the captions, encoder, and camera view. Across three policies, 180 episodes, and 130 isolated contact onsets, the score decreases for about twenty steps before contact. Mechanism controls indicate that the score mainly tracks resemblance to the scene shared by its captions and changes with caption separation and camera view. A constant-confidence control retains the lower catastrophe-rate point estimate, so policy gains do not establish hazard perception.

1 INTRODUCTION

Policy outcomes alone cannot show whether a safety score detects hazards or merely correlates with failure. The paper audits a frozen CLIP score and finds evidence for a caption-shared scene-template signal whose informativeness varies with captions and viewpoint.

1 INTRODUCTION

  • The audit evaluates a frozen CLIP prompt-margin score on trajectories whose policies never observe it, separating score measurement from policy response.Across three VLM-free policies, 180 episodes, and 130 isolated contact onsets, the score shows a negative deviation lasting about twenty steps.Source: §1 · p. 1
  • −1.37 standard deviations for far hazards versus −0.34 for near hazards; collision-only captions showed no resolved pre-contact deviation.The near-hazard interval includes zero.Source: §1 · p. 2
  • The shared caption direction retains 73% of the pre-contact deviation, while better-separated captions allow the danger score to rise.In ViT-B/32, mean cross-group similarity equals mean within-safe similarity at 0.883; unrelated-scene captions reverse the deviation.Source: §1 · p. 2
  • The pre-contact signal varies by camera: it appears only in FormulaOne’s egocentric view, and no single view is consistently informative across environments.Changing the camera can remove the measured signal without changing the dynamics.Source: §1 · p. 2
  • The constant-confidence arm retains a lower catastrophe-rate point estimate than the no-VLM baseline, so policy gains do not establish perception.Confidence only scales a reward bonus in these arms and never enters the cost channel.Source: §1 · pp. 2–3
  • The paper contributes passive-observer controls, caption-geometry analysis, viewpoint tests, and a closed-loop placebo, and documents four evaluation-stack defects.The stack includes Safety-Gymnasium and MetaDrive.Source: §1 · p. 3

2 RELATED WORK

Prior work uses frozen vision-language similarity as a training signal and evaluates it through policy outcomes. This paper distinguishes that end-to-end evidence from what the score itself measures.

2 RELATED WORK

  • Frozen vision-language signals have been used for rewards and success detection, while policy return or collision rate cannot distinguish perception from other correlations with failure.The cited success-detection work uses in-domain fine-tuning, unlike the frozen setting discussed here.Source: §2 · p. 3
  • Caption-bank similarity measures resemblance to known concepts, and CLIP representation properties make it plausible that safety scores reflect scene resemblance rather than what is about to happen.The paper’s negation and off-topic controls show the first two cited representation properties in a safety score treated as a hazard detector.Source: §2 · p. 3
  • This paper examines proxy-measurement validity before optimization, whereas perception-error certificates require state estimates or error bounds that the audited score does not provide.The score’s domain of validity is unknown and depends on viewpoint in the tested environments.Source: §2 · p. 3
  • The study uses standard constrained-MDP solvers and benchmark stacks, reports intervals using seeds as units, and notes an effect below the design’s minimum detectable difference.The relevant shaping precedent is that a restricted-form shaping term cannot change the optimal policy.Source: §2 · p. 3

3 MEASUREMENT PROTOCOL

The protocol measures a frozen scorer on trajectories it cannot influence, then examines policy effects separately. It uses matched event comparisons and controlled rescoring and camera views.

3 MEASUREMENT PROTOCOL

  • The scorer is a passive observer of trajectories, separating what it measures from how a policy responds; policy effects are tested separately in closed loop.Source: §3 · p. 4
  • CLIP ViT-B/32 averages safe- and danger-caption similarities into a margin, then maps the margin to confidence κ; κ is not two-sided confidence without recentering.The margin is positive on 99% of egocentric FormulaOne steps, and fp16 reduces similarities to about 17 distinct values.Source: §3 · p. 4
  • FormulaOne uses ten trained policies across three constrained-RL algorithms and 60 deterministic episodes per policy; controls rescore stored frames or render identical states from multiple cameras.Generalisation experiments cover three additional environments, while the closed-loop experiment trains new policies.Source: §3 · p. 4
  • Contact onsets follow 20 contact-free steps, and each onset is compared with a same-episode contact-free placebo using episode-cluster bootstrap intervals.The study summarizes lags −15, −10, and −5 and additionally matches hazard distance, bearing, frontal distance, and frontal time-to-contact.Source: §3 · p. 4

4 WHAT CAN THE SCORE MEASURE?

The paper contrasts direct hazard detection with a scene-template explanation for changes in the prompt margin. Caption geometry, hazard matching, and viewpoint tests distinguish their predictions.

4 WHAT CAN THE SCORE MEASURE?

  • A prompt-margin interpretation as relative safety or danger evidence requires its changes to reflect hazard content, although both caption groups describe a car on a racetrack.A frame departing from that shared setting could affect both groups even without a visible hazard.Source: §4 · p. 4
  • Hazard detection predicts stronger responses near or ahead of hazards and in collision captions; a scene-template score predicts sensitivity to track layout, vehicle alignment, and view.Matching hazard geometry, changing captions, and rendering identical states from different views test these predictions.Source: §4 · p. 4
  • The safe and danger groups can diverge only along the difference between their text embeddings, so poor separation can make both similarities decrease together.A danger-score increase should follow caption geometry, not scorer class alone.Source: §4 · p. 4
  • An epoch-averaged scalar-dual cost term whose occupancy-level mean is affine in true cost can be absorbed into a retuned budget without adding statewise information.This conditional result does not imply that the visual score is uninformative.Source: §4 · p. 4

5 RESULTS

Across passive-observer evaluations, the pre-contact score deviation survives hazard-geometry matching but varies with camera view and caption geometry. In closed-loop evaluation, constant confidence retains a lower catastrophe-rate point estimate, so policy improvement does not establish hazard perception.

5.1 THE PRE-CONTACT SIGNAL SURVIVES GEOMETRIC MATCHING

  • Across five matching specifications, pre-contact deviation remained −0.131 to −0.153 versus −0.160 before matching, retaining 98%–100% of observations.The deviation began about 20 steps before contact; κ varied non-monotonically with hazard distance, motivating nonparametric matching.Source: §5.1 · p. 5

5.2 THE SIGNAL DEPENDS ON VIEWPOINT

  • Across ten policies, the egocentric deviation was −0.447 [−0.706, −0.204]; the estimate decreased from −0.800 with three policies, and CPO’s within-algorithm result was unresolved.Neither score variability nor camera distance explained the pattern, but apparent hazard size was not measured. ViT-L/14 retained the egocentric advantage while reversing the near-overhead result.Source: §5.2 · p. 5

5.3 NO CAMERA VIEW IS CONSISTENTLY INFORMATIVE

  • The pre-contact deviation appeared only in FormulaOne’s egocentric view, was unresolved in MetaDrive’s top-down view, and reversed across views in other environments.PointGoal1’s paired overhead-to-egocentric contrast was +0.579 [+0.211, +0.957]; changing hazard appearance did not support the proposed visibility explanation under the prespecified criterion.Source: §5.3 · pp. 5–6, Table 1 · p. 6

5.4 CAPTION GEOMETRY GOVERNS THE DANGER-SCORE INCREASE

  • Negating the safe captions preserved the deviation at −0.507 [−0.844, −0.170], while unrelated-scene captions reversed it to +0.441 [+0.031, +0.857].The deployed grouping ranked third among 70 balanced partitions (p = 0.043), and a score using the captions’ shared direction retained 73% of the deployed margin. Caption separation explained danger-score increases more consistently than scorer class; whether the ViT-L/14 increase reflects hazard perception remains untested.Source: §5.4 · pp. 6–7

5.5 THE DEVIATION IS STRONGEST WHEN HAZARDS ARE DISTANT

  • Deviation was −1.369 [−1.785, −0.958] for the farthest-distance tercile versus −0.342 [−1.065, +0.334] for the closest, but strata differences were not tested.The far-hazard estimate included 44 episodes, compared with only eight for no hazard ahead; the estimates should not be interpreted as formal contrasts.Source: §5.5 · p. 7

5.6 CLOSED-LOOP GAINS PERSIST WITH CONSTANT CONFIDENCE

  • Constant confidence reduced catastrophe rate by 0.106 vs no-VLM (p = 0.036), below the 0.133 minimum detectable effect; deployed and constant differed by 0.016 (p = 0.71).The constant and deployed arms had rates of 0.176 and 0.192, respectively; their difference was not detected, which is not evidence of equivalence. No secondary comparison survived Benjamini–Hochberg correction.Source: §5.6 · p. 7

6 FOUR DEFECTS IN THE EVALUATION PIPELINE

The evaluation pipeline contains four defects spanning scenario handling, traffic, hardware, and logging. These can invalidate intended comparisons or make logged return fail to measure task performance.

6 FOUR DEFECTS IN THE EVALUATION PIPELINE

  • Scenario indices alias held-out scenarios to training scenarios, so absolute catastrophe rates are not held-out estimates.The defect was held constant across closed-loop arms to preserve their relative comparison.Source: §6 · p. 8
  • Traffic is re-randomized at every reset, making paired evaluation under this configuration require fixed traffic.For one policy and scenario, two rollouts differed at all 1,000 steps; disabling random traffic made them bit-identical.Source: §6 · p. 8
  • Deterministic rollouts depend on the hardware group, so replay validation is restricted to verified hardware.Five replays reproduced exactly on the reference hardware group, while none of seven replays on other tested groups did.Source: §6 · p. 8
  • Logged return includes the shaping bonus, which contributed 99.1%–99.8% of logged return in nine FormulaOne-L2 runs without confidence gating.Recomputed environment returns overlapped no-VLM baselines, so the pipeline reports environment-only return separately.Source: §6 · p. 8

7 LIMITATIONS AND CONCLUSION

For the tested CLIP ViT-B/32 construction, the evidence supports a scene-template explanation, but the results are bounded by the tested models, environments, and policies. The authors recommend validating safety scores as measurements rather than treating them as semantic ground truth.

7 LIMITATIONS AND CONCLUSION

  • The empirical scope covers specific configurations, not all VLM safety signals or trained policies; the closed-loop experiment uses one environment, algorithm, and placebo family.The egocentric-only pattern is specific to FormulaOne; PointGoal1 shows a view difference, CarButton1 does not, and no view is consistently informative across environments.Source: §7 · p. 8
  • The tested CLIP ViT-B/32 construction supports a scene-template explanation: the pre-contact deviation is stronger with distant hazards, and a shared-caption-direction score retains 73% of its magnitude.The deviation is not detected in captions naming collisions without track descriptions; the informative camera view also varies across environments.Source: §7 · p. 8
  • Safety scores should be validated as measurements before policy gains are attributed to hazard perception.The authors identify caption geometry, viewpoint sensitivity, passive-observer controls, and constant-confidence baselines as direct tests.Source: §7 · p. 8

A FORMAL STATEMENTS

The appendix states formal properties of the prompt margin, rendering information, and an affine occupancy-level cost term. It also describes the scorer inputs and notes that the cost result depends on an untested assumption.

A FORMAL STATEMENTS

  • For balanced caption partitions, the prompt margin is built from a frozen image encoder and normalized caption embeddings, and it is unchanged by translating all caption embeddings equally.The construction therefore discards the captions’ shared direction.Source: A FORMAL STATEMENTS · p. 10
  • A score using only the captions’ shared direction recovers 73% of the margin’s pre-contact effect on nine FormulaOne passive-observer policies in the egocentric view.The pre-contact association is therefore not localized in the prompt-margin contrast.Source: A FORMAL STATEMENTS · p. 10
  • If one deterministic rendering is a measurable function of another, the latter contains at least as much information about contact within k steps.The paper does not establish that any tested rendering is recoverable from another.Source: A FORMAL STATEMENTS · p. 10
  • Under the stated affine occupancy-level assumption, adding the cost term changes the feasible set only through an effective budget, not through added statewise information.For α = 0, the budget shifts by −bβ; this result concerns expected occupancy measures, not identical optimization dynamics.Source: A FORMAL STATEMENTS · p. 11
  • The affine cost result requires the relation to hold for every feasible occupancy measure, a condition the experiments neither test nor establish.It is a conditional failure mode for methods that average a signal before a scalar dual update, not evidence that the measured scores satisfy the condition.Source: A FORMAL STATEMENTS · p. 11

B PROOFS

  • The proofs derive the margin property by cancellation under balanced partitions and the rendering result from the data processing inequality.The cost result follows by rewriting the shaped expected cost as a rescaled original cost plus an offset.Source: B PROOFS · p. 11

C THE SCORERS AND THEIR INPUTS

  • The scorer evaluation covers two frozen CLIP checkpoints and Qwen2-VL-7B across five scorer-and-prompt combinations.This section specifies how each combination processes frames and captions.Source: C THE SCORERS AND THEIR INPUTS · p. 11

C.1 WHAT CLIP RECEIVES

CLIP scores the same rendered frame through image–text similarity, with prompt wording and scorer configuration specified for the FormulaOne tests. Supplementary checks report stable matching estimates, caption-partition and hazard-geometry results, and closed-loop comparisons.

C.1 WHAT CLIP RECEIVES

  • CLIP uses image–text cosine similarity on the 256 × 256 rendered frame, whereas Qwen2-VL uses yes/no next-token probabilities.CLIP resizes the frame to 224 × 224; the two CLIP backbones use different patch grids, which the paper does not test as a cause.Source: §C.1 · pp. 11–12, Figure 3 · p. 12, Figure 4 · p. 12

D FORMULAONE PROMPTS AND SCORING RULES

  • FormulaOne prompt comparisons vary caption sets and scorer configurations, including the deployed eight-caption bank, two fixed group labels, and negated, absurd, and off-topic rewrites.The bank arm retains mildly ungrammatical questions to avoid changing grammar alongside the prompt set.Source: D FORMULAONE PROMPTS AND SCORING RULES · pp. 13–14

E.1 NONPARAMETRIC MATCHING SPECIFICATIONS

  • Five nonparametric matching specifications yield deviations from −0.131 to −0.153, versus −0.160 unmatched, with 98–100% of pre-contact points in usable cells.The estimates are not explained by sample attrition.Source: §E.1 · p. 15, Table 4 · p. 15

E.2 EXHAUSTIVE CAPTION PARTITIONS

  • The deployed caption split ranks third among 70 balanced partitions, while scoring all eight captions without a split retains 73% of its effect.The partition distribution is centred below zero but includes both signs.Source: §E.2 · p. 15, Table 5 · p. 15, Figure 5 · p. 16

E.3 MODERATION BY HAZARD GEOMETRY

  • The pre-contact deviation is descriptively larger when the nearest hazard is far and when nothing lies ahead, opposite the ordering predicted by a hazard detector.The table reports strata from three passive-observer policies with logged hazard geometry.Source: §E.3 · p. 15, Table 6 · p. 16

E.5 CLOSED-LOOP ARMS IN FULL

  • All three VLM-shaped closed-loop arms have lower point estimates than the VLM-free baseline on both safety measures, with no detected difference between constant and frame-dependent confidence.The table notes that κ scales only the reward bonus and that scenario aliasing makes absolute rates optimistic, while between-arm differences stand.Source: §E.5 · p. 15, Table 6 · p. 16

E.4 EXHAUSTIVE PARTITION DISTRIBUTION

  • The partition distribution has a mean of −0.113 and spans −0.620 to +0.510, so a sizeable minority of caption splits reverse sign.The deployed split ranks third of 70; removing the contrast retains 73% of its effect.Source: §E.2 · p. 15, Figure 5 · p. 16

E.6 THE LANGUAGE PATHWAY IN FULL

Caption and scorer controls show that the signal depends on language geometry, while its pre-contact pattern varies across hazards, camera views, and caption wording. Cross-environment tests do not support a general rule that a particular camera view is informative.

E.6 THE LANGUAGE PATHWAY IN FULL

  • The deployed caption split ranks third among 70 partitions, and the all-caption score retains 73% of its effect; track-related captions carry the fall, while collision-only captions show no resolved deviation.Under ViT-B/32, both group means fall before contact; the danger-group rise is not general across encoders and banks.Source: §E.6 · p. 16

E.7 MODERATION BY HAZARD GEOMETRY, PLOTTED

  • The FormulaOne deviation appears only in the egocentric view and is a plateau rather than a countdown; it is largest for far hazards and when nothing lies within the frontal cone.The authors do not test differences between geometry strata.Source: Figure 6 · p. 17

E.8 VIEWPOINT GENERALISATION AND THE DEPICTION TEST

  • The egocentric-only pattern fails to generalize: CarButton1 shows effects in three views without detected viewpoint differences, while PointGoal1 shows an overhead-only effect.Raising hazard height changes what cameras see but not trajectories; the prespecified causal test does not support the depiction account.Source: §E.8 · pp. 17–18

E.9 PRE-CONTACT DEVIATION BY CAMERA

  • The viewpoint table reports κ’s coefficient of variation by camera; the chase camera is most variable yet carries no resolved event-locked effect.The caption identifies ten policies across three algorithms and 178 episodes matched across four views.Source: §E.9 · p. 18, Table 8 · p. 18

E.10 CAPTION-BANK GEOMETRY, CAPTION REWRITES, AND PLACEBO STABILITY

  • For ViT-B/32, the mean cosine between positive and negative captions equals the within-positive mean to three decimals, indicating little separation between the two caption groups.On fixed frames, the absurd caption set still falls, while only the off-topic set reverses; under ViT-L/14, the danger-group rise survives every placebo draw.Source: §E.10 · pp. 18–19

E.11 THE FRAMES, THE PROMPTS, AND WHAT EACH SCORER REPORTED ON THEM

  • The example follows one isolated contact onset selected without using any score, with replayed rewards and costs checked against the stored trajectory before frames are rendered.The figure presents the same frames to three scorers and includes a detached overhead view that no scorer sees.Source: §E.11 · p. 19

F ENVIRONMENT AND EVALUATION DEFECTS

The evaluation stack has four documented defects affecting scenario selection, replay consistency, hardware reproducibility, and reported returns. Replay checks and environment-only return reporting are used to guard against invalid results.

F ENVIRONMENT AND EVALUATION DEFECTS

  • Four defects are documented: scenario-index aliasing maps intended held-out cases onto training scenarios, traffic re-randomization breaks paired replays, replay depends on hardware group, and VLM bonuses dominate logged return in some runs.In nine FormulaOne-L2 runs without confidence gating, the bonus contributes 99.1%–99.8% of logged return while recomputed environment returns overlap the no-VLM range.Source: F ENVIRONMENT AND EVALUATION DEFECTS · pp. 19, 22
  • The illustrated approach frames come from a 1024 × 1024 replay render for legibility, while scorers receive the identical simulator states rendered at 256 × 256.The plotted scorer traces are z-scored within episode; the lower row averages 124 of 130 isolated contacts with complete windows.Source: Figure 7 · p. 20, Figure 8 · p. 21

G LIMITATIONS

The paper’s conclusions are bounded by environment-specific camera effects, a small set of scorers, and the limited scope of its closed-loop test. It also leaves the predictive account of informative views unresolved.

G LIMITATIONS

  • The egocentric-only result occurs in FormulaOne, not the two additional environments, and no view is consistently privileged or predicted in advance.The tested hazard-visibility explanation does not meet its prespecified support criterion.Source: G LIMITATIONS · p. 22
  • Three scorers are too few to characterize generative scorers, and ViT-L/14 was not tested with the two-label prompts.The near-overhead sign reversal for the larger encoder remains unexplained.Source: G LIMITATIONS · p. 22
  • The reported band results are egocentric and rely on one placebo draw per episode; caption rewrites and partitions were tested only with ViT-B/32 in that view.The near-hazard estimate is sensitive to the placebo draw, and ViT-B/32’s two-label row was not redrawn.Source: G LIMITATIONS · p. 22
  • The four-camera tests use three MuJoCo tasks with rendering style fixed within each task, leaving viewpoint confounded with rendering style and hazard appearance across environments.Source: G LIMITATIONS · p. 22
  • Open-loop trajectories make the signal interpretable but do not directly test a policy trained on it; the closed-loop exception covers one environment, algorithm, and placebo family.The first-person deviation is modest, and the authors do not claim it suffices as a safety signal.Source: G LIMITATIONS · p. 22
  • The proposition constrains an epoch-averaged safety term only if its occupancy-level mean is affine in true cost for every feasible occupancy measure; that condition is not tested for the prior construction.The stated limitation is conditional and does not challenge that construction’s reported safety improvements.Source: G LIMITATIONS · p. 22
Loading 2610.09517v1…