PaperlayerView exact revision on arXiv

Source-linked AI summary

Cosine Misleads: Auxiliary Losses Reshape Vision Language Models, Not Their Latents

XiuYu Zhang, Junfeng Fang, Zhenkai Liang

arXiv:2606.05753v1cs.CV

TL;DR

LVR research assumes that better alignment between supervised visual latents and their targets yields better answers, but evidence for that assumption is limited. The paper tests five variants with PRISM, combining probes and latent corruption, and finds that cosine alignment can run opposite to accuracy because training reshapes shared parameters while the latents are bypassed.

  • Problem

    LVR uses cosine similarity or MSE as both latent-training losses and quality metrics, assuming better target alignment implies a better answer.

  • Method

    The paper evaluates five LVR variants with PRISM, which probes answer decodability at two model positions and corrupts latents during generation.

  • Results

    Cosine alignment correlates negatively with V∗Bench accuracy across all five variants (Pearson r = −0.94), while latent corruptions shift accuracy by at most four points.

  • Takeaways & Limitations

    Cosine measures latent fidelity but not whether the model uses the latent; the auxiliary loss instead reshapes the VLM through shared parameters.

  • Takeaways & Limitations

    Evidence comes from one base model, one fine-tuning corpus, selected perturbations, and fine-grained perception benchmarks, so other settings may differ.

Abstract

from arXiv · show

Latent visual reasoning (LVR) inserts supervised latent tokens between perception and answer generation in vision-language models (VLMs). The field uses alignment between these latents and their visual targets, i.e., cosine similarity or mean squared error (MSE), as both the training loss and the quality metric, assuming that better alignment yields a better answer. We test this with a designed matrix of five LVR variants and find the assumption inverted: cosine alignment is negatively correlated with accuracy across all five (r=-0.94). To explain this, we introduce PRISM, a pair of inference-time diagnostics: a linear probe that asks where the answer is decodable, and a corruption test that asks whether the latent is load-bearing. The supervised latents are largely bypassed. Corrupting them shifts accuracy by at most four points. The answer is decodable downstream of the latent but not at it, and the size of this decodability gap predicts how much each variant relies on its latent under perturbation. Consistent with an Information Bottleneck reading of the loss, the auxiliary objective reshapes the language model via shared parameters rather than via the latent variable it nominally optimizes.

1 Introduction

The paper tests whether latent-target alignment predicts VLM answers and finds that cosine alignment can mislead: the supervised latents are often bypassed, while shared-parameter training still changes accuracy.

  • Motivation: LVR inserts supervised continuous latent tokens between visual perception and answer generation, using alignment with visual targets as both loss and quality metric.The alignment objectives include MSE and cosine similarity.
  • The alignment assumption: Pearson r = −0.94 between cosine alignment and V∗Bench accuracy across five variants, reversing the assumed relationship.Progressive scaffolding raises cosine alignment by 40% over baseline, lowers reconstruction error from 3.79 to 1.55, and loses 13 V∗Bench points.
  • PRISM: PRISM combines a latent corruption test with linear probes to determine whether latents are load-bearing and where answer information is decodable.The diagnostics complement alignment metrics by testing causal reliance and internal answer decodability.
  • PRISM: At most four V∗Bench points shift under latent truncation, noise, or swapping across all five variants, indicating that the supervised latents are largely bypassed.For the worst variant, zeroing the latents improves accuracy.
  • Where the signal resides: The answer is decodable at the answer-decoding state but not at the feedback latent, and this contrast predicts responses to latent perturbation.Despite bypassing the latents, the five variants still differ by 13 V∗Bench points.
  • Interpretation: The auxiliary objective reshapes the VLM through gradient flow into shared parameters rather than through the latent variable it nominally optimizes.Under the paper’s Information Bottleneck reading, cross-entropy pressures whichever computation produces the answer.

2 Related Work

The related work connects LVR alignment objectives with probing and faithfulness research, motivating tests that distinguish representation decodability from actual causal use.

  • Latent visual reasoning and alignment: LVR studies supervise latent tokens to reconstruct visual embeddings, while concurrent methods vary the alignment losses used against visual targets.Reported choices include MSE, cosine, REPA, and related alignment formulations.
  • Probing and faithfulness: Linear probing evaluates whether a target is predictable from frozen hidden states, but probe accuracy can reflect classifier capability rather than property presence.This motivates control-task selectivity when interpreting probe results.
  • Probing and faithfulness: Faithfulness research shows that reasoning explanations may not match a model’s computation, so interventions can test whether reasoning traces are causally relied upon.The paper applies this logic to latent reasoning by combining probing with corruption.
  • Information bottleneck: The Information Bottleneck frames representation learning as compressing inputs while preserving target information, whereas cosine and MSE optimize only one side of that trade-off.The paper uses this framework as interpretive scaffolding for cosine–accuracy dissociations.

3 The Information Bottleneck view of LVR

The paper interprets LVR losses through the Information Bottleneck: reconstruction aligns an intermediate state to a low-rate visual target, while cross-entropy can place answer relevance elsewhere in the shared model.

  • 3.1 The LVR loss: LVR supervises its hidden state with a reconstruction loss alongside next-token prediction, while cosine variants replace or supplement MSE with alignment objectives.These objectives pull the LVR-position state toward the visual target and feed generated latents back autoregressively at inference.
  • 3.2 The IB objective: The Information Bottleneck represents a trade-off between compressing input information and preserving information about the answer.An IB-optimal representation has small I(X; Z) and large I(Z; Y).
  • 3.2 The IB objective: The LVR setting distinguishes the loss-computed state Zch, the feedback variable Zfb, and the answer-decoding state Hans read by the language-model head.Without a projection, Zch and Zfb are identical; PRISM probes Zfb and Hans separately.
  • 3.3 The LVR loss is an indirect IB objective, and cosine measures only one side of it: MSE indirectly encourages compression by anchoring Zch to a low-rate annotated visual embedding, whereas cosine mainly constrains direction rather than magnitude or orthogonal components.Neither objective directly bounds mutual information.
  • 3.3 The LVR loss is an indirect IB objective, and cosine measures only one side of it: Cross-entropy pressures whichever computation produces the answer, without requiring that the supervised latent carry the relevant information.This allows answer relevance to settle in other parts of the model.
  • 3.3 The LVR loss is an indirect IB objective, and cosine measures only one side of it: PRISM compares answer decodability at the post-latent answer state and feedback variable, then relates that gap to accuracy changes under latent perturbation.Small perturbation effects indicate that the latent is bypassed.
  • 3.3 The LVR loss is an indirect IB objective, and cosine measures only one side of it: Removing the latent shifts accuracy by at most four points, showing that cosine can measure target anchoring without measuring whether the latent carries computational load.The decodability gap between the answer-decoding state and feedback variable is proposed to measure that missing dimension.

4 PRISM: a replacement diagnostic for cosine

PRISM replaces latent-target alignment with inference-time tests that locate answer information and measure whether supervised latents are load-bearing. It combines linear probes of answer decodability with corruption interventions, while framing the training variants by their Information Bottleneck relationship.

  • Axis 1: linear probes: PRISM probes answer decodability at the answer-decoding state and at the feedback variable passed through the latent loop.Logistic regression estimates cross-validated answer accuracy at both positions.
  • Variant matrix: The five variants span reconstruction-only supervision and an IB-consistent noise regularizer, with reconstruction methods using MSE toward low-rate targets without directly bounding I(X; Z).N-LVR adds Gaussian noise while keeping the target clean, approximating a local-smoothness penalty on the encoder.
  • Information-theoretic interpretation: The Barber–Agakov bound motivates using held-out cross-entropy from a representation as a lower bound on its answer-relevant information.The variational predictor q(Y | rep) estimates answer information from the representation.
  • Axis 1: linear probes: The decodability gap G = accprobe(a) − accprobe(b) contrasts answer information downstream with information carried by the feedback variable.Both probe accuracies remain informative because the gap and its components carry different information.
  • Axis 2: faithfulness corruption: PRISM corrupts latents by truncation, Gaussian noise, or random-donor swaps, then measures the signed change in benchmark accuracy.Truncation tests whether latents are used at all, noise tests robustness, and swapping tests sample-specific content.

5 Experiments

Across five LVR variants, cosine alignment moves backward relative to accuracy, while corruption and probing show that supervised latents are often bypassed and answer-relevant information emerges downstream. The probe contrast further predicts how much variants rely on latents under perturbation.

  • 5.1 The LVR variant matrix and setup: The five variants share a training setup while varying supervision application, targets, or latent-position organization.The shared objective is LLVR = LCE + λ · MSE(h_t, v_t) with λ = 0.1 throughout.
  • 5.2 Cosine inverts the ordering it claims to measure: r = −0.94: cosine alignment and V∗Bench accuracy move in opposite directions across the five variants.Progressive scaffolding raises cosine alignment by 40% and lowers reconstruction error from 3.79 to 1.55, but loses 13 V∗Bench points.
  • 5.2 Cosine inverts the ordering it claims to measure: Cosine cannot distinguish LVR and N-LVR despite nearly identical values, because their swap responses have opposite signs: +1.6 versus −0.5.Cosine also correlates positively with corruption response: r = +0.75 for truncation and r = +0.84 for small noise (σ = 0.1).
  • 5.3 The latents are largely bypassed at inference: Every latent intervention shifts V∗Bench accuracy by at most four points across variants, and zeroing P-LVR-3 latents improves accuracy.N-LVR is the only variant whose own latents reliably outperform random-donor latents under swap.
  • 5.4 The probe contrast localizes where the answer lives: Answer-decoding probe accuracy tracks V∗Bench at r = +0.98, whereas feedback-variable probe accuracy does not, at r = +0.20.The decodability gap G ranges from 14.1 to 36.6 percentage points and predicts V∗Bench at r = +0.86.
  • 5.5 The probe contrast predicts latent reliance: The probe contrast predicts latent reliance under perturbation, with G correlating at r = −0.77 for truncation and r = −0.93 for small noise.A large gap indicates answer information has settled downstream while the latent still plays a structural role in the computation path.

6 Discussion

The supervised latents can be bypassed without preventing the variants from differing in performance, because the auxiliary loss reshapes the VLM through shared parameters.

  • The supervised latents are largely bypassed, yet the variants still differ by 13 V∗Bench points.This shows that bypassing the nominally optimized latent does not eliminate performance differences among variants.
  • The auxiliary loss reshapes the VLM through gradient flow into shared parameters rather than solely through the latent variable.
  • An auxiliary loss can act through a path it does not name.

Limitations

The evidence is bounded by a single base model, corpus, and benchmark family, while both diagnostics measure limited aspects of representation and latent reliance.

  • Evidence comes from one base model and one fine-tuning corpus, so per-variant numbers could shift with larger backbones or controlled-init replicates.
  • The benchmarks emphasize fine-grained perception, so holistic reasoning tasks may exhibit different cosine–accuracy patterns.
  • Linear probes measure linear decodability, not necessarily which information downstream layers actually use.
  • The corruption test covers truncation, three noise levels, and donor swaps but does not exhaust the perturbation space.
  • Probe selectivity is bounded using a random-label control, with position-(a) selectivity above 25% and probe-(b) selectivity under 10% across variants.

A.5 Robustness of the Probe-(a) Result

Robustness checks support the probe-(a) relationship with accuracy across variants, benchmarks, controls, and perturbations, while the small number of variants limits precision.

  • +0.980 is the Pearson correlation between probe-(a) accuracy and V∗Bench accuracy across five variants.The relationship remains descriptive because only five variant-level data points contribute.
  • r ≥0.97 after removing any single variant, indicating that no one variant drives the probe-(a)/V∗Bench relationship.
  • +0.929 is the partial correlation between probe-(a) and V∗Bench accuracy after controlling for question-token content.The passage characterizes this as descriptive evidence rather than a formal conditional mutual-information estimate.
  • The 95% bootstrap interval for r(probe-(a), V*B) is [+0.694, +0.993], while the five-variant design leaves meaningful uncertainty in correlation magnitude.
  • Probe-(a) correlates with BLINK at +0.96 and MMVP at +0.92, supporting benchmark-general answer decodability.
  • Position (a) correlates strongly with V∗Bench, whereas position (b) correlates at only +0.20, distinguishing answer-decoding from latent-feedback states.
  • Corruption is applied before K/V projections while image embeddings, prompt tokens, and decoder weights remain fixed between clean and corrupted runs.
  • Variant ranking by swap delta is invariant across three donor seeds, with absolute deltas varying within ±1.0 V∗Bench point.

B.5 Why Three Tests

The three corruption tests distinguish whether latents are used, whether their content is sample-specific, and how robustly answers tolerate perturbation.

  • Truncation tests whether latents are used at all, swapping tests sample-specific latent content, and noise tests robustness across perturbation magnitudes.
  • −0.89 is the correlation between truncation delta and V∗Bench accuracy, while swap delta correlates more weakly at −0.66.More-negative truncation deltas, indicating greater benefit from own latents, predict higher V∗Bench accuracy.
  • The corruption analysis holds image embeddings, prompt tokens, and decoder weights identical between clean and corrupted runs.

C Training Hyperparameters

All variants share the principal training setup and budget, while their auxiliary supervision and staging schedules define the experimental differences.

  • The shared setup uses Qwen2.5-VL-3B-Instruct, Visual-CoT-438k, AdamW, learning rate 1e−5, cosine scheduling, and bf16 precision.
  • All five variants use effective batch size 64 and 2500 training steps, despite differing in how the batch is achieved.
  • LVR uses a single latent block with λlvr = 0.1 and MSE reconstruction against teacher-forced box embeddings.
  • N-LVR adds zero-mean Gaussian noise with σ=0.3 to teacher-forced visual embeddings during training, without inference-time noise.
  • D-LVR resumes the baseline at step 1500 and removes reconstruction loss for the remaining 1000 steps, leaving answer-text cross-entropy supervision.
  • P-LVR-2 uses context and target stages, while P-LVR-3 adds a free intermediate stage with diversity regularization λdiv = 0.01.
  • Each variant requires approximately 5–12 GPU-hours, with diagnostic reruns adding about one GPU-hour per variant per benchmark.

D Evaluation Protocol

Evaluation covers three existing visual benchmarks and uses fixed decoding, probe, corruption, and held-out cosine/MSE protocols. The study relies on public artifacts without collecting new data or human-subject information.

  • Benchmarks: V∗Bench contains 191 fine-grained visual-reasoning questions, while MMVP contains 300 paired-question visual-shortcoming examples.
  • Benchmarks: BLINK evaluation aggregates accuracy across five subsets totaling 697 perception-oriented questions.
  • Inference: All models use greedy decoding, a 512-token maximum, unchanged image resolution, and autoregressive feedback of generated LVR tokens.
  • Diagnostics: Linear probes and faithfulness corruption use the full V∗Bench evaluation set with stratified 5-fold cross-validation, without a held-out generalization set.
  • Alignment metrics: Cosine and MSE are computed on the same held-out Visual-CoT validation slice for all variants under teacher-forced LVR generation.
  • Artifacts and data: Experiments use existing public datasets, benchmarks, models, and code, without redistributing original corpora or collecting new data, annotators, or personally identifying information.
Loading 2606.05753v1…