PaperlayerView exact revision on arXiv

Source-linked AI summary

GGSS: Geodesic-Gated Spherical Steering for Inference-Time Debiasing of Generative Vision-Language Models

Yiqun Sun, Junyu Chen, Pengfei Wei, Lawrence B. Hsieh

arXiv:2608.25375v1cs.CYcs.CLcs.CV

TL;DR

Generative VLMs can express demographic bias despite controlled visual content, and existing inference-time methods largely target static or CLIP-like representations. GGSS learns a counterfactual spherical bias subspace and applies gated, norm-preserving geodesic steering to visual tokens. Across four models, it achieves the lowest average bias while keeping MMStar accuracy within ±0.6 p.p. of the unsteered baseline.

  • Problem

    Generative VLMs produce demographic disparities under controlled visual changes, while existing inference-time debiasers were largely designed for static or CLIP-like models.

  • Method

    GGSS discovers a counterfactual spherical bias subspace and uses adaptive token gating with norm-preserving geodesic steering during inference.

  • Results

    GGSS achieves the lowest average bias on all four evaluated models, with significant reductions on three of four backbones, while MMStar accuracy stays within ±0.6 p.p. of baseline.

  • Takeaways & Limitations

    The results support adaptive geodesic steering as a practical inference-time primitive for mitigating bias in generative VLMs while retaining capability.

  • Takeaways & Limitations

    The evaluation does not establish generalization to all architectures, languages, domains, bias types, protected attributes, intersectional groups, or multilingual prompts.

Abstract

from arXiv · show

Generative vision-language models (VLMs) are increasingly used in human-centered settings, yet they can produce demographically biased outputs even when images differ only in controlled attributes such as perceived race or gender. However, existing inference-time debiasers were largely designed for static embeddings or CLIP-like models rather than generative VLMs. We propose GGSS---Geodesic-Gated Spherical Steering---a norm-preserving intervention that discovers a counterfactual bias subspace on the unit hypersphere, steers visual tokens along geodesic arcs, and uses an adaptive gate to focus correction on tokens that carry stronger demographic signal. We evaluate four generative VLMs against ten adapted inference-time debiasing baselines and prompt-based mitigation under a single operating-point protocol across categorical, pairwise, and occupation-gender bias tests, while also measuring general visual-language capability. GGSS achieves the lowest average bias on all four models, significant on three of four backbones under paired permutation tests, while preserving MMStar accuracy within +/- 0.6 p.p. of the unsteered baseline. Code is available at https://github.com/dukesun99/GGSS.

1 Introduction

Generative VLMs can produce demographic disparities despite controlled visual content, while existing inference-time debiasers largely target static or CLIP-like embeddings. GGSS addresses this gap with selective, norm-preserving spherical steering and achieves the lowest average bias across four evaluated models while preserving capability.

  • VLMs can produce systematically different outputs across demographic groups even when underlying visual content is controlled.
  • Existing inference-time debiasers largely target CLIP-like models that summarize inputs with a single global embedding.
  • GGSS discovers a counterfactual bias subspace, applies norm-preserving geodesic rotation, and adaptively gates steering by each token’s bias magnitude.
  • GGSS is evaluated on four generative VLMs against ten adapted inference-time baselines, prompt mitigation, and structural ablations under a unified protocol.
  • GGSS achieves the lowest average bias on all four models, with significant reductions on three backbones and per-task reductions up to 96%, 84%, and 61%.MMStar accuracy remains within ±0.6 p.p. of baseline.
  • The contributions include a unified comparison suite and evaluation under one best-avg-α protocol while preserving MMStar capability.

2 Related Work

Prior work documents social biases across VLM representations and behaviors, while debiasing methods have emphasized training-based changes or inference-time interventions developed mainly for CLIP-style systems. GGSS adapts geometric steering to image-side activations and adds token-adaptive gating.

  • Prior studies report VLM disparities across gender, race, age, nationality, religion, and other attributes in representations and downstream behaviors.
  • Inference-time mitigation is motivated by its applicability to frozen models without retraining.
  • Most VLM debiasing methods target CLIP-style dual encoders, whereas inference-time methods modify prompts or embedding geometry at test time.
  • GGSS instantiates geometric concept steering for image-side activations in generative VLMs and adds a bias-norm-calibrated token-adaptive gate.

3 The GGSS Framework

GGSS uses offline counterfactual discovery to learn spherical steering quantities, then applies gated geodesic updates to visual tokens during inference. The update removes learned protected coordinates while exactly preserving each token’s original norm.

  • Framework overview: GGSS separates offline discovery from online inference, reusing a protected-attribute basis, spherical reference point, and gate statistics for new inputs.
  • Token-level geodesic steering: GGSS rotates token directions along spherical geodesics with Slerp, then restores the original radius before passing tokens to the frozen VLM.
  • Counterfactual bias subspace discovery: Counterfactual images vary protected attributes while holding fixed contextual factors such as identity, occupation, pose, clothing, background, and lighting.
  • Counterfactual bias subspace discovery: The discovery stage pools normalized visual activations and estimates a multi-dimensional protected-attribute subspace from tangent-space shifts using SVD.
  • Counterfactual bias subspace discovery: The global reference point is the Fréchet mean of discovery representatives, and its learned subspace parameterizes the steering geometry without a supervised probe.
  • Token-level geodesic steering: Each visual token is normalized, mapped to tangent space, decomposed into protected and clean components, and assigned a gate from discovery bias-magnitude statistics.
  • Token-level geodesic steering: Larger protected-coordinate magnitudes receive stronger steering, while low-bias tokens remain closer to their original directions.
  • Token-level geodesic steering: The update orthogonally removes learned protected-attribute coordinates and preserves token norms exactly, while bias reduction remains an empirical question.

4 Experiments

Experiments compare GGSS with adapted inference-time debiasers and prompt mitigation across four generative VLMs, bias tasks, and MMStar capability preservation. GGSS is the most consistent method, reducing bias substantially while retaining general capability.

  • Main results: GGSS attains the lowest average bias on all four evaluated models against external baselines.The comparison uses one best-avg-α operating point per method and model.
  • Main results: Up to −96%, −84%, and −61% reductions occur on Nurse/Doctor, MCQ, and 2AFC, respectively.The reported examples are LLaVA-1.6-Vicuna-7B Nurse/Doctor, LLaVA-1.6-Vicuna-7B MCQ, and Pixtral-12B 2AFC.
  • Main results: GGSS keeps MMStar accuracy within ±0.6 p.p. of baseline across steering evaluations.The gate confines intervention to directions carrying targeted bias and leaves other visual tokens largely untouched.
  • Main results: GGSS reductions are statistically significant on three of four backbones, while its MMStar changes are indistinguishable from the unsteered model in all eight runs.On LLaVA-Vicuna, the Nurse/Doctor reduction alone has p < 10^-4; INLP (sph.) significantly damages Pixtral MMStar by −6.1 pp.
  • Robustness checks: Cross-fitted α selection keeps GGSS near the top on every model and reduces bias in every held-out fold.It matches the paper’s α on six of eight folds.
  • Baseline comparisons: Prompt mitigation is inconsistent and can amplify bias, whereas LEACE is competitive on Pixtral but less consistent across backbones.Vicuna MCQ increases by +85% under prompt mitigation; LEACE records −26% to −31% on Qwen3-VL.
  • Capability and attribute retention: At α = 0.5 on Qwen3-VL, race recognition remains within 4 pp of baseline while 55% of MCQ reduction is realized.The targeted attribute can therefore be retained more closely at moderate α or with steering gated off.
  • Ablations and sensitivity: The full gated-Slerp method reaches JSD 2.41 (−84%) on Qwen3-VL-4B MCQ, improving over gated non-Slerp JSD 4.53 (−69%).Disabling the gate gives JSD 5.61, matching the ungated spherical hard-projection variant.

5 Conclusion

GGSS debiases generative VLMs by discovering counterfactual spherical bias subspaces, applying calibrated token gates, and steering visual tokens with norm-preserving Slerp. Across four models and three bias protocols, it achieves the lowest average bias while keeping MMStar capability indistinguishable from the unsteered model.

  • Conclusion: GGSS discovers counterfactual spherical bias subspaces, applies a calibrated token gate, and uses norm-preserving Slerp for inference-time debiasing.The framework is designed for generative VLMs and combines subspace discovery, token selectivity, and on-sphere steering.
  • Conclusion: GGSS attains the lowest Avg ∆% on all 4 models across three bias protocols against ten external steering baselines and prompt-based mitigation.The comparison evaluates four generative VLMs under a common operating-point protocol.
  • Conclusion: GGSS is significant on three of four backbones while keeping MMStar indistinguishable from the unsteered model.Matched-gate ablations attribute robustness at scale to on-sphere steering and selectivity to the adaptive gate.

Limitations

The evaluation establishes GGSS only within four VLMs, three demographic-bias protocols, and MMStar, primarily for perceived race and gender in image-conditioned settings. Deployment also requires model-dependent steering calibration and careful auditing because stronger steering can reduce attribute retention and does not guarantee fairness or robustness under distribution shift.

  • Scope: The evaluation does not establish generalization to all architectures, languages, domains, or bias types.It covers four generative VLMs, three demographic-bias protocols, and MMStar, focusing on perceived race and gender in image-conditioned settings.
  • Calibration: GGSS requires selecting a model-dependent steering strength from held-out bias measurements.The paper selects one best-avg-α per method and model from {0.25, 0.5, 0.75, 1.0, 1.5}; a tuning-free rule remains open.
  • Deployment: At stronger steering, targeted attribute reportability can be reduced, so deployments should choose α from the bias–retention trade-off or disable steering when appropriate.The paper cautions that lower benchmark bias is not complete fairness or safety and recommends broader auditing, human review, and monitoring.

Ethics Statement

The paper frames GGSS as a dual-use intervention for reducing demographic bias without retraining, while emphasizing that steering can also amplify sensitivity or remove useful attributes. Its labels are operational constructs, and responsible deployment requires deliberate operating-point selection, oversight, and auditing.

  • Ethics Statement: GGSS is designed to reduce demographic bias in deployed VLMs without retraining, but the same steering mechanism can amplify demographic sensitivity.At high steering strength, the intervention can shade from debiasing into attribute removal.
  • Ethics Statement: Deployments should measure both bias and attribute retention on the target task before fixing a steering strength.The paper releases trade-off measurements to support deliberate operating-point selection.
  • Ethics Statement: Perceived race and binary gender labels are operational constructs assigned by dataset curators, not representations of self-identification or full human identity diversity.The paper cautions against interpreting results as claims about any individual’s identity.
  • Ethics Statement: Reduced benchmark bias is not equivalent to fairness because steering does not remove biased model knowledge and may behave differently under distribution shift.Consequential deployments should retain human oversight and independent auditing.
  • Ethics Statement: The intervention operates on visual-token activations, using spherical normalization, Fréchet means, tangent-space shifts, and norm-restoring steering operations.At inference time, tokens are processed independently and their original norms are restored after steering.

A.4 Ablation Variants

The ablation suite compares Euclidean and spherical pooled or per-token SVD variants, classifier and concept-erasure baselines, and implementation choices under a shared evaluation protocol. Results show that gating and Slerp each contribute to GGSS performance, while external baselines are less consistent across models and prompt mitigation can amplify bias.

  • A.4 Ablation Variants: Pooled SVD (Eucl.) removes spherical geometry and Slerp/gating, while pooled SVD (sph.) retains spherical operations but replaces Slerp with hard projection and disables the gate.The Euclidean variant does not preserve token norms; the spherical hard-projection variant sets gi ≡1.
  • A.4 Ablation Variants: Per-token SVD variants preserve token-position information during discovery but are more expensive and dominated by pooled variants on most model–task cells.Both Euclidean and spherical per-token variants then apply standard null-space projection at inference.
  • A.4 Ablation Variants: Classifier-based methods adapt image-level pooled representations with RBF-SVM or multinomial logistic regression, while INLP and LEACE provide linear concept-erasure baselines.INLP iteratively projects classifier directions into their null space; LEACE uses a closed-form least-squares projection, with spherical variants operating in tangent space.
  • A.4 Ablation Variants: All methods are compared at α ∈{0.25, 0.5, 0.75, 1.0, 1.5} using best-α reporting per method and model, with held-out discovery occupations and exact-match MMStar evaluation.GGSS-specific hyperparameters and selected operating points are summarized separately.

B.3 Per-Model Full Results and MMStar

The evaluation uses best-α operating points with uncertainty tests and held-out cross-fitting, while comparing bias reduction against MMStar capability preservation. GGSS’s reported capability changes remain small, with detailed statistical support for headline claims.

  • 95% bootstrap confidence intervals use 10,000 resamples over items and base identities, while paired sign-flip tests compare methods for every reported cell.
  • −96% for the LLaVA-Vicuna Nurse/Doctor reduction has p < 10^-4, while the MMStar within ±0.6 p.p. claim has per-question CIs within ±1.9 pp and exact McNemar p ≥0.09.
  • Cross-fitted α selection matches the paper’s choice on six of eight folds, and test-fold reduction is negative in every fold on every model.
  • MMStar accuracy per capability dimension stays within the ±6 pp binomial noise band of the unsteered model across all four backbones.Each capability dimension uses n = 250 questions.

B.7 MME Results

The additional MME evaluation shows small aggregate changes under GGSS, while targeted steering preserves recognition of the nontargeted demographic attribute and α controls the removal–retention trade-off. Fairness prompting is inconsistent and can amplify bias.

  • MME perception changes from 1693.9 to 1685.6 (−0.4%), while reasoning changes from 631.1 to 617.1 (−1.8%) under GGSS on Qwen3-VL-4B.No MME subcategory collapses.
  • Steering one demographic attribute leaves recognition of the other attribute intact on all four backbones.
  • Instruction prompting is inconsistent across models and tasks and can amplify bias, whereas GGSS reduces bias consistently on every model.

B.10 LEACE Baseline Results

LEACE is evaluated as a serious baseline at the same projection layer in two variants, including one combined with GGSS’s calibrated gate. It can win on Pixtral but is less consistent across backbones than GGSS.

  • LEACE is evaluated at the same projection layer in a direct spherical port and a closed-form subspace variant combined with the calibrated gate.
  • LEACE wins the Pixtral bias column at its operating point while preserving Pixtral MMStar by −0.8/−0.9 pp, with McNemar p ≥0.15.
  • −26% to −31% on Qwen3-VL illustrates LEACE’s lower consistency across backbones, whereas GGSS retains the best four-backbone mean and worst case.

B.11 Matched-Gate Geometry Ablation

The matched-gate ablation compares Euclidean and spherical steering under the full protocol and examines gate calibration, discovery-prompt invariance, and alternative evaluation controls. Results show geometry differences are often within sampling noise, while calibration choices do not change the paper’s conclusions.

  • Matched-Gate Geometry Ablation: At 4B and 7B, the Euclidean-versus-spherical geometry toggle is within sampling noise in both gate conditions; on Qwen3-VL, point estimates favor Euclidean steering.
  • Matched-Gate Geometry Ablation: 1.51 versus 2.41 is the lowest measured value for one Euclidean-plus-renormalization Table 3 cell, with paired p ≈0.75.
  • Gate Calibration: Qwen3-VL per-token bias norms have median 0.196 versus 0.113 for pooled representatives, yielding mean gate 0.85 and 72% of tokens gated above 0.9.
  • Discovery-Prompt Invariance: Alternative discovery prompts produce subspaces with principal-angle cosines ≥0.9997 and identical end-to-end steering results because visual-token activations precede text-prompt interaction.
  • Scope: The artifacts are intended for research auditing of frozen generative VLMs, not deployment claims without additional human review and domain-specific safety testing.
Loading 2608.25375v1…