Source-linked AI summary
Immuno-VLM: Immunizing Large Vision-Language Models via Generative Semantic Antibodies for Open-World Trustworthiness
Xiang Fang, Wanlong Fang, Wei Ji
TL;DR
LVLMs can force unknown anomalies into known categories because semantic abstraction lacks explicit negative knowledge, creating an open-world trustworthiness problem. Immuno-VLM applies immunological negative selection by generating semantic antibodies in latent space and training a rejection mechanism around them. The framework reports state-of-the-art OOD detection, including over 16% improvement over standard zero-shot baselines and 83.4% AUROC on ImageNet-O versus 70.1% for Energy Score.
Problem
LVLMs lack explicit negative knowledge and may force unknown anomalies into known categories, creating the Open-World Trustworthiness Paradox in open-world deployment.
Method
Immuno-VLM generates semantic antibodies as near-distribution textual outliers and uses a rejection adapter with vaccination loss to separate them from known prototypes.
Results
Immuno-VLM achieves state-of-the-art OOD detection, improving adversarial semantic-shift detection by over 16% and reaching 83.4% AUROC on ImageNet-O versus 70.1% for Energy Score.
Takeaways & Limitations
The framework shifts OOD detection from reactive discrimination toward proactive immunization by actively defining boundaries around known classes.
Takeaways & Limitations
The theoretical guarantees assume bi-Lipschitz semantic alignment and generated antibodies that form a δ-cover of the relevant semantic boundary.
Abstract
from arXiv · showhide
Large Vision-Language Models have achieved unprecedented success in zero-shot recognition by aligning visual features with broad semantic concepts. However, this semantic abstraction creates a critical vulnerability in open-world deployment: the ``Hubris of Semantics'', where models force-fit unknown anomalies into known categories with high confidence due to the lack of explicit negative knowledge. To address this \textit{Open-World Trustworthiness Paradox}, we propose \textbf{Immuno-VLM}, a bio-inspired framework that adapts the biological principle of \textbf{Immunological Negative Selection} to high-dimensional latent spaces. Departing from traditional Open-Set Recognition methods that rely on passive density estimation or inefficient pixel-space outlier generation, Immuno-VLM leverages the generative reasoning of Large Language Models to actively hallucinate ``Semantic Antibodies'', textual descriptions of near-distribution outliers (e.g., look-alikes, contextual anomalies) that effectively bound the decision space of known classes.Extensive experiments on ImageNet-1K and four challenging OOD benchmarks reveal that Immuno-VLM establishes a new state-of-the-art.
1. Introduction
LVLMs generalize through semantic abstraction but can force unknown anomalies into known classes, creating the Open-World Trustworthiness Paradox. Immuno-VLM addresses this vulnerability by using biological negative selection and hallucinated semantic antibodies to define unknown boundaries.
- Motivation: LVLMs achieve zero-shot generalization by aligning visual features with broad semantic concepts.This capability extends recognition beyond closed-set categories through natural-language prompts.
- Motivation: The Open-World Trustworthiness Paradox arises when robust semantic abstraction becomes unsafe in unconstrained, safety-critical environments.The problem is framed as a transition from benchmark success to deployment in settings such as autonomous driving and medical diagnostics.
- Motivation: The Hubris of Semantics forces anomalous inputs toward known categories because dense retrieval favors similarity to available text concepts.This creates fragility in distinguishing known concepts from unknown anomalies despite robustness to covariate shifts.
- Biological Analogy: Biological negative selection protects against unknown pathogens by generating candidates and eliminating those that bind to Self.Immuno-VLM adopts this principle as a model for defining known concepts rather than memorizing the entire external world.
- Proposed Direction: Semantic antibodies describe near-distribution look-alikes and contextual anomalies, then form a dense boundary around known classes in the shared embedding space.Examples include “wolf” and “coyote” for “dog,” plus descriptions such as “dog melting” and “dog made of clouds.”
- Contribution: Immuno-VLM reports state-of-the-art performance and over 16% improvement in detecting adversarial semantic shifts over standard zero-shot baselines while preserving in-distribution recognition.The framework treats Large Language Models as a Computational Thymus for generating diverse near-distribution outliers.
2. Related Work
The related work passage organizes out-of-distribution detection into four methodological families: classification-based, density-based, distance-based, and reconstruction-based methods.
- OOD Detection Taxonomy: Classification-based and density-based methods constitute two established families for out-of-distribution detection.The passage lists classification-based methods alongside density-based methods as distinct categories.
- OOD Detection Taxonomy: Distance-based methods identify out-of-distribution samples through distances in a representation or feature space.They are listed as a separate category from classification, density, and reconstruction approaches.
- OOD Detection Taxonomy: Reconstruction-based methods form a fourth category in the cited out-of-distribution detection taxonomy.The taxonomy distinguishes reconstruction from classification, density, and distance-based approaches.
3. The Proposed Immuno-VLM Framework
Immuno-VLM defines open-world recognition in a shared vision-language space, generates semantic antibodies to cover near-OOD boundaries, and trains the model to reject them while preserving known-class structure. Its theory links antibody coverage and alignment quality to bounded open-space risk, while inference compares affinity to known prototypes against reactivity to antibodies.
- 3.1. Problem Formulation: Open-world recognition treats test data as a mixture of known and unknown distributions, with open-space risk measuring unknown samples misclassified as known classes.The unknown distribution has disjoint labels from the known distribution, and its pixel-space support is unbounded.
- 3.2. Semantic Antibodies and Covering Bound: Semantic antibodies form a δ-cover of the semantic boundary, and enforcing margin m > δ bounds near-boundary false positives by alignment error.The theorem states that coverage density δ and visual-text alignment quality ϵalign control the resulting risk.
- 3.3. Phase 1: Antigen Profiling via Bayesian Hyperspherical Estimation: The class prototype is a semantic interpolation of visual evidence and the text embedding on the unit hypersphere, correcting visual noise with linguistic stability.The optimization places the prototype on the geodesic connecting the visual centroid and text embedding.
- 3.4. Phase 2: Generating Semantic Antibodies via Manifold Sampling: Random negatives become effectively orthogonal to class prototypes as latent dimension increases, so Immuno-VLM samples near-OOD concepts from the semantic manifold instead.The paper specifically identifies d = 512 or d = 768 as settings where random vectors provide negligible boundary-tightening gradients.
- 3.4. Phase 2: Generating Semantic Antibodies via Manifold Sampling: The framework filters generated antibodies for safety and utility, then uses a lightweight rejection adapter and push-pull vaccination loss to separate known samples from antibody directions.The repulsion term enforces an angular sterile zone of width m while the attraction term maintains compact known-class structure.
- 3.6. Inference: The Trustworthiness Score: The Differential Immunity Score accepts samples only when affinity to a known prototype exceeds reactivity to the nearest antibody, with EVT providing per-class thresholds.The theoretical guarantee states that increasing antibody density while keeping the adapter simple bounds error on unseen unknown objects.
4. Experiments
Experiments evaluate Immuno-VLM’s trustworthiness, preservation of in-distribution accuracy, scalability, robustness, semantic components, and computational efficiency. Results show stronger OOD rejection, slight ID improvement, stable operating ranges, and low overhead.
- Main Results: 83.4% AUROC on ImageNet-O, improving over Energy Score’s 70.1%, while FPR95 falls to 28.5% from MSP’s 61.3%.ImageNet-O contains adversarial near-OOD examples designed to fool visual classifiers.
- Main Results: Figure 6 shows dense known-class clusters, antibody perimeter rings, and unseen open-world samples occupying the carved-out empty space for linear separation.The visualization compares embedding spaces before and after vaccination.
- Sensitivity Analysis: Performance peaks at m = 0.2 within the stable m ∈[0.1, 0.3] range, whereas m ≥0.5 causes Manifold Collapse and a 4.2% ID-ACC drop.At m = 0.0, AUROC is 76.2%; the optimal margin corresponds to roughly 37° angular separation.
- Ablation Studies: Removing semantic filtration causes either label confusion with a 12% ID-ACC drop or weak rejection at 72.1% AUROC, while the active band 0.3 < sim < 0.7 targets hard distinctions.The ablation supports the role of semantically selected antibodies rather than generic noise or regularization alone.
- Computational Efficiency: Vaccination takes approximately 3.5 hours on one A100, adapter inference adds 0.04 ms/image, and latency overhead remains below 0.5%.The rejection adapter is a lightweight MLP, while the stated training cost is for ImageNet-1K over 10 epochs.
5. Conclusion
Immuno-VLM reframes open-world out-of-distribution detection as proactive immunization by applying Immunological Negative Selection in foundation-model latent spaces. Experiments report state-of-the-art performance while preserving in-distribution recognition.
- Immuno-VLM operationalizes Immunological Negative Selection within foundation-model latent spaces to shift OOD detection from reactive discrimination to proactive immunization.
- Immuno-VLM achieved state-of-the-art performance on ImageNet-O and OpenOOD benchmarks.
- Immuno-VLM improved detection of adversarial semantic shifts by over 16% compared to standard zero-shot baselines while preserving in-distribution recognition.