Source-linked AI summary
vLLM Hook v0: A Plug-in for Programming Model Internals on vLLM
Ching-Yun Ko, Pin-Yu Chen
TL;DR
vLLM’s limited access to model internals constrains inference-time programming of deployed models. vLLM Hook adds configurable passive probing and active intervention, and the paper presents examples spanning monitoring, steering, and selective retrieval.
Problem
vLLM does not support accessing or modifying important internal states, limiting inference-time monitoring, adjustment, and control of deployed models.
Source: §1 · p. 1Method
vLLM Hook is a plug-in that uses a configuration file to select internal states for passive probing or active intervention during vLLM model generation.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The examples position vLLM Hook as a developer kit for programming internal states in deployed vLLM models.
Source: §1 · p. 3Programming many states or computationally intensive functions can affect usability through latency and memory costs, so developers are advised to use minimal hooks.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Modern artificial intelligence (AI) models are deployed on inference engines to optimize runtime efficiency and resource allocation, particularly for transformer-based large language models (LLMs). The vLLM project is a major open-source library to support model serving and inference. However, the current implementation of vLLM limits programmability of the internal states of deployed models. This prevents the use of popular test-time model alignment and enhancement methods. For example, it prevents the detection of adversarial prompts based on attention patterns or the adjustment of model responses based on activation steering. To bridge this critical gap, we present vLLM Hook, an opensource plug-in to enable the programming of internal states for vLLM models. Based on a configuration file specifying which internal states to capture, vLLM Hook provides seamless integration to vLLM and supports two essential features: passive programming and active programming. For passive programming, vLLM Hook probes the selected internal states for subsequent analysis, while keeping the model generation intact. For active programming, vLLM Hook enables efficient intervention of model generation by altering the selected internal states. In addition to presenting the core functions of vLLM Hook, in version 0, we demonstrate 3 use cases including prompt injection detection, enhanced retrieval-augmented retrieval (RAG), and activation steering. Finally, we welcome the community's contribution to improve vLLM Hook via https://github.com/ibm/vllm-hook.
1 Overview of vLLM Hook
vLLM Hook addresses vLLM’s lack of access to or control over important model internals by adding configurable passive and active programming. The section outlines a Build–Probe–Program workflow, example applications, and trade-offs in using the plug-in.
1 Overview of vLLM Hook
- vLLM Hook is an open-source plug-in that enables programming internal states of vLLM models, addressing a gap that limits inference-time monitoring, adjustment, and control.The states include attentions, attention heads, and activations.Source: §1 · p. 1
- Passive programming probes and saves selected internal states for later analysis without changing model generation; active programming alters selected states to intervene in generation.Source: §1 · pp. 1–2
- The project illustrates passive monitoring with attention-based prompt-injection detection, contrasting its in-model guardrail with cascaded designs that use an additional moderation model.Source: §1 · p. 2
- Active programming supports model steering, while passive programming supports selective retrieval by activating chosen components, including attention heads for information-retrieval reranking [TLFS25].Source: §1 · p. 2
- The model-specific, task-oriented Config workflow is Build, Probe, then Program: design the function and identify states, specify them in a Config, then apply it to the vLLM model.The Build step occurs upstream and does not require vLLM.Source: §1 · p. 3
- Designing the programming function and identifying states during Build is outside the current scope of the vLLM Hook project.Source: §1 · p. 3
- Programming too many states or using computationally intensive functions can reduce usability, including through latency and memory costs; developers are advised to minimize hooks.Source: §1 · p. 3
2 Core Functions of vLLM Hook
vLLM-Hook combines a worker for programming model internals during inference with an analyzer for optional analysis of captured signals, configured through a Config File. Its worker hooks selected modules in the vLLM runtime, while analyzers can reconstruct statistics such as selective attention weights.
2 Core Functions of vLLM Hook
- The worker and analyzer divide responsibilities: the worker programs within the vLLM pipeline, while the analyzer evaluates captured signals using behavior specified by a Config File.The configuration can specify layers and attention heads, steering vectors, and statistic-reduction methods.Source: §2 · pp. 3–4
2.1 Worker
- Workers subclass the vLLM GPU worker, install PyTorch forward hooks after model loading, and record or modify selected internal states during inference.The example worker selects attention modules and caches query/key tensors for chosen layers and heads.Source: §2.1 · p. 5
2.2 Analyzer
- Analyzers retrieve worker-saved statistics and reassemble them, with an example recomputing selective attention from saved keys and queries for prompt-injection risk evaluation.Figure 3 depicts the wrapper orchestration: HookLLM initializes the model, generate runs the forward pass, and analyze performs optional in-model analysis.Source: Figure 3 · p. 4, §2.2 · p. 6
3 Demonstrations of vLLM Hook
The demonstrations show selective access to model internals for prompt-injection monitoring, activation steering, and attention-based retrieval. They describe attention scores for injection risk, steering vectors for instruction following, and selected-head weights for document relevance.
In-model monitoring of prompt injection attacks
Model steering via activation steering
4 How to Contribute to vLLM Hook?
The paper invites community contributions to expand vLLM Hook’s functions, use cases, and demonstrations. It directs contributors to submit changes through the repository and recommends extending workers and analyzers without modifying the core orchestrator.
4 How to Contribute to vLLM Hook?
- Community contributions are invited to expand vLLM Hook’s core functions, use cases, and demonstrations, with significant contributors potentially invited to co-author future technical reports.Source: §4 · p. 7
- The contribution workflow is to fork the repository, create a branch, commit and push changes, then open a pull request for admin approval.Source: §4 · p. 7
- Contributors are encouraged to define new workers or analyzers, avoid modifying the core orchestrator hook, and include examples and documentation.Source: §4 · p. 7