Source-linked AI summary
ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models
Karan Goyal, Afreen Hossain, Debojyoti Das, Vishal Bhutani
TL;DR
Contextual entrainment in vision-language models remains largely uncharted, with no purpose-built instrument for studying it directly. This paper introduces ENTRAP-VL, a manually curated dual-stream dataset and taxonomy that enables rigorous investigation of textual and visual contextual entrainment.
Problem
Contextual entrainment in vision-language models remains largely uncharted, and the field lacks a controlled, taxonomically organized instrument to study it directly.
Method
The paper introduces ENTRAP-VL, a manually curated 1,500-item dataset organized by association and truth axes across textual and visual streams.
Results
ENTRAP-VL provides a dual-stream taxonomy, dataset, documentation, and evaluation protocols for investigating contextual entrainment in VLMs.
Takeaways & Limitations
The paper supports studying VLM entrainment as a distinct dual phenomenon driven independently by textual and visual context.
Takeaways & Limitations
At 1,500 English-only items, ENTRAP-VL is a focused diagnostic rather than a large-scale benchmark, so results concern its specific instantiated conditions.
Abstract
from arXiv · showhide
Contextual entrainment is the tendency of a model to let auxiliary context in its input pull its output, independently of whether that context is relevant, true, or even meaningful. Recently, it has been identified and given a mechanistic account in unimodal language models. Whether and how it manifests in vision-language models (VLMs) is, by contrast, largely unexamined, and the field lacks a purpose-built instrument with which to investigate it. We take the position that studying contextual entrainment in VLMs requires more than porting an existing text-only benchmark to the multimodal setting: it requires a taxonomically structured, dual-modality instrument whose conditions are constructed around the item at hand (the depicted image in the textual stream, the textual query in the visual stream). We argue that the move to VLMs is substantive rather than incremental. It makes entrainment a dual phenomenon, drivable independently by textual and by visual context, and it opens a veracity distinction (context that is false of the depicted scene yet possible in the world) that has no counterpart in the unimodal, world-knowledge-only formulation of prior work. To make this position concrete and actionable, we introduce ENTRAP-VL (ENTRainment Assessment Probe for Vision and Language), a manually curated dataset of 1,500 items across eight categories, organized by a taxonomy that spans two axes, i.e., the association of context with the item and its relationship to truth, and split into a textual-entrainment stream (eight context conditions) and a visual-entrainment stream (three context conditions). We do not claim to measure entrainment in any particular model; we provide the instrument, the taxonomy that motivates it, and the evaluation protocols it enables, so that the community can investigate the phenomenon rigorously. We will release the dataset and its documentation publicly.
1 Introduction
The paper argues that contextual entrainment in vision-language models is an uncharted, dual-modality phenomenon requiring a purpose-built, taxonomically structured instrument. ENTRAP-VL provides that resource while making no claims about entrainment’s presence, mechanisms, or scaling in any particular VLM.
- Motivation: Auxiliary passages, captions, or images intended to support visual question answering can instead pull model outputs away from their unconstrained answers.This failure surface arises when retrieval-augmented or injected content influences generation unintentionally.
- Problem: Unlike prior unimodal work, VLM contextual entrainment remains largely uncharted and is distinct from, rather than identical to, distraction by irrelevant visual context.Existing VLM robustness and mechanistic-interpretability studies address adjacent phenomena involving irrelevant context or visual attention.
- Taxonomy: The proposed instrument treats VLM entrainment as dual, with context associated independently with the depicted image in the textual stream or the textual query in the visual stream.Its taxonomy organizes eight context conditions along association with the item and relationship to truth.
- Taxonomy: The taxonomy distinguishes context false of the depicted scene yet possible in the world from context false in the world, termed contradictory and counterfactual.These distinctions are native to perceptual grounding and require manual, item-aware curation rather than templatic generation.
- Scope: ENTRAP-VL is a behavioral evaluation resource, not a training or fine-tuning dataset, and the paper claims no finding about entrainment in any VLM.The authors reserve questions about presence, absence, mechanism, and scaling for future investigation using the instrument.
2 Background and Related Work
Prior work establishes contextual entrainment as a mechanistic tendency in language models and examines related distraction, knowledge-conflict, sycophancy, hallucination, and visual-context robustness phenomena in VLMs. ENTRAP-VL addresses the resulting gap with a structured probe that separates textual and visual sources of context-induced pull.
- Contextual entrainment: Language models over-prefer context-present tokens, including random or counterfactual ones, through entrainment heads rather than mere retrieval relevance.Counterfactual context exerts a stronger pull than factual context, and ablating entrainment heads attenuates the effect.
- Distraction and interpretability in VLMs: Visual distractors reduce VLM accuracy as irrelevant context grows, motivating comparison with textual distraction while leaving context-induced pull insufficiently characterized.The reported degradation is steep and often logarithmic as distractor images are added.
- Distraction and interpretability in VLMs: ENTRAP-VL is a structured stimulus set for probing context-induced pull, explicitly separating visual and textual sources rather than measuring interpretability or distraction length.Its design targets the missing multimodal instrument for studying contextual entrainment.
- Knowledge conflict and context faithfulness: Knowledge-conflict research shows that models can accept coherent counter-evidence yet favor memory-consistent content, while multimodal models often favor parametric knowledge over external evidence.These findings motivate distinguishing contextual entrainment from broader conflict and faithfulness behavior.
- Sycophancy and hallucination probing: Multimodal sycophancy captures textual pull from asserted user opinions, whereas entrainment also includes bare entity names and irrelevant true statements without opinions.Object-hallucination probing additionally informs ENTRAP-VL’s methodology.
3 Position: VLM Contextual Entrainment Needs a Purpose-built Instrument
VLM contextual entrainment requires a purpose-built, taxonomically organized instrument rather than a relabeled unimodal benchmark because auxiliary context can pull outputs through distinct textual and visual channels. Perceptual grounding also introduces a scene-relative contradiction condition that is absent from unimodal, world-knowledge-only formulations.
- Position: The multimodal setting therefore changes contextual entrainment in ways that demand a new, taxonomically organized instrument rather than simply adding images to a text benchmark.The position is defended by the dual context channels, the scene-relative veracity distinction, and the limits of templatic construction.
- Dual-modality entrainment: VLMs require separate textual- and visual-entrainment streams because auxiliary context can enter through either modality as structurally distinct sources of pull.One stream pairs an image-based query with textual context; the mirror stream pairs a textual query with visual context.
- Scene-relative veracity: A scene referent creates three veracity levels: true of the scene, contradictory to the scene but possible in the world, and counterfactual in the world.For example, a doormat reading “departure” contradicts an image showing “welcome” while remaining globally possible, unlike a statement that the mat reduces weight when stepped on.
- Perceptual grounding: The contradictory condition is native to perceptual grounding because it tests whether injected context overrides what the model sees, not consistency with a prior textual description.A described-scene substitute would test consistency with an earlier assertion, which is a different phenomenon.
- Instrument design: Templatic generation cannot construct image-specific contradictions or scene-associated distractors because both conditions depend on the particular depicted scene.Fixed relation templates control surface form and scale easily, but they cannot express these scene-dependent conditions.
4 A Taxonomy of Dual Contextual Entrainment
The taxonomy defines contextual entrainment along independent association and veracity axes, with scene-based relatability and three graded truth relations. Applied asymmetrically across modalities, it yields eight textual conditions and three visual conditions, supporting complementary probes of dual entrainment.
- Taxonomy axes: The taxonomy’s two independent axes classify context by scene-based association—relatable or random—and by veracity—true, contradictory, or counterfactual.Relatability depends on semantic association with the depicted scene rather than the item’s nominal category; contradictory context is false of the scene but possible in the world.
- Taxonomy axes: A Taj Mahal example separates dataset category, query subject, and distractor similarity: scene-related kites, drones, and bats are relatable despite differing from “birds,” whereas fish, cats, and dogs are random.The relatable distractors share the salient property of flying, while the random distractors are taxonomically closer to birds but unrelated to the scene.
- Textual stream: The textual stream instantiates all eight combinations because text can independently vary association and veracity.Each condition contains three statements, and short forms are restricted to distractors to create a controlled contrast between scene-associated and unassociated bare triggers.
- Visual stream: The visual stream contains three conditions because imagery expresses association more readily than veracity, causing the veracity axis to collapse.This asymmetry makes the visual and textual streams complementary rather than redundant probes.
- Probe construction: Three statements per condition enable item-level entrainment estimates through within-item variance, while curation reduces surface-form and low-level image-statistics confounds.Distractors avoid subword overlap with answers, and relatable images are matched in tonal register where possible.
5 ENTRAP-VL
ENTRAP-VL is a manually curated, dual-stream dataset probing contextual entrainment across eight categories and taxonomically structured textual and visual context conditions. Its textual stream varies statements around fixed image-query pairs, while its visual stream varies candidate images around fixed world-knowledge queries.
- Dataset organization: ENTRAP-VL comprises two mirror streams spanning eight organizational categories: creatures, electronics, fruits, household_objects, humans, monuments, natural_landscapes, and vehicles.Category labels identify the principal subject but do not constrain query type or relatability.
- Textual stream: 800 textual-stream items pair each image and query with long- and short-form answers and eight conditions containing three statements each.The image is the perceptual referent, while textual context varies across fixed image-query pairs.
- Textual stream: 19,200 context-injections result from the textual stream’s 800 × 8 × 3 statement-level design.The stream contains 100 items per condition in each of the eight categories.
- Textual stream: Textual items cover diverse manually curated query types, including activity recognition, attribute identification, image text reading, incidental-element detection, and counting.This diversity is not a taxonomy axis; the taxonomy classifies context conditions rather than query types.
- Visual stream: 700 visual-stream items pair fixed world-knowledge queries with three candidate images: relatable_true, relatable_distractor, and random_distractor.The humans category is excluded, leaving seven categories with 100 items each; the image is the accompanying variable.
6 What the Instrument Enables
ENTRAP-VL provides protocols for measuring textual and visual entrainment without itself reporting model measurements. Its taxonomy supports per-condition metrics, reliability estimates, within-stream comparisons, and direct cross-stream comparisons of textual versus visual pull.
- Measurement protocols: The textual protocol compares answers under each injected context condition with no-context outputs and ground truth, while the visual protocol holds the textual query fixed and varies image conditions.A shift toward injected content constitutes the corresponding entrainment signal.
- Measurement protocols: All three visual conditions are candidate entrainment conditions, differing in the type of image content available to pull the model.relatable_true may pull through fixation on a non-answer feature, whereas distractor conditions may pull toward a competing depicted entity.
- Derived measurements: Per-item, per-condition output distributions support probabilities for injected content and ground-truth answers, plus changes relative to no-context or no-image references.Ratios or differences between these probabilities operationalize pull strength.
- Derived measurements: Three statements per textual condition enable internal reliability estimates from within-item variance, while shared three-condition structures support comparable visual measurements across categories.These structures provide repeated observations for assessing measurement consistency.
- Cross-condition comparisons: The taxonomy enables comparisons by veracity, association, and form within the textual stream, including relatable_contradictory versus relatable_counterfactual contexts.It also supports relatable_distractor versus random_distractor comparisons and their short variants.
- Cross-stream comparison: The dual structure enables a within-model comparison of whether textual and visual contexts entrain the model to comparable degrees.This directly implements the instrument’s dual-modality objective.
7 Limitations and Ethics
ENTRAP-VL is a focused, English-only diagnostic whose findings concern only the specific conditions it instantiates. Its curation involves subjective judgments, while its human-image items and responsible-use framing impose safeguards against unsupported inference and leaderboard-style interpretation.
- Scale and locale: At 1,500 items, ENTRAP-VL is a focused diagnostic rather than a large-scale benchmark, and its English-only scope limits generalization.The three-statements-per-condition design mitigates item-level size constraints but does not substitute for scale.
- Curation subjectivity and validation: Curators and leads make judgments about relatability, local versus global falsehood, and clean distractors, introducing subjectivity into dataset construction.The released artifact is checked for schema completeness, image resolution, watermarking, identifier uniqueness, and trigger-answer disjointness.
- People in the dataset: Human-category queries target directly observable actions, physical states, or occupations, avoiding person identification and inference of protected or internal attributes.Affect-adjacent wording is answered with observable action rather than inferred feeling.
- Responsible use: ENTRAP-VL is a diagnostic instrument, not a leaderboard or accuracy benchmark, and its context conditions measure pull rather than universally prescribed correctness.Reduced pull under distractors indicates the measured behavior is reduced, whereas large pull indicates the phenomenon the instrument is designed to detect.
8 Conclusion
The paper argues that contextual entrainment in vision-language models is a distinct dual phenomenon requiring a purpose-built, taxonomically structured instrument. ENTRAP-VL provides that instrument, with documentation and protocols for rigorous community study.
- Conclusion: VLM entrainment is independently drivable through visual and textual context, introducing a scene-relative veracity distinction absent from text-only formulations.The paper presents this duality as a substantive distinction motivating a dedicated multimodal instrument.
- Conclusion: ENTRAP-VL is a manually curated, dual-stream, taxonomically organized dataset designed to operationalize this position.Its structure embodies the claim that VLM entrainment requires a purpose-built probe.
- Conclusion: The dataset includes documentation and protocols needed to probe contextual entrainment rigorously.These materials are intended to support systematic investigation rather than assert results for any particular model.
- Conclusion: The authors will release ENTRAP-VL publicly so the community can investigate whether and how VLMs are entrained by context.The release is framed as enabling further study of an open question.