Source-linked AI summary
Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Models
Yuliang Cai, Mohammad Rostami, Jesse Thomason
TL;DR
VLMs can mishandle negated visual questions, answering as if negation were absent. SSP turns shared question structure into reusable skeleton-and-strategy prompts, and the paper reports consistent gains across NegVQA and DNVQA and across model scales.
Problem
VLMs can answer negated visual questions as though the negation were absent, undermining reliability on VQA involving absence, exclusion, or logical opposition.
Source: §1 · p. 1Method
SSP extracts a question skeleton, uses same-structure negative examples to generate a concise strategy, and prepends the reusable skeleton–strategy pair to the test prompt.
Source: §1 · p. 2, §3 · p. 4, §3.3 · p. 6Results
SSP consistently improves performance across VLM scales on NegVQA and DNVQA, achieving the best reported performance across evaluated models.
Source: §4.1 · pp. 7–8, §5 · p. 11Takeaways & Limitations
Caching skeleton–strategy pairs preserves comparable accuracy while substantially reducing strategy-generation overhead.
Source: §4.2 · p. 9
Abstract
from arXiv · showhide
Despite the strong performance of Vision-Language Models (VLMs) on a wide range of visual question answering (VQA) tasks, these models consistently struggle to understand negation and produce incorrect answers when questions involve negated clauses. To address this limitation, we propose Skeleton-and-Strategy Prompting (\textbf{SSP}), a training-free, in-context learning method that improves VLM negation understanding capabilities without any parameter updates. Given a negation question, our method first abstracts the underlying question structure into a skeleton, retrieves a small set of same-skeleton questions from a lightweight question pool, then prompts the VLM to analyze their shared negation pattern and synthesize a single-sentence answering strategy. The skeleton and strategy are prepended to the test sample to guide the model correctly tackle the negation problems. Experiments on multiple negation VQA benchmarks show that SSP achieves state-of-the-art performance on negation-focused VQA tasks while remaining computationally efficient.
1 Introduction
VLMs often answer negated visual questions as though negation were absent, despite strong performance on VQA. SSP addresses this gap by representing shared question structure and supplying reusable negation-aware guidance.
1 Introduction
- VLMs can respond as if negation were absent, producing incorrect answers to questions about absence, exclusion, or logical opposition.The introduction frames this as a reliability issue for visual question answering.Source: §1 · p. 1
- Prior negation work primarily targets CLIP image-text alignment, while many approaches rely on computationally expensive full-model fine-tuning.Source: §1 · p. 1
- For a negation question, SSP guides the VLM to select the answer that differs from the visual content, unlike plain and rule-based prompting.Source: Figure 1 · p. 2
- SSP abstracts a negation question into a skeleton, retrieves same-skeleton examples, and prompts the VLM to synthesize a reusable answering strategy.The skeleton and strategy are prepended to the test sample; caching allows reuse of previously calculated skeletons.Source: §1 · p. 2
2 Related Works
Prior multimodal in-context learning commonly relies on instance-level examples, while visual negation research has emphasized alignment tasks and training-based adaptation. SSP is motivated by the gap between those approaches and negation VQA, where question structure matters.
2 Related Works
- The related-work discussion places SSP at the intersection of multimodal in-context learning and VLM negation understanding, addressing limitations in both directions.Source: §2 · p. 3
2.1 In-Context Learning for Visual Question Answering
- Multimodal in-context learning adapts models at inference time, but existing approaches primarily transfer knowledge through explicit examples and leave the reasoning pattern implicit.Prior VQA prompting work includes caption-and-demonstration methods, zero-shot reasoning, and exemplar selection.Source: §2.1 · p. 3
2.2 Negation Understanding for VLM
- Negation research has documented challenges for language models, while VLM work has mainly focused on CLIP-based matching and training-based adaptation rather than reasoning-heavy VQA.The paper targets negation-aware VQA with a training-free in-context learning method.Source: §2.2 · pp. 3–4
3 Methodology
SSP builds reusable skeleton–strategy pairs for negation VQA and adds them to the original image-question prompt. Its pipeline constructs a negative example pool, extracts question structure, generates or retrieves a strategy, and conditions the target VLM on both.
3 Methodology
- SSP is a training-free inference-time framework that identifies a question type and reuses its cached skeleton and strategy when available.For a new type, it extracts a skeleton and generates a strategy using negative VQA samples from the example pool.Source: §3 · p. 4
3.1 Example Pool Construction
- The example pool derives negative question-answer pairs from VQA-v2 examples by rewriting questions into negated formats and switching the correct answer label.The pool supplies varied negation patterns and visual reasoning cases for synthesizing strategies.Source: §3.1 · pp. 4–5
3.2 Skeleton Extraction
- Skeleton extraction preserves interrogative and negation structure while replacing question-specific content with placeholders X and Y.For example, “Which animal is not standing?” becomes “Which X is not Y?”Source: §3.2 · p. 5
3.3 Strategy Generation and Caching
- For an uncached skeleton, SSP selects negative VQA pairs with a similar negation structure and has the VLM generate a one-sentence strategy, then stores the pair for reuse.Caching recurring question formats reduces redundant strategy generation and inference overhead.Source: §3.3 · pp. 5–6
- On NegVQA, SSP has the highest reported accuracy among the compared prompting methods across six VLMs and evaluated model scales.Source: Figure 3 · p. 6
3.4 Skeleton and Strategy Prompting
- The final prompt prepends the question skeleton and reasoning strategy to the original image-question pair, and the VLM answers conditioned on both.Source: §3.4 · p. 6
4 Experiments
Experiments evaluate SSP on NegVQA and DNVQA, including questions with implicit negation, and compare it with prompting baselines. Ablations examine the contributions of matched skeletons and strategies, efficiency, example-pool size, sample count, and generated strategies.
4 Experiments
- Evaluation covers NegVQA across 20 VQA datasets and a 1,000-pair DNVQA benchmark designed for implicit negation, using accuracy across six VLMs.The model set spans GPT-4o, three 8B-level models, and two 2B-level models; baselines are Plain Prompt, CoT Negation, and four attached examples.Source: §4 · pp. 6–7
4.1 SSP Outperforms Baselines on NegVQA and DNVQA
- SSP achieves the highest accuracy across evaluated models on NegVQA and DNVQA, including DNVQA questions expressing implicit negation.On DNVQA, SSP exceeds the strongest baseline for every model, by 1.93 points on InternVL3-2B to 4.29 points on Qwen3VL-2B.Source: Figure 4 · p. 7, §4.1 · pp. 7–8
- SSP outperforms Plain Prompt and attached demonstrations on NegVQA; CoT Negation is less consistent, reducing Qwen3VL-2B accuracy from 51.0 to 29.8.SSP also performs better than attaching four examples on all evaluated models.Source: §4.1 · pp. 7–8
4.2 SSP Components are Complementary
- Both skeleton and strategy contribute to performance, and randomly matched skeleton-strategy pairs score below matched SSP.Strategy Only outperforms Skeleton Only but still falls short of full SSP on InternVL3-8B and Qwen3VL-8B; the random variant scores 76.1 versus 79.1 on InternVL3-8B and 71.0 versus 75.5 on Qwen3VL-8B.Source: §4.2 · p. 9
- Reusing cached strategies changes accuracy by at most 0.4% while reducing generation time about 43 times and generated-token count about 42–45 times.For 1,000 NegVQA samples, cached strategies require 27.19–51.45 seconds and generate 423–489 tokens, versus 1,182.17–2,237.84 seconds and 18,914–20,317 tokens for per-sample generation.Source: Table 2 · p. 8, §4.2 · p. 9
4.3 Effective of Example Pool Size
- 79.1% accuracy is reached with a 256-example pool; increasing the pool to 512 provides no further benefit.Accuracy rises from 76.2% with 32 examples to 76.8% with 64 and 78.6% with 128.Source: §4.3 · pp. 9–10, Table 3 · p. 9
4.4 Effect of Sample Number
- 79.1% accuracy is reached with three sampled examples per strategy, with no further improvement from four or five.Accuracy is 77.4% with one example and 78.3% with two.Source: §4.4 · p. 10
4.5 Generated Strategy Visualization
- Skeletons replace instance-specific objects, attributes, or actions with placeholders while preserving interrogative and negation structure; strategies describe reasoning for that question type.The visualized strategies should focus on the negated condition and avoid answering the positive version.Source: §4.5 · p. 10, Table 4 · p. 10
5 Conclusion
SSP is a training-free framework that improves negation understanding in VQA. It consistently improves performance across benchmarks and model scales, with both its skeleton and strategy contributing to results.
5 Conclusion
- SSP consistently improves performance across NegVQA and DNVQA benchmarks and VLMs of different scales, including large, 8B-level, and 2B-level models.The conclusion characterizes the method as an effective and efficient direction for improving robustness on negation-based visual reasoning tasks.Source: §5 · p. 11
- Ablations show that both the skeleton and strategy components contribute to SSP’s final performance.Analyses of example-pool size and sample number also report strong results with a compact set of negative VQA examples.Source: §5 · p. 11
- SSP abstracts each question into a reusable negation-aware skeleton and associates it with a concise reasoning strategy.It caches skeleton-strategy pairs by question type to avoid repeated strategy generation and reuse lightweight guidance across test samples.Source: §5 · p. 11
- SSP is a training-free, inference-time framework for improving negation understanding in visual question answering.Source: §5 · p. 11
A.1 Strategy Extraction Pipeline
The strategy extraction pipeline uses same-skeleton negative VQA examples to elicit a concise rule for answering questions with that structure.
A.1 Strategy Extraction Pipeline
- When a skeleton is absent from the cache, the pipeline selects K negative VQA pairs sharing that skeleton and prompts the VLM to generate a one-sentence strategy.Source: §A.1 · p. 15
- Each example prompt supplies an image, a negative question, and the correct answer alongside an incorrect answer.The strategy prompt combines four such example prompts with an instruction.Source: §A.1 · p. 15
- The instruction asks for a one-sentence rule that identifies the correct option for negative VQA questions without naming example-specific objects or scenes.Source: §A.1 · p. 15
A.2 DNVQA Benchmark Construction
DNVQA evaluates implicit negation, where questions express negative intent without explicit negation words. Its construction rewrites positive-form VQA examples as soft-negation questions with alternative answers.
A.2 DNVQA Benchmark Construction
- DNVQA includes implicit-negation questions whose negative intent is conveyed without explicit markers such as “not,” “no,” or “without.”Examples include asking what is “missing” or “absent” from a table.Source: §A.2 · p. 15
- Construction starts from positive-form VQA examples and rewrites questions using soft-negative expressions while avoiding explicit negation words.The rewriting preserves the image and answer space while changing the reasoning requirement from positive recognition to implicit negation understanding.Source: §A.2 · pp. 15–16
- The rewritten question’s correct answer is an alternative candidate that fails the original positive condition but satisfies the new soft-negative question.Generated examples are retained only when they preserve visual grounding, express negative or exclusion-based intent without explicit markers, and have the required alternative correct answer.Source: §A.2 · p. 16