Source-linked AI summary
SPOOK: Sparse Pixel Optimization for Off-Manifold Knowledge in Vision Encoders
Ankit Gupta, Akshat Pandya, Christoph Adami, Emily Dolson
TL;DR
The relationship between embedding similarity and visible semantics is unclear for inputs far from the natural-image manifold. SPOOK probes this with black-box sparse edits to blank canvases, finding targeted alignment across OpenCLIP and DINOv3. The resulting model-specific triggers matter for systems that rely on fixed-encoder similarity, while their success is brittle to common transformations.
Problem
How much visible structure vision encoders need to assign a target representation, and whether similarity remains high without recognizable content, is unclear off the natural-image manifold.
Source: §1 · p. 1Method
SPOOK uses black-box, query-based pixel-channel edits from a blank canvas, accepting edits that increase similarity to a chosen target.
Source: §1 · p. 2Results
Takeaways & Limitations
Model-specific triggers can induce targeted failures in fixed-encoder systems for retrieval, filtering, indexing, or nearest-neighbor classification.
Source: §5 · p. 10SPOOK retrieval is model-specific and brittle to compression, cropping, blur, and resizing, so exact-input success does not establish robustness.
Source: §4.4 · p. 9
Abstract
from arXiv · showhide
Modern vision encoders are widely used as semantic representations, but the relationship between embedding similarity and human-visible semantics can become unclear far from the natural-image manifold. We probe this regime in a black-box setting by applying sparse pixel edits to blank canvases with a simple pixel-level hill-climbing procedure. Across ImageNet and Flickr30k, we evaluate whether these edited blank canvases can retrieve chosen targets in multimodal contrastive encoders and self-supervised visual encoders. On OpenCLIP, inputs optimized toward class prompts achieve strong top-1 and top-5 targeted retrieval performance; on DINOv3, inputs optimized toward class prototypes achieve high target alignment and near-top prototype ranks. These results show that sparse edits on otherwise blank, visually non-semantic inputs can enter target semantic basins in both language-aligned and self-supervised visual representation spaces. Our findings identify a shared off-manifold failure mode of modern vision encoders and suggest sparse blank-canvas optimization as a simple probe of representation geometry.
1 Introduction
Embedding similarity is routinely treated as semantic evidence, but its meaning for nearly blank, off-manifold inputs is unclear. The paper introduces a black-box sparse-edit probe and finds that such inputs can achieve target alignment across modern vision encoders.
1 Introduction
- SPOOK starts from a blank canvas and accepts random pixel-channel edits only when they increase similarity to a chosen target, without model weights or gradients.Targets include class prompts, caption prototypes, clean-image embeddings, and DINOv3 image-class prototypes; evaluation asks whether the intended target outranks competitors.Source: §1 · p. 2
- Across OpenCLIP and DINOv3, sparse non-semantic inputs achieve high targeted retrieval or alignment, sometimes matching or exceeding clean-image similarity while changing under 1% of pixel-channels.Figure 1 illustrates the effect for “Irish water spaniel”: optimized canvases score above natural examples under their corresponding encoders.Source: §1 · p. 2, Figure 1 · p. 2
- Pixel interpolation is non-monotonic: target similarity rises with sparse edits, drops sharply when the SPOOK image is mixed with a clean image, then recovers near the clean endpoint.This suggests sparse and clean images reach target alignment through distinct pixel-space routes.Source: §1 · p. 2
- Controls indicate that the effect requires target-specific optimization, is strongest from low-entropy initialization, and persists across CLIP patch sizes and DINOv3 ViT scales.The findings distinguish optimized inputs from blank canvases or random sparse pixels.Source: §1 · pp. 2–3
2 Related Work
Prior work studies model failures on natural-image perturbations, classifier-fooling inputs, and broad CLIP prompt similarity. SPOOK instead tests targeted retrieval from sparse blank canvases, including in image-only representations.
Embedding geometry and off-manifold behavior
- The study spans multimodal CLIP-style embeddings and self-supervised image-only representations, which support downstream semantic tasks through different routes [1] [2] [3] [4] [5].The modality-gap literature describes systematically different regions for image and text embeddings in contrastive models [22].Source: §2 · p. 3
Adversarial examples, patches, and sparse perturbations
- Unlike attacks that perturb an existing natural image, SPOOK synthesizes a sparse non-semantic image to win targeted retrieval against competing embeddings.Prior settings include adversarial perturbations [6] [7] [12], sparse pixel-constrained attacks [16], and adversarial patches [8].Source: §2 · p. 3
Fooling images and sparse blank-canvas attacks
Fooling contrastive image–text models
Embedding geometry and off-manifold behavior
- Weak transfer between CLIP and DINOv3 and non-monotonic interpolation suggest sparse and natural images can reach similar target alignment through distinct routes.This places SPOOK among work showing that high confidence or embedding similarity need not imply human-recognizable semantic content.Source: §2 · pp. 3–4
3 Problem Setup and Method
SPOOK optimizes a blank canvas toward a chosen target embedding using sparse pixel-channel edits and cosine-based retrieval. It uses target-specific embeddings and evaluates both target rank and edit sparsity.
3.1 Targeted sparse embedding spooking
- Targets include CLIP class-caption embeddings, Flickr30k caption prototypes, and DINOv3 ImageNet class prototypes formed by averaging 50 validation-image embeddings per class.Flickr30k prototypes average the five caption embeddings associated with each target image.Source: §3.1 · p. 4
3.2 Target objectives
- CLIP text or image targets and DINOv3 image prototypes use cosine-similarity objectives between normalized embeddings.For DINOv3, targets are image-class prototypes; the listed ablation also considers an equally weighted CLIP joint objective.Source: §3.1 · p. 4, §3.2 · p. 5
3.1 Targeted sparse embedding spooking
- For a 224 × 224 RGB image, a few hundred edited pixel-channels modify substantially less than 1% of the input.The full input contains 150,528 pixel-channels.Source: §3.1 · p. 5
3.3 Sparse hill-climbing
- Each target is optimized independently by sampling an unedited pixel-channel and replacement value, accepting only edits that improve the objective.Accepted edits remain and rejected edits are discarded, yielding sparse images using only query access to the target similarity score.Source: §3.3 · p. 5
- Evaluation reports target cosine, ASR@k, target rank, and pixel-channel sparsity, with ranks computed among 1,000 targets.Pixel edits are percentages of the 224 × 224 × 3 input channels; cosine, rank, and edit rows are reported as mean / median.Source: §3.3 · pp. 5–6, §3.1 · p. 5
- Table 1 uses white-canvas initialization, a 1,000-target retrieval bank, and ViT-B/16 encoders across its settings.Bold cosine entries identify the highest target cosine among blank, SPOOK, and clean images for each setting.Source: Table 1 · p. 6
4 Experiments and Results
Across three CLIP and DINOv3 settings, sparse edits turn blank canvases into strong target matches, while interpolation reveals a distinct route to alignment from clean images. Controls show that results depend on target-specific optimization and initialization, and transfer and transformation brittleness constrain their scope.
4.1 Main targeted retrieval results
- A few hundred accepted pixel-channel edits yield near-top retrieval among 1,000 targets while modifying well below 1% of the input across all three settings.CLIP caption-target images can reach target cosine comparable to or higher than clean references with under 0.3% of pixel-channels changed on average; DINOv3 also achieves high retrieval success.Source: §4.1 · p. 6, Figure 2 · p. 7
4.2 Interpolation analysis: sparse and clean images follow distinct routes to target alignment
- Target similarity rises from blank to SPOOK, often drops sharply along SPOOK-to-clean mixtures, then recovers near the clean endpoint.This non-monotonic profile suggests that sparse SPOOK images and clean images reach target alignment through distinct pixel-space routes.Source: §4.2 · p. 7, Figure 3 · p. 8
4.3 Internal controls and ablations
- Across DINOv3 scales, spookability persists but success rates vary, while OpenCLIP ViT-B/16 and ViT-B/32 both achieve high targeted retrieval.The results span architectures and scales rather than a single checkpoint, with DINOv3 variation indicating dependence on representation details.Source: §4.3 · p. 8, Table 3 · p. 9, Table 2 · p. 9
- Matched-edit random sparse images remain near chance with median ranks near the middle of the 1,000-target bank, while SPOOK achieves median rank 1 in all three settings.Because the random control matches the number of edited pixel-channels, the contrast supports target-specific optimization as essential.Source: §4.3 · p. 8
- White canvases give the strongest results, black canvases remain partially spookable, and dense random initialization nearly eliminates the effect.The authors interpret this as sparse target-aligned signal being overwhelmed by a dense random background.Source: §4.3 · p. 8
- CLIP caption targets are easiest to spook; image targets are harder, while joint objectives are intermediate on ImageNet and nearly match caption-only performance on Flickr30k.The authors suggest text targets define broader or more accessible target basins than individual image embeddings.Source: §4.3 · p. 8
4.4 Transfer, SPOOK robustness, and experiment runtime
- SPOOK images transfer weakly across models and are brittle to compression, cropping, blur, and resizing, so exact-input success is not evidence of robustness.The main 1,000-target runs take roughly 2–3 hours on one NVIDIA H200 GPU despite a 50,000-query budget per target.Source: §4.4 · p. 9
5 Conclusion
SPOOK produces sparse, visually non-semantic inputs that align with chosen targets in OpenCLIP and DINOv3. The conclusion is that embedding alignment can diverge from visible semantic content, with implications for systems relying on fixed encoders.
5 Conclusion
- SPOOK’s sparse, colored-dot images achieve high targeted retrieval success in OpenCLIP and DINOv3 across ImageNet and Flickr30k.They retrieve class prompts, caption prototypes, and image prototypes while modifying less than 1% of pixel-channels.Source: §5 · p. 9
- Embedding-space target alignment can be decoupled from human-visible semantic content, and weak cross-model transfer suggests these are largely model-specific triggers.For fixed-encoder retrieval, filtering, indexing, or nearest-neighbor classification, such triggers are sufficient to induce targeted failures.Source: §5 · p. 10
- SPOOK and clean images can reach similar target alignment through distinct routes: sparse off-manifold features versus natural visual content.The paper presents sparse blank-canvas optimization as a probe for this gap in embedding similarity.Source: §5 · p. 10
A Technical Appendices and Supplementary Material
The appendix supplements the main paper’s targeted-retrieval evidence with analyses intended to help interpret SPOOK across datasets, encoders, and target types.
A Technical Appendices and Supplementary Material
- The appendix adds qualitative examples, diagnostic visualizations, and auxiliary measurements to complement the main experimental evidence.These materials examine method behavior across datasets, encoders, and target types.Source: A Technical Appendices and… · p. 13
A.1 Additional experimental setup details
The experiments test three targeted-retrieval settings across OpenCLIP and DINOv3, using specified target banks, default inputs, and optimization budgets. Additional setup defines reference scores, pixel-channel sparsity, controls, and ablations.
A.1 Additional experimental setup details
- The main settings are CLIP/ImageNet, CLIP/Flickr30k, and DINOv3/ImageNet, with 1,000 targets, 224 × 224 RGB inputs, white-canvas initialization, and 50,000 candidate edits per target by default.OpenCLIP uses ViT-B/16 with its text encoder, and DINOv3 uses ViT-B/16; backbone and scale variants are tested in ablations.Source: §A.1 · p. 13
- ImageNet CLIP targets are class-prompt embeddings, DINOv3 targets are class image prototypes, and Flickr30k targets are caption prototypes formed from normalized caption embeddings.Auxiliary Flickr30k experiments also use paired image embeddings as targets.Source: §A.1 · p. 13
- Clean and blank reference scores are defined by target type, comparing the unmodified white canvas or corresponding clean images with the target embedding.For Flickr30k, the clean score uses the paired image and caption prototype; for ImageNet, it averages class-image similarity to the prompt or prototype.Source: §A.1 · p. 13
- Sparsity counts edited pixel-channels, and each target is optimized independently by retaining only edits that increase its target objective.A 224 × 224 RGB image has 150,528 editable channels; accepted edits remain permanent, making each target trajectory monotonic in optimized score.Source: §A.1 · p. 13
- Controls match each target’s edit count with random edits, while ablations vary initialization, CLIP objective or backbone, and DINOv3 scale; reported results aggregate five seeds unless stated otherwise.Transfer tests evaluate optimized images on another encoder without further optimization, and robustness tests apply image transformations while retaining the original retrieval bank.Source: §A.1 · pp. 13–14
A.2 Qualitative SPOOK Examples
Figures 8–10 provide qualitative examples across the main experimental settings, pairing each target’s natural reference and blank initialization with its optimized SPOOK image. The examples show sparse, non-natural inputs attaining high model-specific target alignment.
A.2 Qualitative SPOOK Examples
- Figures 8–10 show representative qualitative examples for the main experimental settings, with each row comparing images for the same target.The three images are a natural reference, white initialization canvas, and optimized SPOOK image.Source: §A.2 · p. 14
- The score below each image is the target cosine similarity for its model and dataset condition.Source: §A.2 · p. 14
- The optimized examples are visually sparse and non-natural to human observers despite achieving high target alignment in the evaluated model’s embedding space.The examples support interpreting the images as exploiting model-specific sparse directions rather than human-semantic visual content.Source: §A.2 · p. 14
A.3.1 Interpolation paths reveal distinct sparse and natural routes
Interpolation analyses compare target similarity along paths connecting the white canvas, optimized SPOOK image, and real target image. Their profiles indicate that sparse and natural inputs can reach target alignment through distinct pixel-space routes.
A.3.1 Interpolation paths reveal distinct sparse and natural routes
- Figure 3 summarizes target-cosine profiles along interpolation paths among the white canvas, SPOOK image, and corresponding real target; Figures 5–7 show example grids.Source: §A.3.1 · p. 14
- Target similarity generally rises from white canvas to SPOOK image, then often drops sharply when SPOOK is mixed with the real image before recovering near the natural endpoint.The direct white-to-real path supports that the SPOOK endpoint is not merely an intermediate on ordinary interpolation to a natural image.Source: §A.3.1 · p. 14
- The non-monotonic profiles suggest sparse off-manifold and natural-image routes to target alignment, without implying that SPOOK images contain hidden human-recognizable semantics.The visualizations complement the main interpolation analysis in Figure 4.Source: §A.3.1 · p. 14
A.3.2 Cross-model transfer: SPOOKs are model-specific
Cross-model transfer is weak both across encoder families and between CLIP backbones, indicating that SPOOK images are model-specific triggers rather than model-independent semantic depictions. The stated threat model is targeted retrieval against a fixed deployed encoder.
A.3.2 Cross-model transfer: SPOOKs are model-specific
- Table 4 covers runtime for three main settings, each run using 1,000 targets and 50,000 candidate pixel-channel queries per target on one NVIDIA H200 GPU.Runtime per query per target is defined by dividing total runtime by 50,000 × 1,000 candidate evaluations.Source: Table 4 · p. 15, §A.3.2 · p. 15
- DINOv3 ViT-B/16 SPOOKs achieve 85.5% ASR@1 on their source model, but only 0.2% ASR@1 and 0.8% ASR@5 on CLIP ViT-B/16.Conversely, CLIP ViT-B/16 SPOOKs achieve 0.6% ASR@1 and 2.1% ASR@5 on DINOv3 ViT-B/16.Source: §A.3.2 · p. 15
- Transfer is also weak between CLIP backbones: ViT-B/16 SPOOKs reach 1.1% ASR@1 and 3.9% ASR@5 on ViT-B/32, while ViT-B/32 SPOOKs reach 0.4% ASR@1 and 2.0% ASR@5 on ViT-B/16.The paper characterizes these images as model-specific off-manifold embedding triggers, not model-independent semantic depictions.Source: §A.3.2 · p. 15
- A model-specific trigger can induce targeted retrieval failures in systems that deploy a fixed encoder, without needing to be a universal semantic image.Source: §A.3.2 · p. 15
A.3.3 Robustness to Simple Image Transformations
SPOOK images perform well in the untransformed targeted-retrieval setting, but are brittle to common image transformations. The discussion distinguishes encoder-measured target alignment from visible semantics and describes its mechanisms, scope, limitations, and possible next steps.
A.3.4 Runtime analysis
- The main 1000-target experiments complete in roughly 2–3 GPU-hours on a single NVIDIA H200 GPU using batched optimization.Each run uses 50,000 candidate pixel-channel queries per target.Source: §A.3.4 · p. 15
A.4.1 Embedding similarity is an operational score, not a guarantee of visual semantics
- Embedding similarity measures encoder alignment, not guaranteed human-visible semantics; off-manifold inputs can rank highly without recognizable target evidence.The authors frame this as a failure of treating raw embedding similarity as an unqualified semantic score, not conventional misrecognition of a natural image.Source: §A.4.1 · pp. 15–16, §A.4 · p. 15
A.4.2 Why low-entropy canvases matter
- White and black low-entropy canvases let accepted edits define most nontrivial image content, whereas dense random backgrounds largely erase the effect.SPOOK constructs a targeted sparse signal through cumulative accept/reject optimization, rather than assigning semantics to arbitrary dense noise.Source: §A.4.2 · p. 16
A.4.3 Model-specific triggers and the threat model
- Weak cross-model transfer makes SPOOK images model-specific embedding triggers, with the most direct risk in exact-input pipelines relying on one deployed encoder.The discussion names image search, prompt-based filtering, curation, deduplication, indexing, and nearest-neighbor classification as relevant settings.Source: §A.4.3 · p. 16
A.4.4 Relation to interpolation
- Target similarity rises from blank canvas to SPOOK image, drops sharply in SPOOK–clean mixtures, then recovers near the clean endpoint, indicating distinct pixel-space routes.The authors clarify that this does not mean sparse dots contain hidden human-recognizable semantics.Source: §A.4.4 · p. 16, Figure 4 · p. 17
A.4.5 Limitations
- SPOOK is strongest when the optimization and evaluation encoders match; its images are also transformation-brittle, query-intensive, and visually non-natural.The authors say these constraints narrow the immediate threat model but do not remove the central finding.Source: §A.4.5 · pp. 16–17
A.4.6 Future directions
- Future directions include stronger priors for natural-looking inputs, systematic transfer studies, and defenses such as multi-encoder agreement or semantic consistency checks.The authors also caution against using raw embedding similarity as the sole decision rule in high-stakes filtering or indexing.Source: §A.4.6 · p. 17
A.3.3 Robustness to Simple Image Transformations
- Mild JPEG compression or small center crops sharply reduce ASR@1 and ASR@5 across CLIP and DINOv3, so untransformed retrieval success does not establish transformation robustness.The test applies JPEG compression, center cropping, Gaussian blur, and down-up resizing to SPOOK images for a fixed random subset of 64 targets per condition.Source: §A.3.3 · p. 15, Table 9 · p. 26