PaperlayerView exact revision on arXiv

Source-linked AI summary

TDDN: Text-aligned Diffused DINO Network for Puzzle Understanding

Harsha Patnala, Debopriyo Banerjee, Ayush Sunil Munot, Somak Aditya

arXiv:2609.07937v1cs.CVcs.AI

TL;DR

Structured visual reasoning requires fine-grained perception that current VLMs often lack, in part because CLIP-based encoders trade local detail for semantic alignment. The paper fuses DINOv3 and CleanDIFT into DiffusedDINO and aligns it with RoBERTa-L to create TDDN; TDDN matches CLIP on retrieval while outperforming it on dense prediction.

  • Problem

    VLMs often lack the fine-grained visual perception required for structured reasoning, while CLIP-based encoders trade local spatial detail for semantic alignment.

    Source: §1 · p. 2

  • Method

    The paper fuses DINOv3 and CleanDIFT into DiffusedDINO, then aligns it with RoBERTa-L using frozen backbones and lightweight trainable components.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 3

  • Results

    TDDN outperforms CLIP on every reported segmentation benchmark, including ADE20K (18.11 vs. 5.20 mIoU) and Puzzle Perception (22.51 vs. 11.04).

    Source: §5.2 · p. 7

  • Takeaways & Limitations

    TDDN-predicted regions provide spatial signals that improve puzzle-based visual question answering in frozen VLMs without weight updates.

    Source: §1 · p. 2

Abstract

from arXiv · show

Structured visual reasoning, such as image puzzles, demands fine-grained visual perception, an ability current Vision Language Models (VLMs) lack. VLMs built on CLIP-based ViT backbones trade fine-grained detail for high-level semantics, and we show this loss propagates downstream. To recover it, we fuse DINOv3 and CleanDIFT representations into a perception encoder (DiffusedDINO) and align it with RoBERTa-L, yielding a text-aligned model TDDN that preserves this perceptual advantage: with frozen backbones and only $\sim$590K alignment pairs, TDDN matches CLIP on image-text retrieval, surpassing it on three of four settings. It does so while more than tripling CLIP's dense-prediction accuracy (ADE20K 5.20 $\to$ 18.11 mIoU, COCO-Stuff 7.35 $\to$ 24.44), despite CLIP's massive training corpus. TDDN leads on segmentation benchmarks among general-purpose contrastive encoders, including SigLIP$\,$2. We further introduce Puzzle Perception, a segmentation and visual question answering dataset that probes fine-grained spatial understanding, on which TDDN doubles CLIP's segmentation accuracy (11.04 $\to$ 22.51 mIoU).

1 Introduction

The paper targets weak fine-grained visual perception in VLMs as a bottleneck for structured reasoning. It introduces a text-aligned fusion of DINOv3 and CleanDIFT that retains dense-perception advantages while supporting language alignment.

1 Introduction

  • CLIP-based vision encoders trade local spatial detail for semantic alignment, and VLMs can hallucinate on orientation, counting, and viewpoint.Scaling ViT capacity and data leaves seven of nine reported failure patterns unresolved.Source: §1 · p. 2
  • The proposed fusion draws on complementary features: self-supervised representations provide globally coherent semantics, while diffusion features resolve fine spatial boundaries.Prior work cited by the paper found simple fusion of Stable Diffusion and DINOv2 features outperformed either representation alone.Source: §1 · p. 2
  • With frozen backbones and ∼590K alignment pairs, TDDN matches CLIP on image-text retrieval while substantially surpassing it on dense prediction.The fused encoder combines CleanDIFT boundary detail with DINOv3 tokens; the paper also reports improved open-vocabulary segmentation and keypoint matching.Source: §1 · p. 2
  • TDDN-predicted regions improve puzzle-based visual question answering in frozen VLMs without weight updates.The paper frames these regions as actionable spatial signals for downstream puzzle solving.Source: §1 · p. 2

2 Related Work

Prior work documents persistent spatial-understanding failures in VLMs and explores visual encoders, low-resource alignment, and task-specific datasets. The paper positions its work at the intersection of these strands.

Spatial Understanding Failures

  • Visual puzzle benchmarks expose spatial failures across puzzle types and planning settings, with evaluated models remaining below human performance.The cited work suggests these failures can propagate to higher-level reasoning and motivates improving visual representations.Source: §2 · p. 2

Visual Encoders

  • CLIP’s global image-text objective lacks fine-grained spatial supervision; prior approaches add hard negatives, local alignment, diffusion augmentation, or segmentation encoders.Self-supervised encoders improve patch-level semantics, while diffusion features provide complementary dense geometric correspondences.Source: §2 · p. 3

Vision-Language Alignment in low-resource condition

  • Low-resource alignment work uses frozen encoders and lightweight components, including Q-Formers and methods that align language to self-supervised backbones.The paper follows this line by freezing its visual and text backbones and training lightweight attention and projection layers.Source: §2 · p. 3

Datasets for Visual Understanding

  • Existing visual-understanding datasets separate pixel-level segmentation from compositional or puzzle-based question answering, limiting joint evaluation.Puzzle Perception combines segmentation and visual question answering in puzzle environments.Source: §2 · p. 3

3 Methodology

DiffusedDINO fuses frozen CleanDIFT and DINOv3 features to combine spatial precision with semantic structure. TDDN aligns this representation with text using frozen backbones and lightweight trainable components.

3.1 Vision-Language Alignment

  • TDDN pairs DiffusedDINO as its vision encoder with frozen RoBERTa-L, training lightweight heads, MLP, and attention blocks for language grounding.The design aims to protect fine-grained sensitivity while adding language alignment.Source: §3.1 · p. 3

3 Methodology

  • DiffusedDINO combines CleanDIFT’s fine spatial detail and object boundaries with DINOv3’s semantically organized, patch-consistent features.Both pretrained backbones are frozen, restricting learning to the fusion mechanism.Source: §3.1 · p. 3, §3 · p. 3

3.1 Vision-Language Alignment

  • The alignment objective combines InfoNCE with a STRUCTURE regularizer that preserves pretrained latent-space geometry under low-data conditions.The regularizer is anchored on frozen DINOv3 features, while the objective and text encoder remain fixed across alignment conditions.Source: §3.1 · p. 4

3 Methodology

  • Table 1 summarizes Puzzle Perception statistics for structured visual reasoning tasks with segmentation and puzzle-based visual question answering.The caption specifies eight questions for Chess and four for N-Queens under the PVQA task.Source: Table 1 · p. 5

4 Datasets

The paper evaluates TDDN across vision-language alignment, segmentation, classification, keypoint matching, and puzzle-based VQA. It introduces Puzzle Perception to test dense segmentation and question answering in structured visual domains.

4.1 Existing Datasets

  • TDDN training and evaluation cover vision-language alignment, segmentation, classification, keypoint matching, image-text retrieval, and puzzle-based VQA.Alignment uses approximately 590K image-caption pairs: MS-COCO-2014 captions and aesthetic-filtered LAION-5B images recaptioned with Gemma-3-27B-it.Source: §4 · p. 5, §4.1 · p. 5
  • Evaluation spans public benchmarks for classification, image-caption retrieval, segmentation, and keypoint matching, using official splits.The listed benchmarks include Food-101, Flickr30K, ADE20K, and SPair-71k.Source: §4.1 · p. 5

4.2 Puzzle Perception

  • Puzzle Perception combines dense segmentation and puzzle-based VQA across Maze, Chess, Tower of Hanoi, and N-Queens.Images are 512×512 with 30 pixel-level classes; PVQA targets Chess and N-Queens.Source: §4.2 · p. 5

5 Experiments

The experiments test whether fused DINOv3 and CleanDIFT features improve visual representation quality, retain their advantages after text alignment, and support puzzle understanding. TDDN surpasses CLIP on segmentation benchmarks and retrieval in three of four settings, while its predicted regions improve puzzle-question accuracy for frozen VLMs with headroom.

5 Experiments

  • The experiments assess frozen representations on Puzzle Perception segmentation, SPair-71k keypoint matching, and ImageNet-1k classification, alongside alignment benchmarks and region-guided VLM tests.Representation comparisons include CLIP-ViT-L/14, DINOv2/v3, Stable Diffusion, CleanDIFT, and text-aligned DINOv3; the representation tasks use linear probes, nearest-neighbor matching, and k-NN.Source: §5 · p. 5, §5.1 · p. 5, §5.2 · pp. 6–7

5.1 Representation Quality Analysis

  • +3.37 mIoU, +4.16 PCK, and +2.98 classification: TDDN exceeds CLIP ViT-L/14 across the three evaluated perception tasks.DDN′ also exceeds the strongest single-encoder baseline by +1.95 mIoU and +3.19 PCK over CD′, and +0.69 classification over DN.Source: Table 2 · p. 6, §5.1 · p. 6
  • At patch level, DDN′ is closest to CD′ (CKA 0.96, PWCCA 1.00), while globally it is closest to DN (CKA 0.81, PWCCA 0.99).The analysis attributes patch structure to CD′ and global semantics to DN; DDN′ also has the best joint alignment-uniformity position, at 0.79 and −2.12.Source: §5.1 · p. 6, Figure 3 · p. 6

5.2 Vision–Text Alignment

  • 18.11 vs. 5.20 mIoU on ADE20K: TDDN outperforms CLIP on every segmentation benchmark and matches or exceeds SigLIP 2 on all five.TDDN also leads on Cityscapes, COCO-Stuff, and SPair-71k against FG-CLIP 2, which performs better overall using a separately trained dense-feature head.Source: Table 3 · p. 7, §5.2 · p. 7
  • TDDN outperforms CLIP ViT-L/14 on three of four retrieval settings, including COCO T2I (24.0 vs. 18.53), but trails on Flickr30K I2T (85.3 vs. 87.7).The reported comparison is despite 590K versus 400M training pairs.Source: §5.2 · p. 7, Table 3 · p. 7

5.3 Region-guided VLM Perception with TDDN

  • TDDN-predicted regions recover a substantial portion of oracle CRG gains when puzzle-question accuracy improves, especially for models with headroom.On Chess, Qwen2.5-VL-7B gains +5.6 with oracle regions and +3.8 with TDDN regions; near-ceiling Gemma-4-31B changes by +1.5 and −0.4, respectively.Source: §5.3 · p. 8, Table 4 · p. 8

6 Conclusion

TDDN combines CleanDIFT and DINOv3 in DiffusedDINO, then aligns the representation with language while retaining its fine-grained perceptual strengths. The notation distinguishes text-aligned models and representations, feature granularity, and features before versus after fusion.

6 Conclusion

  • TDDN pairs CleanDIFT and DINOv3 in DiffusedDINO, then aligns the fused representation with language to retain its fine-grained perceptual advantage.The conclusion also reports introducing Puzzle Perception and using Contrastive Region Guidance to improve frozen VLM performance on puzzle questions.Source: §6 · p. 8

A.1 Notation

  • The notation distinguishes patch, global CLS, and patch-mean representations, as well as features before fusion and after its self-attention blocks.The subscripts v and t identify visual and text modalities, respectively.Source: §A.1 · p. 12
  • TDDN denotes text-aligned DDN, while TDN denotes text-aligned DN without feature fusion.Source: §A.1 · p. 12

A.2 Construction of Vision Representations

The vision representations fuse spatially aligned CleanDIFT and DINOv3 features, while alignment refines frozen visual and text representations with trainable attention blocks. The appendices specify shared training settings and the procedures used to evaluate segmentation.

A.2 Construction of Vision Representations

  • With patch size 16, an H×H input yields an N×N grid where N=H/16; diffusion components are projected to a common dimension before fusion.For vision-only representations the projections use PCA; during text alignment, learned MLP projection heads replace them.Source: §A.2 · p. 12
  • Fusing CleanDIFT decoder layers 2+5+8 outperforms every individual layer on SPair-71k keypoint matching.The reported metric is keypoint-weighted PCK@0.1 on bounding-box crops.Source: Table 5 · p. 12, §A.2 · p. 13

A.3 Text Alignment Architecture for TDN

  • TDN refines DINOv3 features with two trainable self-attention blocks and forms its image embedding from global and mean-pooled patch representations.Its text branch refines frozen RoBERTa-L token representations, masked-mean-pools them, and applies a trainable linear projection.Source: §A.3 · p. 13

A.4 Additional Vision–Language Alignment Details

  • TDN and TDDN share the alignment pipeline; symmetric InfoNCE aligns image and text embeddings, while STRUCTURE preserves pretrained neighborhood structure using frozen DINOv3 as its geometric anchor.Source: §A.4 · p. 13

B.1 Implementation of Vision-Language Alignment

  • Alignment uses frozen pretrained backbones and trains the vision/text modules, with approximately 80M trainable parameters and about 15 hours of training on four A100 GPUs.TDDN extracts CleanDIFT activations from decoder layers 2, 5, and 8 and resamples them to a 21×21 grid.Source: §B.1 · p. 14

B.2 Open-Vocabulary Segmentation

  • Open-vocabulary segmentation assigns each patch its highest-similarity class using class-name text embeddings; MaskCLIP readout is used for models without supervised final patch representations.FG-CLIP 2 instead uses its dedicated dense-feature head, which adds a task-specific component unavailable to the other models in this comparison.Source: §B.2 · pp. 14–15

B.3 Zero-shot and Few-shot Experiments

The experiments describe zero-shot and few-shot classification procedures, including richer class descriptions and a nested pool of labeled examples. They also specify a Contrastive Region Guidance evaluation that uses predicted or oracle regions to steer frozen VLMs on puzzle questions.

B.3 Zero-shot and Few-shot Experiments

  • Zero-shot classification averages text embeddings from 80 standard OpenAI prompt templates into class prototypes and predicts by cosine similarity to the image embedding.Source: §B.3 · p. 15
  • CuPL supplies 50 generated visual descriptions per class as richer text supervision, while Tip-Adapter combines cache-based scores from labeled image embeddings with text logits.The few-shot cache uses K∈{1,2,4,8,16}; its lower-shot subsets are nested within higher-shot settings.Source: §B.3 · p. 15

B.4 Contrastive Region Guidance (CRG) Experiment

  • CRG evaluates multiple-choice perception questions on Chess and N-Queens, excluding Maze because its fixed start marker and dominant walls and paths make region guidance uninformative.The evaluation uses 800 Chess boards across eight questions and 100 N-Queens boards across four questions.Source: §B.4 · p. 16
  • CRG compares the original image with conditions that black out either the oracle region or a TDDN-predicted region, and reports macro-averaged question accuracy and changes relative to the original.Because CRG requires white-box access to logits, the evaluation is limited to open-weight VLMs.Source: §B.4 · p. 16

C.1 Global-Level Representation Analysis

At the global level, the fused representation remains closest to DINOv3, while patch-level features are closest to CleanDIFT. Puzzle visualizations show that fusion combines boundary precision with coherent object regions, with TDDN retaining tile separation on Tower of Hanoi inputs.

C.1 Global-Level Representation Analysis

  • DDN′ has global CKA 0.81 and PWCCA 0.99 with DN, versus 0.61 and 0.90 with CD′; at patch level it is closest to CD′ (CKA 0.96, PWCCA 1.00).The analysis describes CD′ as supplying fine-grained patch structure and DN as anchoring global semantics.Source: §C.1 · p. 16
  • DDN′ achieves 83.80 ImageNet-1k k-nearest-neighbor top-1 accuracy versus CLIP’s 73.10, and TDDN retains 76.08 after text alignment.The analysis says DDN′ closely tracks DN on Alignment and Uniformity, with lower values better on both axes.Source: §C.1 · p. 16

C.2 PCA Visualization of additional Puzzles

  • On maze, chess, and Tower-of-Hanoi visualizations, DDN combines CD’s sharp boundaries with DN’s coherent object signal; TDDN preserves distinct Tower-of-Hanoi tiles.The figure visualizes patch features by projecting them onto the top three principal components and mapping the result to RGB.Source: §C.2 · p. 16, Figure 5 · p. 17

C.3 Zero-shot and Few-shot experiments

Few-shot adaptation recovers most of TDDN’s zero-shot classification deficit, while its limited alignment-data diversity constrains broad category coverage. Seen-class analysis tests whether performance differs for classes represented in the alignment corpus.

Zero-shot classification

  • TDDN scores 78.40 versus CLIP’s 77.79 on CIFAR-100, but 9.4 versus 39.0 on GTSRB; larger pretraining corpora retain an advantage on three other benchmarks.The paper attributes the broad-class gap to limited diversity in TDDN’s 590K alignment pairs.Source: §C.3 · p. 18

Seen-class analysis

  • Seen classes require every content-word lemma in a class name to appear at least 10 times in the alignment corpus; top-1 accuracy is reported with CuPL prompts.Class names and captions are lemmatized and indexed to identify represented concepts.Source: §C.3 · p. 18, Table 10 · p. 18

Few-shot adaptation

  • With K=16, TIP-Adapter lifts TDDN to 86.8 on CIFAR-100, 86.2 on Food-101, 90.5 on Caltech-101, and 36.3 on GTSRB, surpassing CLIP on two tasks.The few-shot setting uses template-based prompts.Source: §C.3 · p. 18, Table 11 · p. 18
  • A small amount of task-specific supervision closes most of the gap, supporting limited alignment-data diversity—not visual representation quality—as the primary zero-shot classification constraint.The cited few-shot results support the paper’s data-coverage hypothesis.Source: §C.3 · p. 18

C.4 Region-guided VLM Perception with TDDN

Contrastive Region Guidance improves puzzle-question accuracy across diverse VLMs, but gains vary by model and task. TDDN-predicted regions track oracle guidance, making the VLM’s ability to use region information a key limitation.

C.4 Region-guided VLM Perception with TDDN

  • Guidance gains depend on both model and task: Gemma-4-12B-it is flat on N-Queens, while Qwen3.6-35B-A3B degrades on Chess but gains +7.0 on N-Queens.The Chess changes are −0.8/−1.2 for Gemma and −2.1/−3.8 for Qwen3.6-35B-A3B; its N-Queens gains are +7.0/+8.8.Source: §C.4 · p. 18
  • TDDN guidance tracks oracle guidance: it recovers substantial gains, sometimes matching or exceeding them when oracle regions help, while neither helps when oracle guidance fails.The authors identify VLM ability to exploit region-level information, rather than TDDN region quality, as the main limitation.Source: §C.4 · p. 18
  • CRG improves performance across diverse VLMs, with the largest oracle gains reaching +6.4 on Chess and +7.0 on N-Queens.Qwen2.5-VL-7B-Instruct and Qwen3.6-27B improve on both splits.Source: §C.4 · p. 18, Table 12 · p. 19

D.1 Training and Evaluation Data

The paper pairs frozen-backbone alignment and standard benchmark evaluations with Puzzle Perception, a benchmark combining segmentation and puzzle understanding. Its appendices document data variation, model resources, prompting, and qualitative CRG outcomes.

D.1 Training and Evaluation Data

  • Puzzle Perception uses noise-free, pixel-level masks and Chess and N-Queens question–answer pairs for CRG; alignment uses about 591K recaptioned image–text pairs without puzzle-specific supervision.Downstream evaluations follow standard benchmark train/test splits.Source: §D.1 · p. 19, Table 13 · p. 20

D.2 Puzzle Perception vs Other Datasets

  • Puzzle Perception uniquely combines dense segmentation with VQA-based puzzle understanding, unlike natural-image segmentation datasets or puzzle VQA datasets alone.The unified benchmark supports evaluating fine-grained grounding and puzzle understanding on the same structured scenes.Source: §D.2 · p. 19, Table 14 · p. 20

D.3 Variations in Puzzle Perception

  • Puzzle instances vary in appearance and configuration while preserving structure; the variations include maze textures, chess boards, Hanoi configurations, and N-Queens boards and colors.The four task passages specify semantic classes and, for N-Queens, board sizes from 8×8 to 11×11.Source: §D.3 · pp. 19–20

E Online Resources

  • The paper lists model checkpoints, repositories, and licenses to support reproducibility and future comparisons.Experiments combine public vision encoders, language models, and multimodal models from open and proprietary ecosystems.Source: E Online Resources · p. 20

F.1 Prompt template

  • For qualitative CRG examples, InternVL3-8B receives the board image followed by a puzzle question and must answer with one of two options.The prompt describes the 8×8 board and the light/hollow versus dark/solid piece convention.Source: §F.1 · p. 20

F.2 Qualitative Examples

  • In four qualitative successes, TDDN masks the oracle region and CRG corrects the answer; failures arise when the oracle region does not help or TDDN mislocalizes.Figure 6 displays the source board and oracle- and TDDN-masked conditions.Source: §F.2 · p. 20

G Recaptioning LAION Samples

The recaptioning appendix describes a short caption-generation prompt and illustrates how generated captions correct inaccuracies and add visually grounded detail.

G.1 Prompt Template

  • Gemma-3-27B-it is prompted to describe each image in 20–30 words, without a preamble or introduction.The prompt is applied to approximately 590K LAION-5B Aesthetics samples.Source: §G.1 · p. 21

G.2 Recaptioned Examples

  • Recaptioning corrects semantic errors and adds finer-grained objects, actions, and scene context to LAION captions.Examples include replacing “a cat” with a wallaby and identifying museum-displayed NASA astronaut suits.Source: §G.2 · p. 21, Table 16 · p. 23
Loading 2609.07937v1…