PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometric Similarity in VLM Low-Level Vision Representations

Shao-Jun Xia, Huixin Zhang, Zhen Lei, Anlan Sun, Yuner Zhang, Xiaoyang Chen

arXiv:2610.00848v1cs.CVcs.AI

TL;DR

Low-level VLM representations remain poorly understood despite their growing role in all-in-one restoration, and the field lacks evidence about their shared geometric organization. GeoSim analyzes 24 tasks across five categories through four complementary representation-analysis levels. The results show meaningful task structure and broadly shared sparse features within models, alongside limits to cross-model geometric agreement.

  • Problem

    How VLMs organize low-level task representations and whether different tasks share geometric structure remain poorly understood, despite the relevance of shared representations to low-level vision.

    Source: §1 · pp. 1–2

  • Method

    GeoSim analyzes task-conditioned representations across 24 low-level tasks using global similarity, local geometry, sparse feature decomposition, and topological verification.

    Source: §1 · pp. 2–3

  • Results

    Low-level tasks show meaningful geometric organization and within-model sparse-feature reuse, while cross-model universality is substantial but limited.

    Source: §4.2.2 · p. 8, §4.3 · p. 10

  • Takeaways & Limitations

    The analysis offers guidance for developing VLMs for all-in-one image restoration and a benchmark for future interpretability, transfer-learning, and alignment research.

    Source: §5 · p. 10

    The study analyzes one selected layer per model, uses standardized prompts and a transformer-dominated model set, and may confound task comparisons with dataset differences.

    Source: §4.3 · p. 10

Abstract

from arXiv · show

Vision-language models (VLMs) have emerged as powerful candidates for universal vision backbones, with representative architectures including autoregressive (AR) models and diffusion transformers (DiTs). Yet, adapting them efficiently for all-in-one low-level image restoration remains a challenge. Crucially, the field lacks an understanding of how VLMs organize hidden-layer representations and whether these structurally distinct paradigms share a common geometric organization for pixel-level perception. Such shared organization is a prerequisite for building highly transferable, unified restoration VLMs and adapters. In this paper, we systematically investigate representational similarity across 24 low-level tasks spanning 5 categories. We propose GeoSim, a unified four-level framework that analyzes task-conditioned representations from global similarity, local geometry, sparse feature decomposition, and topological verification perspectives. Our formulation applies to the analysis of hidden states in AR models and feature maps in DiTs across same- and cross-task/model settings. Our results reveal the organizing principles of low-level visual representations while exposing their limits in cross-task and cross-model agreement. Ultimately, GeoSim provides an interpretability lens for probing latent transferability in low-level vision and diagnosing model limitations in task- or model-specific scenarios.

1 INTRODUCTION

Low-level vision requires representations that balance pixel statistics, frequency, local structure, and perceptual context, but their organization across tasks remains poorly understood. GeoSim studies this organization across tasks and models through complementary geometric analyses.

1 INTRODUCTION

  • Low-level tasks must balance fine-grained pixel statistics, frequency components, and local image structures with high-level perceptual context.Shared representations are increasingly used to transfer knowledge across diverse perceptual tasks.Source: §1 · p. 1
  • How low-level task representations organize, cluster, and share mechanisms across tasks and models remains unclear.This limits understanding of why transfer varies across tasks and how parameter sharing or unified representation spaces should be designed.Source: §1 · pp. 1–2
  • GeoSim characterizes low-level task representations through global similarity, local geometry, sparse feature decomposition, and topological verification.The framework supports comparisons across tasks, layers, architectures, and model families.Source: §1 · p. 2
  • The study benchmarks 24 low-level vision tasks and tracks representation geometry across depth, quantifying global, local, and cross-model agreement.Its measures include cosine similarity, dCor, NNGS, LID, PAR, and topological analyses.Source: §1 · p. 2
  • An SAE-based pipeline, combined with mutual information and cross-model MMD, distinguishes task-specific from cross-task shared sparse representations.The broader motivation is to understand low-level geometric structure beyond semantic concepts and language alignment.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work advances unified restoration, task-transfer analysis, geometric representation analysis, and SAE interpretability, but leaves the geometric organization of low-level task representations insufficiently characterized.

All-in-one image restoration

  • Unified restoration research uses VLM representations, diffusion priors, and autoregressive visual token generation, while the learned representations' internal organization remains largely unexplored.Component-wise tuning evidence identifies the visual encoder, with additional gains from the alignment projector, as relevant to distortion recognition.Source: §2 · p. 3

Task transferability and task relationships

  • Taskonomy, Task2Vec, and later work characterize task relationships and transfer, but provide limited insight into how transferability appears in VLM hidden-representation geometry.Other studies examine positive and negative transfer or cross-task in-context learning.Source: §2 · p. 3

Geometric representation analysis and SAEs

  • Geometric VLM analyses largely focus on semantic concepts, language alignment, or individual geometric properties, leaving an integrated analysis of low-level perceptual tasks absent.SAEs have been used to interpret individual VLM concepts; this work applies them to shared and task-specific sparse representations across tasks and architectures.Source: §2 · pp. 3–4

3 PROPOSED FRAMEWORK

GeoSim organizes low-level VLM representation analysis into four levels, moving from task-level similarity through local geometry and sparse features to cross-model topological comparison.

3.1 FRAMEWORK OVERVIEW

  • GeoSim combines global similarity, local geometry, sparse feature decomposition, and cross-model universality to characterize low-level visual task representations.The four levels form a progression from task organization to local geometry, feature mechanisms, and cross-model consistency.Source: §3.1 · p. 4

3.2 LEVEL 1: GLOBAL SIMILARITY STRUCTURE

  • Level 1 builds layer-wise task similarity matrices from inter-task and intra-task cosine similarity, and uses dCor to compare distance structure across models.The stable layer is selected where consecutive task-similarity matrices change little; the task-relation graph visualizes consensus dissimilarities with MDS.Source: §3.2 · pp. 4–5

3.3 LEVEL 2: LOCAL GEOMETRIC STRUCTURE

  • Level 2 measures local complexity with within-task LID, cross-model neighborhood overlap with NNGS, and geometric alignment with Procrustes residuals.A smaller PAR indicates stronger agreement up to an orthogonal transformation.Source: §3.3 · p. 5

3.4 LEVEL 3: SPARSE FEATURE DECOMPOSITION

  • Level 3 trains one ReLU SAE per VLM across tasks, then uses activation rates, task selectivity, mutual information, and MMD to characterize feature use.The task-balanced training and held-out evaluation use a common learned dictionary within each model; MMD compares activation-rate distributions without matching individual features across models.Source: §3.4 · p. 6

3.5 LEVEL 4: TOPOLOGICAL VERIFICATION

  • Level 4 projects stable-layer task representations with fixed-configuration UMAP and compares persistent-homology diagrams for H0 components and H1 loops across models.This topology analysis is a supplementary cross-check.Source: §3.5 · p. 6

4 EXPERIMENTS

Across 24 low-level tasks, GeoSim experiments show stable within-task geometry, mechanism-linked task neighborhoods, and cross-model agreement that is stronger globally than locally.

4.1 EXPERIMENTAL SETUP

  • The benchmark covers 24 tasks in five perceptual families, using fixed task prompts across models and samples from representative low-level vision datasets.For artifact removal, colorization, and inpainting, degraded inputs are synthesized from clean images.Source: §4.1 · p. 7

4.2.1 CONDITION I: SAME TASK, SAME MODEL

  • Mean cosine similarities of 0.80–0.96 at the selected analysis layer show aligned instruction-induced displacements across images, while local task complexity varies by task.Task LID ordering is consistent across models (ρ = 0.80), with Denoising lowest in every model.Source: §4.2.1 · p. 7, Figure 2 · p. 8

4.2.2 CONDITION II: SAME TASK, DIFFERENT MODELS

  • Only 6 of 21 model pairs show lower same-task cross-model activation-rate MMD than the matched within-model, different-task reference.Task-specific features account for 0.9–15.1% of live features, while shared-feature fractions range from 13.4% to 72.4%, indicating broad within-model reuse.Source: §4.2.2 · pp. 7–8, Figure 3 · p. 8

4.2.3 CONDITION III: DIFFERENT TASKS, SAME MODEL

  • The consensus graph identifies 36 reliable task relations, including 20 cross-family links, with Removal and Photometric Correction densely connected and many Generation/Enhancement tasks more isolated.The within- versus across-family gap is only 0.135 SD, below typical cross-model disagreement of 0.71 SD.Source: Figure 5 · p. 9, §4.2.3 · p. 9

4.2.2 CONDITION II: SAME TASK, DIFFERENT MODELS

  • Mean dCor is 0.77, while mean chance-adjusted NNGS is 0.27 across 21 same-task model pairs, indicating stronger global than local agreement.Held-out normalized PAR exceeds the mean-prediction baseline of 1.0 for 16 of 21 pairs; models lack a common coordinate frame.Source: §4.2.2 · p. 7, Figure 4 · p. 9

4.2.4 CONDITION IV: DIFFERENT TASKS, DIFFERENT MODELS

  • Same-family task pairs rank 3.1 percentile points worse than cross-family pairs on average, with the same trend in 16 of 21 model pairs.The consistently close task/model combinations are pair-specific rather than evidence that same-family tasks are generally closer.Source: §4.2.4 · p. 9, Figure 6 · p. 10

4.3 DISCUSSION

  • Low-level task geometry is meaningful and mechanism-centric, while cross-model universality is substantial but limited to stronger global than local agreement.Sparse features are reused across tasks within models, but feature-level sharing across models needs explicit validation.Source: §4.3 · p. 10
  • Cross-model universality is mainly relational; sparse-feature analysis may reveal recurring visual primitives, but cross-model feature sharing requires explicit validation.The study notes that selected-layer analysis, standardized prompts, dataset differences, and a transformer-dominated model set limit generalization.Source: §4.3 · p. 10

5 CONCLUSION

The conclusion presents GeoSim as a four-level framework for analyzing low-level visual task representations and their cross-task relations. It establishes a benchmark and offers insights to guide VLM development for all-in-one image restoration.

5 CONCLUSION

  • The study presents a comprehensive geometric analysis of how VLMs represent low-level visual tasks and relate cross-task representations.Source: §5 · p. 10
  • GeoSim combines global similarity, local geometry, sparse feature decomposition, and topological verification in a four-level analysis framework.Source: §5 · p. 10
  • The work establishes a benchmark for future research on representation interpretability, transfer learning, and alignment across low-level tasks.Source: §5 · p. 10
  • The analysis offers guidance for developing VLMs for all-in-one image restoration.Source: §5 · p. 10

A.1 IMAGE RESTORATION AND TASK CONDITIONING

Work on image restoration has progressed from shared backbones for multiple degradation types to ingredient-oriented decomposition and task conditioning through prompts or language instructions.

A.1 IMAGE RESTORATION AND TASK CONDITIONING

  • Early restoration methods used shared CNN- or Transformer-based representations to model multiple degradation types.Examples include AirNet and TransWeather.Source: §A.1 · p. 18
  • IDR introduced an ingredient-oriented formulation that disentangles degradation factors into reusable restoration components.Source: §A.1 · p. 18
  • PromptIR uses learnable prompts for degradation-specific adaptation, while InstructIR uses human-readable language instructions instead of task-specific embeddings.Source: §A.1 · p. 18

A.2 TASK TRANSFERABILITY

Prior work has examined task transferability, geometric analysis of representations, and sparse autoencoders across language, vision, and vision-language models. Together, these lines of work provide context for analyzing cross-task representation structure.

A.2 TASK TRANSFERABILITY

  • Taskonomy was extended to zero-shot task transfer, and VLM studies examined multimodal-task transferability and whether benchmark performance reflects evaluation bias rather than task competence.Source: §A.2 · p. 18

A.3 GEOMETRIC REPRESENTATION ANALYSIS

  • Geometric analysis methods compare activation spaces through global similarity, cross-model alignment, intrinsic geometry, or activation distributions; topological RSA varies between distance geometry and neighborhood structure.Source: §A.3 · p. 18

A.4 SPARSE AUTOENCODERS ACROSS DOMAINS

  • Sparse autoencoders have been applied across language, vision, and VLMs to extract sparse concepts and localized features, while later work cautions that interpretable features need not be steerable and labels alone do not specify local versus global evidence.Source: §A.4 · p. 18

B.1 OVERVIEW OF THE ANALYSIS LEVELS

GeoSim characterizes low-level visual task representations through four complementary levels, moving from global task organization to local geometry, sparse features, and topological checks. The levels examine different aspects of similarity and provide complementary ways to assess whether observed structure is robust to analysis choices.

B.1 OVERVIEW OF THE ANALYSIS LEVELS

  • GeoSim’s four levels examine global task organization, local geometry, shared and task-specific sparse features, and cross-model geometric universality.Source: §B.1 · pp. 18–19

B.2.1 GLOBAL AND LOCAL GEOMETRY

  • Level 1 compares global task similarity within and across VLMs and tracks how task organization changes through network depth; Level 2 tests local neighborhoods and manifold geometry beyond global averages.Source: §B.2.1 · p. 19

B.2.2 SPARSE FEATURES AND TOPOLOGICAL VERIFICATION

  • Level 3 uses task-dependent SAE feature activations to interpret geometric patterns, but depends on SAE training choices; Level 4 adds a topological check that is less dependent on a particular feature decomposition.Source: §B.2.2 · p. 19

B.3.1 LEVEL 1: GLOBAL SIMILARITY STRUCTURE

  • The Level 1 cross-model similarity procedure computes pairwise Euclidean distances for the distance-correlation statistic.Source: §B.3.1 · p. 20

B.3.2 LEVEL 2: LOCAL GEOMETRIC STRUCTURE

  • Level 2 estimates local intrinsic dimensionality for each task representation space using maximum likelihood and averages across samples and sampling repeats.Source: §B.3.2 · p. 20

B.3.3 LEVEL 3: SPARSE FEATURE DECOMPOSITION

  • At a stable layer, each VLM uses one SAE shared across low-level tasks, allowing feature use to be compared in a common dictionary; task-pair binary activation patterns are compared with mutual information.Source: §B.3.3 · p. 20

B.3.4 LEVEL 4: TOPOLOGICAL VERIFICATION

  • Using UMAP-preprocessed point clouds, the analysis constructs Vietoris–Rips filtrations, summarizes H0 and H1 persistence diagrams, and compares models’ diagrams for the same task with r-Wasserstein distance, which reduces to bottleneck distance as r → ∞.Source: §B.3.4 · p. 21

C.1 DATASETS

The dataset suite supplies paired data for 24 low-level vision tasks across restoration, removal, generation and enhancement, reconstruction, and photometric correction; the feature protocol extracts task-conditioned representations and selects a stable analysis layer.

C.1.1 RESTORATION

  • Restoration spans real-world and synthetic paired benchmarks, including compression-artifact removal, deblurring, denoising, deraining, desnowing, and underwater restoration.The selected examples include LIU4K, DPDD, GoPro, SIDD, Rain14000, Snow100K, and UIEB.Source: §C.1.1 · pp. 21–22

C.1.2 REMOVAL

  • Removal benchmarks cover lens flares, raindrops, reflections, and shadows, with real-world datasets represented for each task.The datasets are Flare7K++, Raindrop Clarity, SIR2, and ISTD.Source: §C.1.2 · pp. 22–23

C.1.3 GENERATION AND ENHANCEMENT

  • Generation and enhancement use colorization, harmonization, inpainting, low-light enhancement, day-to-night style transfer, and edge detection data.ADE20K RGB images, iHarmony4, Natural Images, LOL, Night2Day, and BSDS500 supply these task examples.Source: §C.1.3 · p. 23

C.1.4 RECONSTRUCTION

  • Reconstruction uses DIV2K low-/high-resolution pairs for super-resolution and HDRTV1K SDRTV/HDRTV pairs for dynamic-range conversion.Source: §C.1.4 · p. 23

C.1.5 PHOTOMETRIC CORRECTION

  • Photometric correction draws on white-balance, multi-exposure contrast-enhancement, and scene-relighting benchmarks.Rendered WB, SICE, and the VIDIT-based AIM 2020 relighting track provide these data.Source: §C.1.5 · p. 24

C.2 FEATURE EXTRACTION PROTOCOL

  • For AR models, fixed task and neutral prompts produce cached hidden states whose difference defines the task-conditioned representation; Qwen-Image-Edit instead uses task-minus-neutral spatial means of condition-image feature maps.Images are EXIF-oriented, converted to RGB, and resized to 512 × 512 before extraction.Source: §C.2 · p. 24
  • The analysis layer is selected from a smoothed task-similarity change curve using both a change threshold and a task-margin criterion, with a flagged fallback if none qualifies.The stable range is sought within 15–95% of network depth; SAE layer selection uses a separate resampling protocol.Source: §C.2 · p. 25

C.3 SAE TRAINING DETAILS

The SAE protocol fits one model-specific sparse autoencoder per model, evaluates feature activity on held-out codes, and reuses the resulting codes for cross-model comparisons without assuming aligned feature coordinates.

C.3 SAE TRAINING DETAILS

  • SAE layer selection uses 50 within-task bootstrap resamples, distinct from the frozen sampling repeats and topological resamples.Source: §C.3 · p. 25
  • Each model's SAE is trained on representations from all 24 tasks after a base-content-identity 70/30 split, with 500 training samples per task and shared normalization.Paired-control samples remain in the same split across tasks, and replacement is used only when necessary.Source: §C.3 · p. 25
  • SAEs use an 8d feature dimension and are selected by explained-variance and sparsity criteria, then checked across three final-fit seeds and train–evaluation consistency.Eligible λ candidates require explained variance ≥0.80 and active fraction ≤0.05; final fits also undergo dead-feature and cross-seed checks.Source: §C.3 · p. 25
  • Held-out active features are labeled task-specific, shared, or intermediate by TSI thresholds, while TSI, MI, and MMD are excluded from λ selection.Source: §C.3 · p. 25
  • Because per-model SAE feature coordinates are unaligned, cross-model MMD compares activation-rate distributions; Conditions II and IV reuse the trained SAEs and codes.Condition IV uses reciprocal task assignments and ranks results within each model pair.Source: §C.3 · p. 25

D.1 MAPPING BETWEEN CONDITIONS AND ANALYSIS LEVELS

  • Table 4 links each evaluation condition to its GeoSim analysis level and metrics, while the supplementary figures cover dimensionality, task organization, SAE, and topology analyses.Source: Table 4 · p. 26, §D.1 · p. 26, §D.2 · p. 26

D.2.1 CONDITION I: SAME TASK, SAME MODEL

At the selected analysis layer, task-wise local dimensionality has a nearly consistent ordering across models, while instruction-induced similarity varies across network depth.

D.2.1 CONDITION I: SAME TASK, SAME MODEL

  • Condition I varies input images while holding task and model fixed, measuring instruction-induced displacement over ten repeats of 100 distinct images per task.Source: §D.2.1 · p. 26
  • Within-task consistency varies across layers: five models start at 0.95 or above, Qwen-Image-Edit starts lower for some tasks, and Emu3.5 rises sharply before later separation.Thus, the 0.80–0.96 interval at the selected analysis layer does not describe the network as a whole.Source: §D.2.1 · p. 26
  • Mean pairwise Spearman ρ = 0.80 for task LID ordering across models, with Denoising lowest in all seven.LID uses the Levina–Bickel estimator at k = 10 on the selected layer; the authors interpret only ordering given downward bias at n = 100 in several-thousand-dimensional space.Source: §D.2.1 · p. 26, Figure 9 · p. 27

D.2.2 CONDITION II: SAME TASK, DIFFERENT MODELS

For the same task across models, agreement is stronger in global distance structure than in local neighborhoods or held-out coordinate alignment; lower-dimensional tasks are a systematic exception, and SAE and topology analyses provide additional comparisons.

D.2.2 CONDITION II: SAME TASK, DIFFERENT MODELS

  • Mean dCor is 0.77, chance-adjusted NNGS is 0.27, and held-out PAR is 1.049, with 16 of 21 model pairs above its 1.0 mean-prediction baseline.The models share a coarse ordering of distances without sharing local neighborhoods or a coordinate frame.Source: §D.2.2 · pp. 27–28
  • ρ = +0.853 for LID versus PAR and ρ = −0.764 versus neighborhood overlap; Denoising and Motion Deblurring have residuals of 0.74 and 0.83, while every task with LID above 15 exceeds 1.1.Figure 10 plots each task's seven-model mean LID against cross-model agreement; the authors suggest fewer operative directions leave less room for model differences.Source: §D.2.2 · p. 28
  • Six model pairs meet the same-task cross-model activation-rate MMD criterion after seed aggregation, but only three meet it consistently across all three seeds.The comparison uses a matched within-model, different-task baseline; TSI and feature–task MI instead measure within-model selectivity.Source: §D.2.2 · p. 28
  • The Level-4 analysis reports H0 and H1 2-Wasserstein and bottleneck distances for 16 model-pair/task combinations, with 2-Wasserstein agreeing in direction as a complementary topological check.Distances are averaged over ten sampling runs of 100 samples per task, with run-to-run variation shown for 2-Wasserstein.Source: §D.2.2 · pp. 28–29, Figure 11 · p. 28

D.2.3 CONDITION III: DIFFERENT TASKS, SAME MODEL

Within a model, Removal tasks form a coherent block across models, while Generation/Enhancement shows no consistent block. The consensus task graph also contains cross-family relations, and a supplementary comparison examines different tasks in different models without paired data or aligned coordinates.

D.2.3 CONDITION III: DIFFERENT TASKS, SAME MODEL

  • The Removal block remains coherent in every model, whereas Generation/Enhancement shows no consistent block.Source: §D.2.3 · p. 29
  • 36 of 276 task pairs pass the consensus threshold, including 20 that cross a family boundary.The threshold is 0.714 standard deviations, set to the mean disagreement across models.Source: §D.2.3 · p. 29
  • Defocus–Motion Deblurring (+1.66), Deraining–Desnowing (+1.63), and Reflection–Shadow Removal (+1.61) are the strongest consensus edges, each linking tasks with a shared degradation mechanism.Colorization and Harmonization fall below their model-average similarity to every other task; Inpainting misses the threshold by 0.009 and is described as undecided rather than isolated.Source: §D.2.3 · p. 29
  • The MDS layout’s first two coordinates carry 31.5% of structure and correlate 0.772 with true dissimilarity, so positions indicate grouping rather than precise metric distance.Coordinates summarize a task’s profile against all other tasks, while an edge represents one pairwise value.Source: §D.2.3 · p. 30

D.2.4 CONDITION IV: DIFFERENT TASKS, DIFFERENT MODELS

  • Condition IV compares 10 reciprocal candidates across different tasks and models using screening ranks and H0/H1 topological distances, as a supplementary unpaired analysis.Figure 13 shows raw and scale-normalized views; its dots are 2-Wasserstein means and its bars show bootstrap intervals and the range between task-assignment means.Source: §D.2.4 · p. 30, Figure 13 · p. 30

Related papers

Loading 2610.00848v1…