Source-linked AI summary
Text-Guided Visual Dependency Graph Learning with Cross-Modal Attention Priors
Fei Wang, Yutong Zhang, Yang Ye, Jinxian Chen, Wang Wenshuai, Xiong Wang
TL;DR
CM-GLasso addresses the limited study of interpretable conditional-dependence structures for multimodal visual-linguistic features. It builds cross-modal attention priors and jointly estimates shared and class-specific sparse precision components, achieving competitive results across eight benchmarks, including strong matched-protocol classification and segmentation performance.
Problem
Interpretable conditional-dependence structures for multimodal visual-linguistic features remain largely unexplored, despite challenges from unstable class covariances, misaligned modality priors, and separate graphs that miss shared structure.
Method
CM-GLasso renders text as images for shared SigLIP 2 encoding, distills prototype-indexed attention into graph priors, and jointly estimates common and class-specific precision matrices with a convex ADMM objective.
Results
Across eight benchmarks, CM-GLasso is competitive or superior; under matched control, it reaches 91.97% average classification accuracy and 74.75% VOC and 64.01% ADE20K segmentation mIoU.
Takeaways & Limitations
The method provides explicit sparse common-specific graph topologies that complement foundation-model features and support precision-based classification and topology-aware segmentation.
Takeaways & Limitations
The method has ADMM convergence limitations, heuristic attention priors, heterogeneous task-specific evaluation protocols, and uncharacterized scaling to very large vocabularies or denser patch grids.
Abstract
from arXiv · showhide
Estimating interpretable conditional-dependence structures from multimodal visual-linguistic features remains largely unexplored. We propose CM-GLasso (Cross-Modal Graphical Lasso), a framework that bridges vision-language representation learning and sparse Gaussian Graphical Models. CM-GLasso introduces three key components: (i) a text visualization strategy that renders class-attribute descriptions as images and processes them through the same SigLIP-2 vision encoder as natural images, yielding prototype-indexed patch-level attention footprints in a shared feature coordinate system; (ii) a cross-attention distillation mechanism that condenses high-dimensional patches into a small set of semantic graph nodes, whose attention-footprint similarities yield cross-modal structural priors for non-uniform L1 penalization; (iii) a joint ADMM formulation that estimates shared and class-specific precision components within a single convex objective, avoiding the need to first estimate and then decompose separate class-wise graphs. The learned sparse graph topologies directly support a parameter-free, precision-based classification rule and a lightweight topology-aware segmentation head. Extensive experiments on eight benchmarks demonstrate that CM-GLasso achieves competitive or superior performance compared with strong feature-based and task-specific baselines. Under the matched controlled protocol, it attains the highest average classification accuracy (91.97%) and the highest segmentation mIoU among the controlled baselines on VOC (74.75%) and ADE20K (64.01%), while also yielding explicit sparse conditional-dependence graphs with common-specific decomposition.
1 Introduction
CM-GLasso addresses instability, missing cross-modal alignment, and the inability of separate class-wise graphs to distinguish shared from class-specific structure. It uses shared visual encoding, attention-derived priors, and joint precision estimation to support graph-based inference.
- Three difficulties limit multimodal Graphical Lasso: unstable class covariances when nc < p, misaligned modality dimensions, and failure to separate invariant from class-specific structure.
- CM-GLasso renders class descriptions as images and processes them with natural images through the same SigLIP 2 vision encoder.
- Prototype-indexed cross-attention distills patch tokens into p graph nodes whose footprint overlap forms a class-conditioned p × p prior for sparsity penalties.
- A single convex objective jointly estimates shared precision and class-specific deviations, with eBIC selecting prior strength and ADMM solving the program.
- The resulting sparse graphs directly support generative classification and topology-aware segmentation.
2 Related Work
Prior work develops sparse precision estimation, shared-specific graphical models, vision-language alignment, feature adaptation, and contextual vision tasks. CM-GLasso combines these strands by using shared prototype-indexed attention geometry to construct explicit graph priors and topology.
- GLasso and related methods improve sparse precision estimation through non-uniform penalties and eBIC-guided selection.
- CSSL and joint graphical models separate shared structure from condition-specific structure, but prior work does not provide CM-GLasso’s vision-language attention priors.
- CLIP and SigLIP 2 align visual and linguistic representations, while CoOp, TIP-Adapter, and DINOv2 probes provide strong adaptation baselines.
- Unlike visual text compression, CM-GLasso renders short offline class descriptions to obtain patch-level attention footprints rather than compressing tokens or decoding with an online LLM.
- Existing segmentation and prompt-tuned classification methods model contextual interactions without exposing sparse conditional-dependence graphs; CM-GLasso supplies explicit common-specific topology.
3 Methodology
CM-GLasso builds aligned cross-modal graph representations, estimates common and class-specific sparse precision structures jointly, and uses the learned graphs for classification and segmentation.
- Unified Representation: CM-GLasso renders class descriptions as images and processes them with natural images through the same SigLIP 2 vision tower.The shared pathway provides comparable prototype indices across modalities without online LLM decoding.
- Graph Nodes: Learnable prototypes distill patch tokens into semantic graph nodes whose attention footprints align node indices across modalities.The distilled nodes provide the observations used by GLasso, while prototype footprints support cross-modal prior construction.
- Cross-Modal Priors: Attention-overlap priors weight sparsity without fixing graph support, leaving edge selection to the likelihood and common–specific decomposition.The prior is therefore a penalty preference rather than a hard graph structure.
- Nonparanormal Transformation: Rank-based Gaussianization is applied because transformer nodes are non-Gaussian, but joint Gaussianity remains a modeling assumption with residual non-Gaussianity.The held-out marginal Shapiro–Wilk pass rate rises to approximately 88%, while the transformation is used diagnostically rather than as proof of the assumption.
- Unified Optimization: A single convex ADMM procedure jointly estimates shared Θcom and class-specific S(c) components while maintaining positive-definite precision variables.eBIC selects prior strength, and k∗=0 yields prior-free joint GLasso–CSSL rather than forcing auxiliary structure into the estimate.
- Multi-task Heads: The resulting fixed precision graphs support generative classification and topology-aware segmentation through graph-derived attention decoding.Segmentation training fits the decoder while keeping the estimated graph fixed, so segmentation labels do not refit the precision matrices.
4 Experiments
Experiments evaluate CM-GLasso across eight classification and segmentation benchmarks under controlled and task-specific protocols, alongside ablations, sensitivity analysis, and qualitative inspection. The method achieves strong controlled results while exposing dataset-dependent gains and interpretable graph-guided behavior.
- Evaluation setup: Eight benchmarks cover classification on CIFAR-10/100, CUB-200-2011, and Caltech-256, and segmentation on PASCAL VOC, ADE20K, MS COCO, and Kvasir-SEG.
- Evaluation setup: The controlled evaluation uses training-only graph estimation, validation-only hyperparameter selection, frozen test settings, and identical resolution, splits, and feature protocols across baselines.
- Controlled results: 91.97% average classification accuracy is highest under the matched protocol, while CM-GLasso improves the two-stage graph baseline by 6.58 mIoU on VOC and 12.55 mIoU on ADE20K.
- Controlled results: DINOv2 remains strongest on CIFAR-10 and Caltech-256, indicating that CM-GLasso’s controlled gains are concentrated in structure-sensitive settings rather than universal across datasets.
- Task-specific results: Task-specific results reach 0.9283 accuracy on CUB-200, 0.9426 on CIFAR-100, 0.6401 mIoU on ADE20K, and 0.7475 mIoU on VOC-2012, but heterogeneous published protocols limit causal comparison.
- Ablations and interpretation: Ablations support shared-encoder rendering, cross-attention nodes, Gaussianization, joint optimization, and common–specific precision components, while qualitative maps emphasize objects and contours over background clutter.
5 Conclusion
CM-GLasso combines interpretable common–specific graph structure with competitive performance, especially in structure-sensitive settings. Its practical scope is tempered by computational, modeling, and validation limitations.
- Conclusion: CM-GLasso exposes reusable common dependencies and class-specific deviations for inspecting how cross-modal evidence changes class-specific structure.The same precision matrices support classification and segmentation without refitting the encoder.
- Conclusion: The method is intended to complement strong foundation-model features with an explicit conditional-dependence layer rather than replace feature learning.
- Practical Implications: Graph fitting is offline, with feature extraction and model selection dominating cost while fixed-k∗ ADMM remains relatively small.Table 6 reports complexity and runtime on one NVIDIA A800.
- Limitations: Attention overlap is a heuristic structural prior, prototype nodes need not match human concepts, and task-specific comparisons use heterogeneous published protocols.Only the matched comparison isolates the graph estimator’s effect; learned diagnostics should be validated against domain knowledge.