PaperlayerView exact revision on arXiv

Source-linked AI summary

MSGAT: Multi-Head Spiking Graph Attention with Similarity-Space Fusion for Image-Text Retrieval

Xintao Zong, Wenxuan Liu, Jianhao Ding, Zhaofei Yu, Tiejun Huang

arXiv:2610.11526v1cs.CVcs.AI

TL;DR

Spiking image-text retrieval must capture structural and multi-granularity semantic relationships, but existing SNN approaches leave some of these aspects underexplored and feature-level fusion faces a representation mismatch. MSGAT combines dynamic spiking graph attention with similarity-space fusion. Across Flickr30K and MSCOCO, it outperforms ANN methods and SNN baselines, with approximately 55% lower theoretical graph-module energy at two time steps.

  • Problem

    Existing spiking ITR methods rely on local alignment and additional soft-label supervision, leaving semantic relation reasoning and global alignment underexplored; sparse discrete spike features also differ from continuous global features.

    Source: Introduction · pp. 1–2

  • Method

    MSGAT performs dynamic multi-head spike-driven graph reasoning over visual and textual graphs, while Sim-Fuse integrates coarse- and fine-grained matching relations in similarity space.

    Source: Introduction · p. 2, Multi-head Spiking Graph Attention · p. 4

  • Results

    Across Flickr30K and MSCOCO, MSGAT outperforms ANN-based methods and existing SNN retrieval baselines; at two time steps, theoretical graph-module inference energy is reduced by approximately 55%.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    With only two time steps, MSGAT achieves performance comparable or superior to its ANN counterpart, indicating potential for efficient cross-modal retrieval.

    Source: Main Experimental Results · p. 5

Abstract

from arXiv · show

Spiking neural networks (SNNs) offer an energy-efficient computing paradigm through sparse event-driven computation, showing great potential for efficient multimodal learning. However, applying SNNs to high-level multimodal tasks, such as image-text retrieval (ITR), remains challenging, since sparse spike representations make it difficult to capture semantic structures required for cross-modal alignment. Existing spiking ITR methods rely on local alignment and additional soft-label supervision during training, while lacking awareness of structural and multi-granularity relationships. To address these issues, we propose a Multi-head Spiking Graph Attention Network (\textbf{MSGAT}) for structural modeling and equip it with dynamic attention heads to capture complementary relational patterns and enable spike-driven graph reasoning and aggregation. However, within a two-branch multi-granularity fusion framework, the fine-grained spike representations generated by MSGAT are sparse and discrete, whereas the global representations are continuous, making conventional feature-level fusion susceptible to interference across heterogeneous representations. Therefore, we introduce \textbf{Sim-Fuse}, a similarity-space fusion alignment strategy integrating coarse- and fine-grained matching relations while avoiding direct fusion of heterogeneous representations. Experiments on Flickr30K and MSCOCO show our method outperforms ANN methods under matched settings and existing SNN retrieval baselines. Moreover, with only two time steps, our SNN achieves comparable or superior performance to its ANN counterpart while reducing theoretical module-level energy by 55\%. The code is provided in the Supplementary Materials.

Introduction

Spiking image-text retrieval needs structural semantic reasoning and multi-granularity alignment, but sparse discrete spike features and continuous global features complicate direct fusion. MSGAT uses dynamic multi-head graph reasoning and Sim-Fuse, and experiments report stronger retrieval results with two time steps and lower theoretical module energy.

Introduction

  • Existing spiking ITR methods rely on local alignment and additional soft-label supervision, while leaving semantic relation reasoning and global alignment underexplored.Structural dependencies among visual and linguistic units provide cues for distinguishing fine-grained semantic correspondence.Source: Introduction · p. 1
  • Fine-grained SNN representations are sparse and discrete, whereas GPO global representations remain continuous, so feature-level fusion may interfere with joint optimization.The paper motivates similarity-space fusion as a way to integrate coarse- and fine-grained matching relations without direct feature-level fusion.Source: Introduction · p. 2
  • MSGAT constructs spiking visual and textual graphs for spike-driven structural reasoning, using dynamic attention heads to capture complementary relational patterns.Sim-Fuse integrates coarse- and fine-grained matching relations in similarity space rather than fusing heterogeneous feature representations.Source: Introduction · p. 2
  • Under comparable feature-extraction settings, MSGAT outperforms ANN-based methods and existing SNN retrieval baselines on Flickr30K and MSCOCO, with approximately 55% lower theoretical graph-module energy at two time steps.At T=2, performance is comparable or superior to the ANN counterpart.Source: Introduction · p. 2

Related Works

Prior ITR work uses graph structures to model semantic relations and fine-grained matching, while SNN research has developed spike-driven attention and graph-learning approaches. MSGAT is positioned as a multimodal retrieval method with dynamic multi-head graph attention and multi-granularity fusion.

Related Works

  • Prior ITR graph methods model relations over regions, tokens, objects, attributes, or semantic fragments, with approaches including graph convolution, structured matching, and cross-embedding graphs.VSRN, GSMN, CSGM, HREM, and CORA are examples.Source: Image-Text Retrieval · p. 2
  • CMSF established SNN feasibility for ITR, but relies on soft-label supervision and is described as susceptible to semantically irrelevant nodes in self-attention.MSGAT is presented as addressing these limitations through spike-driven graph reasoning, dynamic attention, and multi-granularity fusion.Source: Image-Text Retrieval · p. 2

Spiking Neural Networks

  • SNN research spans surrogate-gradient training, spike-driven self-attention, and graph learning, while MSGAT targets multimodal ITR with dynamic multi-head attention and softmax-free spike aggregation.Examples cited include Spikformer and its variants, and graph methods such as GSAT.Source: Spiking Neural Networks · p. 2

Proposed Method

MSGAT constructs spiking visual and textual graphs, applies spike-driven graph reasoning for fine-grained structural semantics, and preserves coarse-grained information through GPO. Its fused similarity supervises the two pre-fusion similarity matrices, while the LIF neuron supplies threshold-based spiking dynamics.

Proposed Method

  • MSGAT reasons over spiking visual and textual graphs to obtain fine-grained structural semantics, while GPO preserves coarse-grained information; a detached fused similarity supervises both pre-fusion matrices through KL-divergence alignment.The framework converts node features into spikes, performs neighborhood-aware aggregation, then fuses coarse- and fine-grained similarities into a multi-granularity target.Source: Proposed Method · p. 2, Figure 2 · p. 3

Spiking Neuron

  • The LIF neuron accumulates input currents into membrane potentials and emits a spike when the threshold is reached; firing resets the potential, otherwise it remains at the accumulated value.The membrane time constant is τ.Source: Spiking Neuron · p. 3

Spiking Graph Construction

The method projects pre-extracted visual-region and word features into a shared dimension, constructs visual and textual graphs, and converts node features into spikes. Textual edges combine syntactic dependencies with semantic similarity, while the Spike Encoder applies time-specific transformations and spiking neurons.

Spiking Graph Construction

  • Faster R-CNN visual-region features and BERT word features are mapped by a linear layer into a shared embedding dimension D.The inputs have dimensions 2048 for visual regions and 768 for word features.Source: Spiking Graph Construction · p. 3

Spiking Textual Graph

  • The textual graph combines parsed syntactic dependencies with semantic edges between nodes whose cosine similarity exceeds threshold θw.The semantic adjacency is binary, and the combined matrix incorporates both syntactic and semantic relations.Source: Spiking Textual Graph · p. 3
  • The visual graph connects detected regions according to pairwise cosine similarity, reflecting that instances of the same object category may have similar embeddings.The visual node-level similarity threshold is θv.Source: Spiking Visual Graph · p. 3

Spike Encoder

  • The Spike Encoder layer-normalizes floating-point node features, replicates them across T time steps, and applies learnable time-specific scale and bias before spiking-neuron conversion.The time-step parameters are γt and βt.Source: Spike Encoder · p. 3

Multi-head Spiking Graph Attention

MSGAT applies multi-head spiking graph attention to model source-target relationships between nodes and aggregate information from graph neighborhoods. Its dynamic attention and spike-driven aggregation operate on sparse binary representations.

Multi-head Spiking Graph Attention

  • MSGAT projects spiking node features into query, key, and value tensors, then divides them into H attention heads.Each head has dimension d = D/H.Source: Multi-head Spiking Graph Attention · p. 4
  • Each head constructs a vector-valued response for every node pair to compute dynamic source-target attention.This follows GATv2’s dynamic source-target interaction formulation.Source: Multi-head Spiking Graph Attention · p. 4
  • MSGAT applies ReLU to attention responses, masks them with the adjacency matrix, and row-normalizes them for neighborhood aggregation.A small epsilon prevents instability when sparse responses make the denominator zero.Source: Multi-head Spiking Graph Attention · p. 4
  • It concatenates the outputs of the dynamic heads and converts the aggregated representation into spikes with a LIF neuron.The concatenated representation is passed through LIF to produce binary spike outputs.Source: Multi-head Spiking Graph Attention · p. 4

Similarity-Space Fuse Alignment Strategy

Sim-Fuse aligns coarse- and fine-grained matching in similarity space rather than fusing their heterogeneous features directly. It combines branch similarities and uses the fused scores to guide alignment of the separate branches.

Similarity-Space Fuse Alignment Strategy

  • Sim-Fuse avoids feature-level fusion because sparse, discrete MSGAT representations are misaligned with continuous GPO representations.The paper reports this discrepancy makes feature-level fusion suboptimal.Source: Similarity-Space Fuse Alignment Strategy · p. 4
  • The fine-grained branch matches words to their most relevant regions and regions to their most relevant words.These directional matches are computed from a word-region similarity tensor.Source: Similarity-Space Fuse Alignment Strategy · p. 4
  • The coarse-grained branch uses GPO to form global image and text embeddings and computes their cosine similarity.The fused similarity is a learnable weighted combination of coarse- and fine-grained similarities.Source: Similarity-Space Fuse Alignment Strategy · p. 4
  • Stop-gradient fused matching distributions supervise the fine- and coarse-grained distributions with KL-divergence alignment in both image-to-text and text-to-image directions.The paper describes the branch similarities and fused similarity as a collaborative refinement process.Source: Similarity-Space Fuse Alignment Strategy · p. 4

Training Objectives

Training combines a hubness-aware retrieval loss with objectives at the node and structure levels. These objectives address sparse-input noise and supervise local, global, and fused similarities.

Training Objectives

  • The training objective improves a hubness-aware loss, treating diagonal similarity entries as positive pairs and off-diagonal entries as negatives.The temperature scale and margin are denoted by µ and γ.Source: Training Objectives · pp. 4–5

Node-level Objective

  • A node-level auxiliary objective is applied before the Spike Encoder to stabilize inputs against unreliable sparse activations that could propagate through graph reasoning.It applies the retrieval loss to node similarities computed from visual and textual nodes.Source: Node-level Objective · p. 5

Structure-level Objective

  • After MSGAT, the structure-level objective sums losses on fine- and coarse-grained similarities to capture local and global structural information.A fused-similarity loss with learnable weights jointly optimizes both branches.Source: Structure-level Objective · p. 5

Experiments

MSGAT is evaluated for image-text retrieval on Flickr30K and MSCOCO using Recall@K and R@Sum. It reports higher overall retrieval scores than compared methods and lower theoretical inference energy for the graph-reasoning module.

Datasets Metrics and Settings

  • Flickr30K and MSCOCO contain 31,783 and 123,287 images, respectively, each paired with five captions; evaluation uses Recall@1, @5, @10, and their sum.Flickr30K uses 29,000/1,000/1,000 train/validation/test images; MSCOCO uses 113,287/1,000/5,000.Source: Datasets Metrics and Settings · p. 5

Main Experimental Results

  • R@Sum reaches 539.0 on MSCOCO 1K (5-fold) and 529.8 on Flickr30K 1K, outperforming all compared methods.The results are reported for a single model, unlike the cited multi-model ensemble methods.Source: Main Experimental Results · p. 5
  • 0.985 mJ theoretical inference energy is reported for MSGAT, 54.9% lower than GATv2.The reported cost excludes the replaceable Faster R-CNN and BERT feature extractors; MSGAT also reduces ACs by 10.5% compared with SSA.Source: Figure 3 · p. 5, Efficiency Analysis of MSGAT · p. 5
  • With T=2, MSGAT surpasses its ANN counterpart by 2.0 and 2.6 R@Sum points; on MSCOCO 5K it achieves the best R@Sum of 452.0.On MSCOCO 5K, the paper says it outperforms its ANN counterpart across most metrics.Source: Main Experimental Results · p. 5

Ablation Studies

The ablations compare fine-grained branch designs, fusion strategies, training objectives, time steps, and attention-head counts. MSGAT with Sim-Fuse performs best in the reported fusion comparison, while the studies identify favorable objective, temporal, and head configurations.

Ablation on Sim-Fuse Strategy

  • The Sim-Fuse objective combines coarse- and fine-grained similarities, with structure-level supervision improving individual similarities before the final objective reaches 529.8 R@Sum.The passage reports that LKL aligns the branch similarities with the detached fused similarity distribution.Source: Ablation on Training Objectives · pp. 6–7

Ablation on MSGAT Module

Ablation on Time Steps

  • At T=2, MSGAT reaches its best R@Sum of 529.8; without Sim-Fuse, performance at T=1 falls to 509.7.Increasing time steps beyond T=2 does not improve the best result, and the authors describe MSGAT as robust to different temporal configurations.Source: Figure 5 · p. 7, Ablation on Time Steps · p. 7

Ablation Studies

  • MSGAT reaches 529.8 R@Sum, exceeding the fine-grained branch ablations without that branch (512.7), with SSA (520.4), and with SGCN (524.7).The authors note SGCN cannot model dynamic attention among nodes, while SSA may be disturbed by irrelevant node interactions.Source: Ablation on MSGAT Module · p. 5, Table 3 · p. 7

Ablation on Training Objectives

  • The fused loss alone scores 522.2 R@Sum; adding node-level and structure-level objectives raises it to 525.0 and 526.4, and adding LKL yields 529.8.The node-level objective provides stable spike representations before MSGAT reasoning; LKL aligns the fine- and coarse-grained similarity distributions with the detached fused similarity.Source: Ablation on Training Objectives · pp. 6–7, Table 4 · p. 7

Ablation on Attention Heads

  • Eight attention heads achieve the best R@Sum of 529.8, balancing subspace diversity against per-head spike representation capacity.With two heads, thresholding produces all-zero spike attention outputs, preventing effective neighborhood aggregation and graph information propagation.Source: Table 5 · p. 7, Ablation on Attention Heads · p. 7

Conclusion

MSGAT combines spike-driven graph reasoning with similarity-space alignment for image-text retrieval. On Flickr30K and MSCOCO it outperforms existing SNN retrieval and state-of-the-art ANN methods, and at two time steps exceeds its ANN counterpart while reducing theoretical energy consumption by about 55%.

Conclusion

  • MSGAT combines multi-head spike-driven neighborhood aggregation over visual and textual graphs with Sim-Fuse alignment in similarity space.Sim-Fuse addresses the distribution mismatch between discrete spike and continuous global representations.Source: Conclusion · p. 7
  • Experiments on Flickr30K and MSCOCO show MSGAT outperforming the existing SNN retrieval baseline and state-of-the-art ANN-based methods.Source: Conclusion · p. 7
  • With two time steps, MSGAT exceeds its ANN counterpart GATv2 while reducing theoretical energy consumption by approximately 55%.Source: Conclusion · p. 7
Loading 2610.11526v1…