PaperlayerView exact revision on arXiv

Source-linked AI summary

Mutual Equilibrium: Multimodal Representation Learning through Reciprocal Feedback

Ho-min Park, Byungkon Kang

arXiv:2609.39456v1cs.LGcs.CV

TL;DR

Because single-pass fusion treats features as fixed, it may miss information that complementary modalities provide about one another. MEQ repeatedly updates two modality-specific representations through reciprocal feedback to a fixed point. It improves over concatenation on four of five benchmarks, with mixed comparisons against attention baselines.

  • Problem

    Many multimodal methods concatenate fixed feature descriptors in one feedforward operation, although complementary modalities can change how each input is interpreted.

    Source: §1 · p. 1

  • Method

    MEQ uses two feature extractors to repeatedly exchange hidden-state information and returns their mutually coupled embeddings at a fixed point.

    Source: §2 · p. 3

  • Results

    MEQ improves over simple concatenation on four of five benchmarks, including +14.36 pp on VQA v2, while comparisons with attention baselines are mixed.

    Source: §3.1 · p. 7

  • Takeaways & Limitations

    The authors identify iterated reasoning, rather than modality fusion alone, as MEQ’s strength.

    Source: §3.1 · p. 7

    Further work is needed on the theoretical foundation of iterative refinement under its constraints and its connections to traditional dynamical systems.

    Source: §4 · p. 10

Abstract

from arXiv · show

This work proposes a mutual feedback architecture, MEQ, that refines the two inputs, of possibly different modalities, into a pair of coupled embeddings such that each embedding reflects the information of the other. The core idea is to incorporate continuous interchange of information between the two inputs. This idea leads to a mutual feedback architecture consisting of two components whose outputs are fed back into the other. The final output of this model is defined as the fixed point of this interaction. We provide theoretical analysis that offers interpretation of this model as well as design choices to prevent failure cases. We show the benefits of MEQ through classification and visual grounding tasks spanning various datasets. Quantitatively, our model outperforms or shows competitive performance on concatenation-based multimodal classification problems. Qualitatively, the proposed interactive mechanism allows the model to progressively refine the visual grounding when paired with complementary modality, thus demonstrating the power of mutual feedback under such settings.

1 INTRODUCTION

Existing multimodal approaches often combine fixed descriptors in a single feedforward step, although complementary modalities can change how each input is interpreted. MEQ instead models reciprocal iterative refinement and treats its fixed point as the mutually consistent representation pair.

1 INTRODUCTION

  • MEQ models reciprocal iterative refinement, returning the fixed point where two modality-specific representations become mutually consistent.Each representation reflects information from the other modality.Source: §1 · p. 1

1.1 RELATED WORKS AND CONTRIBUTIONS

  • Unlike concatenation’s single feedforward fusion and methods with finite interactions, MEQ defines cross-modal exchange through a fixed-point process.MulT uses a fixed finite number of interactions.Source: §1 · p. 1, §1.1 · p. 2
  • Unlike multimodal deep equilibrium work based on a weighted sum of features, MEQ focuses on the coupled dynamics of two interacting modalities.The comparison is made with Ni et al. (2023).Source: §1.1 · p. 2
  • MEQ formulates mutual interaction as coupled dynamics in representation space, with theoretical properties guiding its algorithm and possible module variants.This representation-space framing is distinguished from multi-agent LLM interaction, while the dynamics are analyzed theoretically.Source: §1.1 · p. 2
  • Experiments are reported to show meaningful equilibria across prediction tasks and value in iterative refinement itself.The introduction frames these as contributions established by the experiments.Source: §1.1 · p. 3

2 MAIN APPROACH

MEQ iteratively couples modality-specific embeddings until they reach a fixed point. Its design uses Jacobian analysis to guide safeguards against instability and collapse, while training combines the task loss with regularizers.

2 MAIN APPROACH

  • MEQ produces coupled embeddings by iteratively updating each modality with information from the other, with the fixed point serving as the refined representation pair.Each feature extractor receives its raw input, its own hidden state, and the other modality’s hidden state.Source: §2 · pp. 3–4
  • In practice, MEQ uses a damped fixed-point iteration truncated after K steps and reads the final iterate rather than a solver-computed root.The paper motivates this choice by numerical sensitivity in the coupled system and describes the iteration as more robust.Source: §2 · pp. 3–4

2.1 THEORETICAL PROPERTIES AND DESIGN CHOICES

  • The proposed templates require each feature extractor to use both the other modality’s hidden state and its own raw input non-trivially.The design separates self- and cross-modal influence, with raw-input feature terms supporting nonzero cross-sensitivity.Source: §2.1 · p. 5
  • Fixed-point collapse can occur when raw-input sensitivity and total influence are both small, leaving the representations nearly independent of their inputs.The Jacobian analysis interprets input sensitivity at the fixed point and relates it to the influence passed between modalities.Source: §2.1 · pp. 4–6

2.2 TRAINING OBJECTIVE

  • The training objective combines task loss with a spectral-radius range penalty and a fixed-point correction term.The correction penalizes residuals so truncated iterates remain close to self-consistent fixed points.Source: §2.2 · p. 6

3 EXPERIMENTS

The experiments evaluate whether MEQ's mutual iterative refinement helps prediction and progressively updates visual grounding. They compare MEQ with fusion baselines across multimodal benchmarks and ablate components on selected tasks.

3 EXPERIMENTS

  • The experiments target the value of iterative feature refinement rather than state-of-the-art performance, using five benchmarks and simple concatenation and attention-based baselines.Attention baselines are parameter-matched to MEQ.Source: §3 · pp. 6–7

3.1 ACCURACY RESULTS

  • MEQ beats simple concatenation on four of five benchmarks, with +14.36 pp on VQA v2; against stronger attention baselines, it leads on three and trails on Hateful Memes.CMU-MOSEI is within seed noise of concatenation.Source: Table 1 · p. 7, §3.1 · p. 7

3.2 ITERATIVE IMPROVEMENT RESULTS

  • On COD10k, MEQ trained on SNLI-VE focuses on hidden objects within the first few iterations in successful cases, but never localizes the object in the failure case.Ground-truth masks allow direct verification of visual grounding.Source: §3.2 · p. 7, Figure 2 · p. 8
  • MEQ's accuracy peaks at 80.0 at k=10, versus 77.2 at k=300; matched-caption entailment probability rises while irrelevant-caption probability falls in success cases.The plotted comparisons show probability over iterations, with read depth K=10 marked.Source: §3.2 · pp. 7–8, Figure 3 · p. 8

3.3 ABLATION STUDIES

  • On VQA v2, removing cross-attention costs 4.20 pp and reducing to one iteration also degrades performance; removing the soft gate costs only 0.54 pp.The learned gate also collapsed to approximately 10^-7 on real inputs, zeroing both cross-modal Jacobians.Source: §3.3 · p. 8, Table 2 · p. 9
  • On SNLI-VE, replacing the cross-modal mixer with gMLP yields indistinguishable accuracy, 75.26% against 75.15%; additional post-iteration mixing changes accuracy by only +0.01 pp.The reported comparisons use parameter-matched variants.Source: §3.3 · pp. 8–9
  • State-layout ablations favor MEQ on VQA v2, while collapsing CMU-MOSEI's three modality states costs 0.84 pp; on Hateful Memes, the layouts do not separate.The VQA v2 margins over the joint-state and feature-sum baselines are 0.35 pp and 11.39 pp, respectively.Source: §3.3 · p. 9
  • Single-modality controls find CMU-MOSEI indistinguishable without audio or video, whereas removing the cross-modal path costs 4.20 pp on VQA v2; the authors attribute the CMU-MOSEI result to its benchmark under their feature pipeline.The controls also separate on Hateful Memes and SNLI-VE; SNLI-VE's text-only score is 70.42, reflecting a known annotation artifact.Source: §3.3 · p. 9, Table 3 · p. 9

4 CONCLUSION AND FUTURE WORKS

MEQ’s contribution is iterative information refinement through mutual feedback, extending beyond multimodal fusion. The conclusion identifies theoretical foundations of iterative refinement under constraints as an open direction and points to broader applications.

4 CONCLUSION AND FUTURE WORKS

  • The paper frames MEQ’s broader contribution as iterative information refinement, beyond its demonstrated use in bimodal fusion.The reported classification performance is better than or on par with baselines.Source: §4 · pp. 9–10
  • The authors identify multi-modal generalization, modular LLMs, agent agreement, and when iteration benefits problems as future directions.Source: §4 · p. 10
  • A central open question is how to ground iterative refinement theoretically under the constraints of each proposed setting.The conclusion connects this need to MEQ’s coupled-dynamical-system structure and calls for further study of its links to traditional dynamical systems.Source: §4 · p. 10
  • The paper reports that every number can be regenerated from released run records, source, and scripts that rebuild tables byte-identically.The described bundle also includes a verifier and SHA-256-indexed checkpoints; it is slated for public release with the final paper version.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The study uses publicly available datasets and no materials derived from human or animal subjects.Source: ETHICS STATEMENT · p. 10

A.1 PROOF OF LEMMA 1

The appendix derives fixed-point Jacobians with the implicit function theorem, interprets them as repeated self- and cross-modal influence, and relates those influences to input sensitivity.

A.1 PROOF OF LEMMA 1

  • The proof applies the implicit function theorem to both fixed points to derive coupled Jacobian equations with respect to an independent variable.The derivation assumes x and y are independent of that variable; otherwise the relevant partial derivatives must be changed.Source: §A.1 · p. 13

A.2 INFLUENCE DYNAMICS

  • The influence expansion combines direct self-influence with cross-modal paths that pass through the other component and return, summing repeated influence paths.The inverse term is interpreted as an infinite sum, so the expression accounts for all possible influence paths.Source: §A.2 · p. 14

A.3 SENSITIVITY

  • Sensitivity analysis asks how a small perturbation to the joint input u = [x; y] changes the fixed point and derives an approximate bound using Taylor expansion.Source: §A.3 · p. 14

A.4 SHARPER CONDITION FOR INVERTIBILITY

This section gives sufficient conditions for invertibility of the total influence term. It contrasts a Jacobian-norm constraint with a sharper condition that requires more computation.

A.4 SHARPER CONDITION FOR INVERTIBILITY

  • Constraining the norm of the total Jacobian is presented as a sufficient alternative that permits fine-grained tuning, while the sharper condition costs more computation.Source: §A.4 · pp. 14–15
  • The proposed sharper sufficient condition for invertibility assumes ρ(G_z*y) < 1 and ρ(F_z*x) < 1.Source: §A.4 · p. 14
  • The proof uses norm bounds on the self-influence terms and a Neumann-sum argument to establish a sufficient invertibility bound for the cross-influence term.The text notes that the norm upper-bounds the spectral radius and uses this because the spectral radius is hard to compute.Source: §A.4 · p. 14

B MORE THAN TWO MODALITIES

The appendix sketches two ways to extend MEQ beyond two modalities: hierarchical pairwise composition and complete-graph interaction. Each comes with a distinct limitation.

B MORE THAN TWO MODALITIES

  • Hierarchical composition repeatedly applies MEQ to modality pairs, then combines each pair’s embeddings into a temporary summary before adding the next modality.The merge order may be random or chosen using domain knowledge.Source: B MORE THAN TWO MODALITIES · p. 15
  • Hierarchical composition risks high-variance performance when modality order is random and adds parameters for the combiner MLP.Source: B MORE THAN TWO MODALITIES · p. 15
  • Complete-graph composition lets every modality interact with every other modality through its individual encoding function.Source: B MORE THAN TWO MODALITIES · p. 15
  • For complete-graph composition, the standard Jacobian analysis no longer yields simple solutions, and the larger Jacobian requires further analysis.The Jacobian has N^2 block Jacobians, so the inverse no longer has the interpretable decomposition given by Lemma 1.Source: B MORE THAN TWO MODALITIES · p. 15

C.1 EXPERIMENTAL SETUPS

The experiments use frozen pretrained image and text encoders, damped fixed-point updates, and benchmark-specific data and evaluation protocols.

C.1 EXPERIMENTAL SETUPS

  • VQA v2 uses 3,129 answer classes and a question-level 9:1 split of val2014; 99.7% of held-out questions share an image with training.The image encoder is frozen, so the exposure reaches the model only through the fusion block; results estimate held-out-question rather than held-out-image performance.Source: §C.1 · p. 15
  • MEQ runs 10 damped fixed-point updates, with a Jacobian range penalty and benchmark-specific mixing settings.Training uses AdamW, 5% linear warmup, and fixed-point tolerance 10^-3; Table 5 specifies β settings.Source: §C.1 · pp. 15–16, Table 5 · p. 16
  • Results are three-seed means on each benchmark’s validation split; VCR uses its first 5,000 validation examples, and CMU-MOSEI combines image with audio before text.The CMU-MOSEI setup is hierarchical because it is tri-modal.Source: §C.1 · p. 16
  • All experiments use frozen pretrained image and text encoders; the MEQ module uses 768-dimensional hidden states and 8 attention heads.The image encoder is frozen CLIP and the text encoder is frozen BERT-base, with per-benchmark variations.Source: §C.1 · pp. 15–16, Table 6 · p. 16

C.2 EXTRA TASK RESULTS

The extra experiments examine SNLI-VE accuracy over iterations, robustness to corrupted Hateful Memes inputs, and results on a COD10k visual-entailment task.

C.2 EXTRA TASK RESULTS

  • SNLI-VE dev accuracy rises steeply over the first three updates and is flat from k=5 onward, so gains occur early rather than by approaching a root.Figure 4 reports dev accuracy at each iteration k.Source: §C.2 · p. 16, Figure 4 · p. 17
  • Under strong additive image noise, MEQ leads by 1.88 pp (σ=0.3; paired 2σ of 1.51 pp over three seeds and five corruption draws).The Hateful Memes robustness test also perturbs images with noise and blur and texts with typos.Source: §C.2 · pp. 16–17, Table 7 · p. 17
  • The COD10k experiment compares directly trained MEQ and baseline architectures on a binary visual-entailment task using images and automatically generated captions.Results average three random seeds; learning rates are selected per architecture for training stability.Source: §C.2 · p. 17, Table 8 · p. 17
  • On VQA v2, the small residual under λf=0.3 is an identity collapse: seed 42 ends at 7.62 × 10^-8 with Jacobian norm 1.000.This reports the residual-penalty effect summarized in Table 9.Source: §C.2 · p. 17

C.3 ANALYSIS EXPERIMENTS

The analysis tests residual-penalty collapse, state design, fixed-point behavior, and inference cost, showing that lower residuals or restored coupling do not necessarily improve accuracy.

C.3 ANALYSIS EXPERIMENTS

  • Removing the residual penalty restores VQA v2 coupling and improves accuracy by 9.04 pp, while a gate floor recovers 2.30 pp without restoring coupling.Table 9 covers the residual penalty; Table 10 reports structural safeguards. Directly restoring cross-modal sensitivity with an off-diagonal-block hinge did not yield a called accuracy gain.Source: §C.3 · p. 18, Table 9 · p. 18, Table 10 · p. 18
  • On CMU-MOSEI, maintaining a state per modality is worth 0.84 pp, while cross-reading is indistinguishable from noise; on VQA v2, removing the cross-modal path costs 4.20 pp.The CMU-MOSEI single-state arm has a residual 25 times smaller and still loses; Table 11 compares state designs.Source: §C.3 · p. 19, Table 11 · p. 19
  • Only CMU-MOSEI is contractive at read depth K=10, and iterating far beyond K degrades the metric on every benchmark.Residuals fall by factors of three to eighty before read depth, supporting an approximate fixed point; Table 12 reports fixed-point behavior.Source: §C.3 · p. 19, Table 12 · p. 19
  • At K=10, MEQ inference costs 2.13× the fastest single-pass baseline in wall clock and 1.87× in FLOPs.Each iteration adds 1.687 GFLOPs per sample; peak memory stays constant with K because the forward carries one state, and a tolerance-driven run reaches the cap of 100.Source: §C.3 · p. 19, Table 13 · p. 20
Loading 2609.39456v1…