PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent-Aligned Reasoning for Multimodal Recommendation

Jiarui Jin, Anyang Ji

arXiv:2609.04645v1cs.IRcs.CLcs.CVcs.LG

TL;DR

Multimodal recommendation with VLMs faces cross-modal dilution, as visual and textual signals attenuate through multi-step reasoning. LARK uses two stages of latent reasoning with vision and reasoning-text alignment, and it consistently improves over baselines across public and industrial datasets and multiple architectures.

  • Problem

    VLM reasoning for recommendation can progressively attenuate both visual and textual signals, while prior latent-reasoning methods had not addressed dense recommendation embeddings requiring preservation of both modalities.

  • Method

    LARK interleaves learnable latent tokens with CoT reasoning, aligns them with a frozen vision encoder, then uses bridge-MLP contrastive training with alignment to CoT hidden states.

  • Results

    LARK consistently outperforms baselines across three public benchmarks, one industrial dataset, and multiple recommendation architectures, with up to 15.1% gains as drop-in feature replacements.

  • Takeaways & Limitations

    The results support using complementary vision and reasoning-text alignment to preserve multimodal information across the recommendation reasoning pipeline.

Abstract

from arXiv · show

Multimodal Vision-Language Models (VLMs) have demonstrated remarkable capabilities in cross-modal understanding, yet a fundamental challenge persists when applying them to recommendation: as representations propagate through multi-step reasoning, both visual and textual signals progressively attenuate - a phenomenon we term cross-modal dilution. To address this, we propose LARK (Latent-Aligned Reasoning frameworK), a two-stage latent reasoning framework with complementary alignment mechanisms within a single VLM. In the first stage, learnable latent tokens are interleaved with multi-step chain-of-thought (CoT) reasoning and explicitly aligned with a frozen vision encoder, serving as visual checkpoints that preserve perceptual details throughout the reasoning chain. In the second stage, the latent representations are projected via a bridge MLP and trained with item-to-item contrastive learning; to prevent the reasoning semantics from fading, intermediate features are aligned with the CoT hidden states from the first stage, anchoring the final embeddings to the model's own reasoning output. Experiments on three public benchmarks and one industrial dataset show that LARK achieves state-of-the-art performance across multiple recommendation architectures, with controlled ablations confirming the distinct contribution of each component.

1 INTRODUCTION

Multimodal recommendation must preserve visual and textual information as VLM reasoning unfolds, because both signals can attenuate over long chains. LARK addresses this with two aligned latent-reasoning stages and shows consistent gains across datasets and recommendation architectures.

  • Motivation: VLM reasoning can suffer cross-modal dilution as visual and textual signals progressively attenuate over longer generated sequences.Visual-token attention has been observed to decay as reasoning chains grow.
  • Motivation: Existing latent-reasoning methods use continuous hidden states but had not addressed recommendation, where dense embeddings must preserve both visual and textual signals.These methods were developed primarily for text-generation tasks.
  • LARK: LARK interleaves learnable latent tokens with multi-step CoT reasoning and aligns them with a frozen vision encoder as visual checkpoints.The first stage also supervises CoT text, while the second stage aligns intermediate features with first-stage CoT hidden states.
  • LARK: LARK projects latent representations through a bridge MLP for item-to-item contrastive learning and anchors final embeddings to the model’s reasoning output.The two stages use complementary alignment mechanisms to counteract signal dilution.
  • Results: LARK consistently outperforms baselines across three public datasets, one industrial dataset, and DeepFM, LightGCN, and SASRec architectures.Controlled ablations isolate component contributions, while attention analyses indicate mitigation of cross-modal dilution.

2 PRELIMINARIES

The paper formulates multimodal recommendation as learning item and user representations from visual, textual, and interaction data. It supplements user-item ranking with Swing-derived item-to-item supervision and situates LARK among multimodal recommendation and latent-reasoning methods.

  • Problem Formulation: Each item has an image and textual description, while binary user-item interactions indicate whether a user interacted with an item.The formulation represents users and items as sets and interactions as a binary matrix.
  • Problem Formulation: Multimodal recommendation learns dense item and user embeddings whose inner product predicts the likelihood of user-item interaction.The item encoder jointly maps visual and textual content into an item representation.
  • Problem Formulation: The model optimizes a Bayesian Personalized Ranking objective over training triples containing one interacted item and one non-interacted item.The objective uses a sigmoid over the difference between positive- and negative-item scores.
  • Item-to-Item Signal: Swing mines item-to-item relevance from users who interacted with both items, then retains top-K similar items as positive supervision.The method down-weights co-occurrences associated with highly overlapping user histories to reduce popularity bias.
  • Related Work: Prior multimodal recommenders combine pre-extracted modality features, modality-specific graphs, or self-supervised alignment, while newer work applies VLMs and latent reasoning.LARK targets the recommendation setting where both visual and textual signals must remain available in dense embeddings.

3 LARK

LARK performs multimodal recommendation as two-stage latent reasoning within one VLM, using complementary alignments to preserve visual details and reasoning semantics. It produces compact item embeddings for downstream recommendation models while supporting offline deployment.

  • Vision-Language Reasoning Stage: LARK interleaves latent-token reasoning with language-mode CoT segments, using continuous hidden-state feedback for latent positions and teacher-supervised token generation for language positions.Latent tokens are inserted before each CoT step according to a fixed template.
  • Item-Item Alignment Stage: LARK serves as an offline multimodal item encoder whose frozen embeddings are consumed by independently updated downstream recommendation models.A projection head mean-pools hidden states and maps them to D-dimensional item embeddings.
  • Vision-Language Reasoning Stage: Vision alignment matches first-stage intermediate features to frozen vision-encoder patch features, anchoring latent representations to a semantically rich visual space.The alignment maximizes patch-wise cosine similarity and is intended to prevent representations from drifting through the reasoning chain.
  • Item-Item Alignment Stage: A Bridge MLP maps first-stage latent hidden states into the embedding space for second-stage processing, which operates on continuous vectors rather than discrete tokens.The second stage derives compact item embeddings from the bridged representations and a task prompt.
  • Item-Item Alignment Stage: Reasoning-text alignment matches second-stage intermediate features to pooled first-stage CoT hidden states, preserving reasoning semantics in the final item embeddings.Stop-gradient prevents this alignment loss from back-propagating into first-stage language generation, while contrastive discrimination and causal attention maintain token-level diversity.

4 EXPERIMENTS

Across four datasets and multiple recommendation architectures, LARK consistently outperforms baselines, while ablations and analyses attribute gains to latent reasoning, dual alignment, and robust representation choices.

  • Overall performance: LARK consistently outperforms all baselines across four datasets and all metrics, regardless of the downstream recommendation model.On In-House, LARKLightGCN reaches R@20 of 0.0548, improving over AlignRec by 10.0%.
  • Overall performance: Multimodal methods outperform pure collaborative filtering, showing the value of incorporating visual and textual signals beyond interaction patterns.AlignRec is the strongest baseline on most datasets, while NoteLLM-2 is competitive or superior on Sports.
  • Ablation study: Removing CoT supervision reduces Baby R@20 by 6.1%, while replacing latent tokens with discrete text causes a further 9.5% drop.The ablation indicates that CoT supervision supports semantically meaningful latent states and continuous representations retain fine-grained information beyond discrete tokens.
  • Ablation study: Removing vision alignment lowers Baby R@20 by 5.5%, removing reasoning-text alignment lowers it by 8.4%, and removing both lowers it by 11.8%.The two objectives are complementary: vision alignment preserves visual information, while text alignment preserves reasoning conclusions.
  • Ablation study: Removing latent reasoning leaves the VLM backbone 17.4% below LARK on Baby R@20, demonstrating gains beyond backbone strength.The comparison directly pools the VLM’s last hidden states for contrastive training instead of using latent tokens and CoT supervision.
  • Sensitivity and design choices: Loss weights near 0.1 perform best, 50% vision masking is optimal, and the VLM’s own ViT outperforms DINOv2 as the alignment target.Weights of at least 0.5 degrade embedding quality; excessive masking discards visual information, while DINOv2 trails the default by 3.9% on Baby R@20.
  • Representation analysis: Attention and CKA analyses show that dual alignment mitigates progressive multimodal signal loss during reasoning.Without alignment, visual CKA falls from 0.82 to 0.45 and textual CKA from 0.78 to 0.49 by the fourth step; LARK keeps visual CKA above 0.72.
  • Additional analyses: LARK yields larger gains for weaker multimodal baselines, including +13.9% R@20 on Baby and +15.1% on Sports over MMGCN.FREEDOM, which already uses feature refinement, improves by smaller but meaningful margins of +5.6% and +5.8%.

5 CONCLUSION AND FUTURE WORK

LARK repurposes a VLM as a multimodal item encoder using latent-aligned reasoning and dual alignment to counteract cross-modal dilution. Across three Amazon benchmarks and one industrial dataset, it consistently improves over state-of-the-art baselines, including up to 15.1% gains as drop-in feature replacements.

  • LARK repurposes a vision-language model as a multimodal item encoder for recommendation.
  • Its two-stage pipeline combines vision alignment in Stage 1 with reasoning-text alignment in Stage 2 to counteract cross-modal dilution.
  • 15.1% gains are achieved over state-of-the-art baselines as drop-in feature replacements across three Amazon benchmarks and one industrial dataset.

A OVERALL ALGORITHM

LARK training alternates between two stages: vision-language reasoning with frozen visual alignment and item-item alignment with reasoning-text regularization. Inference applies the same pipeline to produce final item embeddings.

  • Training alternates between Stage 1 interleaved latent-language reasoning with vision alignment and Stage 2 item-item contrastive learning with reasoning-text alignment.
  • At inference, the two-stage pipeline produces a final item embedding e_i for each item.
  • The algorithm initializes all parameters and samples item batches before executing the vision-language reasoning stage.
  • Stage 1 aligns frozen vision features with latent representations, while Stage 2 updates the model using item-item alignment and weighted CoT, vision, and text losses.

B PROMPTS

LARK generates structured four-step CoT annotations from product images and titles, then uses the resulting item representations in offline inference. The annotations are constrained, quality-checked, and generated once before training.

  • Prompt design: Gemini-2.5-Flash generates four-step CoT annotations covering visual attributes, functional analysis, target audience, and distinguishing features.
  • Prompt design: The prompt takes an item image and title as input and requests structured analysis beginning with a product title field.
  • Inference: At inference, trained components generate item embeddings, which are stored in a vector index for downstream recommendation.
  • Inference: CoT generation runs in a single pass over the entire item catalog before training begins.
  • Prompt design: Each reasoning step is limited to one sentence under 15 words and focuses on the product itself.
  • Data statistics: The generated reasoning chains average approximately 55 tokens across four steps.
  • Quality control: Fewer than 1.2% of items require regeneration after missing-step verification, and fewer than 0.3% produce duplicate chains.
Loading 2609.04645v1…