PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient and Effective Adaptation of Multimodal Foundation Models in Sequential Recommendation

Junchen Fu, Xuri Ge, Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, Kaiwen Zheng, Yongxin Ni, Joemon M. Jose

arXiv:2411.02992v2cs.IRcs.CV

TL;DR

Adapting large multimodal encoders for sequential recommendation remains costly, and IISAN is limited to symmetrical MFMs. IISAN-Versa extends adaptation to symmetrical and asymmetrical encoders, reporting improved performance with larger text encoders and strong results across the evaluated multimodal scenarios.

  • Problem

    Fine-tuning large encoders requires substantial computational resources, while IISAN is limited to symmetrical MFMs and has difficulty accommodating larger text encoders.

    Source: §I · pp. 1–2, §II · p. 3

  • Method

    IISAN-Versa adapts symmetrical and asymmetrical MFMs using intra- and inter-modal side networks, with dimension transformation alignment and group LayerDrop for asymmetry.

    Source: §I · p. 2, A. Overview · p. 3, E. LayerDrop · p. 5

  • Results

    Larger text encoders typically perform better, and IISAN-Versa achieves state-of-the-art performance on the public MicroLens benchmark.

    Source: §I · p. 2

  • Takeaways & Limitations

    IISAN-Versa supports multimodal sequential recommendation across symmetrical and asymmetrical encoders and diverse multimodal-text scenarios.

    Source: §I · p. 2, §VII · p. 12

    Future work should incorporate additional modalities such as audio, raw video, or graph information and explore tasks including VQA and video retrieval.

    Source: §VII · p. 12

Abstract

from arXiv · show

Multimodal foundation models (MFMs) have revolutionized sequential recommender systems through advanced representation learning. While Parameter-efficient Fine-tuning (PEFT) is commonly used to adapt these models, studies often prioritize parameter efficiency, neglecting GPU memory and training speed. To address this, we introduced the IISAN framework, significantly enhancing efficiency. However, IISAN was limited to symmetrical MFMs and identical text and image encoders, preventing the use of state-of-the-art Large Language Models. To overcome this, we developed IISAN-Versa, a versatile plug-and-play architecture compatible with both symmetrical and asymmetrical MFMs. IISAN-Versa employs a Decoupled PEFT structure and utilizes both intra- and inter-modal adaptation. It effectively handles asymmetry through a simple yet effective combination of group layer-dropping and dimension transformation alignment. Our research demonstrates that IISAN-Versa effectively adapts large text encoders, and we further identify a scaling effect where larger encoders generally perform better. IISAN-Versa also demonstrates strong versatility in our defined multimodal scenarios, which include raw titles and captions generated from images and videos. Additionally, IISAN-Versa achieved state-of-the-art performance on the Microlens public benchmark. We release our code at https://github.com/GAIR-Lab/IISAN.

I. INTRODUCTION

The introduction frames practical training costs and IISAN’s restriction to symmetrical encoders as barriers to adapting larger text models. IISAN-Versa extends the framework to symmetrical and asymmetrical MFMs, with reported scaling effects and strong results across recommendation settings.

I. INTRODUCTION

  • Fine-tuning large encoders can require substantial computational resources, while EPEFT methods still face training-time and GPU-memory challenges.The introduction reports that IISAN’s decoupled PEFT reduces GPU memory use and training time per epoch compared with full fine-tuning and EPEFT.Source: §I · p. 1
  • IISAN’s hidden-state merging is limited to symmetrical encoders, making it difficult to explore larger LLM text encoders alongside smaller vision encoders.The paper notes that commonly used vision transformers have fewer than 1 billion parameters, while larger pre-trained transformers generally offer better performance.Source: §I · p. 1
  • IISAN-Versa addresses asymmetry with group layer-dropping and dimension transformation alignment, extending IISAN beyond symmetrical encoders.The framework also adapts titles and captions generated from images and videos in a multimodal-text scenario.Source: §I · p. 2
  • IISAN-VA significantly improves performance with state-of-the-art LLM text encoders, and larger pre-trained text encoders typically perform better.The paper also reports that IISAN-Versa achieves state-of-the-art performance on the public MicroLens benchmark.Source: §I · p. 2
  • IISAN-Versa is presented as a versatile paradigm for adapting symmetrical and asymmetrical MFMs with minimal dimension-transformation layers and LayerDrop.The paper categorizes the symmetrical variant as IISAN-VS and the asymmetrical variant as IISAN-VA.Source: §I · p. 2

II. RELATED WORK

Prior work leaves practical efficiency and asymmetrical multimodal adaptation as open challenges for sequential recommendation. IISAN-Versa addresses these challenges with side-adapted networks for both symmetrical and asymmetrical encoders, then combines their representations for next-item prediction.

II. RELATED WORK

  • Multimodal recommendation methods can outperform offline or single-modality approaches, but end-to-end methods often fully fine-tune large encoders and PEFT work frequently overlooks training speed and GPU memory.DPEFT adoption in recommendation remains limited, and existing methods commonly focus on one modality; IISAN also struggled to incorporate asymmetrical MFMs.Source: §II · p. 3

A. Overview

  • IISAN-Versa uses separate textual and visual intra-modal SANs and an inter-modal SAN, with dimension transformation and group LayerDrop addressing asymmetric encoder dimensions and depths.The framework decouples trainable side-adapted networks from pretrained backbones, enabling hidden-state caching.Source: A. Overview · p. 3

B. Problem Formulation

  • For next-item prediction, text and image representations from pretrained encoders feed three side-adapted towers, whose outputs are fused into item embeddings for a sequential encoder.The three outputs are text, image, and inter-modal representations.Source: B. Problem Formulation · p. 4

C. Intra-model Side Adapted Network

IISAN-Versa builds decoupled side-adapted networks from SAN blocks, using intra-modal adaptation for each modality and inter-modal fusion to combine their information. For asymmetrical encoders, it aligns embedding dimensions and merges hidden states through gating.

C. Intra-model Side Adapted Network

  • Each side-adapted network contains SAN blocks built from fully connected upsampling and downsampling layers; the adapter block is implemented decoupled from the backbone.A learnable gate fuses textual intra-modal information, and the visual SAN uses similar operations.Source: C. Intra-model Side Adapted Network · p. 4
  • Intra-SAN applies to symmetrical and asymmetrical frameworks, with SANB input and output dimensions enlarged as needed for larger encoders.The first SAN block takes text embeddings; the visual SAN uses similar operations.Source: C. Intra-model Side Adapted Network · p. 4

D. Inter-model Side Adapted Network

  • For asymmetrical encoders, group LayerDrop converts unequal hidden-state counts to a symmetrical form, while a dimension transformation layer maps higher-dimensional LLM embeddings to lower-dimensional vision outputs.The transformation takes an input feature in din dimensions to an output in dout dimensions, where din > dout.Source: D. Inter-model Side Adapted Network · p. 4
  • The inter-modal SAN fuses hidden states from both modalities and uses a gate to sum information from its last SAN block.The paper names the symmetrical and asymmetrical variants IISAN-VS and IISAN-VA.Source: D. Inter-model Side Adapted Network · p. 4

E. LayerDrop

LayerDrop reduces SAN-block redundancy and aligns encoder depths, while caching reuses pretrained hidden states to avoid repeated backbone passes. Training uses an in-batch debiased cross-entropy objective.

E. LayerDrop

  • LayerDrop reduces SAN blocks and aligns asymmetrical encoders; symmetrical models drop the first hidden state from each pair of transformer blocks, while asymmetrical models group LLM layers evenly.For asymmetrical models, the selected group size is chosen to maximize cross-layer interaction across the LLM.Source: E. LayerDrop · p. 5

F. Caching Strategies

  • Caching stores and reuses item hidden states from pretrained backbones, reducing repeated forward passes during training.For asymmetrical IISAN-Versa, caching avoids moving the LLM onto GPUs, supporting adaptation with limited resources.Source: F. Caching Strategies · p. 5

G. Training Objectives

  • Training uses the in-batch debiased cross-entropy loss adopted in academic and industrial systems.The loss passage defines item popularity, the user's interacted-item set, and the batch as terms in its formulation.Source: G. Training Objectives · p. 5

IV. EFFICIENCY ANALYSIS

The efficiency analysis contrasts full fine-tuning, embedded PEFT, and IISAN-Versa’s decoupled PEFT, including its cached variant. It argues that decoupling reduces backward computation, while caching can also reduce forward computation.

IV. EFFICIENCY ANALYSIS

  • IISAN-Versa decouples backbone hidden states from standalone PEFT modules, reducing backward computation through large foundation models.The cached variant stores hidden states on disk rather than recalculating them, further optimizing the forward pass.Source: §IV · p. 6
  • Caching hidden states on disk avoids recalculating them during training and can make the forward pass more efficient.The cached results can be reused in subsequent training.Source: §IV · p. 6
  • Embedded PEFT still traverses the foundation model, so it does not substantially reduce the computational load of forward and backward propagation.The added PEFT-module cost is generally negligible under the analysis assumption that transformer blocks are much larger than PEFT modules.Source: §IV · pp. 5–6

V. EXPERIMENT SETUP

The section introduces the experimental setup topics covered for IISAN-Versa.

V. EXPERIMENT SETUP

  • The section outlines the datasets used to evaluate IISAN-Versa.It also covers caption generation, evaluation methods, and implementation details.Source: §V · p. 6
  • Caption generation is included among the experimental setup topics.The section also covers datasets, evaluation methods, and implementation details.Source: §V · p. 6
  • Evaluation methods and implementation details are part of the experimental setup.The section also introduces the datasets and caption generation process.Source: §V · p. 6

A. Dataset

The setup evaluates IISAN-Versa on Amazon review datasets and MicroLens, including multimodal text scenarios built from image and video captions. It describes the evaluation protocol and implementation choices.

A. Dataset

  • The study uses three Amazon review datasets and MicroLens, with a one-million-interaction sample from NINERec to expand evaluation scale.MicroLens includes raw text, images, and video content; items are represented by raw modality information.Source: A. Dataset · p. 6

B. Multimodal Text Generation

  • For MicroLens multimodal text scenarios, LLaVA generates captions from cover images and video frames for evaluation alongside original text.The study samples three evenly spaced frames per video and uses separate LLaVA models for cover-image and video captioning.Source: B. Multimodal Text Generation · pp. 6–7

C. Performance Evaluations.

  • Evaluation reserves the final sequence item for testing and the penultimate item for validation, using HR@10 and NDCG@10 against the complete item set.Earlier interactions train the model; unless otherwise stated, results are reported on the test set.Source: C. Performance Evaluations · p. 7

D. Implementation Details.

  • Implementation uses pretrained Hugging Face encoders and selects hyperparameters on validation performance, with experiments run on one A6000 GPU.The sequential encoder uses 64 dimensions, two Transformer blocks, and two attention heads; the Llama-3-70b model uses GPTQ 4-bit quantization due to limited computing resources. Figure 4 organizes Scientific-dataset Hit@10 results by text-encoder scale and group averages.Source: D. Implementation Details · p. 7, Figure 4 · p. 7

VI. EXPERIMENT

IISAN-Versa’s experiment section poses questions about encoder scale, benchmarks, efficiency, components, multimodal scenarios, and hyperparameters. For the larger-text-encoder comparison, IISAN-VA improves on IISAN-VS across all three Amazon datasets.

VI. EXPERIMENT

  • The experiment frames RQ1–RQ7 around larger text encoders, scaling, public benchmarks, efficiency, component effects, multimodal text scenarios, and hyperparameters.RQ1 asks whether a larger text encoder improves asymmetrical IISAN-Versa over the symmetrical version.Source: §VI · p. 7

A. IISAN-Versa Performance with larger text encoders (RQ1)

  • IISAN-VS outperforms full fine-tuning and the Adapter, LoRA, and BitFit baselines across the three Amazon datasets.Those baselines were applied only to the symmetrical model because of their high computational cost.Source: A. IISAN-Versa Performance with larger… · p. 8
  • +6.31% HR@10 and +6.12% NDCG@10 for Scientific, +4.83% and +2.23% for Instrument, and +3.13% and +3.72% for Office are IISAN-VA gains over IISAN-VS.The comparison uses BERT-large for IISAN-VA and BERT-base for IISAN-VS, with ViT-base fixed as the image encoder.Source: A. IISAN-Versa Performance with larger… · p. 8, Table IV · p. 8

B. Scaling Effect (RQ2)

IISAN-Versa generally benefits from scaling text encoders, while larger vision encoders offer less benefit alongside strong text encoders. On MicroLens, it surpasses prior approaches and outperforms SASRec across reported accuracy metrics while using less GPU memory.

B. Scaling Effect (RQ2)

  • Llama3-70b-GPTQ achieves the strongest performance, and results generally improve as text encoders scale across the evaluated models.Average performance improves with each text-model scale group; most instruction-tuned LLMs do not outperform their non-instruction-tuned counterparts, except Mixtral 8x7b.Source: B. Scaling Effect (RQ2) · p. 8, Table V · p. 8
  • Larger vision encoders provide limited gains with strong text encoders, while EVA-CLIP-18B boosts performance with lightweight text models.In some cases, ViT-large or ViT-huge reduce performance, especially alongside Llama3-70B-GPTQ.Source: B. Scaling Effect (RQ2) · pp. 8–9

C. Performance on Public Benchmark (RQ3)

  • IISAN-VS and IISAN-VA surpass existing approaches on MicroLens, and IISAN-VA outperforms IISAN-VS; the framework also surpasses the previous state of the art.Both IISAN-Versa variants outperform existing approaches; the best models’ improvements over GRU4RecV are marked significant in Table VII.Source: Table VII · p. 9, C. Performance on Public Benchmark (RQ3) · pp. 9–10

B. Scaling Effect (RQ2)

C. Performance on Public Benchmark (RQ3)

D. Efficiency Analysis (RQ4)

IISAN-Versa reduces GPU memory use and training time relative to EPEFT approaches, with memory savings especially apparent at larger batch sizes. Caching trades manageable on-disk storage for efficiency, and dropping unnecessary layers can further reduce storage and training costs.

D. Efficiency Analysis (RQ4)

  • IISAN-Versa reduces GPU memory usage as batch size grows and significantly improves training speed compared with EPEFT methods.The comparison fixes BERT-base and ViT-base as text and vision encoders and tests batch sizes from 1 to 32.Source: D. Efficiency Analysis (RQ4) · pp. 10–11
  • Caching Llama-70b-GPTQ hidden states for Scientific requires 26 GB, reducible to 2.4 GB by retaining six layers.The caching strategy trades on-disk memory for training-time and GPU-memory efficiency.Source: D. Efficiency Analysis (RQ4) · p. 10
  • Using only six essential Llama-70b-GPTQ layers significantly reduces training time when CPU loading and processing otherwise become the bottleneck.Full embeddings increase training cost; GPU memory also rises with text-encoder size and can be slightly reduced by offloading unnecessary layers.Source: D. Efficiency Analysis (RQ4) · p. 10
  • Caching incurs additional on-disk memory, but the costs are generally manageable and can be lowered by caching only useful layers after layer dropping.The table caption defines its LayerDrop method values as the number of blocks retained.Source: Table X · p. 10, D. Efficiency Analysis (RQ4) · p. 11

E. Ablation Studies (RQ5)

The ablations find that combining intra- and inter-modal adaptation gives the best performance, while layer dropping and gated fusion shape the balance between efficiency and modality use.

E. Ablation Studies (RQ5)

  • Using intra-modal and inter-modal SANs together gives the best performance for both IISAN-VS and IISAN-VA.Either SAN alone reduces effectiveness relative to using both, though each improves on frozen backbones.Source: E. Ablation Studies (RQ5) · pp. 11–12
  • Text gate weights exceed 0.5 while visual weights range from 0.26 to 0.42, with IISAN-VA relying more on text than IISAN-VS.Figure 9 visualizes the gate values at the optimal checkpoint.Source: E. Ablation Studies (RQ5) · p. 11
  • LayerDrop improves IISAN-VS performance by 0.1% on both HR@10 and NDCG@10, with six even-numbered blocks giving the best performance-efficiency balance.For IISAN-VA, the 6+6 structure has the best overall mean performance; exhaustive testing of all layer combinations was not performed.Source: E. Ablation Studies (RQ5) · p. 11

F. Performance on Multimodal Text (RQ6)

IISAN-Versa handles multimodal text combinations, with titles and video captions the most informative among those tested, though raw cover images with titles perform best. Its variants are robust to hyperparameter changes, and the paper identifies additional modalities as future work.

F. Performance on Multimodal Text (RQ6)

  • Combining raw titles and video captions gives the best performance among tested multimodal-text combinations, but titles with raw cover images perform best overall.The experiments use two BERT-base models to encode raw titles and generated captions from videos or cover images.Source: F. Performance on Multimodal Text (RQ6) · p. 12

G. Hyper-parameter Sensitivity Analysis (RQ7)

VII. CONCLUSION AND FUTURE WORK

  • The conclusion identifies audio, raw video, and graph information as additional modalities for future research.It also proposes exploring IISAN-Versa for visual question answering and video retrieval.Source: §VII · p. 12

Related papers

Loading 2411.02992v2…