PaperlayerView exact revision on arXiv

Source-linked AI summary

VIP5: Towards Multimodal Foundation Models for Recommendation

Shijie Geng, Juntao Tan, Shuchang Liu, Zuohui Fu, Yongfeng Zhang

arXiv:2305.14302v2cs.IRcs.AIcs.HCcs.LGcs.MM

TL;DR

Recommendation research has often treated visual, textual, and personalization information separately, despite foundation models offering a shared interface. VIP5 combines those modalities through personalized prompts and tunes lightweight adapters while freezing the P5 backbone. The paper reports stronger results than baselines across three task groups, with savings in training time and memory.

  • Problem

    Visual, textual, and personalization approaches have developed separately, leaving a need for a shared model to process these modalities across recommendation tasks.

    Source: §1 · pp. 1–2

  • Method

    VIP5 combines visual, textual, and personalization inputs in multimodal personalized prompts and tunes lightweight adapters while keeping the P5 backbone fixed.

    Source: §1 · p. 2

  • Results

    VIP5 beats strong baselines on three task groups while saving substantial training time and memory.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported performance and efficiency trade-off makes foundation models easier to train and deploy for recommendation.

    Source: §1 · p. 2

    VIP5 relies on training-data quality and diversity, and existing biases may lead to biased and unfair recommendations.

    Source: Limitations and Future Work · p. 9

Abstract

from arXiv · show

Computer Vision (CV), Natural Language Processing (NLP), and Recommender Systems (RecSys) are three prominent AI applications that have traditionally developed independently, resulting in disparate modeling and engineering methodologies. This has impeded the ability for these fields to directly benefit from each other's advancements. With the recent development of foundation models, large language models have emerged as a potential general-purpose interface for unifying different modalities and problem formulations. In light of this, we propose the development of a multimodal foundation model (MFM) considering visual, textual, and personalization modalities under the P5 recommendation paradigm, thus named VIP5 (Visual P5), to unify various modalities and recommendation tasks. This will enable the processing of multiple modalities in a shared architecture for improved recommendations. To achieve this, we introduce multimodal personalized prompts to accommodate multiple modalities under a shared format. Additionally, we propose a parameter-efficient training method for foundation models, which involves freezing the P5 backbone and fine-tuning lightweight adapters, resulting in improved recommendation performance and increased efficiency in terms of training time and memory usage. Code and data of VIP5 are available at https://github.com/jeykigung/VIP5.

1 Introduction

VIP5 brings visual, textual, and personalization information together in a shared recommendation model. It uses multimodal prompts and lightweight adapters, and the paper reports stronger results across three task groups with lower training time and memory use.

1 Introduction

  • Foundation models offer a route to unify visual, textual, and personalized information, while these areas have historically developed across separate research communities.P5 and M6-Rec had already used instructional prompts to support multiple recommendation tasks under a shared model and objective.Source: §1 · p. 1
  • VIP5 maps multimodal information into a shared tokenized space and keeps the large language model backbone fixed while tuning lightweight adapters.The paper contrasts this approach with pretraining that updates all backbone parameters.Source: §1 · p. 2
  • VIP5 beats strong baselines on three task groups while saving substantial training time and memory.The framework combines multimodal learning with parameter-efficient tuning.Source: §1 · p. 2

2 Related Work

Related work covers prompt-learning approaches, large recommendation models, and multimodal recommendation methods. These lines of work provide context for VIP5’s combination of task prompts, personalization, and multimodal inputs.

Prompt Learning

  • Prompt learning ranges from discrete prompt search to continuous soft prompts and natural-language instruction prompts for downstream tasks.Soft prompts are harder to interpret than discrete prompts, while instruction prompts describe tasks directly in natural language.Source: §2 · pp. 2–3

Large Recommendation Models

  • Recommendation models have used personalized prompts, behavior sequences, and instruction-based objectives to represent users and items or unify tasks.P5 and OpenP5 unify recommendation tasks through natural-language instructions in a shared encoder-decoder architecture and joint objective.Source: §2 · p. 3

Multimodal Recommendation

  • Multimodal recommendation uses content as side information, generates visual explanations, or models intra-item and inter-item semantic structure.Examples include visual features for matching and methods that fuse visual, semantic, and collaborative information.Source: §2 · p. 3

3 VIP5 Paradigm with Multimodal Personalized Prompts

VIP5 combines textual, visual, and personalization information in multimodal prompts and uses adapters for parameter-efficient tuning. The prompts convert image features into item-associated tokens, while an encoder-decoder Transformer processes inputs and generates task outputs.

3.1 Multimodal Personalized Prompts

  • VIP5 uses multimodal personalized prompts to connect textual, visual, and personalization information through a foundation model as a general-purpose interface.Figure 2 depicts textual and image inputs to an encoder-decoder Transformer that produces responses or recommendation decisions.Source: §3.1 · p. 3, Figure 2 · p. 4
  • A visual encoder extracts image features, and a two-layer mapping network converts each feature into k image tokens appended to corresponding item tokens.The combined item and image tokens form the multimodal personalized field M.Source: §3.1 · pp. 3–4
  • VIP5 creates 29 prompts for sequential recommendation, direct recommendation, and explanation, substituting the multimodal field for the item field.Sequential and direct recommendation prompts can contain multiple multimodal personalized fields.Source: §3.1 · p. 4

3.2 Parameter-efficient Tuning with Adapters

  • The encoder-decoder model processes tokenized multimodal sequences using position, whole-word, and category embeddings before encoding and decoding them.The category embedding distinguishes textual from visual tokens.Source: §3.2 · p. 4
  • VIP5 inserts adapters after self-attention and feed-forward modules before LayerNorm, and also places adapters between decoder cross-attention and LayerNorm.The backbone parameters are frozen while lightweight adapter modules are updated, reducing trainable parameters, training time, and memory use.Source: §3.2 · pp. 4–5
  • The adapters are bottleneck fully connected layers with an activation, using down-sampling and up-sampling layers around a residual connection.The adapter placements and bottleneck structure are specified for encoder modules and decoder cross-attention.Source: §3.2 · pp. 4–5
  • VIP5 trains with conditional next-token generation after encoding multimodal prompts, conditioning decoder predictions on prior generated tokens and input text.Sequential and direct recommendation use beam search for candidate lists, while explanation generation uses greedy decoding.Source: §3.2 · p. 5

4 Experiments

The experiments compare VIP5 across three recommendation task groups and four datasets, then examine adapter tuning and visual prompt choices. VIP5 generally compares favorably with baselines, while visual gains vary by task and tuning choices trade performance against efficiency.

4.1 Experimental Setups

  • Experiments use four Amazon datasets across sequential recommendation, direct recommendation, and explanation generation, with seen and zero-shot unseen prompts for evaluation.The paper compares against P5 and task-specific baselines, using HR@k and NDCG@k for recommendation and BLEU and ROUGE for explanation generation.Source: §4.1 · pp. 5–6

4.2 Performance on Different Task Groups (RQ1)

  • VIP5 beats all sequential-recommendation baselines on four datasets, wins on Sports for direct recommendation, and outperforms baselines on explanation tasks with multimodal prompts.For direct recommendation, VIP5 is marginally lower on HR@10 for Toys, Beauty, and Clothing but exceeds baselines on other metrics; with Prompt C-12, it surpasses P5 across datasets and metrics.Source: Table 2 · p. 6, Table 3 · p. 6, §4.2 · p. 7

4.3 Ablation on Parameter-efficient Tuning (RQ2)

  • Adapters in both self-attention and cross-attention blocks achieve better or comparable results to full fine-tuning, while saving 21.2% training time and 18.1% memory.The comparison is on Toys; adapter tuning performs better for Prompt C-12 and comparably for Prompts A-9 and B-8.Source: §4.3 · pp. 7–8, Figure 4 · p. 8, Table 5 · p. 9
  • Adapter reduction rates of 4 and 8 are suitable across all three task groups; lowering the rate increases bottleneck size and trainable parameters.The adapter reduction-rate ablation uses ResNet-101 features and a single image token.Source: §4.3 · p. 8

4.4 Ablation on Visual Components (RQ3)

  • Visual signals improve direct recommendation and explanation generation across datasets, but bring no obvious gains for sequential recommendation.The prompt comparison uses zero versus one image token, a reduction rate of 8, and ResNet-101 features; explanation gains are especially notable on Toys.Source: Figure 3 · p. 7, §4.4 · p. 8
  • Five image tokens work best for Prompts A-9 and B-8, while two work best for C-12; five tokens take 60.8% more time than two.The paper selects two image tokens as the default considering the performance-time trade-off.Source: §4.4 · p. 8, Figure 6 · p. 8
  • Visual-component tests vary downsample rate and encoder type; stronger encoders improve direct recommendation but do not consistently improve sequential recommendation or explanation generation.For explanation generation, hint words and their compatibility with visual embeddings vary across encoders; VIP5 remains better than the best baseline under most encoders.Source: §4.4 · pp. 8–9, Figure 5 · p. 8, Figure 7 · p. 8

5 Conclusions

VIP5 combines visual, language, and personalization information through multimodal prompts and parameter-efficient adapter tuning. Experiments support its effectiveness across recommendation tasks, while future work includes broader modalities and prompt strategies.

5 Conclusions

  • VIP5 unifies vision, language, and personalization in a parameter-efficient multimodal foundation model for recommendation.The model integrates visual signals with text and personalization information through multimodal personalized prompts.Source: §5 · p. 9
  • Adapter-based tuning updates a small proportion of parameters and offers a trade-off among recommendation performance, training efficiency, and memory usage.The conclusion characterizes the approach as improving performance while using less training time and memory.Source: §5 · p. 9
  • Extensive experiments support VIP5’s effectiveness and indicate that multimodal information is helpful for various recommendation tasks.Future work includes scaling the backbone, adding modalities, and exploring prompt strategies such as chain-of-thought prompting.Source: §5 · p. 9

Limitations and Future Work

The authors identify fairness, transparency, modality scalability, and LLM efficiency as limitations or areas for future work.

Limitations and Future Work

  • Training-data quality and diversity constrain fairness, as existing biases may lead to biased and unfair recommendations.The authors suggest investigating bias mitigation, data representativeness, and fairness.Source: Limitations and Future Work · p. 9
  • VIP5 lacks inherent transparency, which may hinder users’ trust in its recommendations.Enhancing transparency and explainability is identified as future work.Source: Limitations and Future Work · p. 9
  • Extending VIP5 to modalities such as audio or video remains a challenge.The authors identify efficient incorporation of additional modalities as an area for investigation.Source: Limitations and Future Work · p. 9
  • The efficiency work is an initial attempt focused on parameter-efficient tuning; pretraining, fine-tuning, and prompt-based inference efficiency remain future investigation areas.The authors note that real-world service latency should remain small.Source: Limitations and Future Work · p. 9

Appendix

The appendix presents multimodal personalized prompts for three recommendation task groups, with 29 prompts listed across them.

Appendix

  • The appendix lists 29 multimodal personalized prompts covering three recommendation tasks.The prompts are presented in Figures 8–10.Source: Appendix · p. 13
  • Figure 8 presents multimodal personalized prompts for Task Group A, Sequential Recommendation.Source: Figure 8 · p. 13
  • Figure 9 presents multimodal personalized prompts for Task Group B, Direct Recommendation.Source: Figure 9 · p. 14
  • Figure 10 presents multimodal personalized prompts for Task Group C, Explanation Generation.Source: Figure 10 · p. 15
Loading 2305.14302v2…