PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Training Towards Robust Multimedia Recommender System

Jinhui Tang, Xiaoyu Du, Xiangnan He, Fajie Yuan, Qi Tian, Tat-Seng Chua

arXiv:1809.07062v4cs.IRcs.MM

TL;DR

Deep-feature multimedia recommenders can be highly sensitive to small, purposeful image perturbations, raising a robustness gap in preference prediction. The paper proposes AMR, which trains the recommender against loss-increasing perturbations, and reports improved robustness and effectiveness across two real-world multimedia recommendation tasks.

  • Problem

    The robustness of deep multimedia representations and their impact on recommendation performance have received little investigation, despite evidence that small image perturbations can severely reduce recommendation accuracy.

  • Method

    AMR jointly trains a multimedia recommender and an adversary that adds perturbations to content to maximize the recommendation loss.

  • Results

    Experiments on image recommendation and visually-aware product recommendation show positive effects from adversarial learning and demonstrate AMR’s effectiveness.

  • Takeaways & Limitations

    Adversarial learning can improve the robustness and effectiveness of multimedia recommender models against purposeful content perturbations.

  • Takeaways & Limitations

    AMR introduces two crucial hyper-parameters, ϵ and λ, whose values must be carefully tuned because extremes can disrupt training or limit robustness gains.

Abstract

from arXiv · show

With the prevalence of multimedia content on the Web, developing recommender solutions that can effectively leverage the rich signal in multimedia data is in urgent need. Owing to the success of deep neural networks in representation learning, recent advance on multimedia recommendation has largely focused on exploring deep learning methods to improve the recommendation accuracy. To date, however, there has been little effort to investigate the robustness of multimedia representation and its impact on the performance of multimedia recommendation. In this paper, we shed light on the robustness of multimedia recommender system. Using the state-of-the-art recommendation framework and deep image features, we demonstrate that the overall system is not robust, such that a small (but purposeful) perturbation on the input image will severely decrease the recommendation accuracy. This implies the possible weakness of multimedia recommender system in predicting user preference, and more importantly, the potential of improvement by enhancing its robustness. To this end, we propose a novel solution named Adversarial Multimedia Recommendation (AMR), which can lead to a more robust multimedia recommender model by using adversarial learning. The idea is to train the model to defend an adversary, which adds perturbations to the target image with the purpose of decreasing the model's accuracy. We conduct experiments on two representative multimedia recommendation tasks, namely, image recommendation and visually-aware product recommendation. Extensive results verify the positive effect of adversarial learning and demonstrate the effectiveness of our AMR method. Source codes are available in https://github.com/duxy-me/AMR.

1 INTRODUCTION

Multimedia data can enrich recommendation by exposing visual signals, but deep feature-based systems introduce robustness concerns. The paper demonstrates this vulnerability and proposes adversarial training, with experiments showing AMR improves multimedia recommendation.

  • Motivation: Deep neural networks have shifted multimedia recommendation toward learned image features rather than tags or handcrafted visual representations.This trend aims to leverage multimedia content that can reveal user preferences beyond collaborative-filtering behavior alone.
  • Robustness problem: Small, purposeful image perturbations can substantially alter recommendation rankings while remaining nearly imperceptible to humans.In the VBPR example, perturbations with ϵ = 0.007 cause the interacted image to rank below non-interacted images.
  • Adversarial Multimedia Recommendation: AMR trains the recommender against an adversary that perturbs multimedia content to maximize recommendation loss.The adversarial and model objectives form a minimax-style training process intended to reduce the perturbations’ impact on predictions.
  • Evaluation: Experiments on personalized image recommendation and visually-aware product recommendation verify positive effects from adversarial learning and AMR’s effectiveness.The evaluation uses Pinterest image data and Amazon product data.
  • Robustness problem: The paper identifies vulnerability as a previously underemphasized issue in state-of-the-art multimedia recommenders using DNN-based feature learning.The authors connect this issue to possible instability in image representations or in the recommender’s use of small feature fluctuations.

2 PRELIMINARIES

The paper builds AMR on latent-factor recommendation and VBPR, which combines collaborative and visual signals for personalized ranking. It motivates adversarial training by showing that small image perturbations can cause large ranking changes in VBPR.

  • Latent Factor Model: Latent Factor Models represent users and items with latent vectors and estimate preference through their inner product.The functions fU and fI map users and items into a shared latent space.
  • Latent Factor Model: Matrix factorization is the simplest latent-factor instantiation, assigning embedding vectors to users and items when only identity information is available.The embedding size is denoted by K.
  • Multimedia representations: Multimedia recommendation incorporates content features into the item representation, which can alleviate cold-start issues for out-of-sample items.A content transformation can map deep image features into the latent space, allowing item representations even without prior interactions.
  • Adversarial extension: AMR uses VBPR as its building block and applies adversarial learning to improve robustness against perturbations in multimedia content.The paper positions VBPR as a state-of-the-art multimedia recommendation method and AMR as an adversarial extension.
  • Visual Bayesian Personalized Ranking: VBPR extends latent-factor recommendation by combining collaborative filtering with image-based preference modeling and optimizes the BPR pairwise ranking loss.The loss assumes observed user-item interactions should receive higher scores than unobserved counterparts.
  • Vulnerability of VBPR: VBPR is vulnerable because small pixel-level perturbations can cause large ranking changes through either unstable image features or sensitivity to feature fluctuations.The paper presents both mechanisms as possible explanations and links the issue to weak generalization of the overall recommender.

3 ADVERSARIAL MULTIMEDIA RECOMMENDATION

AMR improves multimedia recommendation robustness by training a VBPR-based model against adversarial perturbations applied to deep image features. The method uses constrained worst-case perturbations and a minimax optimization procedure while avoiding pixel-level perturbation difficulties.

  • Adversary Construction: AMR applies adversarial perturbations to image deep feature vectors rather than raw pixels, avoiding end-to-end DNN training.The DNN remains a fixed feature extractor, while the recommender updates the latent-space transformation.
  • Adversary Construction: The feature-level perturbation improves robustness in the recommender’s latent-space representation Eci, but not the quality of the fixed deep image representation ci.This design makes learning more efficient because the DNN is neither updated nor involved in adversary construction.
  • Adversary Construction: The adversary constructs worst-case perturbations by maximizing the BPR loss over training data.An L2 constraint ||∆i|| ≤ϵ limits perturbation magnitude and prevents increasing the loss through unbounded perturbation scale.
  • Model Optimization: AMR trains model parameters to minimize both the original BPR loss and the adversary’s loss, with λ controlling the adversarial contribution.Setting λ to 0 removes the adversary’s impact and reduces the method to VBPR; the formulation is expressed as a minimax game.
  • Model Optimization: AMR’s effectiveness depends on tuning ϵ and λ: overly large values can disrupt training, whereas overly small values limit robustness and generalization improvements.These two hyper-parameters are additional to those in VBPR.
  • Optimization Algorithm: The method uses SGD because batch gradient descent over BPR’s many pairwise training instances can be time-consuming and slow to converge.Algorithm 1 applies stochastic updates using sampled training instances (u, i, j).

10 end

AMR alternates between learning adversarial perturbations that maximize instance loss and updating model parameters to resist them. It requires pre-training and increases training cost relative to VBPR while preserving prediction-time complexity.

  • Learning Adversarial Perturbations: AMR learns worst-case perturbations by maximizing the BPR loss for each sampled instance.The perturbations are obtained for the involved positive and negative items under an ϵ-constraint.
  • Learning Model Parameters: The model then minimizes an objective with fixed perturbations to learn parameters resistant to adversarial multimedia perturbations.Parameter updates use gradient descent, with Adagrad used in the experiments.
  • Training Procedure: Pre-training is critical because adversarial training improves generalization only after the model reaches reasonable performance.Without pre-training, ordinary training is sufficient to improve parameters and adversarial training can slow convergence.
  • Time Complexity Analysis: AMR and VBPR have the same prediction-time complexity because they use the same prediction model.Their difference lies in the training loss rather than the prediction computation.
  • Time Complexity Analysis: AMR costs about twice VBPR’s theoretical training complexity and empirically takes about three times as long per epoch on both datasets.The theoretical costs are 4×Of +4×Ob+2×Oadv +Ou for AMR versus 2×Of +2×Ob+Ou for VBPR.

4 EXPERIMENTS

The experiments evaluate whether AMR improves recommendation performance, generalization, and robustness, and examine how its key adversarial hyper-parameters affect performance.

  • Research Questions: RQ1 asks whether AMR can outperform state-of-the-art multimedia recommendation methods.
  • Research Questions: RQ2 examines the effect of adversarial training on model generalization and robustness.
  • Research Questions: RQ3 studies how the adversarial hyper-parameters ϵ and λ affect recommendation performance.
  • Experimental Organization: The section first presents experimental settings and then reports results addressing the three research questions.

4.1 Experimental Settings

Experiments use Pinterest and Amazon data under a leave-one-out evaluation protocol, comparing AMR with popularity, collaborative-filtering, and visually-aware baselines.

  • Datasets: The experiments use Pinterest for image recommendation and Amazon women-category data for visually-aware product recommendation.Each item has one image, and both user-item interaction matrices are highly sparse.
  • Evaluation Protocol: The leave-one-out protocol assigns one randomly selected interaction per user to testing and the remainder to training.About 52.6% of Pinterest test items and 45.9% of Amazon test items are cold-start items.
  • Evaluation Protocol: Evaluation samples 999 non-interacted items and ranks the test item among 1,000 candidates using HR@N and NDCG@N.The ranking list is truncated at N for top-N recommendation evaluation.
  • Compared Methods: AMR is compared with POP, MF-eALS, MF-BPR, DUIF, and VBPR.The baselines cover popularity, collaborative filtering, and visually-aware recommendation approaches.
  • Implementation and Tuning: Visually-aware methods use the same ResNet-50 extractor, and all models use mini-batch Adagrad with batch size 512.The embedding size is fixed to 64; AMR additionally tunes ϵ and λ after selecting VBPR settings.

4.2 Performance Comparison (RQ1)

AMR consistently outperforms the evaluated baselines on both datasets and all reported metrics. Its gains are especially large for cold-start Pinterest items, while gains are smaller on Amazon.

  • Overall Comparison: Personalized models substantially outperform POP on both datasets, with the largest reported relative improvement reaching 280% on Pinterest.The comparison includes MF-eALS, MF-BPR, VBPR, and AMR.
  • Baseline Comparison: VBPR generally outperforms MF-eALS, MF-BPR, and DUIF, indicating benefits from incorporating multimedia features into collaborative filtering.The passage also reports that DUIF performs worse than MF-BPR and MF-eALS despite using the same visual features as VBPR.
  • Overall Comparison: AMR consistently outperforms all baselines on every metric across both datasets.Table 2 reports top-N performance for N ∈{5, 10, 20}; asterisks indicate statistically significant improvements for p < 0.05.
  • Cold-Start Performance: On Pinterest cold-start items, AMR improves over VBPR by 74% in HR@10 and 101% in NDCG@10.For non-cold items, the corresponding improvements are 6.2% and 2.2%.
  • Dataset Differences: AMR’s improvements over VBPR are smaller on Amazon, where popularity and collaborative-filtering effects are stronger.The authors report that AMR still improves significantly over VBPR according to the t-test.

4.3 Effect of Adversarial Training (RQ2)

Adversarial training improves VBPR’s generalization and robustness. AMR improves testing performance and suffers smaller ranking losses than VBPR under adversarial perturbations.

  • Generalization: AMR further improves Pinterest VBPR’s best NDCG from 0.116 to 0.123 and HR from 0.183 to 0.203.Normal continued VBPR training did not improve performance and could decrease it through overfitting on Amazon.
  • Robustness: Adversarial perturbations severely damage VBPR, whereas random perturbations decrease performance only slightly.VBPR-grad applies adversarial perturbations to image features; VBPR-rand applies random perturbations.
  • Robustness: Perturbation effects increase sharply with scale: ϵ = 0.01 is inconspicuous, while ϵ = 0.5 causes a fatal drop.These observations inform the perturbation scale used in AMR.
  • Robustness: Across perturbation scales, AMR has a much smaller relative performance drop than VBPR.On Amazon at ϵ = 0.05, VBPR drops 8.7% versus 1.4% for AMR; at ϵ = 0.2, VBPR drops more than 67.7%.
  • Robustness: At ϵ = 0.1, AMR has fewer large impacts and lower mean and variance of rank drops than VBPR on Pinterest and Amazon.On Amazon, most samples show no rank changes under perturbation for AMR.

4.4 Hyper-parameter Exploration (RQ3)

The authors examine perturbation scale ϵ and adversary weight λ while fixing other hyper-parameters. Performance improves up to task-dependent optima, then overly large settings damage training.

  • Perturbation scale: Increasing ϵ improves performance below the optimum but causes a rapid performance drop above it.The authors interpret this as utility for controllable perturbation scales and damage from overly large perturbations.
  • Adversary weight: The authors similarly vary λ, the adversary’s weight, while keeping other hyper-parameters fixed at roughly optimal values.The reported λ trend follows the same general pattern as ϵ: improvement below a threshold and degradation beyond it.

5 RELATED WORK

Related work spans multimedia recommendation, deep visual representations, and adversarial learning. AMR applies adversarial training to image features in multimedia ranking, distinguishing it from prior recommender robustness methods.

  • Multimedia Recommendation: Multimedia recommendation integrates rich visual features with collaborative filtering, with VBPR projecting DNN-derived features into visual predictors.DNN features replaced earlier reliance on annotated tags and low-level representations as deep representation learning advanced.
  • Multimedia Recommendation: AMR’s novelty is making multimedia recommender models less vulnerable to worst-case perturbations in input features.The paper presents this as the first implementation and evaluation of the idea for multimedia recommendation.
  • Adversarial Learning: AMR adapts adversarial learning from pointwise classification to multimedia recommendation’s pairwise ranking loss.The method can directly apply to pairwise-ranking methods, while pointwise-loss methods require adapting the loss in model optimization.
  • Adversarial Learning: Adversarial learning constructs perturbations that degrade performance and trains the model to perform well under their effect.This work follows prior demonstrations that adversarial training can improve model robustness.
  • Adversarial Learning: Unlike APR, which perturbs embeddings, AMR perturbs image features to improve tolerance for noisy multimedia inputs.Profile-injection robustness methods address a different attack surface and are described as orthogonal to AMR.

6 CONCLUSION

The paper introduces AMR to improve multimedia recommender robustness through adversarial training. Experiments on two real-world datasets demonstrate adversarial learning’s utility and AMR’s effectiveness, while future work extends the method to other models and richer contexts.

  • AMR simultaneously trains the recommender and an adversary that applies purposeful image perturbations, producing more robust and effective model parameters.
  • Experiments on two real-world datasets demonstrate the utility of adversarial learning and the strength of AMR.
  • AMR is presented as a generic blueprint for improving content-based recommender models beyond the model explored in this paper.
  • Future work includes extending AMR to attention-based neural recommenders and incorporating time, location, and user personality contexts.
Loading 1809.07062v4…