PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing the Context: Enhancing Recommender Systems with Image-Derived Contextual Signals

Tal Cordova, Tomer Geva, Moshe Unger

arXiv:2610.08407v1cs.IR

TL;DR

Recommender systems have used images to enrich item or user representations rather than infer situational context. This paper introduces ICE-Fuse to extract and integrate image-derived context into recommendation. Image context does not outperform established signals alone, but improves them when combined, while semantic analysis indicates distinct information from reviews.

  • Problem

    Images in recommender systems have mainly enriched item or user representations rather than been used to infer situational context.

    Source: §1 · p. 1

  • Method

    ICE-Fuse extracts image context and integrates it into a recommendation system that uses RGCL.

    Source: §2 · p. 2, §5 · p. 5

  • Results

    Image context does not outperform established contextual signals as a standalone predictor, but consistently improves them when combined, and image- and text-derived context capture largely distinct concepts.

    Source: §5 · p. 6

  • Takeaways & Limitations

    The findings support treating image-derived context as complementary to existing contextual sources.

    Source: §5 · p. 6

    Context quality is assessed indirectly through downstream performance in the absence of ground-truth labels, and available structured context is limited to city and coarse temporal features.

    Source: §5 · p. 6

Abstract

from arXiv · show

Contextual information, capturing the circumstances of a user-item interaction, is central to recommender systems. Prior work draws context from location, time, or reviews, but not images; multimodal recommender systems mainly use images to enrich item or user representations, not identify situational context. We propose a new representation of context derived from images, spanning physical, social, and modal categories learned via a vision-language model. We introduce ICE-Fuse, a pipeline for evaluating this representation that fuses these categories and integrates them into a context-aware recommender system, using TripAdvisor data and Review-aware Graph Contrastive Learning as the recommendation algorithm. Image context does not outperform established signals standalone, but improves them combined, indicating complementary information. Semantic analysis shows image- and review-derived context capture distinct aspects of the interaction, positioning images as complementary context.

1 Introduction

The paper addresses the lack of image-derived situational context in recommender systems, where images have mainly enriched user or item representations. It proposes ICE-Fuse to extract and evaluate image context, finding that it complements established signals when combined.

1 Introduction

  • Prior recommender systems use images to enrich item or user representations rather than infer situational context.Existing contextual signals include location, time, and reviews.Source: §1 · p. 1
  • Images can preserve formality and ambience details that textual reviews may lose through retrospective memory.Examples include casual attire, a candlelit table, or a lively versus quiet room.Source: §1 · p. 1
  • The proposed image-context representation spans physical, social, and modal categories, inferred from visual cues.For example, the number of plates can indicate a social gathering.Source: §1 · p. 1
  • ICE-Fuse prompts a vision-language model for category-specific descriptions, fuses them, and integrates the resulting context into a recommender.The fusion uses a gating mechanism.Source: §1 · p. 1
  • The paper introduces an evaluation pipeline comparing image context alone and in combination with other signals for rating prediction.Its stated contributions also include identifying image-specific contextual features.Source: §1 · p. 2
  • Image context does not outperform other context signals alone, but can improve predictive performance when combined, suggesting complementary information.The authors report that image context captures features distinct from other contextual sources.Source: §1 · p. 2

2 Method

ICE-Fuse extracts category-specific situational descriptions from review images, projects and fuses them into interaction context, then integrates that context into a recommender. Its evaluation compares image-derived context with established contextual signals and direct image embeddings.

2.1 Context Fusion, Integration and Rating Prediction

  • The fused context is integrated into RGCL during training to refine user and item representations, which support rating prediction without images at inference.A lightweight fully connected network maps the joint interaction representation to a scalar rating.Source: §2 · p. 2, §2.1 · pp. 2–3

2 Method

  • For each interaction image, a vision-language model describes physical, social, and modal context while avoiding item-content details unless they indicate situation.The descriptions are encoded and projected through category-specific trainable MLPs before fusion.Source: §2 · p. 2

2.1 Context Fusion, Integration and Rating Prediction

  • ICE-Fuse uses a gating network to weight category representations and fuse them into one interaction-context vector.The network's softmax weights vary across user-item interactions.Source: §2.1 · p. 2

2.2 Alternative Context and Content Signals

  • To evaluate image context, comparisons use a no-context baseline, city-date and text-based contextual signals, and a content signal consisting of a direct image embedding via CLIP without VLM-based interpretation.City-date has 29 one-hot features; text-based context uses 52 expert-identified phrases mapped to nine features.Source: §2.2 · p. 3

3 Evaluation

The evaluation uses TripAdvisor restaurant reviews from five European cities and a global time-based train-test split. Image context and direct image embeddings are encoded and projected for rating-prediction experiments.

3 Evaluation

  • The filtered dataset contains 39,956 reviews from 1,441 users and 1,341 restaurants, including 22,220 image-bearing reviews.It spans Madrid, Barcelona, Moscow, Istanbul, and Gijón from 2010–2019; users and restaurants had to meet a seven-image-review threshold.Source: §3 · p. 3
  • Image-context descriptions use Gemma 3 12B IT and e5-small-v2, while direct image embeddings use CLIP ViT-B/32.All contextual and content representations are projected to 64 dimensions.Source: §3 · p. 4
  • The global time split trains on 2010–2016 interactions and tests on 2017–2019, with a randomly selected validation set.Training uses Adam and early stopping on validation loss, combining rating-prediction MSE with RGCL contrastive losses.Source: §3 · p. 4

4 Results

Image context improves established signals when combined, although it does not surpass city-date or text context as a standalone predictor. Semantic analysis and category ablations further characterize the distinct information captured by image context and the contribution of its categories.

4.1 Recommendation Performance

  • Image context outperforms no-context but does not outperform city-date or text-based context as a standalone predictor.The standalone comparison confirms image context carries recommendation-relevant signal without exceeding established contextual signals.Source: §4.1 · p. 4, Table 2 · p. 5
  • Adding image context significantly reduces RMSE with city-date context and image embeddings; its text-context gain is not statistically significant.Table 3 reports before-and-after combination performance; significance uses paired two-sided Wilcoxon signed-rank tests.Source: §4.1 · p. 4, §4 · p. 4, Table 3 · p. 5

4.2 Semantic Complementarity of Image and Review Context

  • Image and review contexts overlap on a few concepts, but images capture fine-grained ambience while reviews capture meal timing and relational specifics.Examples of image-specific context include lighting and outdoor seating; review-specific examples include brunch and work colleagues.Source: §4.2 · p. 5

4.3 Context Category Ablation Analysis

  • With all three image-context categories active, the gate weights are relatively balanced and recommendation results are best; in pairs, it favors modal context.Physical context receives the smallest share when paired with another category.Source: §4.3 · p. 5

5 Conclusions

The paper extends established contextual signals with image-derived context and evaluates it through ICE-Fuse. Image context does not outperform established signals alone but improves them in combination, while capturing distinct aspects of interactions.

5 Conclusions

  • ICE-Fuse acquires physical, social, and modal context from review images and integrates it into a recommendation system.The paper presents image context as an extension to established contextual sources such as location, timestamp, and text-derived context.Source: §5 · pp. 5–6
  • Image-based context does not outperform established contextual signals as a standalone predictor, but consistently improves them when combined.The paper also describes the resulting improvements as modest but meaningful [7].Source: §5 · p. 6
  • Semantic analysis finds that image- and text-derived context capture distinct concepts, supporting images as a complementary information source.The paper identifies image-specific contextual features that are not present in other contextual sources.Source: §5 · p. 6
  • Context quality is assessed indirectly through downstream performance, and the study reports an absence of ground-truth labels.Available structured context is limited to city and coarse temporal features, with signals such as time of day unavailable.Source: §5 · p. 6
  • The authors plan to evaluate image-based context across additional datasets, domains, and algorithms, and to study which visual cues matter most.Source: §5 · p. 6

A Prompts

The prompts operationalize physical, social, and modal context as separate descriptions inferred from visible cues in restaurant-review images. Each asks for concise free text and a no-context response when meaningful cues cannot be determined.

A.1 Physical Context Prompt

  • Physical context covers time, position, activity, weather, light, and temperature in the dining situation.Source: §A.1 · p. 7
  • The physical prompt targets environmental cues such as indoor or outdoor seating, lighting, seasonal or weather clues, surroundings, decor, crowdedness, and ambiance.Source: §A.1 · p. 7

A Prompts

  • For each category, the prompt requests a concise free-text description and specifies “no_context” when no meaningful context can be determined.It also instructs the model to avoid speculative wording and certain image-narration phrases.Source: §A.1 · p. 7, §A.2 · p. 8

A.2 Social Context Prompt

  • Social context describes who is around the diner, including whether they are alone or in a group and the dining configuration.Source: §A.2 · p. 7
  • The social prompt looks for visible people, group composition, relationships, body language, table settings, children, and formal or casual interaction.Source: §A.2 · p. 8

A.3 Modal Context Prompt

  • Modal context represents the diner’s state of mind, goals, mood, and purpose or occasion for the visit.Source: §A.3 · p. 8
  • The modal prompt uses cues such as celebrations, aesthetic framing, and visual indicators of calm, lively, romantic, fun, or festive moods.Source: §A.3 · p. 8
Loading 2610.08407v1…