PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Processing for Offline Evaluation in Recommender Systems: a Survey

Alberto Carlo Maria Mancino, Angela Di Fazio, Danilo Danese, Matteo Attimonelli, Daniele Malitesta, Antonio Ferrara, Claudio Pomo, Tommaso Di Noia

arXiv:2609.31696v1cs.IRcs.LG

TL;DR

Offline recommender evaluation depends on data-processing choices, but these practices have received less systematic attention and can vary across studies. This survey characterizes data processing across recommendation domains and finds that splitting-protocol labels may conceal materially different experimental conditions, affecting how reported performance should be interpreted.

  • Problem

    Data-processing choices and heterogeneous dataset versions or splits can complicate comparisons between recommender-system studies.

    Source: Introduction · pp. 7–8

  • Method

    The survey systematically reviews offline-evaluation data processing, including dataset practices and the reporting of filtering and splitting decisions.

    Source: Introduction · p. 8, Interplay Between Transformations · p. 102, §5.1 · p. 115

  • Results

    Splitting protocols with different names may use similar allocation rules, while protocols sharing a label may differ in split units, temporal constraints, eligibility, or validation treatment.

    Source: §5.1.4 · p. 119

  • Takeaways & Limitations

    Reported performance should be interpreted as a property of the model–protocol combination, and split provenance matters when evaluating its assumptions.

    Source: §5.1.4 · p. 119

    The iterative use of user–item filters is rarely specified, making it impossible to determine whether the resulting datasets satisfy simultaneous core consistency.

    Source: Interplay Between Transformations · p. 103

Abstract

from arXiv · show

Offline evaluation is the dominant experimental paradigm in recommender systems research, enabling reproducible and cost-effective comparisons on historical interaction data. Yet, while considerable attention has been devoted to recommendation models and evaluation methodologies, the data processing decisions that precede model training have received less scrutiny. These decisions determine the information available to recommendation algorithms and can affect the comparability and reproducibility of experimental results. This survey provides a systematic, cross-domain characterisation of data processing practices for the offline evaluation of recommender systems. We examine the data-centric pipeline, from dataset selection and interaction representation to data preparation, multimodal feature extraction, and train-validation-test splitting. Our analysis spans recommendation paradigms, including collaborative, sequential, session-based, graph-based, knowledge-aware, context-aware, multimodal, federated, cross-domain, contrastive-learning, and LLM-based recommendation. Beyond reviewing existing practices, we introduce a unified framework and taxonomy for describing data transformations and feature-extraction strategies, distinguishing data preparation from the extraction of representations from multimodal side information. Our empirical analysis reveals a landscape dominated by a narrow set of dataset-level transformations, particularly support-driven filtering, while representation-dependent transformations remain less common. We further identify substantial heterogeneity in how auxiliary information is prepared and represented, as well as inconsistencies in the specification of data splitting protocols, where similar labels may conceal different experimental conditions.

Introduction

Offline evaluation is widely used, but data-stage decisions and inconsistently documented side-information pipelines complicate fair comparison and interpretation. This survey maps data selection and processing across recommendation domains, clarifies key terms, and identifies gaps in prior cross-domain coverage.

Introduction

  • Offline evaluation has methodological weaknesses involving dataset–task and metric–task mismatch, leakage, biased negative sampling, and limited statistical testing, which hinder comparability and may weaken validity.Its real-world relevance also depends on whether historical-log outcomes predict live performance.Source: Introduction · pp. 7–8
  • Earlier pipeline stages have received less systematic attention, while heterogeneous preparation of side information and feature extraction complicates comparisons of model innovations.The survey addresses this gap with an overview of datasets and preprocessing practices, aiming to support more transparent and reproducible research.Source: Introduction · p. 8, §1.1 · p. 9

1.1. Strategy for Literature Search

  • The survey spans established and emerging recommendation domains, using recent survey articles as entry points for broad coverage.Its scope includes collaborative, sequential, session-based, graph-based, knowledge-based, context-aware, contrastive, federated, cross-domain, multimodal, and LLM-based recommendation.Source: §1.1 · p. 9
  • The authors selected recent, cited surveys from ranked venues, screened their papers for relevance and offline evaluation, and assembled a corpus of 517 unique papers from 2001–2024.The survey selection used 2020–2025 publication dates, at least 30 citations, and Q1 journal or A/A*-conference status; the authors released retrieval and filtering code.Source: §1.1 · pp. 9–10, §1.2 · p. 10, §1.3 · p. 11

1.3. Terminology and Conceptual Scope

  • About 75% of the surveyed literature appears in A*/A conferences or Q1 journals, with conference publications predominating; non-ranked venues are particularly common in LLM-based recommendation.Venue rankings are assigned using the publication year, with multidisciplinary journals assessed by their highest relevant Computer Science or Information Systems ranking.Source: §1.3 · p. 11
  • Data processing covers pre-training operations such as cleaning, filtering, normalisation, feature extraction, and splitting; data preparation is the subset modifying dataset structure, content, or representation.The survey treats preprocessing and data processing as interchangeable terms.Source: §1.3 · pp. 12–13
  • Multimodal feature extraction turns heterogeneous side information into machine-readable representations; the survey covers representation generation and reuse, not multimodal model architectures.Offline evaluation uses historical interactions without direct user involvement or live deployment, and is the default meaning of evaluation in the survey.Source: §1.3 · pp. 13–14

1.4. Survey Context and Related Surveys

  • Prior surveys and studies address recommendation broadly, particular domains, or isolated data and evaluation issues, but do not provide systematic cross-domain coverage of data processing.The authors characterize their work as the first systematic cross-domain survey to include multimodal feature-extraction pipelines.Source: §1.4 · pp. 14, 16

The Recommendation Pipeline

Offline evaluation uses historical interactions to support controlled experimentation, while the dataset pipeline determines what information models receive and how their results are assessed. The section surveys interaction data, side information, preparation, splitting, and evaluation settings rather than model-specific methods.

2.1 Evaluation Paradigms in Recommender Systems

  • Offline evaluation trains and assesses models on historical interactions, typically withholding data for validation and testing to simulate user behaviour.It enables controlled, repeatable, cost-effective experiments, although it abstracts from exposure bias, feedback loops, and recommendations’ effects on later behaviour.Source: §2.1 · p. 19

2.2 Interactions and Data for Offline Evaluation

  • Interaction records capture user–item events and may include action, timestamp, or contextual attributes, with explicit and implicit signals supporting different interpretations.Implicit actions are indirect evidence of preference, and an absent interaction does not necessarily indicate negative preference.Source: §2.2 · p. 20, §2.3 · p. 21

2.3 Augmenting Recommendation Data

  • Side information complements interaction logs with descriptions of users or items, including text, images, audio, video, and external knowledge sources.It exists independently of a particular interaction, unlike event attributes such as timestamps or reviews generated during a user–item event.Source: §2.3 · p. 22, §2.4 · p. 23

2.4 Dataset Preparation

  • Dataset preparation transforms interaction logs and side information through dataset-level and representation-dependent operations, alongside modality-specific feature extraction.Dataset-level operations include filtering and rating transformations; representation-dependent operations reorganize interactions as sequences, sessions, graphs, or contextual data.Source: §2.4 · pp. 23–24

2.5 Dataset Splitting

  • Splitting strategies include random, time-aware, and benchmark-specific protocols, each defining the interactions available for training and performance assessment.Random splits may place future interactions in training while earlier ones enter testing; time-aware splits preserve chronology, while benchmark-specific protocols may encode hidden assumptions.Source: §2.5 · pp. 25–26

2.6 Transductive and Inductive Evaluation

  • Transductive evaluation uses fixed users and items, whereas inductive evaluation tests generalization to unseen entities; the section then frames recommendation as learning a scoring function from processed data.The survey focuses on datasets and processing, not model-specific architectures or optimization procedures.Source: §2.6 · p. 26, §2.7 · p. 27

Recommendation Datasets

Recommendation datasets are structured experimental objects whose properties shape the conditions and interpretation of offline evaluation. The section distinguishes dataset domains from task formulations and proposes a taxonomy spanning interaction structure, information sources, and dataset organization.

3.1 The Dataset as a Structured Experimental Object

  • A recommendation dataset’s structure, information, and assumptions define the conditions for training, evaluation, and comparison, so it is not a neutral input.The paper defines it as users, items, and interactions, optionally enriched with attributes and side information, organized to support recommendation-model training and evaluation.Source: §3.1 · pp. 29–30

3.1.1 Beyond Interaction Data: Side Information

  • Side information complements sparse interactions with descriptions or relations that can provide perspectives unavailable from interaction records alone.Reviews may function as interaction records or as side information when their text is processed into semantic representations.Source: §3.1.1 · p. 31, the recommendation pipeline · p. 31

3.2 Recommendation Domains and Task Formulations

  • Domains shape available entities, interaction semantics, and information modalities, but tasks arise from how interaction data is structured rather than from the domain itself.A single dataset can support multiple paradigms, and observed domain–task associations reflect common practices rather than strict relationships.Source: §3.2 · pp. 32–34

3.3 A Taxonomy of Recommendation Datasets

  • The proposed taxonomy has three complementary dimensions: structural representations, information sources, and dataset organization.A dataset may combine, for example, a sequential representation, descriptive side information, and cross-domain or federated organization.Source: §3.3 · p. 34

3.3.1 Structural Representations

  • Structural representations provide alternative ways to organize and interpret interaction records, including graphs, sequences, sessions, and context-dependent interactions.These perspectives emphasize different components of the same underlying interaction data.Source: §3.3 · p. 34, §3.3.1 · p. 34, Graph-Based Representation · p. 34

Recommendation Datasets

This section characterizes recommendation datasets by interaction structure, information sources, and global organization, and examines their use across domains and benchmarks alongside practices for dataset availability, identification, and auxiliary-information release. It also considers datasets as evolving data ecosystems in which auxiliary information is integral to content-aware, multimodal, and knowledge-enhanced recommendation.

Recommendation Datasets

Side Information

  • Datasets distinguish interaction-event attributes from side information about entities; side information may describe users or items or encode explicit relationships among them.Descriptive information includes metadata and media, whereas structured knowledge connects entities through semantic relationships.Source: Interaction Data · p. 38, Side Information · pp. 39–40
  • Descriptive information may be released as raw, human-interpretable content or as pre-computed embeddings and feature vectors ready for experiments.With raw attributes, feature extraction is left to later pipeline stages; pre-computed representations have already undergone extraction.Source: Side Information · p. 39

3.3.3 Dataset Organisation

  • Dataset organization has independent domain-scope and distribution dimensions: datasets may span one or interconnected domains and be centralized or distributed across local clients.Federated datasets keep interaction data on decentralized clients while sharing intermediate updates or aggregated information; benchmark choices also shape which application scenarios receive research attention.Source: §3.3.3 · p. 40, Domain Scope · p. 41, Data Distribution · p. 41

3.4.1 Dataset Frequency and Long-Tail Distribution

  • The analysis covers 517 papers, 1,497 paper–dataset pairs, and 270 distinct datasets; papers use approximately 2.9 datasets on average.Each distinct dataset appears in approximately 5.5 paper–dataset pairs, indicating both substantial diversity and reuse of some datasets.Source: §3.4.1 · p. 43
  • Among frequently used datasets, major benchmark families, challenge releases, and other sources feature prominently; Yelp 1 is challenge-derived, and some papers leave MovieLens versions unspecified.The analysis treats dataset versions as distinct entries but does not distinguish preprocessing variants of the same underlying source.Source: §3.4.1 · pp. 43, 45
  • Reuse barriers include proprietary or request-only access, broken links, and highly specialized domains, limiting reproducibility and broader adoption.The authors present these as recurring patterns, not an exhaustive explanation of the long-tail distribution.Source: §3.4.1 · pp. 45–46
  • 46.1% of identified datasets appear in only one publication, while a small fraction is repeatedly reused, revealing a fragmented, highly skewed usage distribution.The concentration around a limited set of benchmarks facilitates model comparison but may bias evaluation toward particular domains, data characteristics, and interaction patterns.Source: §3.4.1 · pp. 43, 45, Figure 3.3 · p. 46

3.4.2 Domain Distribution of Recommendation Datasets

  • Movies research is concentrated around a few reused benchmarks, while e-commerce has many distinct datasets and domain attention remains uneven.Movies includes 25 distinct datasets, with usage focused on MovieLens and Amazon Movies and TV; Douban datasets are widely used for cross-domain recommendation. Movies, Fashion, and Games rely on relatively few benchmarks, whereas E-commerce, Point of Interest, and Music have greater dataset diversity; some domains have only a handful of datasets, and “Other” contains single-dataset domains.Source: §3.4.2 · pp. 47–49

3.4.3 Benchmark Adoption Across Research Topics

  • Benchmark adoption varies by research topic: general-purpose datasets span multiple areas, while task-specific datasets dominate some specialized fields.Yelp is used extensively in graph-based, contrastive, and sequential recommendation but rarely in cross-domain, session-based, and federated work; MovieLens is common in federated studies despite not being designed for that setting. YooChoose and Diginetica dominate session-based recommendation, and InCarMusic and LDOS-CoMoDa are used almost exclusively for context-aware recommendation. MovieLens-1M spans collaborative, sequential, federated, and knowledge-aware recommendation, whereas LLM-based recommendation has no clearly dominant benchmark; adoption reflects both dataset popularity and topic-specific requirements.Source: §3.4.3 · pp. 49, 51, Figure 3.4 · p. 50

Dataset Provenance

  • 59.5% of surveyed datasets are academic and 27.5% industrial; challenge datasets comprise 8.8% academic challenges and 4.2% industrial challenges.The survey classifies datasets by the primary actor responsible for their creation and dissemination, also distinguishing community-created datasets.Source: Dataset Provenance · pp. 52–53, Table 3.2 · p. 53

Dataset Accessibility

  • 70.2% of datasets have a publicly accessible source, while 29.8% cannot be directly retrieved.Of inaccessible datasets, 17.7% are proprietary industrial resources, 11.3% are no longer shared, and 0.8% could not be classified. Among datasets with an identified source, 82.8% have working links and 12.4% have inaccessible links; availability-only-on-request, closed-challenge links, and non-official mirrors account for 1.1%, 2.2%, and 1.6%, respectively. These findings show that accessibility can change over time as repositories and websites disappear or challenge infrastructures close.Source: Dataset Accessibility · pp. 53–54, Table 3.3 · p. 54

Dataset Referencing

  • 78% of datasets are associated with a reference publication, but inconsistent naming, indirect citations, and unspecified versions can leave dataset identity unclear.Ambiguity is especially consequential for evolving families such as MovieLens and Amazon Reviews, and for custom-preprocessed datasets that retain their original names, complicating comparisons across studies.Source: Dataset Referencing · p. 55

Lessons Learned for Reproducible Dataset Usage

  • Reproducible dataset use requires official names, original dataset citations, direct links to official sources, and explicit identification of the version used.Version differences can affect users, items, interactions, and side information; these documentation practices improve traceability and long-term reproducibility.Source: Lessons Learned for Reproducible… · p. 56

3.4.5 Auxiliary Information Sources in Recommendation Datasets

  • Benchmarks provide six recurring categories of auxiliary information, and these resources are predominantly item-centric rather than user-centric.User-side data is usually limited to basic profile attributes, while rich user descriptions are rarely released due to privacy constraints; models often derive user representations from interaction histories or item-side information.Source: §3.4.5 · pp. 57–58
  • Auxiliary information is released in heterogeneous forms, from raw text to externally added knowledge graphs and pre-computed multimedia features.Knowledge-graph alignments and other enrichments can evolve after initial dataset releases, as in MovieLens; TikTok provides pre-computed visual and audio representations that reduce storage and ease sharing but constrain researchers’ choice of encoders. Features may also be derived from raw content or video keyframes, so a modality can be distributed as original data or a processed representation.Source: §3.4.5 · pp. 58–59
  • Datasets increasingly integrate new modalities, semantic resources, and feature representations, making auxiliary information central to model design and experimental evaluation.This evolution involves both dataset creators and the research community.Source: §3.4.5 · p. 59

Data Preparation

Data preparation transforms interaction logs and associated side information before dataset splitting, shaping the data available for model training and evaluation. The chapter formalises these operations and organises them into a taxonomy by what they transform and what criteria guide them.

Data Preparation

  • Data preparation changes dataset structure, content, or representation before model training, and its choices help define the effective experimental benchmark.The chapter treats partition construction separately from these transformations.Source: Data Preparation · p. 61

4.1 A Formal View of Data Preparation

  • Operators P and Ψ separately model transformations of interaction logs and auxiliary side information, and preparation pipelines apply parameterised operators in sequence.Different pipelines on the same raw log can yield substantially different experimental datasets.Source: §4.1 · pp. 63–64

4.2 Taxonomy of Data Transformations

  • The taxonomy classifies transformations by the object they act on and the criterion defining each operation.Interaction-log operations include dataset-level and representation-dependent families.Source: §4.2 · p. 64

4.2. Taxonomy of Data Transformations

  • Dataset-level transformations use support properties such as user or item activity, or interaction attributes such as ratings, timestamps, action types, and context.Support criteria may also depend on the availability of associated side information.Source: §4.2 · pp. 64, 66

Side-information transformations.

The chapter distinguishes interaction-log transformations by whether they operate directly on interaction tuples or require a structural representation. It also treats side-information preparation and feature extraction as separate stages.

Side-information transformations.

  • Side-information processing standardises modality-specific content before extracting feature representations suitable for recommendation models.The survey analyses these stages separately for each modality.Source: Side-information transformations · p. 66

4.3 Interaction-Log Transformations

  • Interaction-log transformations divide into dataset-level operations on interaction tuples and representation-dependent operations requiring structures such as sequences, sessions, or graphs.Table 4.1 overviews dataset-level techniques; representation-dependent operations are organised by their structural representation.Source: §4.3 · p. 66, Table 4.1 · p. 67

4.3.1 Dataset-Level Transformations

  • Dataset-level transformations are either support-driven, using user or item properties, or attribute-driven, using event values such as ratings, timestamps, actions, or context.They modify the composition or interpretation of observed interactions.Source: §4.3.1 · p. 67

Support-Driven Transformations

  • Support-driven operations include activity filtering, subsampling, external-criterion filtering, and deduplication; iterative k-core filtering alternates user and item removal until constraints hold.The resulting interaction log retains users and items meeting the minimum degree constraint.Source: Support-Driven Transformations · pp. 68–71

Attribute-Driven Transformations

Attribute-driven transformations modify interaction values or select interactions according to feedback, time, and action type. Representation-dependent transformations instead require structures derived from the interaction log, such as sequences, sessions, or graphs.

Attribute-Driven Transformations

  • Attribute-driven transformations modify interaction attributes or their interpretation, while support-driven operations select interactions based on participating users or items.Attribute-driven operations generally leave the interaction log’s support unchanged, though some may also remove interactions.Source: Attribute-Driven Transformations · p. 71
  • Explicit-to-implicit conversion retains interactions above a rating threshold as positive evidence and removes below-threshold ratings rather than treating them as explicit negatives.The operation changes both interaction attributes and log support.Source: Attribute-Driven Transformations · p. 72
  • Rating normalisation changes numerical scale while preserving rating order and all interactions; parameters may be computed globally or separately per user.It is used when combining datasets with different rating scales or when models assume normalised inputs.Source: Attribute-Driven Transformations · pp. 72–73
  • Time-window selection removes interactions outside a chosen timestamp interval, whereas action filtering retains only selected interaction types.These transformations operate on temporal information and action attributes, respectively.Source: Attribute-Driven Transformations · pp. 73–74

4.3.2 Representation-Dependent Transformations

  • Representation-dependent operations become meaningful only after interactions are structured as sequences, sessions, graphs, or contextual representations.Their applicability depends on the adopted representation, not solely on the original interaction tuples.Source: §4.3.2 · pp. 74–75

Sequential Representations

Sequential preparation operates on user histories ordered by timestamp. The section describes truncating long histories and filtering out sequences below a minimum length.

Sequential Representations

  • Sequence truncation retains only each user’s most recent L interactions to limit history length, computational cost, and historical information considered by sequential models.It preserves every user while discarding older interactions beyond the specified length.Source: Sequential Representations · pp. 75–76
  • Sequence-length filtering removes user histories shorter than a minimum length because very short histories are assumed to provide insufficient sequential context for predicting future interactions.This criterion applies to ordered sequences rather than directly to the interaction log.Source: Sequential Representations · p. 76

Session Representations

Preparation operators depend on how interactions are represented: sessions are filtered as units, graphs by connectivity, and contextual data by variable selection or discretisation.

Session Representations

  • Session-based preparation acts on complete sessions, commonly filtering sessions outside a predefined length range because very short or extremely long sessions may be unsuitable or anomalous.Unlike sequence-length filtering, this operates on local session contexts rather than complete user histories.Source: Session Representations · p. 77

Graph Representations

  • Graph-specific connectivity filtering retains the largest connected component, unlike activity-based filtering, which depends only on local interaction counts.Activity-based filtering and duplicate removal remain dataset-level transformations because their definitions depend only on the interaction log.Source: Graph Representations · pp. 77–78

Context-Aware Representations

  • Context-aware preparation can select contextual variables to reduce representation dimensionality or discretise continuous variables into categories.These transformations are defined on contextual representations rather than directly on the original interaction log.Source: Context-Aware Representations · pp. 79–80

4.4 Side-Information Transformations

The framework separates side-information preparation, which preserves modality, from feature extraction, which produces numerical representations. It organises preparation decisions and extractor families, then applies these principles across modalities.

4.4 Side-Information Transformations

  • Side-information transformations comprise modality-preserving data preparation followed by feature extraction into a representation usable by recommendation models.The framework applies to side information generally, while this section focuses on multimodal content.Source: §4.4 · p. 80

4.4.1 Data Preparation

  • Data preparation selects information, standardises its format, and structures it for extraction while preserving the original modality and semantic content.Structuring determines the computational unit presented to the extractor, while selection sets the information boundary of the pipeline.Source: §4.4.1 · pp. 81–83

4.4.2 Feature Extraction

  • The taxonomy classifies extractors by how their representation function is obtained: hand-crafted, frozen pre-trained, end-to-end trained, or dataset-provided.Frozen pre-trained extractors are described as dominant; end-to-end training offers task adaptation at higher computational cost and with reduced transferability.Source: §4.4.2 · pp. 84–85

4.4.3 Modality-Specific Transformations

  • Preparation and extraction principles apply across textual, visual, audio, video, and knowledge-graph information, with concrete operations varying by modality.The section overviews modality-specific techniques and directs readers to their detailed descriptions in the text.Source: §4.4.3 · pp. 86–87, Table 4.4 · p. 86

Textual Information

Textual pipelines select and standardise textual fields, then structure them for an extractor. The literature spans several extractor families, with frozen pre-trained language models now dominant and input choices important to reproducibility.

Data Preparation

  • Textual preparation specifies which fields to retain and whether multiple fields are processed separately or concatenated, including their ordering and separators.Standardisation removes markup and artefacts while preserving lexical and semantic content; structuring may use word-level or subword tokenisation, truncation, and padding.Source: Textual Information · pp. 87–88

Feature Extraction

  • Textual extraction has ranged from early hand-crafted lexical descriptors and jointly trained neural encoders to the now-dominant frozen pre-trained language models.Some datasets also distribute pre-computed textual representations for direct use.Source: Textual Information · p. 88
  • Reproducibility for transformer-based text extraction requires reporting the checkpoint, tokenizer, maximum sequence length, pooling strategy, and textual input fields.Tokenisation, truncation, and padding determine the input processed by the extractor.Source: Textual Information · p. 88

Visual Information

Visual and audio pipelines make modality-specific preparation choices before extraction. Frozen pre-trained extractors are prominent, while datasets may also supply ready-made representations.

Visual Information

  • Visual preparation specifies whether items use one image or several, standardises image format and resolution, and organises retained images for extraction.With multiple images, the pipeline may retain the collection or construct fixed-size batches for aggregation.Source: Visual Information · p. 89
  • Frozen pre-trained visual encoders dominate current multimodal recommendation practice, with newer studies converging on ResNet-50 and Inception variants while CLIP adoption remains less widespread.Dataset-provided visual features are also common; Amazon benchmark collections distribute pre-computed CNN features alongside raw images.Source: Visual Information · p. 90

Audio Information

  • Audio preparation separates embedded audio when needed, standardises signal properties, and segments recordings into units for feature extraction.For spectrogram pipelines, standardisation can include Short-Time Fourier Transform, Mel-filterbank projection, and logarithmic compression.Source: Audio Information · p. 91
  • Audio extraction is narrower than textual or visual extraction: frozen pre-trained encoders dominate, and TikTok and Kwai datasets provide pre-computed acoustic embeddings.End-to-end audio encoders are only sporadically adopted; VGGish is the most frequently used audio encoder in the cited survey.Source: Audio Information · pp. 91–92

Video Information

Video pipelines must specify how frames or clips are selected, standardised, and temporally organised before feature extraction. Knowledge-graph representations range from fixed embeddings to task-optimised encoders, while cross-modal preparation must address alignment, missing modalities, and consistent splitting.

Video Information

  • Video preparation specifies temporal sampling, frame standardisation, and whether extractors receive independent frames, fixed-length clips, or ordered sequences.Pipelines should report clip duration, sampling strategy, temporal stride, and policies for videos of different lengths.Source: Video Information · pp. 92–93
  • Frozen pre-trained extractors predominate for video, while end-to-end trained encoders are less common and dataset-provided representations remain relatively uncommon.Frozen approaches include image encoders applied to sampled frames and dedicated video encoders; most studies extract features directly from released video content.Source: Video Information · p. 93

Knowledge Graph Information

  • Knowledge-graph preparation selects a source and relevant subgraph, standardises identifiers and relations, and structures graph data for the chosen extractor.Common selection steps include linking catalogue items to entities and extracting local neighbourhoods; structures may be triples, k-hop neighbourhoods, or task-specific subgraphs.Source: Knowledge Graph Information · pp. 94–95
  • Knowledge-graph features may use fixed pretrained embeddings, task-optimised graph encoders, or benchmark-provided embeddings that reduce control over extraction.Hand-crafted graph descriptors are rarely used in modern knowledge-aware recommendation.Source: Knowledge Graph Information · pp. 95–96

4.4.4 Cross-Modal Considerations

  • Cross-modal pipelines must align assets to entities, define missing-modality handling, specify whether multiple assets are retained, and keep each entity's assets in one data split.Silently dropping items with missing modalities may substantially overestimate multimodal contributions, while graph-based imputation recovers much of the lost performance.Source: §4.4.4 · pp. 96–97
  • Cross-modal extractors such as CLIP and ViLBERT encode multiple modalities jointly, but surveyed recommendation literature remains largely dominated by modality-specific feature extraction.These models are classified as frozen pre-trained extractors, rather than replacements for the preparation, extraction, and recommendation-modelling stages.Source: §4.4.4 · pp. 97–98

4.5 Empirical Trends in Data Preparation

The empirical analysis codes interaction-log transformations and side-information processing across surveyed studies, finding concentrated use of a small set of routine operations alongside substantial reporting gaps. K-core filtering dominates, and many studies do not document transformation details explicitly.

Reporting Practices and Reproducibility

  • 202 of 517 papers (approximately 39%) omit or indirectly specify interaction-log transformations, limiting reproducibility and comparability.The survey records 315 papers (approximately 61%) as explicitly describing operations and, where applicable, their configuration.Source: Reporting Practices and Reproducibility · p. 99

Prevalence of Transformation Techniques

  • User k-core (28.1%) and item k-core (25.3%) are the most frequent transformations, together accounting for over half of observed operations.The top 10 of approximately 60 identified techniques account for the vast majority of applications.Source: Prevalence of Transformation Techniques · p. 100, Figure 4.2 · p. 101
  • Iterative k-core filtering is virtually absent from top-tier results, leaving unclear whether reported one-pass user and item filters satisfy both thresholds simultaneously.A one-pass filter on one side can lower counts on the other, so the resulting data may not meet the intended core requirements.Source: Prevalence of Transformation Techniques · p. 100
  • Binarisation is the third most common practice, appearing in 14.9% of cases; session length filtering (4.8%) and sequence truncation (2.8%) are linked to sequential and session-based recommendation architectures.User alignment (2.0%) is reported for cross-domain tasks.Source: Prevalence of Transformation Techniques · pp. 101–102
  • Reported interaction-log operations may understate preprocessing because some studies use already-processed dataset releases, while metadata-driven item filtering occurs in only 2.1% of cases.The survey notes that transformations embedded in a dataset version may not be reported as experimental pipeline steps.Source: Prevalence of Transformation Techniques · p. 102

Interplay Between Transformations

Transformation combinations cluster around user and item k-core filtering, often paired with binarisation, while iterative execution is rarely reported. Dataset-level operations dominate the surveyed distribution, and preparation pipelines are typically short.

Interplay Between Transformations

  • User and item k-core filtering co-occur in 429 instances, alongside binarisation in 141 and 111 instances, respectively.This combination forms a common multi-stage pipeline that reduces interaction sparsity and converts remaining interactions to implicit feedback for Top-N tasks.Source: Interplay Between Transformations · p. 102
  • Iterative k-core execution is explicitly reported in almost no cases, so one-pass filtering may leave user or item counts below the intended threshold.The survey identifies this ambiguity as a barrier to determining whether datasets satisfy core requirements and to exact replication.Source: Interplay Between Transformations · pp. 103–104

Structural Nature of Transformations

  • Dataset-level transformations make up approximately 90.3% of cases, with support-driven operations at 73.7% and attribute-driven operations at 16.6%.Representation-dependent transformations account for about 9.7%; session adaptations occur in 5.1%, sequential pruning in 3.5%, and context-related transformations in 1.1%.Source: Structural Nature of Transformations · pp. 104–105

Complexity of Data Preparation Pipelines

  • Preparation pipelines average 2.22 techniques per paper, with two operations most common; four or more are a minority and 8–10-operation pipelines are outliers.The two-operation peak corresponds to k-core filtering and binarisation; approximately 90 papers use only one technique.Source: Complexity of Data Preparation Pipelines · pp. 106–107

4.5.2 Empirical Trends in Side Information Transformations

  • The side-information analysis counts disclosed extractor choices and representation provenance, but does not infer unreported selection, standardisation, or structuring decisions.Undisclosed or absent modality pipelines are reported separately from declared extractor choices.Source: §4.5.2 · pp. 107–108

Corpus and method

The encoder analysis manually codes multimodal feature choices across 130 paper–dataset pairs from 48 studies. Frozen pre-trained extractors dominate disclosed choices, while dataset-shipped representations and less common extraction strategies remain distinct categories.

Corpus and method

  • The analysis covers 48 studies using multimodal side information, comprising 130 paper–dataset pairs from the 517-paper survey corpus.Each pair was coded for dataset, available modalities, encoder choices, extractor specification, and whether representations were extracted by the study or supplied with the dataset.Source: Corpus and method · p. 108
  • Paper–dataset pairs are the unit for setting-dependent outcomes, while whole papers are the unit for paper-level reporting outcomes.Modality-specific observations are counted separately, so one pair can contribute visual, textual, audio, or knowledge-graph observations.Source: Corpus and method · p. 108, Encoder adoption by modality · p. 109

Encoder adoption by modality

  • Visual extractors are declared for 110 of 129 pairs with visual information, textual extractors for 110 of 127 pairs with textual information, and audio or knowledge-graph extractors for 19 pairs.Table 4.5 reports disclosed feature extractors by category using modality-specific paper–dataset observations.Source: Encoder adoption by modality · p. 109, Table 4.5 · p. 109
  • Frozen pre-trained extractors account for 73 of 110 declared visual, 82 of 110 textual, and 12 of 19 other-modality extractors.Dataset-shipped representations are also substantial for visual information, while end-to-end training is less frequent and hand-crafted extraction occurs only for textual information.Source: Encoder adoption by modality · p. 110
  • Among frozen pre-trained extractors, ResNet-50 accounts for 32 of 73 visual observations, while BERT and Sentence-BERT jointly account for 54 of 82 textual observations.VGGish appears in 10 of 12 declared frozen pre-trained audio observations, all associated with TikTok or Kwai.Source: Encoder adoption by modality · p. 110, Figure 4.6 · p. 111, Figure 4.7 · p. 112
  • Dataset-shipped representations account for 28 of 110 declared visual extractors and 11 of 110 textual extractors, and are supplied with the dataset rather than extracted in the reported pipeline.Unless the study extracts features anew from raw side information, it does not control the encoder or preprocessing procedure; these representations should be reported separately from frozen pre-trained extraction.Source: Encoder adoption by modality · p. 110
  • End-to-end extractors account for 28 modality-specific observations, while hand-crafted extractors appear only in three textual observations using SGRank.No visual, audio, or knowledge-graph observation uses a hand-crafted extractor as its primary representation.Source: Encoder adoption by modality · pp. 110–111
  • A temporal analysis is needed to determine whether end-to-end extractor prevalence changed over the surveyed period.The reported analysis gives a distribution but does not establish its temporal trajectory.Source: Encoder adoption by modality · p. 111

Data Splitting Strategies

This section characterizes offline evaluation splitting protocols, mapping recurring families and design dimensions across recommendation paradigms. It clarifies how protocol choices shape temporal information boundaries and model evaluation, and supports more reproducible and comparable descriptions.

5.1 Taxonomy of Splitting Strategies

  • The taxonomy groups splitting practices into random, temporal, and fixed, inherited, or hybrid families, offering a compact descriptive language for inconsistently presented protocols.Its categories synthesize recurring practices and are not intended as universally established or mutually exclusive names.Source: §5.1 · p. 114

5.1.1 Random (order-agnostic) strategies

  • Random hold-outs allocate interactions globally or within users, while random leave-one-out reserves one test interaction per user; negative sampling separately shapes the ranking task.Global splits do not ensure user or item coverage across subsets, and user-wise coverage depends on sufficient histories or eligibility constraints; random protocols may also train on interactions later than evaluation data.Source: §5.1.1 · p. 114, §5.1 · p. 115

5.1.2 Temporal (time-aware) strategies

  • Temporal protocols include global, user-wise, and session- or window-based splits, with later interactions reserved for evaluation to approximate prediction under information-availability constraints.They reduce direct use of future interactions but require reliable timestamps and may intensify sparsity, cold-start effects, and distribution shift; user-wise cutoffs need not share a global boundary.Source: §5.1.2 · p. 116, §5.1 · p. 117

5.1.3 Fixed, inherited, Hybrid, and undocumented protocols

  • Fixed or inherited splits can support comparison when public and precisely identified, but their original allocation may be unrecoverable; hybrid protocols combine rules, while incomplete reporting impedes reproducibility.Undocumented procedures can prevent determining whether an evaluation is random, temporal, fixed, or hybrid, limiting methodological comparability.Source: §5.1.3 · p. 118, §5.2 · p. 119

5.1.4 Discussion

  • Protocols sharing a label can differ substantially, so performance and methodological comparisons require the allocation details and split provenance, not protocol names alone.The assessment setting determines training information, predicted behaviour, and deployment resemblance; untraceable fixed or inherited splits leave assumptions unknown.Source: §5.1.4 · p. 119, §5.2 · p. 122

5.2. Dimensions of Variation Across Splitting Protocols

  • Splits vary by grouping unit, ordering, allocation and repetition, temporal cutoff, validation design, entity coverage, and provenance, so a broad family label alone cannot establish equivalent evaluation settings.Candidate-set construction and negative sampling are separate from splitting, though they further affect interpretation of performance estimates.Source: §5.2 · pp. 120–122

5.3 A unified formalisation of splitting protocols

  • The unified formalisation treats a protocol as an assignment of observed interactions to training, validation, test, or exclusion labels, regardless of how the assignment is constructed.Construction may combine grouping, ordering, temporal cutoffs, selection, filtering, and repeated resampling decisions.Source: §5.3 · pp. 122–124
  • The formalization distinguishes random from ordered interactions, hold-out from leave-one-out, global cutoffs from temporal blocks, and single from repeated or cross-validation assignments.Repeated hold-out aggregates estimates from independent random assignments, whereas K-fold cross-validation makes each eligible observation or group test data in one fold; hybrid protocols can combine assignment stages.Source: §5.3 · pp. 124–126
  • Interaction-derived transformations must be fitted using training data or information available before the temporal cutoff to avoid validation or test information entering training.Static, exogenous attributes may be restricted to entities in each sublog, but time-varying or interaction-derived features must respect information available at prediction time.Source: §5.3 · pp. 127–128
  • Protocol descriptions should record split provenance and treatment of excluded or unseen entities, while candidate-set construction and negative sampling remain separate evaluation choices.Post-split transductive filtering is optional because retaining unseen users or items may be intentional in inductive or cold-start evaluation.Source: §5.3 · pp. 127–128

5.4.1 Overall prevalence of splitting families

  • Random global or user-wise hold-out accounts for 35.8% of paper–dataset pairs, random leave-one-out 21.2%, temporal leave-one-out 14.5%, and temporal hold-out 12.3%.Fixed or inherited benchmark splits account for 8.4%, while hybrid, custom, and cross-validation procedures together account for 4.1%. Random protocols do not impose a common temporal information boundary, unlike protocols designed to predict future behaviour.Source: Table 5.1 · p. 129, §5.4.1 · pp. 129–130

5.4.2 Paradigm-dependent conventions

  • Collaborative filtering and cross-domain recommendation predominantly use random protocols, while sequential recommendation concentrates on temporal leave-one-out for next-item prediction.Examples include collaborative filtering He et al., 2017; Rendle et al., 2009; Sedhain et al., 2015 and cross-domain recommendation Shu et al., 2018; sequential protocols commonly reserve each user’s latest interaction for testing and the penultimate for validation Kang and McAuley, 2018; Rendle et al., 2010; Sun et al., 2019a.Source: §5.4.2 · p. 130
  • Session-based recommendation favours global temporal hold-out, graph-based work has a 30% combined share of fixed/inherited and hybrid/custom procedures, and LLM-based practice remains heterogeneous.Session studies often reserve the latest temporal block, such as a day or week, for evaluation Hidasi et al., 2016; Wu et al., 2019. The graph-based 30% is a combined category and does not mean all cases are inherited benchmark splits.Source: §5.4.2 · p. 131

5.4.3 Temporal trends: aggregate changes in protocol adoption

  • Temporal protocols become more prevalent from approximately 2016, alongside greater visibility of fixed and inherited benchmark splits.The aggregate trend is descriptive: it may reflect both changing practices within paradigms and shifts in the corpus’s representation of sequential and session-based research.Source: §5.4.3 · pp. 131–132, Figure 5.1 · p. 133

5.4.4 Protocol reporting and reproducibility

  • Not Declared reporting is comparatively low in sequential recommendation and higher in session-based, multimodal, and LLM-based work, limiting reproducibility and comparability.Transparent reporting should specify the allocation rule, grouping unit, ordering criterion, validation design, entity-coverage policy, and split provenance.Source: §5.4.4 · pp. 132–133

Task-Aware Offline Evaluation

Offline evaluation is a comparison between complete experimental pipelines, not just models: dataset, preparation, side-information, and splitting choices shape the task and the interpretation of results. Credible progress requires these assumptions and protocols to be explicit, task-aware, and reproducible.

Task-Aware Offline Evaluation

  • A dataset’s domain, feedback, temporal coverage, sparsity, entities, and side information encode assumptions; even the same named dataset may differ by version, subset, or representation.Selection should be motivated by the intended recommendation scenario, not treated only as a conventional benchmark choice.Source: Task-Aware Offline Evaluation · pp. 134–135
  • Filtering, feedback transformations, sequence construction, and side-information extraction alter available evidence; choices of modalities, extractors, pretraining sources, and provenance can materially affect the task.Established practices may be adopted without their exact implementation being disclosed; the rationale and implementation should be explicit and reproducible.Source: Task-Aware Offline Evaluation · p. 135
  • Random and user-wise splits are common for ranking unobserved interactions, while temporal splits are more prevalent when sequential or session-based recommendation centers time.No split is universally preferable; allocation, grouping, ordering, validation, unseen-entity handling, and provenance should be specified for reconstruction.Source: Task-Aware Offline Evaluation · p. 135
  • When datasets, transformations, or protocols differ or remain partly described, offline metric gains cannot automatically be attributed to modelling, and familiar dataset names do not guarantee comparability.Visible pipeline details are also necessary for later researchers to replicate or meaningfully extend an experiment.Source: Task-Aware Offline Evaluation · pp. 135–136
  • Offline reports should specify dataset source and version, interaction semantics, transformations, side-feature provenance, and complete train-validation-test protocols, with code, configurations, and splits released when possible.Future work includes versioned datasets and reusable pipeline specifications, reporting standards, robustness checks, and protocols attentive to deployment conditions.Source: Task-Aware Offline Evaluation · pp. 136–137

Related papers

Loading 2609.31696v1…