Sequential Multimodal Evidence Optimization for Product Media Ranking in E-Commerce

Prasenjit Dey; Frank McIntyre; Arnab Sinha

Source-linked AI summary

Sequential Multimodal Evidence Optimization for Product Media Ranking in E-Commerce

Prasenjit Dey, Frank McIntyre, Arnab Sinha

arXiv:2608.15662v1Aug 16, 2026cs.LG

TL;DR

E-commerce customers must sequentially inspect heterogeneous product media, making it important to order evidence so they can reach purchase decisions efficiently. SMEO learns prefix utility and optimizes a survival-weighted autoregressive ranking policy, improving estimated conversion by 5.5% and reducing swipes-to-conversion by 15% over an existing baseline.

Problem
Customers sequentially accumulate evidence from product media, creating a need to order multimodal assets so they find decision-relevant information with less effort.
Method
SMEO learns consumed-prefix conversion utility and trains an autoregressive policy with survival-weighted optimization to front-load decision-relevant media.
Results
5.5% higher DR-CVR and 15% fewer swipes-to-conversion than the existing baseline are reported under offline off-policy evaluation.
Takeaways & Limitations
SMEO surfaces decision-relevant product media earlier and shows gains in estimated conversion and swipe efficiency across offline baseline comparisons.
Takeaways & Limitations
The approach is scoped to evidence available in the consumed media prefix and models logged stopping depth as endogenous.

Abstract

from arXiv · show

On modern e-commerce stores, customers consume ordered slates of heterogeneous product media, such as images, videos, and 3D renders, before making purchase decisions. Existing media-ranking systems often optimize myopic engagement proxies such as clicks or dwell time, even though product media assets are cooperative informational components of the same item that together help customers find the information they need through sequential interaction. We present Sequential Multimodal Evidence Optimization (SMEO), a two-stage utility-guided framework for customer-oriented media sequencing. SMEO first learns a trajectory utility model from consumed media prefixes to estimate how ordered evidence helps customers reach a purchase decision, while mitigating position-bias and variable-depth imbalance in logged data. Recognizing that customer attention is a limited resource, it then trains an autoregressive ranking policy with survival-weighted reward-to-go that prioritizes the most decision-relevant information early, so customers can find what they need with less effort. By decoupling utility learning from policy optimization, SMEO enables stable offline learning from biased logs and post-hoc media attribution without explicit media-level labels. Evaluated offline on large-scale e-commerce sessions using doubly robust off-policy estimation, SMEO improves estimated conversion by 5.5% and helps customers reach a purchase decision with 15% fewer swipes than existing baselines.

1 Introduction

SMEO treats product-media ranking as sequential multimodal evidence optimization: heterogeneous assets cooperatively build purchase confidence, but customers may abandon before reaching deeper decision-relevant media. Its two-stage utility-guided framework learns conversion utility from consumed prefixes and trains a policy to front-load decision-relevant media, improving offline conversion estimates while reducing swipes.

Motivation: Customers sequentially accumulate evidence from heterogeneous product media, yet early failures can cause abandonment before deeper decision-relevant assets are revealed.Mobile viewport constraints require customers to swipe or click through ordered media slots.
Limitations of Existing Methods: Existing methods optimize myopic engagement proxies such as clicks or dwell time, which may not provide the specifications needed for terminal conversion.The persuasive utility of one asset can depend strongly on which complementary media the customer viewed first.
Problem Formulation: SMEO formulates media ranking as cooperative sequential evidence optimization under variable stopping depths and sparse terminal rewards.Multiple assets jointly shape a single purchase, while media-level attribution is unobserved in logs.
SMEO Framework: SMEO learns trajectory utility from consumed prefixes with position-aware learning and survival-based depth balancing, then trains an autoregressive policy using survival-weighted reward-to-go.The utility model is frozen before policy training, and the framework supports post-hoc attribution through counterfactual masking without explicit media-level conversion labels.
Results: 5.5% improvement in DR-CVR and 15% reduction in swipes-to-conversion over the existing baseline were achieved in rigorous offline off-policy evaluation.The evaluation used large-scale offline data and off-policy estimation.

2 Related Work

Prior sequential recommendation and slate-ranking methods model competing catalog items and item-level engagement, whereas SMEO sequences cooperative media assets for one product under sparse terminal feedback. Its delayed-outcome attribution also relates to Multi-Touch Attribution, while targeting sequentially viewed media rather than next-item engagement or full-slate relevance.

Sequential Recommendation and Slate Optimization: Sequential recommenders model user histories to predict future item interactions, while slate-ranking methods capture dependencies among catalog items through listwise scoring or autoregressive generation.Examples include DIN, DIEN, SASRec, DLCM, and Seq2Slate.
Sequential Recommendation and Slate Optimization: SMEO instead models cooperative media assets for a single product, optimizing sequentially viewed media under sparse terminal labels rather than next-item engagement or full-slate relevance.Its feedback setting differs from conventional catalog recommendation, where items are competing alternatives and feedback is typically item-level engagement.
Credit Assignment and Counterfactual LTR: Attributing delayed outcomes across sequential interactions relates SMEO to Multi-Touch Attribution, which assigns conversion credit across marketing touchpoints.The supplied passage introduces this connection but ends before detailing the cited MTA methods.

3 Problem Formulation

The problem formulates product media ranking as sequential ordering under customer stopping, where only a variable-length consumed prefix and terminal purchase outcome are observed. It targets both conversion-maximizing media sequencing and latent, context-dependent attribution of individual media contributions.

Problem Setup: Each product has a multimodal media catalog, and a policy presents a feasible ordered slate of its images, videos, and 3D renders.The slate is represented as an ordering of the product’s media assets.
Problem Setup: Customers consume media sequentially and may stop early, so logs contain only a consumed prefix ending at an endogenous stopping depth.The consumed prefix consists of the assets viewed up to the observed stopping point.
Observation Model: Assets after the observed stopping depth are unobserved and should not receive direct credit for the terminal purchase outcome y_s ∈ {0, 1}.The formulation therefore combines variable-depth trajectories with sparse terminal rewards.
Optimization Objectives: The objectives are to learn product-conditioned orderings that maximize expected terminal conversion and estimate each asset’s marginal contribution despite unobserved media-level labels.Contribution may depend on surrounding media context and ordering, distinguishing the task from standard listwise ranking.

4 Proposed Methodology · 4.1 Multimodal Asset Representation · 4.2 Trajectory Utility Model

SMEO represents each product asset with multimodal visual, modality, historical, and interaction features, then learns a position-aware trajectory utility model from consumed prefixes under variable-depth supervision. The model mitigates position-bias leakage and balances stopping depths while restricting credit to evidence actually viewed.

4 Proposed Methodology: SMEO first constructs multimodal asset representations, then learns a trajectory utility model from logged consumed prefixes before using the frozen utility for policy training and media attribution.The framework later trains an autoregressive pointer-network policy with the frozen utility as a reward model.
4.1 Multimodal Asset Representation: Each asset representation fuses visual content, modality type, historical priors, and in-session interaction signals.Historical priors include aggregate statistics such as average dwell time and click-through rates, while training-only interaction signals include normalized dwell time, click count, and zoom count.
4.1.1 Visual Content Embedding: Visual features use frozen modality-specific encoders, with outputs projected to d_v = 256 and a learnable modality embedding of d_e = 32 capturing distinct media roles.The encoders are CLIP ViT-L/14 for images, VideoMAE with mean-pooled frame features for video, and Uni3D for 3D point clouds and VTO meshes.
4.1.2 Interaction Features and Train-Serve Decoupling: Feature masking replaces training-time interaction signals with a learned mask token with probability p_drop = 0.3 and always masks them at inference.Masked examples force prediction from visual content and historical priors alone, while position-aware composite tokens encode evidence-accumulation order with d = 256.
4.2.1 Causal Sequence Encoder and Position-Aware Learning: The trajectory utility model is an L-layer causal Transformer whose contextualized states represent cumulative evidential belief for each consumed prefix.Position-aware tokens form the observed prefix sequence consumed by the utility model.
4.2.1 Causal Sequence Encoder and Position-Aware Learning: Position-Aware Learning decomposes utility into content and position-only logits, uses 10% dropout on positional features, and discards the position tower during policy optimization and inference.This nuisance pathway reduces position-bias leakage from biased logged data while the frozen reward model uses PAL-adjusted content utility.
4.2.2 Variable-Depth Supervision: Utility supervision is restricted to each session’s observed consumed depth, preventing assets viewed later from receiving direct credit for an earlier prefix.Supervising all prefixes can incorrectly credit a weak image for a conversion observed only after a later video was consumed.
4.2.3 Stopping Depth Balancing: Survival-based depth reweighting addresses shallow-prefix dominance by upweighting rare deep prefixes, while clipping controls variance.The trajectory balancing weight uses the empirical probability of reaching each depth.

4.3 Autoregressive Pointer Network Policy

SMEO uses an autoregressive pointer-network policy to construct media rankings directly from asset embeddings. Order-agnostic encoding and masked decoding produce content-driven, valid permutations of each product’s media assets.

Policy architecture: The policy is a pointer network whose output vocabulary is the input asset set, enabling ranking across products without retraining on fixed item IDs.It operates on asset embeddings rather than fixed item IDs.
Policy architecture: The encoder represents all assets without positional embeddings, so representations depend on content and mutual context rather than catalog order.The interaction mask token m_τ is used to construct representations.
Autoregressive decoding: A 2-layer causal Transformer decoder autoregressively selects the ordering by attending to previously selected asset representations and scoring each unselected asset with pointer attention.At step t, causal self-attention produces decoder state s_t before candidate scoring.
Autoregressive decoding: Masked softmax over unselected assets guarantees that the decoder outputs a valid permutation.The policy begins from a learnable initial state and uses learnable projections for scoring.

4.4 Policy Optimization with Survival-Weighted Reward-to-Go

SMEO optimizes media orderings using survival-weighted marginal utility, prioritizing conversion-informative evidence before likely customer drop-off. Reward-to-go policy optimization with a self-critical baseline provides localized, lower-variance learning while respecting the fixed primary-image slot.

Utility-based policy objective: SMEO scores generated orderings using frozen pre-serving content utility, replacing in-session interactions with a learned mask token.Policy optimization therefore uses only information available before carousel interaction.
Utility-based policy objective: Marginal utility removes constant base conversion utility, so uninformative assets contribute approximately zero reward instead of benefiting from longer prefixes.The prefix utility is initialized to the empirical base conversion rate.
Survival-weighted reward: Survival weighting assigns each position’s marginal utility according to the probability that customers reach it, rather than stop exactly there.Because reach probability decreases with carousel depth, identical gains receive greater reward when surfaced earlier, targeting reduced swipe depth.
Policy-gradient optimization: Reward-to-go REINFORCE gives each decoding decision credit only for future rewards, unlike terminal REINFORCE’s shared slate reward.SMEO uses a greedy self-critical sequence-training baseline to reduce variance.
Policy-gradient optimization: The greedy baseline uses stop-gradient deterministic decoding and requires one additional rollout per batch, which is practical for small asset sets.The baseline is computed from the current policy via argmax decoding.
Policy constraints and regularization: SMEO optimizes slots from t = 2 onward because the primary product image is fixed at t = 1, while entropy regularization uses λ_H = 5 × 10^-4.This preserves product-page layout conventions and helps prevent premature policy collapse under noisy offline advantages.

4.5 Media Contribution Estimation

Section 4.5 introduces offline media diagnostics using a frozen content-based utility model and leave-one-out masking to estimate context- and ordering-dependent asset contributions. Evidence accumulation curves complement these scores with interpretable signals for media prioritization, replacement, and requests, while primary evaluation remains conversion-oriented.

Utility-based diagnostics: The frozen content-based utility model enables offline media diagnostics with the PAL tower removed and in-session interactions masked.These diagnostics estimate media contribution without relying on explicit media-level labels.
Utility-based diagnostics: Leave-one-out masking estimates each media asset’s contribution under a given ordering, helping identify high-value or redundant media and media types to surface earlier.The score is context- and ordering-dependent because the remaining assets are re-indexed after masking.
Evidence accumulation: Evidence accumulation curves E_t(π) = U(π≤t) visualize how quickly an ordering builds conversion utility before likely drop-off.The diagnostics provide category audit and merchandising teams with interpretable signals for prioritizing, replacing, or requesting media assets; primary evaluation uses observed conversion-oriented metrics.

5 Experiments

Experiments use large-scale, temporally held-out e-commerce session logs to compare SMEO with engagement-, conversion-, listwise-, and autoregressive-ranking baselines using offline conversion and early-swipe metrics. Results indicate that conversion-aware sequential modeling front-loads informative evidence, while position-aware utility learning and policy design drive the gains.

Experimental setup: The evaluation uses approximately 150M anonymized mobile sessions spanning millions of products and multiple categories, with candidate slates capped at N_max = 15.Sessions include ordered media, per-asset interactions, stopping depth, and terminal purchase outcomes; evaluation uses a temporally held-out test set.
Experimental setup: SMEO is compared with Heuristic, CTR, CLTR-CVR, Listwise-MM, and Seq2Slate-Adapt using SNIPS-CVR, DR-CVR, and MSC.The baselines isolate engagement optimization, independent conversion scoring, listwise context, and terminal-outcome autoregressive reinforcement learning.
Main results: Engagement proxies are insufficient: CTR delivers marginal conversion lift and almost no swipe-efficiency improvement, whereas conversion objectives and inter-asset context improve DR-CVR.CLTR-CVR remains limited by independent asset scoring and unmodeled sequence redundancy, while Listwise-MM adds inter-asset context.
Ablation study: Removing PAL reduces downstream DR-CVR by 1.0% relative to full SMEO, showing that position-aware learning is important for utility estimation.The ablation removes the PAL tower during Stage 1 training, forcing the utility model to absorb historical position effects into the content encoder.
Evidence front-loading: SMEO leads on independently predicted early-prefix utility before likely customer drop-off, while Listwise-MM and Seq2Slate-Adapt partially catch up later.This indicates that SMEO primarily improves earlier placement of conversion-informative media rather than merely increasing full-slate utility.
Interpretability: SMEO learns media-type-specific persuasion roles without supervision: demonstration assets matter for operation, motion, or fit-in-use, while specification and ingredient images matter in high-scrutiny categories.Specification and ingredient images contribute 0.078±0.017 versus 0.022±0.009 in high-scrutiny categories, with p<0.01.

6 Batch Inference and Serving

SMEO supports offline, horizontally scalable batch generation from static product signals, with low-latency serving and fallback availability. Its attribution analysis finds reduced utility from visually redundant assets and category-specific roles for different media modalities.

Batch inference: SMEO generates media orderings offline from static visual embeddings, historical priors, and product/media metadata, avoiding dependence on real-time customer state at page load.A distributed batch pipeline uses precomputed embeddings and scales horizontally, with refresh cost linear in catalogue size.
Serving: Generated orderings are retrieved by product identifier through a single distributed key-value-store lookup with sub-5 ms latency, while unavailable orderings fall back to a default.This serving design preserves availability when a generated ordering is missing.
Attribution: ρpartial = −0.61 (p< 0.001) indicates that visually redundant assets contribute less incremental utility after controlling for position-depth confounding.The correlation is based on partial Spearman correlation controlling for asset position t.
Attribution: 3D assets contribute more in spatial categories than elsewhere, with 0.091±0.019 versus 0.028±0.011 (p<0.01), where geometry, scale, and room-fit evidence matter.The attribution pattern is category-specific and reflects distinct merchandising roles for media.

7 Conclusion

SMEO treats product media as sequential, cooperative evidence and combines consumed-prefix conversion utility learning with survival-weighted autoregressive policy optimization. Offline evaluation shows gains over multiple baseline families in estimated conversion and swipe efficiency, while diagnostics provide further analysis.

7 Conclusion: SMEO is a two-stage framework that treats product media as sequential, cooperative evidence for purchase decisions.It learns consumed-prefix conversion utility and optimizes a survival-weighted autoregressive policy.
7 Conclusion: SMEO surfaces the most decision-relevant media early, before likely customer drop-off.This follows from combining consumed-prefix utility learning with survival-weighted autoregressive policy optimization.
7 Conclusion: Offline evaluation shows consistent gains over heuristic, pointwise, listwise, and autoregressive baselines in estimated conversion and swipe efficiency.The conclusion reports improvements across both outcome and attention-efficiency measures, without specifying their numerical values here.

8 Gen-AI Usage Disclosure

Generative AI tools were used only to refine language in author-written text, while all substantive manuscript content and interpretations remained the authors’ own.

8 Gen-AI Usage Disclosure: Generative AI was used solely to improve grammar, style, clarity, and readability of text originally authored by the paper’s authors.No manuscript sections were generated by AI systems.
8 Gen-AI Usage Disclosure: The authors retained responsibility for the manuscript’s conceptual contributions, technical content, experimental design, data analysis, and interpretations.This included the abstract, related work, methodology, experimental results, and conclusions.