Source-linked AI summary
Does This Moment Justify the Recommendation? Counterfactual Behavior-Grounded Evidence Retrieval for Personalized Video Recommendation
Xin Liu
TL;DR
Personalized video recommendation and temporal grounding respectively score videos and localize query-relevant moments, but neither alone establishes whether a moment is evidence for recommending a video to a particular user. The paper introduces a counterfactual benchmark and framework to separate localization from evidence existence and test response to replacement. CBGER reports gains in Pair Accuracy and Intervention Consistency, while its MRR difference from QD-DETR is not statistically significant.
Problem
Video-level recommendation scores provide little temporal evidence, while localization does not determine whether a moment supports recommending a video to a particular user.
Source: §1 · p. 1Method
CBGER-10K pairs factual and counterfactual videos that replace one behavior-supported segment, and CBGER uses structured counterfactual learning to separate segment localization from video-level evidence estimation.
Source: §1 · pp. 1–2Results
CBGER achieves 0.4432 MRR, 0.6977 Pair Accuracy, and 0.6987 Intervention Consistency; versus QD-DETR, its MRR difference is not significant while Pair Accuracy improves by 0.1103.
Source: §1 · p. 2Takeaways & Limitations
The results show that strong temporal localization does not necessarily imply reliable personalized evidence existence, supporting evaluation of Whether alongside Where.
Source: §7 · p. 8Automatically constructed silver labels may carry semantic and matching biases, and the MicroLens histories and FineVideo candidates measure behavior-supported evidence rather than actual exposure, preference, or causal engagement.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Personalized video recommendation predicts user preference at the video level, while temporal video grounding localizes query-relevant moments. However, strong localization does not establish whether the retrieved moment constitutes valid evidence for recommending the video to a particular user. We study counterfactual behavior-grounded evidence retrieval, which separates where personalized evidence occurs from whether such evidence exists and evaluates whether model predictions respond consistently when that evidence is replaced. We introduce CBGER-10K, containing 5,000 controlled factual--counterfactual pairs for 3,026 users, where each pair replaces only the focal behavior-supported segment while preserving the user, temporal position, and hard distractors. We further propose CBGER, a compact framework that decouples segment-level localization from video-level evidence estimation and learns both through structured counterfactual supervision. CBGER achieves $0.4432$ MRR, $0.6977$ Pair Accuracy, and $0.6987$ Intervention Consistency across five adapted personalized-highlight and temporal-grounding baselines. Notably, compared with QD-DETR, its MRR improvement is not statistically significant, while Pair Accuracy improves by $11.03$ points. These results show that accurate temporal localization does not necessarily imply reliable personalized evidence existence, motivating explicit evaluation of Whether alongside Where.
1. Introduction
The introduction reframes personalized video grounding as both locating behavior-supported moments and deciding whether they constitute evidence for recommending a video. It presents matched counterfactuals and structured learning to evaluate these dimensions and model responses to evidence replacement.
1. Introduction
- The task separates Where, which localizes behavior-supported evidence, Whether, which tests whether sufficient evidence exists, and Intervention, which tests response to evidence replacement.Source: §1 · p. 1
- CBGER decouples user-conditioned segment localization from video-level evidence estimation, using structured counterfactual learning for localization, pairwise ordering, local intervention response, and distractor invariance.Source: §1 · p. 2
- CBGER achieves 0.4432 MRR, 0.6977 Pair Accuracy, and 0.6987 Intervention Consistency; versus QD-DETR, its MRR difference is not significant, but Pair Accuracy improves by 0.1103.The Pair Accuracy improvement has a 95% CI of [0.0765, 0.1430].Source: §1 · p. 2
- CBGER-10K contains 5,000 factual–counterfactual pairs for 3,026 users, replacing one focal segment while preserving the user, temporal position, and surrounding distractors.The benchmark uses single-slot interventions and user- and source-disjoint splits.Source: §1 · p. 2, Figure 1 · p. 2
- Across five adapted personalized-highlight and temporal-grounding baselines, the work argues localization quality alone does not characterize personalized evidence reliability.Source: §1 · p. 2
2. Related Work
Related work connects CBGER to temporal grounding, personalized highlights, missing-correspondence detection, explainable recommendation, counterfactual evaluation, and behavior-conditioned video benchmarks. It positions CBGER as combining personalized evidence localization with video-level evidence existence and intervention consistency.
2. Related Work
- Temporal grounding and personalized-highlight methods localize or rank relevant moments, but primarily focus on which moment is most relevant within a video.The reviewed grounding work spans proposal-based matching, transformer-based retrieval, and unified grounding–highlight formulations.Source: §2 · pp. 2–3
- Negative-aware grounding tests absent query events, while explainable recommendation and counterfactual video evaluation address reasons for suggestions and dependence on relevant evidence.CBGER combines these directions with behavior-derived user profiles and matched factual–counterfactual timelines.Source: §2 · p. 3
- CBGER-10K builds on MicroLens and FineVideo but targets whether localized content is supported by observed user behavior, using silver labels and controlled, disjoint splits.Source: §2 · p. 3
3. Task and the CBGER-10K Benchmark
The task separates localization of behavior-supported segments from deciding whether a video contains personalized evidence and whether predictions respond to replacing it. CBGER-10K operationalizes this distinction with matched factual–counterfactual timelines, while evaluating its prediction axes separately.
3.1. Task Definition
- The task predicts segment-level evidence scores and a video-level evidence-existence score, evaluating localization, factual-over-counterfactual ordering, and focal-slot score decrease after replacement.Source: §3.1 · p. 3
3.2. Behavior Profiles and Candidate Videos
- MicroLens histories and titles yield behavior profiles from observed consumption, while FineVideo supplies eligible annotated intervals matched to identify focal evidence.Profiles summarize observed consumption rather than psychological ground-truth interests.Source: §3.2 · p. 3, Figure 2 · p. 4
3.3. Semantic Annotation and Matching
- 7,651 valid intervals remain after structured Qwen3-VL annotation filtering, and BGE-M3 matching admits factual evidence only at m(u,v) ≥ 0.30.Construction encoders are not exposed downstream, which instead uses frozen CLIP features; separation reduces direct leakage but does not remove potential teacher bias.Source: §3.3 · p. 4
3.4. Factual–Counterfactual Pair Construction
- Each nine-segment timeline pairs one focal factual segment with a behavior-weaker content neighbor, holding the user, slot, and eight distractors fixed; labels are silver, not engagement-causal.Evidence segments are subject to reuse and per-user contribution limits.Source: §3.4 · pp. 4–5
3.5. Splits and Benchmark Statistics
- Users and FineVideo sources are disjoint across train, validation, and test splits, and all 5,000 pairs pass the stated structural checks.Source: §3.5 · p. 5
3.6. Evaluation Metrics
- Where uses MRR and NDCG@k on factual records, Whether uses Pair Accuracy, and Intervention Consistency checks whether the focal-slot score falls after replacement.The axes are reported separately because accurate localization can coexist with failure to distinguish evidence-present from evidence-weakened videos.Source: §3.6 · p. 5
4. Method
CBGER separates segment localization from video-level evidence estimation, then trains both using matched factual and counterfactual timelines. Its design distinguishes where personalized evidence occurs from whether it exists.
4.1. Overview
- Given a user profile and nine-segment timeline, the shared model outputs segment-level Where scores and video-level Whether scores for factual and counterfactual inputs.Source: §4.1 · p. 5
4.2. Behavior-Conditioned Evidence Scoring
- Frozen CLIP representations feed a lightweight scorer combining residual similarity with relative-temporal correction, producing segment logits whose ranking localizes Where.The CLIP encoders remain frozen while the lightweight scorer is trained.Source: §4.1 · p. 5, §4.2 · p. 5, Figure 3 · p. 6
4.3. Video-Level Evidence Estimation
- Parameter-free mean pooling aggregates segment scores for Whether, evaluating evidence across the controlled timeline without adding parameters.Source: §4.3 · p. 6
4.4. Intervention Consistency
- Intervention consistency measures whether the edited focal moment’s score decreases when its supporting evidence is replaced, unlike Whether’s comparison of video-level scores.The local response is measured at the known edited slot.Source: §4.4 · p. 6
4.5. Structured Counterfactual Learning
- Training jointly supervises factual localization, factual-over-counterfactual video evidence ordering, edited-slot score reduction, and stability across eight unchanged distractors.The invariance term prevents a trivial global shift of counterfactual scores.Source: §4.5 · p. 7
5. Experiments
Experiments compare CBGER with adapted recommendation and grounding baselines, test its components and aggregation choices, and examine intervention responses. CBGER improves evidence-ordering and intervention measures over QD-DETR despite statistically similar localization performance.
5.1. Experimental Setup
- All methods use the same frozen splits, CLIP representations, timelines, and downstream evaluation; results are averaged over three random seeds.Baselines include personalized-highlight and temporal-grounding models, with user behavior profiles replacing language queries for temporal-grounding methods.Source: §5.1 · p. 7
5.2. Main Results
- 0.1103 PairAcc and 0.0470 Intervention gains over QD-DETR are significant, while MRR (.4432 vs. .4231) is statistically similar.CBGER's PairAcc gain over QD-DETR has a 95% CI of [0.0765, 0.1430]; the Intervention gain has a 95% CI of [0.0166, 0.0772].Source: §5.2 · p. 7, Table 2 · p. 8
5.3. Where–Whether Ablation
- Counterfactual training raises MRR from .4274 to .4432 and Intervention from .6830 to .6987; mean pooling boosts PairAcc by 5.8 and 6.8 points without and with training, respectively.Under mean pooling, counterfactual training adds 1.67 PairAcc points; the interaction is not statistically significant.Source: §5.3 · p. 7, Table 3 · p. 8
5.4. How Should Evidence Existence Be Aggregated?
- Mean pooling performs best without additional parameters; max pooling often follows unchanged hard distractors, while aggregating more segments progressively improves PairAcc.A learned pointwise existence head also underperforms.Source: §5.4 · pp. 7–8, Table 4 · p. 8
5.5. Qualitative Intervention Analysis
- Baselines often fail to lower video-level or focal-segment scores after replacement, whereas CBGER preserves factual–counterfactual ordering and lowers the edited-slot score.These cases illustrate that plausible localization alone does not guarantee reliable evidence verification.Source: §5.5 · p. 8
6. Limitations and responsible interpretation
CBGER-10K is a controlled diagnostic benchmark, with limits on label validity, what its data represent, and how broadly its findings generalize.
Data and labels
- Automatically constructed labels may contain semantic and matching biases, and behavior-weaker counterfactual replacements are not guaranteed to be universally irrelevant.The replacement criterion is tied to the benchmark's construction criteria.Source: §6 · p. 8
Interpretation
- MicroLens histories and FineVideo candidates measure behavior-supported evidence, not actual user exposure, preference, or causal engagement.The benchmark should be interpreted as a diagnostic rather than a simulation of deployed recommendation systems.Source: §6 · p. 8
Scope and metrics
- Fixed nine-segment timelines and frozen CLIP representations limit generalization to unrestricted videos and other representations; PairAcc is not calibrated engagement probability.PairAcc measures relative evidence ordering within matched pairs.Source: §6 · p. 8
7. Conclusion
The conclusion presents counterfactual behavior-grounded evidence retrieval as a way to separate where personalized evidence occurs from whether it exists and how predictions respond to replacement. It reports that localization alone does not ensure reliable evidence verification, and that controlled interventions expose this gap.
Conclusion
- The framework separates Where personalized evidence occurs from Whether evidence exists and whether predictions respond consistently to replacement.CBGER jointly learns localization, evidence existence, and intervention consistency through structured counterfactual supervision.Source: §7 · p. 8
- CBGER-10K provides 5,000 controlled factual–counterfactual pairs for evaluating personalized evidence capabilities.Source: §7 · p. 8
- Strong temporal localization does not necessarily imply reliable personalized evidence existence; controlled interventions provide a practical way to expose this gap.Source: §7 · p. 8