Source-linked AI summary
Generate to Explore, Select to Exploit: Aligning LLM-based Headline Generation with Personalized Recommendation
Yi Chen, Rufeng Cheng, Qiang Xie, Tao Li
TL;DR
Static, point-estimated headline generation can miss multimodal user intent when profiles are noisy or sparse. GESE decouples generation for distributional coverage from selection for individual utility; on a commercial platform, it achieved a 2.57% CTR lift and a 0.87% dwell-time lift.
Problem
Method
GESE separates generation of a candidate set for distributional coverage from selective exploitation, with a hierarchical reward combining semantic utility and diversity.
Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4Results
GESE achieved a 2.57% lift in CTR and 0.87% in dwell time over strong baselines, including SFT and DPO.
Source: §1 · p. 2Takeaways & Limitations
The framework treats generative diversity as a hedge against uncertainty in industrial user profiles by generating for the group and selecting for the individual.
Source: §1 · p. 2
Abstract
from arXiv · showhide
In industrial recommendation feeds, presenting a static headline for an item often fails to satisfy the diverse, multimodal interests of the user population, particularly suppressing the needs of long-tail audiences. While Large Language Models (LLMs) have been integrated into recommendation for content understanding or ranking, directly optimizing them to output a single best headline typically leads to mode collapse---converging to generic patterns that satisfy average tastes but miss specific latent intents. To bridge this gap, we introduce GESE (Generate to Explore, Select to Exploit), a framework operating at the system's presentation layer that decouples personalization into generative exploration and selective exploitation. First, we treat the LLM as a probabilistic explorer, utilizing Group Sequence Policy Optimization (GSPO) with a hierarchical reward mechanism to generate a candidate set that maximizes the semantic coverage of potential user interests. Subsequently, a lightweight, real-time feedback-aware selector acts as the exploiter, identifying the optimal realization from the candidate pool based on instant contextual signals. Extensive deployment on a commercial platform with over 100 million daily active users demonstrates that GESE significantly outperforms state-of-the-art baselines, achieving a 2.57% lift in CTR and 0.87% in dwell time. These results validate that decoupling diversity-oriented generation from precision-oriented selection offers a robust blueprint for aligning generative AI with dynamic user utility.
1 Introduction
The introduction argues that static, point-estimated headlines miss diverse user intents, especially under profile uncertainty. GESE separates candidate generation from selection, and commercial deployment outperformed strong baselines on CTR and dwell time.
1 Introduction
- A static headline often misses heterogeneous and multimodal user interests, particularly those of long-tail audiences.The introduction describes point-estimation methods as regressing toward generic patterns and missing specific intent peaks.Source: §1 · pp. 1–2
- The authors argue that SFT and scalar-reward alignment can encourage homogeneous outputs, leaving end-to-end models without a way to hedge against profile noise.They frame generative diversity as a hedge against uncertainty in user profiles.Source: §1 · p. 2
- GESE separates generative exploration from selective exploitation, positioning the LLM to generate diverse hypotheses and a real-time selector to choose among them.The selector uses immediate contextual signals, including session feedback and real-time user meta-tags.Source: §1 · p. 2
- Its GSPO alignment strategy combines generation diversity, predicted CTR, and content faithfulness in a hierarchical reward mechanism.The approach generates the candidate set in one inference pass to reduce inference costs.Source: §1 · p. 2
- GESE outperformed strong baselines, including SFT and DPO, with a 2.57% lift in CTR and 0.87% in dwell time.The deployment was on a commercial recommendation platform with over 100 million DAUs.Source: §1 · p. 2
2 Related Work
Prior recommendation alignment methods are described as favoring average or point-wise outputs, while diversity methods often act only after generation. GESE instead makes diversity intrinsic to generation and assigns exploitation to a separate context-aware selector.
2.1 LLM Alignment in Recommendation
- SFT can mimic average training-data behavior, while DPO-style alignment optimizes a single preferred response and can overcommit when profiles are ambiguous.The related work frames this as a shift from point-wise alignment toward coverage of potential user interests using GSPO.Source: §2.1 · p. 2
2.2 Diverse Generation
- Post-hoc re-ranking cannot recover semantic nuances lost when generation collapses, while lexical diversification may trade off against faithfulness and relevance.GESE structurally separates generator exploration from real-time, context-aware selection.Source: §2.2 · pp. 2–3
3 Proposed Method
GESE reframes personalized headline generation as set-wise exploration followed by context-aware selection, rather than asking an LLM to predict one headline for noisy, multimodal user intent. Its training and online stages jointly produce diverse candidates and select among them using user context and feedback.
3 Proposed Method
- GESE replaces point-wise headline prediction with a candidate set designed to cover potential intents, then uses a real-time-aware ranker to select a realization.The generator focuses on distributional coverage; the ranker selects using real-time contextual signals.Source: §3 · p. 3
3.1 Generative Exploration
- SFT trains Qwen3-14B to emit distinct headline lists in one pass, while a CTR reward model combines contextual semantics with display-bias features.Single-pass generation is intended to reduce inference costs; display features include headline length and punctuation density.Source: §3.1 · p. 3
- GSPO samples output groups and uses group-relative rewards for CTR, diversity, and faithfulness, encouraging distinct candidates instead of repeated safe headlines.Group-wise normalization creates local competition among candidates and pushes exploration across semantic space.Source: §3.1 · p. 4
3.2 Hierarchical Reward Shaping
- The hierarchical reward combines headline-level CTR, faithfulness, and novelty utility with set-wise redundancy penalties, balancing average utility and diversity.Faithfulness is enforced through a discrimination model; set-wise redundancy uses character-set overlap.Source: §3.2 · pp. 4–5
3.3 Online Exploitation
- For cold-start candidates, confidence-aware Thompson Sampling explores against the original title, then terminates exploration when confidence criteria classify candidates as winners or losers.The mechanism uses impression counts, empirical CTR, sample thresholds, and high/low confidence thresholds to govern rotation and stopping.Source: §3.3 · p. 5
- After candidates gain interaction data, a context-aware ranker predicts click probability from user, dynamic-interest, and candidate-semantic features and selects the highest-probability headline.The model learns high-order interactions among user and headline features.Source: §3.3 · pp. 5–6
4 Experiments and Results
Across online and offline evaluations, GESE combines diverse headline generation with feedback-aware selection and outperforms the reported baselines. The experiments also show how diversity and faithfulness constraints affect engagement and candidate quality.
4.1 Experimental Setup
- MSR-50k contains 50,000 resources across graphics, short videos, and micro-videos in over 40 vertical categories; evaluations also used a platform serving about 100 million DAU.The evaluation compared GESE with large proprietary and open-weight models and an SFT-Only baseline, and measured diversity, redundancy, fidelity, and entailment.Source: §4.1 · p. 6
4.2 Online A/B Test
- SFT-Only was 1.5% below the production baseline in CTR; SFT & UCB reached +1.37%, and GSPO & UCB reached +2.28%.The reported comparison attributes the additional increase to replacing the SFT generator with the GSPO explorer.Source: §4.2 · p. 6
- 2.57% higher CTR and 0.87% higher User Dwell Time: full GESE achieved the highest performance across all online metrics.Table 2 reports online A/B testing relative to Raw Headline.Source: Table 2 · p. 6, §4.2 · p. 6
4.3 Offline Evaluation
- GESE scored lower on ROUGE but higher on NLI (80.17) than SFT-Only, whose R-L was 44.09 and NLI score was 75.66.The authors describe this as semantically novel headlines that remain factually consistent with the content.Source: §4.3 · p. 6
- GESE achieved the highest predicted CTR Score (11.38), exceeding the SFT baseline (9.16) and the 10x larger teacher models.The authors attribute this result to hierarchical reward shaping that aligns generation with engagement signals.Source: §4.3 · pp. 6–7
- GESE reduced Self-BLEU from 18.02 to 9.93 and achieved the highest Distinct-N-Gram score (56.20) among the compared models.Table 3 presents offline performance on MSR-50k across Diversity, Quality, and Appeal.Source: §4.3 · p. 6, Table 3 · p. 7
4.4 Ablation Study
- Without the Quality constraint, CTR Score rose to 12.79 while NLI fell to 47.32 and ROUGE-L to 11.65.The authors characterize this as reward hacking: exploiting CTR-model biases at the cost of content consistency.Source: §4.4 · p. 7
- Without the diversity bonus, Self-BLEU rose from 9.93 to 19.73; removing RL reduced appeal, while removing SFT degraded linguistic quality.The reported ablations give SFT-Only CTR as 9.16 and Cold RL Distinct-N-Gram as 37.83.Source: §4.4 · p. 7
5 Conclusion
GESE reframes personalized generation as exploring latent interest modes before selective exploitation, and its commercial deployment confirms gains in engagement and user satisfaction.
5 Conclusion
- Deployment on a commercial platform with 100 million DAUs confirms GESE boosts engagement (CTR) and user satisfaction (Dwell Time).The conclusion describes this as a hedge against aleatoric uncertainty in industrial user profiles.Source: §5 · p. 7
- GESE replaces single-output point estimation with generative exploration via GSPO followed by selective exploitation.It optimizes coverage of latent interest modes rather than token likelihood.Source: §5 · p. 7
- The paper argues that forcing one optimal output under profile uncertainty leads to mode collapse.Source: §5 · p. 7