Source-linked AI summary
A Better Spur Should Start From Each Objective
Shanwen Mao, Hao Zhang, Guangtao nie, Zhiheng Li, Huimu Wang, Sulong Xu, Gu Simiu
TL;DR
MMPO addresses sparse and conflicting rewards, gradient interference, and late-stage metric oscillations in multi-objective reinforcement learning. It combines reward reshaping, priority-aware gradient decoupling, and adaptive constraints, improving e-commerce metrics and generalizing to ToolRL and code generation.
Problem
Multi-objective reinforcement learning faces sparse and conflicting rewards, while direct gradient aggregation can create alignment tax through directional interference.
Method
MMPO uses exposure-based reward debiasing, priority-aware sequential orthogonal projection, and self-prompted constraints to address objective conflicts at data, gradient, and constraint levels.
Results
MMPO consistently improves offline completeness and online business metrics in real-world e-commerce, while delivering competitive performance on ToolRL and code generation.
Takeaways & Limitations
MMPO provides a robust and scalable framework for complex alignment tasks involving multi-objective tradeoffs across e-commerce and broader benchmarks.
Takeaways & Limitations
Evaluation of process fidelity remains open, and underlying foundation models remain vulnerable to advanced adversarial threats.
Abstract
from arXiv · showhide
Real-world Multi-Objective Reinforcement Learning (MORL) often suffers from sparse rewards, reward conflicts, and late-stage reward tug-of-war, causing traditional linear scalarization to experience severe metric oscillations. To address optimization conflicts among multiple objectives in real-world deployment scenarios, we propose Multi-Marginal Preference Optimization (MMPO), a fine-grained framework that intervenes at the data, gradient, and constraint levels rather than relying on coarse-grained global scalarization. Specifically, MMPO performs exposure debiasing to mitigate sparse and biased rewards, applies priority-aware orthogonal projection to decouple conflicting gradients, and introduces self-prompted gradient constraints to prevent dominant objectives from overwhelming weaker ones. Experiments on real-world e-commerce datasets show that MMPO improves training stability and consistently achieves better performance across conflicting metrics. Moreover, it generalizes robustly to broader tasks such as ToolRL and code generation, demonstrating its effectiveness as a practical paradigm for multi-objective alignment.
1 Introduction
The paper frames e-commerce term extraction and explanation as a multi-objective reinforcement learning problem with sparse, conflicting feedback and unstable late-stage optimization. MMPO addresses these issues through objective-specific reward processing, gradient decoupling, and adaptive constraints, with experiments showing improvements in e-commerce and broader RL tasks.
- Problem: E-commerce term extraction and explanation jointly target offline extraction quality and online user-preference metrics.The task extracts high-value product terms and generates concise explanations while optimizing metrics such as completeness, accuracy, coverage, clicks, and likes.
- Problem: Sparse and skewed feedback, conflicting objectives, and late-stage reward tug-of-war make stable MORL optimization difficult.Simple scalarization can blur training signals, while dominant objectives may cause weaker metrics to oscillate or regress.
- Method: MMPO combines exposure-based reward reshaping, orthogonal gradient decoupling, and self-prompted restrictions for objective-specific optimization.These modules intervene at the data and reward, optimization, and dynamic constraint levels rather than relying on coarse global compromise.
- Contribution: MMPO is presented as an independent-objective-first MORL framework for sparse and conflicting rewards in industrial deployments.The framework is designed to resolve systematic conflicts and late-stage metric oscillations.
- Results: Extensive experiments show consistent improvements in offline completeness and online business metrics for real-world e-commerce, with competitive performance on ToolRL and code generation.The reported evaluation spans the industrial e-commerce setting and broader reinforcement-learning tasks.
2 Related Work
Prior MORL work mainly addresses objective conflicts through fixed-weight scalarization, gradient methods, or reward normalization. These approaches offer simplicity or partial conflict reduction but remain limited in adapting to stage-dependent conflicts and coordinating objectives beyond reward aggregation.
- Scalarization: Existing MORL methods commonly use static reward shaping or global scalarization with predefined objective weights.Fixed configurations are simple to implement but can be inflexible when conflict patterns change across training stages.
- Gradient methods: Gradient-based approaches address objective conflicts through gradient-level coordination, but the supplied related-work passage does not detail their full limitations.The passage introduces this category before continuing beyond the excerpt.
- LLM alignment: GRPO aggregates weighted advantages, allowing reward terms to compete and intensify reward conflicts and tug-of-war behavior.Its aggregation mechanism combines reward terms before optimization.
- LLM alignment: GDPO independently normalizes rewards to reduce scale inconsistency and partially alleviate tug-of-war, but remains focused on reward aggregation.The passage states that GDPO does not further consider coordination among data distributions.
3 Method
MMPO formulates product term extraction and explanation generation as MORL with offline quality and online preference objectives that can conflict. It addresses these conflicts through reward refinement, gradient decomposition and priority-aware projection, followed by self-prompted constraints for stable updates.
- Task formulation: The task jointly extracts key product segments and generates explanations while optimizing completeness, precision, coverage, clicks, and likes.Because these objectives may be misaligned and non-monotonic, the task is framed as multi-objective reinforcement learning.
- Challenges: Sparse feedback, skewed user signals, conflicting objectives, and late-stage tug-of-war make naive scalarization unstable.Dominant objectives can overshadow weaker ones, causing metrics to oscillate or regress.
- Data and rewards: MMPO reshapes rewards through exposure stratification, bias correction, and Fourier-based smoothing to improve reward learnability.Exposure correction normalizes feedback across categories and items, while Fourier features produce a smoother mapping from segment features to rewards.
- Objective decomposition: The method decomposes objective gradients and removes preference-gradient components that conflict with higher-priority base gradients.Projection restricts updates to a feasible safe subspace, where conflicting components can be removed without losing valid preference-alignment signals.
- Priority-aware projection: Priority-aware sequential projection establishes basis directions from high-priority base capabilities before orthogonalizing remaining preference objectives.The resulting orthonormal basis reparameterizes aggregated gradients into independent coordinates and provides a transparent space for intervention.
- Self-prompted constraints: Self-prompted gradient constraints define reference coordinates from ideal instruction-following signals and use their envelope to regulate updates.This mechanism supplies an adaptive boundary for each coordinate rather than relying only on heuristic weight adjustments.
4 Experiments
Experiments evaluate MMPO on real-world e-commerce, online business outcomes, ablations, ToolRL, and code generation. MMPO shows stronger multi-objective performance, greater training stability, and competitive generalization across these settings.
- Offline Evaluation: MMPOQwen3-8B ranks first across all three core metrics among 8B-scale baselines, achieving 94.11% Completeness, 73.43% Accuracy, and 22.82% Coverage.Its Coverage is closest to the ideal value of approximately 22%.
- Offline Evaluation: MMPO remains competitive with frontier LLMs on a unified benchmark of 1,500 products across 50 categories, surpassing selected models on Accuracy and Coverage.Its 22.82% Coverage more than doubles the best frontier-model value of 10.69%.
- Offline Evaluation: MMPO converges faster and more stably than GRPO and GDPO, which show frequent fluctuations and late-stage metric collapse.The reported setup uses a 1 × 10^-6 learning rate, batch size 256, and a 0.001 KL penalty.
- A/B Testing: In a two-week online A/B test with 10% traffic, MMPO increased Entry-click GMV by 5.07% and Entry-click Orders by 3.14% over the DeepSeek-V3 control.Additional gains were reported for Unique Visitors, UV Value, Total Interaction Turns, and Active Users, with all improvements statistically significant at the 95% confidence level.
- Ablation Results: Ablations show that Smoothed Reward is essential under sparse rewards, while Subspace Decoupling and Self-Prompt Gradient Limitation jointly produce the strongest full-framework results.Removing Smoothed Reward reduces completeness and accuracy below 5%; the full framework reaches 93.37% completeness and 74.62% accuracy.
- ToolRL and Code Generation: MMPO generalizes beyond e-commerce, improving ToolRL overall accuracy and balancing code-generation accuracy with diversity at higher pass@k values.It raises DS-7B Overall Acc from 33.33% to 43.55% and reaches 57.17% pass@20 on Qwen2.5-Coder-3B.
5 Conclusion
MMPO addresses sparse rewards, gradient conflicts, and metric oscillations in MORL through exposure debiasing, subspace orthogonal decoupling, and self-prompted constraints. Results across e-commerce, ToolRL, and code generation support its robustness for multi-objective alignment.
- MMPO addresses sparse rewards, gradient conflicts, and metric oscillations in Multi-Objective Reinforcement Learning.
- MMPO combines exposure debiasing, subspace orthogonal decoupling, and self-prompted constraints for stable, multi-faceted LLM alignment.
- MMPO significantly outperforms GRPO and GDPO across real-world e-commerce applications, ToolRL, and code generation.
Limitations
MMPO inherits limitations from underlying LLMs and current evaluation ecosystems. Process-fidelity standardization and resilience against advanced adversarial threats remain open challenges.
- Standardizing evaluation of process fidelity remains an open challenge.
- Foundation models remain vulnerable to multimodal adversarial smuggling and multi-turn strategic red-teaming.
- Future work will integrate robust defense mechanisms and process monitoring to improve reliability in adversarial environments.