Source-linked AI summary
Eliciting ESG Preferences for Reinforcement Learning-Based Portfolio Optimization
Giovanni Dispoto, Marcello Restelli, Carmine Ventre
TL;DR
Existing ESG-aware RL approaches often rely on one provider and require unintuitive manual objective weighting. This paper combines three-agency MORL with Gaussian-Process preference elicitation and evaluates it using LLM personas, finding that regional contexts shift inferred preferences between ESG alignment and risk-adjusted performance.
Problem
Existing ESG-aware RL approaches rely on a single provider, while manually weighting conflicting financial and ESG objectives is unintuitive for practitioners.
Method
The framework treats three ESG ratings and financial returns as MORL objectives, then infers manager-specific weights from pairwise portfolio comparisons using Gaussian Processes and LLM personas.
Results
Regional persona contexts shifted inferred preferences: European personas emphasized ESG alignment, whereas Texas personas placed greater weight on risk-adjusted performance.
Takeaways & Limitations
The framework supports preference-conditioned portfolio optimization across divergent ESG objectives and heterogeneous simulated manager preferences.
Takeaways & Limitations
The empirical evaluation uses simulated ESG ratings because proprietary historical ratings are unavailable, omitting idiosyncratic noise and sudden methodological shifts in real data.
Abstract
from arXiv · showhide
Modern portfolio management increasingly demands a balance between traditional risk-adjusted returns and strict Environmental, Social, and Governance (ESG) mandates. Current Reinforcement Learning (RL) approaches typically optimize for a single ESG provider, neglecting the significant divergence in rating methodologies across the industry and the unintuitive nature of manually weighting conflicting objectives. This paper addresses these limitations by formulating ESG-aware portfolio optimization as a Multi-Objective Reinforcement Learning (MORL) problem that simultaneously incorporates ratings from three distinct ESG agencies. To bridge the gap between high-dimensional algorithmic trade-offs and human decision-making, we integrate a Preference Elicitation framework using Gaussian Processes. This system enables practitioners to infer their latent utility functions through intuitive pairwise comparisons of candidate portfolios based on their Sharpe ratios and aggregate ESG scores. We systematically evaluate our framework by employing Large Language Model (LLM) personas to simulate Portfolio Managers operating under varied regional contexts. Empirical results using historical market data reveal that regional backgrounds fundamentally shift the derived preference weights. For instance, European-based personas tend to prioritize ESG alignment over financial returns, while Texas-based personas favor risk-adjusted performance. This work offers a highly adaptable framework that successfully aligns multi-objective algorithmic trading with diverse, real-world human sustainability preferences.
1 Introduction
The paper frames ESG-aware portfolio optimization as a multi-objective RL problem because investors must balance financial performance with divergent sustainability ratings. It adds preference elicitation so managers’ portfolio choices can determine objective weights.
- Institutional investors must balance traditional risk-adjusted performance with strict ESG mandates, creating a multi-dimensional allocation trade-off.
- Single-provider ESG optimization is limited because different agencies use methodologies and weightings whose scores correlate poorly.
- The framework uses three ESG agencies as separate objectives alongside financial returns in a Multi-Objective Reinforcement Learning formulation.
- Gaussian-Process preference elicitation infers a Portfolio Manager’s latent weights from pairwise comparisons of candidate portfolios using Sharpe ratios and aggregate agency scores.
- LLM personas provide heterogeneous synthetic preference oracles for stress-testing elicitation and examining how regional conditioning and prompt design affect selected portfolios.
2 Preliminaries
The preliminaries define RL through Markov decision processes and extend the formulation to multiple conflicting objectives. Preference elicitation then models latent utilities from pairwise choices using Gaussian Processes.
- Reinforcement Learning: An MDP represents sequential decisions with states, actions, transitions, rewards, discounting, and an initial-state distribution, while a policy maps states to action distributions.
- Multi-Objective Reinforcement Learning: An MOMDP outputs a reward vector for multiple objectives, which can be combined through linear scalarization using a preference vector Λ.
- Multi-Objective Reinforcement Learning: The preference-conditioned policy π(a|s, Λ) allows one neural network to learn across the preference space, trained here with PPO.
- Preference Elicitation: Preference elicitation infers a hidden utility function from observed pairwise comparisons instead of requiring users to specify complex objective weights numerically.
- Preference Elicitation: A Pairwise Gaussian Process places a non-parametric Bayesian prior over utility and quantifies predictive uncertainty while modeling potentially non-linear relationships.
- Preference Elicitation: The probit likelihood connects continuous utility differences to discrete choices, with σ representing observation noise from inconsistent decision-making.
3 Related Work
Prior work applies RL and evolutionary optimization to portfolio selection, including ESG-aware objectives, but commonly relies on a single ESG data provider. The paper also builds on LLM-persona methods for behaviorally conditioned preference simulation.
- AI in Finance: RL has been applied to portfolio optimization across multiple algorithms and objective functions.
- Sustainable Finance: ESG-aware algorithmic portfolio studies commonly incorporate scores from isolated providers such as Bloomberg or Morningstar Sustainalytics.
- Sustainable Finance: Garcia-Bernabeu et al. approximate a static Mean-Variance-ESG Pareto frontier with an evolutionary algorithm using Sustainalytics data.
- LLM Personas: Role-play prompting has outperformed standard zero-shot prompting on several benchmarks, supporting persona-conditioned behavioral simulation.
4 Methodology
The methodology trains a preference-conditioned multi-objective portfolio policy and then elicits a manager-specific conditioning vector through pairwise portfolio comparisons. The framework combines Sharpe-ratio optimization with multiple ESG objectives and evaluates behavior across preference settings.
- 4.1 Problem Formulation: The portfolio problem uses a daily, 252-day episodic Multi-Objective MDP with preference vector Λ weighting conflicting financial and ESG objectives.States include market features, asset returns, volatilities, agency ratings, rating momentum, and portfolio information.
- 4.1 Problem Formulation: The reward combines the Sharpe ratio with ESG agency objectives using linear scalarization, where Λ weights the objective vector.The Sharpe component is implemented through a differential Sharpe Ratio reward based on online exponential moving averages.
- 4.2 Policy Optimization: PPO trains a neural policy with a FiLM-inspired feature extractor that conditions market representations on Λ at two network depths.The preference-processing architecture supplies the conditioning information used by downstream conditional layers.
- 4.2 Policy Optimization: Preference vectors are sampled across the trade-off space, including corner cases, while each episode uses a fixed Λ to learn π(a|s, Λ).This dual-sampling strategy exposes the policy to a broad spectrum of objective trade-offs.
- 4.3 Preference Elicitation: Pairwise comparisons of candidate portfolios based on Sharpe ratios and aggregate ESG ratings infer the conditioning vector preferred by each Portfolio Manager.Gaussian Process preference learning replaces direct manual specification of numerical weights.
5 Experiments
The experiments evaluate ESG-aware portfolio optimization, preference elicitation, and LLM-persona conditioning using chronologically separated market-data periods. Results show that regional persona preferences produce different ESG–Sharpe trade-offs, while prompt sensitivity varies across regions.
- Experimental setup: The evaluation separates policy training, preference elicitation, and later testing into disjoint chronological periods to avoid temporal leakage.Policy training uses 2009 to mid-2014, while preference elicitation uses mid-2014 to 2015; the reported out-of-sample metrics use 2015 to mid-2016.
- Experimental setup: The MORL policy is trained on ten curated DJIA constituents using 5.5 years of historical daily closing prices from 2009 to mid-2014.The asset universe includes CAT, MMM, MSFT, CRM, AAPL, AMZN, NVDA, CVX, BA, and JNJ.
- Preference Elicitation: Preference elicitation initializes with 8 random portfolio pairs, fits a PairwiseGP, and uses qEUBO for 20 active-learning steps before extracting the posterior-mean-maximizing preference vector.The elicited vector conditions the already trained generalized policy π(a|s, Λ) for a specific Portfolio Manager.
- Preference Elicitation using LLM-Personas: The LLM-persona experiments simulate Portfolio Managers from Europe, Texas, America, and Asia choosing between candidate portfolios using financial and three-agency ESG metrics.A system prompt assigns regional and professional identity, while a task prompt presents the portfolio comparison and permits an uncertainty response.
- Preference Elicitation using LLM-Personas: The European persona prioritizes environmental standards and social-controversy mitigation, whereas the Texas persona prioritizes risk-adjusted returns over higher niche ESG scores.These responses illustrate how regional framing changes the stated portfolio-selection criterion.
- Preference Elicitation using LLM-Personas: Prompt ablation leaves the main finding unchanged, but Europe- and Asia-conditioned personas are more sensitive to prompt wording than Texas- and America-conditioned personas.Texas and America consistently assign high importance to the Sharpe ratio, while Europe emphasizes λ2 and λ3 and gives negligible weight to λ0.
6 Limitations
The empirical evaluation uses simulated ESG ratings, and the MORL architecture relies on linear scalarization with frictionless-trading assumptions. These choices constrain how directly the framework represents real ESG data and portfolio-market trade-offs.
- Simulated ESG ratings omit the idiosyncratic noise and sudden methodological shifts characteristic of real ESG data.The simulation is calibrated to mirror documented divergence among real-world agencies but does not reproduce all of their dynamics.
- Linear scalarization can struggle to identify optimal policies in non-convex regions of the Pareto frontier.The architecture combines financial and sustainability objectives through a preference vector Λ into a single reward signal.
- The asset-allocation trade-off between Sharpe ratio and targeted sustainability scores can be non-linear for some Portfolio Managers.
- Frictionless trading simplifies the market dynamics represented by the empirical setup.
7 Conclusions and Future Work
The paper proposes an adaptable MORL framework that treats divergent ESG agency scores as distinct objectives alongside risk-adjusted returns. Preliminary results suggest no positional bias in the medium-sized models tested, while prompt ordering remains an open concern.
- The framework treats each ESG agency’s score as a distinct optimization objective alongside traditional risk-adjusted returns.
- The paper proposes a highly adaptable Multi-Objective Reinforcement Learning framework for ESG-aware portfolio optimization.
- Preliminary results indicate that the medium-sized models considered do not suffer from positional bias.
- The medium-sized models’ result contrasts with positional bias reported for smaller models in previous work.
- The effect of portfolio ordering in prompts on elicited preferences remains important to investigate.