Financial Context-Rich Strategy for Actor-Critic based Agents: A Deep Reinforcement Learning Approach for Stock Trading

1 INTRODUCTION

• Conventional strategies may be suboptimal because they do not use all dynamically evolving factors that influence stocks, while quantitative-only decisions omit qualitative features such as sentiment.The paper identifies profitable, risk-minimizing strategies as a central need.Source: §1 · p. 1
• The study incorporates sentiment into actor-critic DRL trading strategies and evaluates its effects alongside differences in agent policy design.The agents include A2C, PPO, TD3, SAC, and DDPG.Source: §1 · pp. 1–2
• The paper frames its contribution as a comprehensive evaluation of distinct DRL agents and an examination of sentiment infusion and policy-design variations in financial decision-making.Source: §1 · pp. 1–2

2 RELATED WORK

• Earlier stock-prediction studies used linear and machine-learning methods, neural networks, text mining, and natural-language processing.Source: §2 · p. 2
• Deep-learning work includes CNN, recurrent-network, and attention-based approaches, while prior studies also establish reinforcement learning and DRL as approaches to stock trading.Source: §2 · p. 2
• The paper extends this research by comparing five DRL agents with distinct policy designs under the same market conditions and testing sentiment infusion.Source: §2 · p. 2

3.1 Stock Market Data

• Five Indian stocks selected by market capitalization cover energy, telecommunications, and banking; the historical dataset spans January 2019 to June 2024 and includes technical indicators, VIX, and turbulence.The selected stocks are Reliance, Adani, Bharti Airtel, HDFC, and ICICI; historical data came from Yahoo Finance.Source: §3.1 · p. 2

3.2 Financial News

• Google RSS feeds supplied multiple headlines per stock and day to provide broader reporting and an average daily market-sentiment estimate.The number of headlines per day ranged from 1 to 10 on average.Source: §3.2 · p. 2

3.3 Sentiment Score

• FinBERT assigns positive, negative, or neutral labels to headlines, and the mean adjusted sentiment score for a stock and date forms its inclination score.The label values are 1, -1, and 0, respectively; the final news dataset has dimensions (4843, 4).Source: §3.3 · pp. 2–3

3.4 Fusion

• The inclination scores are joined to historical stock data by ticker and date, with missing scores set to zero to indicate neutral sentiment.Source: §3.4 · p. 3

4 MODELING

• Trading is modeled as a Markov Decision Process in which an agent acts by interacting with a dynamic environment.Source: §4 · p. 3
• The environment contains states, rewards, and metadata, and sequential agent decisions are represented as lists of states and associated rewards.Source: §4 · p. 3
• Feedback from the changing environment supports agents’ learning to trade through policy optimization.Source: §4 · p. 3

4.1 Stock Market Structure

• The study focuses on stock trading among instruments handled by exchanges.The US exchanges named are NYSE and NASDAQ; the Indian exchanges named are NSE and BSE.Source: §4.1 · p. 3
• The modeled scope is localized between order book and trading, and market data feeds.The real-world setup describes a trader using inbound market data to strategize actions within a stock trading setup.Source: §4.1 · p. 3
• The modeled setup has four components: environment, state, reward, and agent.The agent plays the trader; policy represents trader intuition or technical acumen, while reward is feedback for actions.Source: §4.1 · p. 3

4.2 State

• The use case is modeled as a Markov decision process, in which the state must contain information about current environmental conditions.The paper states that decisions and rewards are assumed to depend on the current state only.Source: §4.2 · p. 3
• The state includes cash balance, stock prices, shares held, technical indicators, and a sentiment-derived inclination score for each stock.The paper says sentiment is included in the state to let the agent make more sentiment-aware decisions.Source: §4.2 · p. 4
• The transition dynamics give the probability of moving to the next state and action from the current state and action.The passage describes this probability as entirely dependent on the current state and action.Source: §4.2 · p. 4

4.3 Environment

• The environment encompasses state, action, and reward and is what the RL agent interacts with.The agent acts based on state, and the environment provides rewards or penalties for its action.Source: §4.3 · p. 4
• The environment's transition dynamics give a probability distribution for the next state given the current state and action.The paper identifies equations 2, 3, and 7 as defining aspects of its environment.Source: §4.3 · p. 4
• Actions receive rewards or penalties from the environment based on the state from which the agent acts.The passage describes the agent as taking actions based on state.Source: §4.3 · p. 4

4.4 Reward

• The baseline reward is the change in asset value between the beginning and end of a time step.Asset value is calculated from cash balance and the number of stocks held; the resulting change is the final reward.Source: §4.4 · p. 4
• The sentiment-influenced reward incorporates the inclination score so sentiment has greater influence on agent actions.The method first calculates asset change, then computes an average individual reward.Source: §4.4 · p. 4
• Each stock's individual reward multiplies its price difference by its inclination score, and the individual rewards are averaged across stocks.The passage defines the price difference using the current price and the price at the beginning of the time step.Source: §4.4 · p. 4
• The paper states that results for the two reward-calculation cases are documented in the experiment section.Source: §4.4 · p. 5

4.5 Agent

• The five selected actor-critic agents split into policy-optimization methods (A2C and PPO) and methods that also use Q-learning (DDPG, SAC, and TD3).Trading actions are buy, sell, or hold, with performance-based rewards or penalties.Source: §4.5 · p. 5

4.5.1 Advantage Actor-Critic (A2C).

• A2C uses an actor to choose actions and a critic to provide feedback; its advantage function estimates action benefit while reducing variance and speeding learning.In the trading experiment, A2C aims to maximize profits by updating its policy from actor actions and critic feedback.Source: §4.5.1 · p. 5

4.5.2 Proximal Policy Optimization (PPO).

• PPO clips the probability ratio between current and old policies to limit rapid policy changes and improve stability.The paper selects PPO to avoid significant trade volume, especially during high market volatility.Source: §4.5.2 · p. 5

4.5.3 Deep Deterministic Policy Gradient (DDPG).

• DDPG is an off-policy, deterministic actor-critic method that uses a replay buffer to store and sample experience and supports continuous action spaces.Its deterministic policy directly maps states to actions.Source: §4.5.3 · pp. 5–6

4.5.4 Twin Delayed DDPG (TD3).

• TD3 addresses Q-learning overestimation bias with target-policy smoothing, delayed policy updates, and clipped double Q-learning.It uses two Q-networks and the minimum prediction for a conservative action value, which the paper associates with reducing trading risk and variance.Source: §4.5.4 · p. 6

4.5.5 Soft Actor-Critic (SAC).

• SAC combines off-policy actor-critic learning with maximum entropy, balancing policy exploration against expected returns.Its trading loop observes technical indicators and an inclination score, executes an action, then calculates a reward and next state.Source: §4.5.5 · p. 6

5 EXPERIMENTS & RESULTS

• The experiments used an initial portfolio of 1 million INR over approximately one year, from June 2023 to June 2024, across three sentiment-infusion phases.The phases were trading without sentiment, partial sentiment infusion, and deep sentiment infusion.Source: §5 · p. 6

5.1.1 Annual Returns.

• Annual returns measure the overall profitability of an RL trading strategy over the investment period.The annual portfolio value is compared with the initial portfolio value.Source: §5.1.1 · pp. 6–7

5.1.2 Sharpe Ratio.

• The Sharpe ratio estimates risk-adjusted returns, with higher values indicating more return per unit of risk.The selected evaluation metrics are intended to distinguish investment preferences such as aggressive return focus and risk aversion.Source: §5.1 · p. 6, §5.1.2 · p. 7

5.1.3 Stability.

• Stability measures consistency in generating profitable returns across market conditions.Source: §5.1.3 · p. 7

5.1.4 Annual volatility.

• Annual volatility measures price fluctuations, while maximum drawdown measures the decline from a local peak to a trough; lower values indicate less risk.For maximum drawdown, the portfolio values at the highest and lowest points are used.Source: §5.1.5 · p. 7, §5.1.4 · p. 7

5.2 Trading without Sentiment

• Without sentiment, PPO led annual returns and Sharpe ratio, while DDPG had the lowest volatility and was presented as a risk-averse option.A2C had the highest maximum drawdown among the agents, at -0.1293.Source: Table 2 · p. 7, §5.2 · p. 7

5.3 Trading with Partial Sentiment Infusion

• With partial infusion, TD3 was recommended for profit maximization and SAC for risk-averse trading; annual returns improved for A2C, DDPG, TD3, and SAC, while PPO declined.The authors say PPO’s policy design may make it conservative when real-time sentiment scores fluctuate.Source: Table 3 · p. 7, §5.3 · p. 8

5.4 Trading with Sentiment Deep Infusion

• Under deep sentiment infusion, DDPG had the highest annual return and Sharpe ratio; DDPG and SAC became more stable, while TD3 became less profitable.The authors attribute TD3’s decline to conservative updates and under-exploration amid increased reward variability.Source: Table 4 · p. 8, §5.4 · pp. 8–9

• A2C and PPO use policy optimization alone, whereas DDPG, TD3, and SAC combine policy optimization with Q-learning principles.Source: §6 · p. 9
• Deep sentiment infusion consistently improved DDPG and SAC, while TD3 declined because of its more conservative value-estimation approach.Source: §6 · p. 9
• The authors recommend Explainable Reinforcement Learning (XRL) techniques[1, 15] to examine how sentiment scores and technical indicators affect agent decisions.Source: §6 · p. 9