Reinforcement learning

Sarod Yatawatta

Source-linked AI summary

Reinforcement learning

Sarod Yatawatta

arXiv:2405.10369v1May 16, 2024astro-ph.IMcs.AIcs.LG

TL;DR

Astronomy relies on expert-driven operational tasks, while applications of reinforcement learning remain limited. This paper surveys deep reinforcement learning for astronomy and shows that hint-assisted agents achieve higher rewards in a challenging walker environment.

Problem
Astronomy has relatively few reinforcement-learning applications spanning telescope operations and data-processing pipelines, motivating a focused overview of its potential uses.
Method
The paper surveys deep reinforcement learning theory, algorithms, practical tools, astronomical applications, and hint-based transfer of existing knowledge to agents.
Results
Hint-assisted SAC and TD3 achieved higher rewards than their unassisted versions in the hard bipedal-walker environment.
Takeaways & Limitations
Reinforcement learning can support efficient, autonomous astronomical agents for monitoring, control, and data processing with minimal human involvement.
Takeaways & Limitations
Hint-assisted reinforcement learning must account for inaccurate hints, with δ controlling the allowed hint–action discrepancy and trust in hint accuracy.

Abstract

from arXiv · show

Observing celestial objects and advancing our scientific knowledge about them involves tedious planning, scheduling, data collection and data post-processing. Many of these operational aspects of astronomy are guided and executed by expert astronomers. Reinforcement learning is a mechanism where we (as humans and astronomers) can teach agents of artificial intelligence to perform some of these tedious tasks. In this paper, we will present a state of the art overview of reinforcement learning and how it can benefit astronomy.

1. Introduction

This section introduces reinforcement learning as a feedback-driven, sequential-action paradigm and motivates its use for astronomy. The paper surveys modern deep reinforcement learning, emphasizing astronomical applications and practical accessibility for new users.

Reinforcement learning, supported by deep neural networks, has achieved breakthroughs in games, chess, Go, robotics, matrix multiplication, and sorting.
Astronomical applications include telescope control for adaptive optics and adaptive reflective surfaces, observation scheduling, and radio astronomical data-processing pipelines.
Unlike supervised learning, reinforcement learning learns through repeated attempts and external feedback while treating tasks as sequences of actions.
The paper provides an overview of modern deep reinforcement learning focused on astronomy, aiming to help new users quickly apply these techniques.

2. Reinforcement learning theory

Reinforcement learning models an agent that repeatedly observes an environment, takes actions, and receives rewards while pursuing a specified objective. Its theoretical formulation uses states, actions, rewards, transitions, and discounted future returns, illustrated through a maze-based Q-learning example.

Agent–environment interaction: An RL agent observes its environment, selects actions toward a specified goal, and receives rewards reflecting the effects of those actions.The interaction repeats as the environment changes and returns updated state or observation information to the agent.
State and action representation: RL represents problems with states and actions that may contain discrete or continuous values, while distinguishing the environment state from the agent’s observation.The state can condense observations by removing directly dependent observables.
Markov decision process: The tuple (S, A, R, P) defines a Markov decision process in which the next-state probability depends only on the current state and action.The transition probability is parameterized as p(st+1|st, at), and rewards are represented as r(st, at).
Future rewards: In an infinite-horizon MDP, a discount factor γ with 0 < γ < 1 produces a finite discounted cumulative reward over future steps.Discounting accounts for uncertainty in the future while ensuring that the reward summation converges.
Q-learning example: The maze example demonstrates crude Q-learning by iteratively updating a state-action Q-table from immediate rewards and the maximum value of the next state.The method is feasible only for low-dimensional discrete state and action spaces; the example uses γ = 0.9.

3. Deep reinforcement learning algorithms

Model-free deep reinforcement learning faces data scarcity, exploration–exploitation trade-offs, and training instability. Replay buffers, Q-learning variants, target networks, and continuous-action methods address these challenges across discrete and continuous action spaces.

Training challenges: Deep RL training is constrained by insufficient interaction data, difficult exploration across high-dimensional spaces, and instability caused by iterative Q-function updates.Real-world environments can be complex and expensive to operate, while poor sampling can cause local-minimum convergence or overfitting.
Experience replay: Replay buffers store transitions (s, a, r, s′) for reuse, enabling off-policy learning and optional prioritization of experiences with large Q-value changes.Mini-batches sampled from the buffer are used for temporal-difference learning, and larger batches generally provide more stable performance subject to available memory.
Discrete action spaces: For discrete actions, Q-learning evaluates all possible next actions and selects the maximum Q-value, but iterative updates can be unstable and overestimate values.Double Q-learning uses two Q-functions, keeping one fixed while updating the other to address this issue.
Deep Q-networks: Deep neural networks represent practical Q-functions, while a lower-cadence target-network update stabilizes training by periodically copying the trained network parameters.The loss is averaged over mini-batches of transitions during optimization.
Continuous action spaces: Continuous action spaces make direct Q-learning infeasible because actions have infinitely many values, so policy functions play a major role in selecting actions.The section identifies value iteration and policy iteration as two dynamic-programming families for continuous-action problems.
Continuous action spaces: TD3 addresses DDPG’s Q-value overestimation by using two Q-functions, each with a corresponding target network.The two functions are parameterized by θ1 and θ2, with corresponding target parameters beginning θ′.

4. Model based reinforcement learning

Model-based reinforcement learning uses a learned environment model to generate training data and support planning, but depends on accurately representing environmental dynamics. PETS exemplifies this approach, while hint-assisted RL incorporates existing models or expert knowledge directly into policy learning.

Model based reinforcement learning: Model-based RL uses an internal dynamics model to generate additional data when real-world data collection is expensive or potentially damaging.Its effectiveness depends on the proxy model accurately representing the environment’s dynamics.
Model based reinforcement learning: Environment models must address aleatoric uncertainty from inherent measurement randomness and epistemic uncertainty from incomplete system information.Examples include thermal noise, quantization, and misrepresentation of the system state.
Model based reinforcement learning: A trained model can generate training data, predict future rewards for model predictive control, or support direct gradient-based policy optimization.These uses correspond to model-based data generation, action-sequence planning, and differentiable policy optimization.
PETS: PETS samples candidate action trajectories through an ensemble dynamics model and uses the cross entropy method to return the action associated with the highest averaged reward.It propagates trajectories with randomly selected ensemble models, retains M elite actions, and updates the candidate-action distribution over N iterations.
Hint assisted RL: Hint-assisted RL incorporates existing models or experienced astronomers’ suggestions by providing hints that replace actions and constrain policy learning.The hint-action distance can be thresholded by δ, allowing the method to account for potentially inaccurate hints.

5. Applications in astronomy

The section outlines practical design considerations for applying reinforcement learning in astronomy and identifies applications spanning observatory operations, resource management, tuning, and archival-data reuse. A bipedal-walker example illustrates algorithm performance and the value of hints in difficult environments.

Practical considerations: State representations require task-specific insight and experimentation, especially when transitions depend on history rather than only the current state.Historical data can be incorporated into the current state, and actions can be represented incrementally when needed.
Practical considerations: Combining heterogeneous inputs requires appropriate scaling or normalization to maintain numerical stability in neural-network models.The state and action may combine information from multiple sources with different dynamic ranges.
Astronomy applications: Potential astronomy applications include planning and control, fair resource allocation under cost and energy constraints, hyper-parameter tuning, and discovering new science from archival data.RL agents can outperform grid-search approaches for regression, classification, and clustering tuning tasks, and can index archives for potential scientific reuse.
Bipedal-walker example: The bipedal walker state has 24 real numbers, while its action space is continuous and 4-dimensional, representing torques applied to four leg joints.The state contains leg-joint positions and body velocities.
Bipedal-walker example: A cumulative reward of 300 or more defines successful walking, with SAC reaching this target while TD3 achieves a slightly lower reward.The comparison uses fixed random seeds, and both easy and hard environments are evaluated.
Bipedal-walker example: Hint-assisted training improves performance in the hard environment for both SAC and TD3, although the TD3 improvement remains much smaller than SAC’s.Hints come from agents trained in the easy environment and are therefore inherently inaccurate; the TD3 source agent does not reach the target reward.

6. Conclusions

The paper overviews deep reinforcement learning algorithms applicable to astronomy and introduces hints as a mechanism for transferring knowledge from existing astronomical methods to RL agents. It argues that data-intensive astronomy requires efficient, autonomous agents and provides public source code for the discussed algorithms.

The paper reviews deep reinforcement learning algorithms directly applicable to various astronomical tasks.The overview is framed around methods that can support astronomy-related operations.
Hints provide a simple mechanism for transferring knowledge from existing astronomical methods to reinforcement learning agents.This mechanism accounts for alternative methods and techniques already used in astronomy.
Data-intensive astronomy requires efficient and autonomous agents to monitor, control, and process data.
Source code implementing all algorithms discussed in the paper is publicly accessible.

Appendix A. Python code for Q-table iteration

Appendix A provides Python code implementing Q-table iteration for a five-state, four-action environment. The code uses γ=0.9, updates Q-values through episodic transitions, handles invalid and terminal states, and runs 100 epochs.

Appendix A. Python code for Q-table iteration: The implementation initializes a 5×4 floating-point Q-table with zeros and sets the discount factor γ=0.9.The Q-table is created with NumPy using float32 values.
Appendix A. Python code for Q-table iteration: Each episode randomly selects an initial state, chooses among actions maximizing the current reward, and transitions according to Sprime.The episode loop continues while t<100.
Appendix A. Python code for Q-table iteration: Invalid next states trigger an error and stop the episode, while terminal state 100 assigns the immediate reward without a future-value term.The reward structure marks invalid transitions as −1 and the terminal transition as 100.
Appendix A. Python code for Q-table iteration: For nonterminal transitions, the code updates Qt[s,a] using the immediate reward plus γ times the maximum Q-value of the next state.The update is Qt[s,a]=R[s,a]+gamma∗(np.max(Qt[sprime])).
Appendix A. Python code for Q-table iteration: The code prints the Q-table, executes episodes for 100 epochs, and prints the resulting Q-table again.Each epoch calls episode(R, Sprime, gamma, Qtable).

Appendix B. Hyperparameters in TD3 and SAC

TD3 and SAC use matching three-layer critic and actor DNN architectures, with SAC splitting the actor output into separate µϕ and log σϕ heads. ReLU activations are used except in final layers, and training uses Adam optimization.

Critic architecture: Both TD3 and SAC use a three-layer critic DNN with dimensions (24 + 4) × 256, 256 × 256, and 256 × 1.The critic architecture is shared across both algorithms.
Actor architecture: Both algorithms use a three-layer actor DNN with dimensions 24 × 256, 256 × 256, and 256 × 4.In SAC, the output layer is divided into two heads for µϕ and log σϕ.
Training configuration: ReLU activation is used in all layers except the last, and Adam gradient descent is used for training.The optimizer is cited as Kingma and Ba (2014).