Learning High-Risk High-Precision Motion Control

• Unlike sequential tasks where later actions can correct errors, billiards shots have irreversible outcomes that demand greater precision.Source: §1 · p. 2
• Billiards represents high-risk, high-precision control: small angle differences can change trajectories, and its low-dimensional variants allow reward landscapes and algorithm behavior to be visualized.Source: §1 · p. 2
• PPO, SAC, and AWR fail even in the simplest billiards variants, motivating analysis of their visualized behavior and opportunities for improvement.Source: §1 · p. 2
• SCOOT produces highly precise shots and alternative strategies, and computes successful billiard shots in a single forward pass rather than relying on iterative optimization.The comparison is with existing solutions based on CMA-ES.Source: §1 · p. 2

• Sampling-based optimization can solve high-risk, high-precision control problems, but its computational cost and solution delay have motivated neural policies that compute control parameters in a single forward pass.Source: §2 · p. 2
• PPO, SAC, and AWR are common DRL methods; MoE methods have been used to model diverse motions, while this work combines AWR-style weighted regression with MoE.Source: §2 · p. 2
• The paper contributes visualizations of reward landscapes intended to explain why and when MoE policies can be useful.Source: §2 · p. 3
• Prior computational-pool approaches include search and trajectory optimization, but real-time viable shots remain an open challenge; one DRL approach treats billiards as sequential decisions.Source: §2 · p. 3

• In the billiards MDP, the state is the initial ball configuration and the action is the cue ball’s launch velocity.Source: §3 · p. 3
• The objective is to maximize expected return over randomly sampled billiard-ball positions, focusing on single optimal shots rather than shot sequences.Source: §3 · p. 3
• Setting γ=0 makes returns equal instantaneous rewards, which can simplify DRL implementation.Source: §3 · p. 3

• The simulator uses one action for a 200-timestep trajectory, reflecting that real billiards actions occur after balls stop and can irreversibly change the game state.The cue-ball placement is parameterized in the state, and the action applies an initial impulse.Source: §4 · p. 3
• The reward combines bounded bonus and penalty terms; pocketing the cue ball yields zero, while shaping guides the target toward a pocket and penalizes cue-ball proximity to pockets.Shaping weights are low relative to the binary pocketing reward, so the maximum reward requires a successful shot.Source: §4 · pp. 3–4
• In one-dimensional state-action views, reward appears as narrow peaks and ridges, so a good policy samples those high-reward actions for each state.Source: §4 · p. 4
• Different pockets and bounce routes make rewards multimodal, while cue-ball and target-ball pocketing can create discontinuities between states.Source: §4 · p. 4

5 PROBLEMS WITH OFF-THE-SHELF DRL

• In the simplest one-dimensional state-and-action case, PPO, SAC, and AWR fail to find viable solutions across the state space.The authors exclude model-based methods because billiards’ contact- and collision-rich transitions make prediction errors consequential for action precision.Source: §5 · p. 4
• PPO fluctuates between reward peaks, while SAC does not adequately reduce state-conditioned action entropy.The authors suggest SAC’s entropy regularization may favor exploration over exploitation, which they consider a poor fit for high-risk, high-precision problems.Source: §5 · p. 4
• AWR without learned action variance captures only rough peak shapes; learned variance fits peaks better but still fluctuates, while replay helps retain rare high-reward actions.With learned variance, AWR has less action noise, possibly because it can use a large buffer of off-policy experience.Source: §5 · p. 4
• The paper uses learned-variance AWR as SCOOT’s baseline and starting point.Figure 4 overviews policy-optimization iterations with a mixture-of-experts policy and elite samples from an off-policy replay buffer.Source: §5 · p. 4, Figure 4 · p. 4

6 STATE-CONDITIONED SHOOTING

• SCOOT retains AWR’s iterative sample-simulate-buffer-fit loop, while replay preserves rare high-return actions but can slow convergence when fitting old, suboptimal actions.The policy is fitted to high-return actions drawn from a multi-iteration FIFO experience buffer.Source: §6 · pp. 4–5

6.1 Elite Samples

• Because low-return samples vastly outnumber high-return samples, SCOOT fits only replay-buffer samples exceeding the value estimate, whose advantages are non-negative.This makes AWR-style exponentiation unnecessary for converting advantages into non-negative sample weights.Source: §6.1 · p. 5

6.2 Distance-Regularized Mixture-of-Experts

• The policy uses state-conditioned Gaussian heads with categorical weights, while Mahalanobis distance regularization discourages overlapping head means.Each head represents a state-conditioned Gaussian distribution; the regularizer penalizes head overlap beyond a set distance threshold.Source: §6.2 · pp. 5–6
• Distance regularization with non-overlapping initialization yields broader action-space coverage and higher final return, although the single-Gaussian elite-sample policy learns faster initially.The method initializes head means across the action space and penalizes overlap between heads.Source: Figure 5 · p. 6, §6.2 · p. 6

6.3 Training Curriculum

• Adding the curriculum improves mean return and models the landscape’s continuous top and bottom ridges; late central-head modality loss does not affect final policy performance.The policy models most high-reward regions by the end of stage two, before component weights are learned in stage three.Source: Figure 6 · p. 6, Figure 7 · p. 7, §6.3 · p. 7

6.1 Elite Samples

• Elite-sample fitting with direct advantage weights improves performance over baseline AWR and removes AWR’s temperature hyperparameter β.Elite samples have non-negative advantages, so exponentiation is unnecessary to obtain non-negative weights.Source: §6.1 · p. 5, Figure 6 · p. 6

6.3 Training Curriculum

• The three-stage curriculum learns action means first, then reduces variance, and finally learns state-conditioned component weights to direct sampling toward heads near local optima.The stages are intended to prevent premature convergence while adapting the policy to reward optima.Source: §6.3 · p. 7

6.4 Implementation Details

• Implementation uses 128 samples per iteration and a 3,200-sample FIFO buffer; 1D experiments use an ~8.7-million-sample lookup table, while billiards figures use real simulation.The lookup table accelerates 1D training and evaluation; billiards figures and videos use the simulator.Source: §6.4 · p. 7

Learning High-Precision Shots

• SCOOT discovers rigid, continuous reward peaks for 1-dimensional shots and can switch strategies, while policies without its exploration features find these peaks less consistently.The reported features are the MoE policy, distance regularization, and curriculum learning.Source: §7 · p. 8

Discovering Diverse Shots

• A four-headed policy follows four major reward peaks and converges to distinct high-return shots, each representing a different strategy for the same input state.The policy's converged action means correspond to the diverse shots.Source: §7 · p. 8
• In this setup, four distinct strategies occur in almost every state; learning how many strategies are viable remains a direction for future work.The paper suggests a per-head value network could predict each strategy's return at inference.Source: §7 · p. 8

Scaling Up

• Around 75% success after 72 hours on an NVIDIA A100, SCOOT performs reasonably well on billiards with cue-ball position varying in both x and y.Occasional failures remain due to imprecise actions.Source: §7 · p. 8
• In the higher-dimensional task, SCOOT discovers local reward peaks and adapts to them incompletely; many failures involve excessive force or slight angle misalignment.Additional RAdam training improves performance gradually, but adaptation slows to an infeasible amount.Source: §7 · p. 8
• No solidly performing policy has yet been achieved for variants with multiple target balls whose positions vary, leaving competent generalization to these higher-dimensional tasks open.The paper conjectures that dimensionality, limited policy capacity, and a more irregular reward landscape may contribute.Source: §7 · p. 8

8 CONCLUSION

• SCOOT is proposed as a DRL algorithm for learning high-precision, high-risk movement control tasks.The paper describes it as inheriting AWR's simplicity with minor modifications.Source: §8 · p. 8
• SCOOT has demonstrated the capability to learn multiple alternative action strategies.The paper reports this capability alongside its proposed use for high-risk, high-precision control.Source: §8 · p. 8
• The billiards evaluation is limited to relatively simple problem variants, which nevertheless challenge commonly used DRL algorithms.The authors present this evaluation as opening avenues for investigating high-risk, high-precision motion control.Source: §8 · p. 8