Episodic Future Thinking Mechanism for Multi-agent Reinforcement Learning

• EFT remains underexplored in multi-agent RL despite its importance for anticipating future events during interactions.Source: §1 · p. 1
• Diverse agent characteristics make future-state prediction difficult because state transitions depend on interacting agents’ actions and behavioral preferences.Source: §1 · pp. 1–2
• Characters represent behavioral preferences encoded by different reward-component weight combinations, such as prioritizing safety or speed.Source: §1 · p. 2
• The multi-character policy captures heterogeneous behavioral patterns and supports character inference from target agents’ observation-action trajectories.Source: §1 · p. 2
• The proposed EFT agent infers other agents’ characters, predicts their actions, simulates future observations, and selects foresighted actions.Source: §1 · p. 2
• The proposed mechanism enhances group rewards across societies with different levels of character diversity.Source: §1 · p. 2

• Existing future-thinking approaches often predict states without modeling each agent’s behavior or incorporating predictions directly into policy selection.Source: §2 · pp. 2–3

Episodic Future Thinking

• The EFT agent predicts future observations from surrounding agents’ predicted actions and selects a foresighted action for the anticipated observation.Source: §2 · p. 3

Model-based Reinforcement Learning

• Model-based RL methods commonly assume few homogeneous agents and full observability, whereas heterogeneous multi-agent interactions provide incomplete and noisy information.Source: §2 · p. 3

Machine Theory of Mind

• Machine Theory of Mind and inverse-learning approaches often operate with few agents, small discrete action spaces, or limited character diversity.Source: §2 · p. 3
• The study explicitly links observation-action trajectories to characters and extends the action space from continuous to hybrid.Source: §2 · p. 3

False Consensus Effect

• The false-consensus comparison tests whether assuming other agents share the observer’s character harms decision-making in heterogeneous societies.Source: §2 · p. 3

3 Character Inference Using Multi-character Policy

• Accurate character inference is required because the inferred character is used to predict future actions within the EFT mechanism.Source: §3 · p. 4

3.1 Problem Formulations for Multi-agent Decision-making

• The MA-POMDP models simultaneous multi-agent decisions with continuous states and observations, hybrid actions, joint action-dependent transitions, and character-conditioned rewards.Source: §3.1 · p. 4
• Characters are K-dimensional vectors that parameterize agents’ reward functions and thereby represent behavioral preferences.Source: §3.1 · p. 4

3.2 Training a Multi-character Policy

• The multi-character policy conditions an actor-critic architecture on continuous observations and characters while producing hybrid actions.Source: §3.2 · p. 4
• A post-processor quantizes the actor’s continuous proto-action to handle the policy’s discrete action component.Source: §3.2 · p. 4

3.3 Inferring Character of Target Agent

• After training, the agent collects a target’s observation-action trajectory and estimates the character maximizing its trajectory log-likelihood.Source: §3.3 · p. 5
• Character inference uses gradient ascent to update the character toward higher action likelihood under the multi-character policy.Source: §3.3 · p. 5

4 Foresight Action Selection Based on Episodic Future Thinking Mechanism

• The EFT mechanism comprises future thinking and action selection modules that simulate subsequent observations before selecting a foresighted action.The future thinking module performs action prediction and next-observation simulation, while action selection uses the simulated observation.Source: §4 · p. 5

4.1 Future Thinking: Step I - Action Prediction

• The agent predicts each neighbor’s action with the trained multi-character policy conditioned on that agent’s inferred character and current observation.The predicted actions are assembled into a set representing the other agents’ upcoming behaviors.Source: §4.1 · pp. 5–6

4.2 Future Thinking: Step II - Next Observation Simulation

• The agent simulates its next observation by applying predicted others’ actions while holding its own action fixed as a null action.This produces the observation after target agents act but before the agent itself acts.Source: §4.2 · p. 6

4.3 Action Selection

• Using the simulated next observation and its own character, the agent obtains an adaptive action that accounts for others’ upcoming behaviors.Algorithm 3 summarizes the procedure, including future simulation, proto-action generation, and state updating.Source: §4.3 · p. 6

5.1 Performance Evaluation: Character Inference

• Over 500 inference iterations, the L1-norm rapidly approaches zero, while longer trajectories reduce the iterations needed for convergence.Figure 3 evaluates convergence using the L1-norm between estimated and true characters, with convergence defined as L1-norm ≤5 × 10^-4.Source: §5.1 · p. 7, Figure 3 · p. 7

5.2 Ablation Study: Character Inference and EFT Modules

• The proposed EFT consistently outperforms without EFT, whereas EFT using an incorrect shared character performs worse when character diversity exists.The results attribute degradation to incorrect action prediction and future-observation simulation, making accurate character inference crucial.Source: §5.2 · pp. 7–8, Figure 4 · p. 8

5.3 Investigating the Effects of Trajectory Noise

• Character inference accuracy decreases as trajectory-noise variance increases, while EFT reward rises with inference accuracy and remains effective at approximately 90% accuracy across diversity levels.The noise study reports accuracy across standard deviations and finds similar reward trends across diversity levels.Source: §5.3 · p. 8, Figure 5 · p. 8

5.4 Assessing Generalizability: Inference on Out-of-Distribution Character

• On unseen character ranges, inference performs well between trained regions and slightly declines outside the trained range while still capturing extreme values near their true values.The evaluation tests training on [0.0, 0.6] and [0.8, 1.0] versus training on [0.2, 0.8].Source: §5.4 · pp. 8–9, Figure 6 · p. 9

5.5 Performance Comparisons

• Across increasing diversity levels, the proposed solution outperforms the evaluated popular MARL algorithms while using only local observations and actions for training.Q-MIX performs worst in the hybrid-action task, whereas centralized-training MARL baselines require all agents’ observations and actions during training.Source: §5.5 · p. 9, Table 2 · p. 9
• Model-based RL methods show an increasing performance gap and higher variance as diversity rises, with Dreamer unable to adapt to high diversity and MBPO producing difficult-to-trust transitions.These results support the difficulty of learning multi-agent dynamics without understanding other agents.Source: §5.5 · p. 9

5.6 Additional Evaluation on MPE and SMAC

• ToMC2 declines as diversity increases and LIAM fails across diversity levels, whereas the proposed solution remains robust; it also achieves the best score in two of three MPE tasks and superior SMAC performance.In SMAC, the proposed method is especially strong in 3s5z_vs_3s6z and 6h_vs_8z, while MAPPO remains competitive in simpler tasks.Source: §5.5 · p. 9, §5.6 · p. 10, Table 3 · p. 10

6 Discussion

• EFT outperforms existing solutions across all diversity levels of the heterogeneous society.Source: §6 · p. 10
• The mechanism predicts other agents’ behaviors and simulates the agent’s future observation using a multi-character policy and inferred characters.Source: §6 · p. 10
• The proposed approach is presented as analogous to human decision-making because it simulates other agents’ upcoming actions.Source: §6 · p. 10
• The experiments include one EFT agent while all other agents lack EFT functionality to keep the problem tractable.Source: §6 · p. 10
• The execution phase assumes stationary agent policies, motivating further study of non-stationary policies for practical deployment.Source: §6 · p. 10

C.1 Algorithm 1

• Algorithm 1 uses 3500 episodes, 3000 timesteps, a replay buffer of 4 × 10^6, batch size 128, and discount factor 0.99.Source: §C.1 · p. 16

C.2 Algorithm 2

• Algorithm 2 uses Adam with learning rate 10^-3, 200 iterations, and 3000 samples.Source: §C.2 · p. 16

Appendix D Post-processor Function in (1)

• The post-processor partitions the continuous action interval [−W, W] into 2W + 1 discrete action values.Source: Appendix D · p. 17
• The post-processor uses min(·, W) to prevent actions from exceeding the upper bound, while a ceiling-and-max formulation can provide the lower-bound alternative.Source: Appendix D · p. 17

Appendix E Derivation of (2)

• The derivation decomposes the character-inference objective using the Markov property and removes transition terms unrelated to the character.Source: Appendix E · p. 18
• The character-inference objective becomes the sum of continuous- and discrete-action log probabilities because the action space is hybrid.Source: Appendix E · p. 18

Appendix F Loss Function for Character Inference

• The inference loss compares observed target actions with actions generated under an estimated character, and errors decrease when the estimate matches the actual character.Source: Appendix F · p. 19

Appendix G Experiments: Autonomous Driving

• The autonomous-driving task uses vehicle positions and velocities as state, neighboring-vehicle information as partial observation, and acceleration plus lane change as hybrid actions.Source: Appendix G · p. 20, §G.1 · p. 20, §G.2 · p. 20

G.3 Action

• Lane-change actions are discrete values −1, 0, and 1, representing outward movement, staying in the same lane, and inward movement, respectively.Source: §G.3 · p. 20

G.4 Reward

• The reward is Rt,i = c1R1 + c2R2 + c3R3 + rfail, combining desired velocity, safe distance, lane-changing, and infeasible-action penalties.Source: §G.4 · p. 20
• The infeasible-action penalty supports safety learning by penalizing unsafe decisions without requiring an accident.Source: §G.4 · p. 20
• The safety-distance term is zero when the following-vehicle gap meets s∗ and negative when it does not.Source: §G.4 · p. 21
• Increasing c1 raises average velocity toward the desired velocity v∗i = 3.5m/s, while lower c1 widens velocity dispersion.Source: §G.4 · p. 22, Figure H1 · p. 22
• Increasing c2 increases relative distance from surrounding vehicles, encouraging safer driving through more space.Source: §G.4 · p. 22, Figure H1 · p. 22
• Decreasing c3 increases lane-changing frequency because c3 weights the penalty for unnecessary movement into lanes with less driving space.Source: §G.4 · p. 22, Figure H1 · p. 22

• Figure I1 compares initial, estimated, and true character parameters and reports three-standard-deviation confidence intervals across ten experiments.Source: Figure I1 · p. 23, Appendix I · p. 23
• The likelihood trajectories converge to the true character values from different initial points.The contour plots show successful inference regardless of initial parameter location.Source: Appendix I · p. 23
• The inferred character parameters generally match the true values, indicating successful character inference from target observation-action trajectories.The estimates are especially accurate for c_j,1 and c_j,3, while c_j,2 becomes inaccurate when c_j,2 ≥1.2.Source: Appendix I · p. 23

J.1 Original Plot of Figure 4

• In scenarios with two or more character groups, the proposed solution consistently achieves the highest average reward, while FCE-EFT achieves the lowest.With one homogeneous character group, the proposed and FCE-EFT solutions perform identically, and both exceed the without-EFT condition.Source: §J.1 · p. 24, Figure J1 · p. 24

J.2 Performance Comparison with Confidence Interval

• The appendix reports the autonomous-driving comparison using a one-standard-deviation confidence interval table.Source: Table J1 · p. 24
• The comparison includes centralized-critic or centralized-value MARL baselines, including MADDPG, MAPPO, and quantized-action Q-MIX.MADDPG uses a centralized Q-function; MAPPO uses a centralized critic; Q-MIX combines individual Q-networks through a mixer.Source: §J.2 · p. 24
• Model-based baselines include Dreamer, which learns through latent imagination, and MBPO, which augments data with short model-synthesized rollouts.Source: §J.2 · p. 24
• Agent-modeling baselines include ToMC2 for intention sharing and LIAM for modeling other agents from local information in partial observability.Source: §J.2 · p. 24

K.1 Task Description

• The experiments use Spread, Adversary, and Tag from the Multi-Particle Environment suite.Source: §K.1 · p. 25
• Spread requires three agents to reach three landmarks without collisions, using distance and collision-penalty rewards.Source: §K.1 · p. 25
• Adversary involves two cooperating agents reaching a true goal while avoiding an adversary that observes relative distances without goal communication.Source: §K.1 · p. 25
• Tag is a predator-prey task with one good agent, three adversaries, and obstacles, where adversaries are rewarded for tagging the good agent.Source: §K.1 · p. 25

K.2 Performance Comparison with Confidence Interval

• A performance-comparison table with confidence intervals accompanies the task description.Source: Table K1 · p. 25

L.1 Task Description

• SMAC agents operate under partial observability, act through movement, attacks, and healing, and are rewarded for damage, kills, and winning.Source: §L.1 · p. 26
• The evaluation uses SMAC scenarios 2s3z, 3s5z_vs_3s6z, and 6h_vs_8z with different ally-enemy compositions.Source: §L.1 · p. 26

L.2 Performance Comparison with Confidence Interval

• The appendix includes a performance-comparison table for the SMAC scenarios.Source: Table L1 · p. 26

NeurIPS Paper Checklist

• The paper states that its claims match the stated scope and reports limitations in Section 6.Source: NeurIPS Paper Checklist · p. 27
• The checklist states that hyperparameters, task descriptions, experimental settings, and compute resources are provided in appendices or specified sections for reproducibility.Source: NeurIPS Paper Checklist · pp. 28–30
• The reported experiments use confidence intervals and IQR to support statistical reliability, with guidance emphasizing explicit error-bar definitions and variability factors.Source: NeurIPS Paper Checklist · pp. 29–30
• The checklist records no human-subjects research, no high-risk release requiring safeguards, and discussion of both positive and negative societal impacts.Source: NeurIPS Paper Checklist · pp. 30–32