Affectively Framework: Towards Human-like Affect-Based Agents

• Games provide diverse play traces, emotional experiences, and modelling modalities, making them suitable stimuli for affective-computing and reinforcement-learning research.The paper contrasts interactive games with static stimuli and notes their use of pixels, game states, and controller inputs.Source: §I · p. 1
• Existing reinforcement-learning agents primarily optimise gameplay success, while no environments incorporate human affect for training emotion-aware gameplaying agents.Source: §I · p. 1
• Developing game environments and sourcing human gameplay and affect demonstrations is time-consuming, creating a need for publicly available environments, benchmarks, and affect corpora.Source: §I · p. 1
• The Affectively Framework incorporates a human-sourced arousal model into the observation space and follows the Open-AI Gym philosophy to support later integration of games and agents.Source: §I · p. 1
• The framework provides three challenging environments based on the arousal video game annotation dataset and baseline experiments using in-game behaviour and human arousal rewards.The package is presented as an easy-to-install research test bed for affect-aware virtual agents.Source: §I · p. 1

• At each time step, the framework maps game-state observations and agent actions to updated states, environment scores, behaviour rewards, affect values, and total rewards.Affect values are generated periodically every 3 seconds using a human affect model.Source: §II · p. 1, Figure 1 · p. 2
• The behaviour reward RB is constructed from environment scores and domain knowledge to help training and mitigate sparse environment rewards.Source: §II · p. 1
• The affect reward RA is derived from affect values and is used here to maximise arousal, while more complex functions could instead imitate affect.Source: §II · p. 1
• The total reward combines normalised behaviour reward and affect reward as Rt = (1 −λ) · n(RB) + λ · RA, with λ controlling their relative importance.Source: §II · p. 1

A. Environments

• Three environments span side-scrolling platforming, first-person shooting, and racing, selected for diversity in genre, action/state space, and gameplay goals.Source: A. Environments · p. 2
• The framework is designed for extension beyond its three environments, which are based on the AGAIN dataset and differ substantially in observation and action spaces.Source: A. Environments · p. 2, Figure 2 · p. 3

Pirates

• Pirates uses an 11×11 entity grid plus seven state properties, with rewards for collected items, rightward traversal, and avoiding death.Its behaviour reward uses item-score changes, Mr = 0.1 for rightward movement, and D = 5 for dying.Source: A. Environments · p. 2

Heist

• Heist combines a 9×9 field-of-view grid with 20 additional observations and mixed discrete-continuous actions for movement, shooting, and aiming.Its behaviour reward adds exploration of new map areas and orientation toward the nearest enemy to kill-based environment rewards.Source: A. Environments · pp. 2–3

Solid Rally

• Solid Rally uses 50 real-valued observations and steering/forward-motion actions for a three-lap race against three AI opponents.Its behaviour reward combines waypoint progress with speed and orientation toward the next checkpoint.Source: A. Environments · p. 3

B. Affect Model

• The model uses k-nearest neighbours with k = 5 to infer increasing or decreasing arousal transitions from consecutive annotated 3-second gameplay windows.Stable arousal transitions are discarded, and features include game-specific variables such as speed, health, and score.Source: B. Affect Model · p. 3
• Every 3 seconds, inverse-distance weighting over the five nearest human transitions produces an affect value from 0 for unanimous decrease to 1 for unanimous increase.The resulting affect reward is shared across environments and maximises arousal-increasing transitions; intermediate time steps receive zero affect reward.Source: B. Affect Model · p. 3

III. EXPERIMENTAL PROTOCOL

• Agents are evaluated while maximising in-game performance, arousal, or a combination, with final environment reward and mean arousal reported at session end.Source: §III · p. 3
• Table II compares RL agents and baselines across three reward functions using 30 evaluation runs and 95% confidence intervals.Bold entries identify the best observed reward for each column.Source: Table II · p. 4

1) Random Agent:

• The random baseline uniformly samples the environment’s action space, and its results are averaged across 30 runs on the same environment.Source: 1) Random Agent · p. 4

2) Humans:

• Human gameplay is evaluated using in-game score and arousal changes across consecutive 3-second windows.Human sessions share the agents’ scoring and ending conditions, including a 2-minute time limit.Source: 2) Humans · p. 4

3) PPO Agent:

• PPO agents are trained for 1 million time steps and evaluated 30 times with exploration turned off.The agents use Stable Baselines defaults and stochastic action prediction during evaluation.Source: 3) PPO Agent · p. 4
• Max. Behaviour sets λ = 0 to optimise only the behavior reward and game performance.This agent is expected to play only to win by maximising RE.Source: 3) PPO Agent · p. 4
• Max. Arousal sets λ = 1 to optimise only the arousal reward derived from human examples.The authors expect this objective may produce unexpected and uncontrollable behaviors.Source: 3) PPO Agent · p. 4
• Blended reward sets λ = 0.5 to optimise behavior and arousal equally as a more human-like objective.The design assumes humans both play to win and seek increased arousal.Source: 3) PPO Agent · p. 4

IV. RESULTS

• Random agents achieved very low RE scores and showed no consistent affect pattern because they quickly became stuck and repeated arousal values.Table II reports final RE normalised to [0, 1] alongside average arousal for PPO agents and baselines.Source: §IV · p. 4
• PPO agents using behavior rewards scored below human demonstrations in every game, with Heist especially difficult because of its complex action space and gameplay.Behavior-based scores were approximately half the average human scores for Solid Rally and Pirates.Source: §IV · p. 4
• RE = 0 in all 30 runs across all three games for arousal-maximizing PPO agents, despite their arousal surpassing random and human baselines.These agents exploited high-arousal regions near the start instead of exploring the environments.Source: §IV · p. 4
• Blended rewards produced inconsistent outcomes: behavior dominated in Solid Rally, while affect dominated in Pirates and encouraged the agent to remain near high-arousal starting areas.The blended agent in Pirates reached RE = 0.106 during its first 50 episodes before converging on the starting area.Source: §IV · p. 4
• In Heist, the blended agent performed poorly relative to humans and only marginally better than the random agent.The authors attribute this result to insufficient training time for learning an effective policy.Source: §IV · p. 5

V. DISCUSSION

• Balancing behavior and arousal rewards is difficult because affect can be deceptive or overpowering, while arousal rewards occur only every 3 seconds.The authors warn that sparse, clock-triggered RA rewards may confuse agents in blended schemes.Source: §V · p. 5
• Clustering gameplay traces into player groups could support agents targeting specific personas, such as expert behavior and arousal.The framework plans to incorporate pre-made player clusters across all three environments.Source: §V · p. 5
• Longer training, hyperparameter tuning, alternative RL algorithms, and additional affect rewards are needed to assess PPO more fully.The paper specifically notes poor behavior from PPO agents that maximise arousal.Source: §V · p. 5

VI. CONCLUSION

• Affectively is compatible with the standard Open-AI Gym API and supports training game-playing agents that account for both behaviour and affect.Source: §VI · p. 5
• The framework introduces three game environments and evaluates them with random and PPO baseline agents across behaviour and affect tasks.Source: §VI · p. 5
• Future studies can use the framework’s software package and test-bed environments to investigate relationships between player behaviour and affect during gameplay.Source: §VI · p. 5

ETHICAL IMPACT STATEMENT

• The publicly available data and environments support scientific reproducibility and future extensions of the study.Source: ETHICAL IMPACT STATEMENT · p. 5