Source-linked AI summary
Chronocooked: A Benchmark for Implicit Interval Timing in Reinforcement Learning Agents
Amrapali Pednekar, Alvaro Garrido-Perez, Yara Khaluf, Pieter Simoens
TL;DR
Existing RL research lacks a simple, standardized benchmark for testing implicit interval timing, where time is unobserved but critical for optimal performance. Chronocooked introduces cooking-based tasks and timing-focused metrics, showing that recurrent models exhibit emergent time-keeping-like behavior while all models remain limited in temporal awareness.
Problem
RL research lacks a simple, standardized benchmark for studying implicit interval timing, where time is unobserved yet critical for optimal performance.
Method
Chronocooked introduces cooking scenarios with implicit timing demands and evaluation metrics targeting interval discrimination, flexibility, generalization, and timing accuracy.
Results
All models fall short in several aspects of temporal awareness, while recurrence produces emergent time-keeping-like behavior and LSTM125 generally outperforms CTRNN125 on timing tasks.
Takeaways & Limitations
Chronocooked provides a controlled benchmark for exposing temporal limitations and assessing time-aware behavior in reinforcement learning agents.
Takeaways & Limitations
The proposed metrics are not an exhaustive characterization of time-aware models, and broader tasks, model settings, and implementations are needed.
Abstract
from arXiv · showhide
This paper presents Chronocooked, a reinforcement learning (RL) benchmark suite for studying implicit interval timing in RL agents. Inspired by Overcooked, the suite comprises cooking scenarios that require temporal decision making. The tasks and reward functions are designed such that temporal information is unobserved yet critical for optimal performance. The environment is intentionally kept simple to enable controlled experiments and support biologically plausible models. Evaluation metrics are designed to expose limitations in timing abilities of RL agents, and we report baselines using a non-recurrent, a recurrent, and a biologically plausible model. This work ultimately aims to underscore the need to incorporate time perception and temporal processing in artificial agents designed for human robot interaction and deployment in time dependent human societies.
1 Introduction
The introduction argues that time should be fundamental to AI agents operating alongside humans, while proposing an RL benchmark for implicit interval timing. The benchmark uses time as a means to achieve task goals and evaluates temporal limitations and human-like timing biases.
- Motivation: Time shapes human perception, coordination, and decision-making, motivating its inclusion as a fundamental characteristic of AI agents designed to function alongside humans.The passage contrasts this motivation with AI systems whose temporal processing is evaluated primarily through final task performance.
- Benchmark contribution: The study introduces an RL benchmark for interval timing in which time-keeping is not the end goal but a means to achieve task goals.Its reward functions and agent states are designed around this implicit temporal decision-making setup.
- Evaluation: The benchmark’s evaluation metrics quantify whether trained models replicate human timing biases and capture characteristics of time-aware models described in psychology and neuroscience.The first purpose relates to smoother human-robot interaction, while the second consolidates key characteristics of time-aware models.
- Task design: The bisection and fixed interval timing tasks are inspired by psychology research, while the other tasks modify them to test different temporal decision-making characteristics.This task family is summarized in Figure 1.
2 Prior work
Prior work either studies timing that emerges in AI models or explicitly engineers time into them, drawing on behavioral, neural, psychological, and neuroscience research. However, the field lacks a simple standardized benchmark for evaluating time-aware agents across tasks and timing aspects.
- Emergent timing: Timing research in AI studies emergent behavioral and neural similarities to human or animal timing, including timing biases, time cells, ramping cells, and oscillations.Emergent timing research examines both behavioral replication and biologically plausible neural activity.
- Time-aware models: Explicitly engineering time into models improves task performance and human-timing replication, but studies usually target specific tasks and timing aspects.Psychology and neuroscience provide theoretical knowledge for designing time-aware AI agents.
- Biological timing models: Neuroscience and psychology have developed biologically plausible timing models, while reinforcement learning is linked to interval timing through temporal difference learning.These models prioritize biological plausibility over task generalization but may serve as components of AI systems incorporating a sense of time.
- Benchmark gap: Across these research threads, the absence of a simple standardized benchmark limits evaluation and comparison of time-aware models.PsychLab included some timing tasks, but Chronocooked extends this direction with implicitly timed scenarios.
- Benchmark design: The proposed benchmark uses timing-literature metrics to quantify human-timing-bias replication and key characteristics of time-aware models in a simple environment.The simple environment facilitates testing models that may be disadvantaged by complex environments.
3 Environment
Chronocooked uses a simplified, single-agent Overcooked environment with compact grid worlds, counters, and task-specific items. The agent prepares and delivers soup through sequential phases, including an oven timing phase governed by an invisible target-duration timer.
- Environment layout: The environment is a simplified Overcooked-AI scenario with a single agent in a 5X3 or 4X3 grid world.It includes three counters: an onion dispenser, an oven, and a delivery counter.
- Environment layout: The primary items are onions and soup, with additional task-specific elements described in later sections.The environment is illustrated in Figure 1.
- Task procedure: The agent prepares and delivers soup by picking up an onion, placing it in the oven, and waiting for cooking to finish.These steps form the ‘pick onion,’ ‘put onion in oven,’ and ‘oven on’ or timing phases.
- Task procedure: The oven starts an internal invisible timer when it receives an onion, linked to a predefined target duration (TD).The timing phase can vary slightly by task, and the oven reaches a ‘ready’ state after TD or when it indicates readiness.
4 Tasks
Chronocooked implements temporal bisection, fixed-interval, peak-interval, and buffered timing tasks using oven-based cooking scenarios with unobserved durations. These tasks evaluate temporal perception, response timing, and temporal control through rewards, psychometric curves, and timing metrics.
- Bisection task: The bisection task trains agents on short and long anchor durations, then tests categorization of intermediate and out-of-range durations.Chronocooked uses separate short and long delivery counters, with the oven duration sampled between anchors with 50% probability; correct delivery earns +1.
- Bisection task: Bisection performance is characterized by psychometric curves plotting P(long) against probe duration, with successful agents approaching 0% at or below short and 100% at or above long anchors.The bisection point distinguishes temporal encoding: the geometric mean indicates logarithmic encoding, whereas the arithmetic mean indicates linear encoding.
- Fixed-interval task: The fixed-interval task requires agents to wait for an invisible target duration before collecting and delivering soup for +1 reward; earlier actions receive 0.The reward structure imposes an explicit lower response bound and an implicit upper bound through discounting.
- Fixed-interval task: First oven check (FOC) records the oven timer when the agent first interacts, measuring timing accuracy, temporal control, and unnecessary oven checks.If the timer has reached the target duration, that oven time is recorded as FOC; otherwise, continued checking is required before successful retrieval.
- Peak-interval task: The peak-interval variant removes post-target reinforcement and extends the target duration to 2× or 5× training values to assess response-rate timing beyond training.Expected response rates increase toward the target duration and then decline, while scalar timing predicts linear scaling of mean and variance with target duration.
- Buffered timing task: The buffered timing variant awards +1 for retrieving soup within TD ± buffer duration, reduces early or late retrieval to 0.1, and supports multiple target durations simultaneously.Each target duration has a distinct oven state.
5 Dual task and uncertainty task
Section 5 introduces dual and uncertainty tasks that combine interval-timing demands with additional reward or oven-transition contingencies. Optimal behavior requires leaving a reward sink in time, or using a manual oven-off button when autonomous transition may fail.
- Dual task: The dual task adds a small reward sink during timing, requiring agents to leave it in time to reach the oven and collect the soup.Retrieving soup too early or too late reduces reward.
- Uncertainty task: The uncertainty task adds a manual oven-off button that makes the oven ready regardless of target duration but reduces reward.During training, autonomous oven transition has a small probability of never occurring.
- Uncertainty task: When autonomous transition fails, the optimal policy is to use the manual oven-off button, testing adaptation from standard timing without uncertainty.The task examines how quickly agents adapt to this condition.
6 Results
Results compare non-recurrent, recurrent, and biologically plausible recurrent PPO agents across psychometric, time-keeping, exact-timing, and multi-timer tasks. Recurrent models show emergent timing behavior, with LSTM125 often strongest overall, while scalar-timing conformity remains incomplete.
- Model architectures: The study evaluates CNN-MLP, CNN-LSTM-MLP, and CNN-CTRNN-MLP agents trained with PPO, with recurrent models sharing actor and critic networks.The CNN-CTRNN-MLP implementation modifies recurrent PPO.
- Psychometric performance: LSTM125 outperforms CTRNN125 on psychometric curves, whereas the two models perform comparably at memory size 256.CTRNN125 has an almost constant Weber fraction, unlike LSTM125, although this consistency varies across other memory sizes.
- Time-keeping behavior: Recurrence produces emergent time-keeping-like behavior, unlike the CNN’s usual FOC of zero and immediate oven interaction.The CNN strategy can still achieve the final soup-delivery goal but does not exhibit time-keeping behavior.
- Scalar timing: CTRNN125 shows an approximately constant Weber fraction in bisection, but recurrent models fail scalar-property tests because CV and mean–standard-deviation relationships are not consistently linear.Mean FOC increases linearly with target duration for both recurrent models, consistent with the mean property of scalar timing.
- Exact timing: LSTM125 generally outperforms the other models in exact timing, while CNN exceeds the minimum reward at the shortest target duration, TD = 4.At TD = 4, LSTM125 and CTRNN125 perform equally well.
- Multi-timer tasks: In multi-timer tasks, LSTM125 generally performs best, while models settle at trained target durations or intermediate values and exhibit approximate timer shifts.The number of distinct shifts in average oven-off-time curves indicates the number of approximate timers stored by a model.
7 Discussion
The discussion concludes that all evaluated models have shortcomings in temporal awareness, while noting that some evaluations are unfair or inherently difficult under the models’ training conditions. It also identifies the benchmark’s limited scope and outlines directions for expanding tasks, scenarios, library support, and model coverage.
- Model limitations: All models fall short in several aspects of temporal awareness, although some shortcomings reflect challenging evaluations under their training conditions.The authors distinguish model limitations from difficulties caused by evaluation conditions.
- Model limitations: Uniform oven-checking behavior on the peak interval timing metric is unsurprising because models were not trained for that scenario.The passage characterizes this comparison as unfair.
- Study limitations: The proposed metrics are a starting point rather than an exhaustive characterization of time-aware models.The authors identify metric coverage as a limitation of the study.
- Future work: Future work could add psychology-based tasks such as temporal reproduction, extend the benchmark to multi-agent scenarios, support libraries beyond SB3, and evaluate more model types.These directions are presented as ways to broaden the benchmark and its accessibility.
A Appendix · B Bisection task
The appendix’s bisection task evaluates psychometric timing behavior across three model types and anchor durations, with Weber fractions summarized across models. The figure plots long-response probabilities and identifies bisection points from sigmoid fits.
- B Bisection task: The bisection task compares psychometric curves for three model types across different anchors.Rows represent model types, while columns represent anchors.
- B Bisection task: The curves plot percentage of long responses against probe durations.The y-axis shows P(long), and the x-axis shows probe test durations.
- B Bisection task: Each anchor is accompanied by its arithmetic mean and geometric mean.These means contextualize the probe durations shown in the psychometric plots.
- A Appendix: The appendix includes the bisection task as a section of the benchmark’s timing analysis.The supplied material identifies this content under Appendix and Bisection task.
- B Bisection task: The plots show run-level probabilities from 11 seeds and the average probability across runs.Gray dots represent individual runs, while black dots represent the across-run average.
- B Bisection task: A sigmoid fit is applied to the average P(long), with a reference line at P(long)=50%.The red line is the sigmoid fit, and the horizontal blue dotted line marks the 50% level.
- B Bisection task: Bisection points are shown in the respective psychometric plots.The bisection points are derived from the plotted timing-response curves.
- B Bisection task: Table 4 reports Weber fractions across models and anchor durations.The table summarizes Weber-fraction measurements by model and anchor duration.
C Fixed interval … D.2 Multiple timers
The multi-timer analyses examine first oven-check timing, exact-timer reward under timing buffers, and multi-timer reward and oven off time across target durations. The figures report variability across runs or model types and identify missing CTRNN data for the exact-timer plots.
- D.2 Multiple timers: First oven check (FOC) distributions are compared across model types and target durations.Standard deviation is shown as error bars on the corresponding barplots.
- D.2 Multiple timers: Average reward is evaluated across different timing buffers for each target duration (TD) in the exact-timer task.Shaded regions represent the standard deviation across runs.
- D.2 Multiple timers: The exact-timer reward plots mark 0.1 as the minimum possible reward for successful task completion.A red dotted line indicates this reference value.
- D.2 Multiple timers: The exact-timer plots omit data from CTRNN.This limitation is stated in the figure note.
- D.2 Multiple timers: Multi-timer performance is presented using reward and oven off time as functions of target duration.The figure reports both metrics across target durations.
- D.2 Multiple timers: Average oven off time is plotted for different target durations (TDs).The figure focuses specifically on average oven off time across TDs.