PaperlayerView exact revision on arXiv

Source-linked AI summary

Cognitively Constrained Partners for Zero-Shot Coordination in Real-Time Partitioned Control

Ash Simmons, George De Ath, Richard M. Everson

arXiv:2610.31726v1cs.LGcs.HC

TL;DR

Agents trained against simulated partners can overfit to artificial conventions, while learning from human feedback or large behavioural datasets is costly. This paper trains agents against partners with physiologically grounded cognitive constraints in a partitioned Lunar Lander task. The constrained agents match the strongest baselines under cross-play, while whether cross-play predicts human teaming remains untested pending the planned study.

  • Problem

    Human feedback and large behavioural datasets make training costly, while purely simulated partners can induce artificial conventions that fail in zero-shot coordination with real people.

    Source: §1 · p. 1

  • Method

    The framework trains agents against partners with physiological constraints on processing delays and perception, and evaluates them in partitioned control against frozen partners and baseline methods.

    Source: §1 · p. 2, §4 · p. 5, §5.2 · p. 6

  • Results

    Under cross-play, constrained agents match PBT and FCP, and are the only agents whose performance improves rather than declines when evaluated away from their training partner pool.

    Source: §6.2 · p. 8, §8 · p. 9

  • Takeaways & Limitations

    The preliminary results support cognitive constraints as a way to generate partners whose behavioural region resembles real players and to train agents that retain cross-play performance across partner-generation procedures.

    Source: §9 · p. 9

    Cross-play uses held-out artificial partners rather than humans, so whether its performance rankings predict human-agent coordination remains untested until the planned study is complete.

    Source: §8 · p. 9

Abstract

from arXiv · show

Effective human-machine teaming requires agents that coordinate with unpredictable human partners. Reinforcement learning from live human feedback or large behavioural datasets is expensive, while purely simulated training overfits to artificial conventions and fails at Zero-Shot Coordination with real people. We propose a framework for training teaming agents against Cognitively Constrained Partners: simulated partners whose diversity emerges from procedural constraints grounded in human physiological literature, specifically reaction delays and perceptual limitations, rather than from reward shaping or historical training checkpoints. In a modified Lunar Lander environment, we report preliminary evidence that these partners play more like real people than unconstrained agents do, and that agents trained against them match the strongest cross-play baselines while being the only ones whose performance does not depend on how their training partners were generated. We also describe a human-participant study designed so that its rankings can be compared directly against cross-play, testing how well that standard evaluation predicts real human-agent teaming.

1 Introduction and Motivation

Human-machine teaming requires agents that coordinate with unpredictable people, but human feedback and behavioural datasets are costly, while purely simulated training can overfit to artificial conventions. The paper proposes cognitively constrained partners and evaluates agents in partitioned control, using cross-play and a planned human study to examine whether cross-play predicts real teaming.

1 Introduction and Motivation

  • Human feedback and behavioural datasets can be prohibitively expensive, while simulated populations may overfit to artificial conventions and fail in zero-shot coordination with people.Existing alternatives include human-data cloning and self-play checkpoints, which do not fully address the gap between artificial partners and human teammates.Source: §1 · p. 1
  • A time-continuous, strictly partitioned control setting forces both partners to coordinate distinct inputs because neither can take over the other's control.The authors position this as a benchmark for zero-shot coordination in a comparatively under-explored domain.Source: §1 · p. 2
  • The proposed framework trains against partners limited by human-like cognitive constraints and compares performance with MARL, PBT, and FCP in held-out cross-play and a planned human study.The study is designed to test whether cross-play, a standard zero-shot coordination measure, predicts coordination with real people.Source: §1 · p. 2

2 Related Work

Prior work addresses continuous teaming through shared autonomy, human proxies, and diverse partner populations, but each approach leaves a gap for partitioned zero-shot coordination. This paper instead generates partner diversity from procedural cognitive constraints.

2 Related Work

  • Shared autonomy approaches range from a POMDP requiring prior dynamics and goals to model-free blended Lunar Lander assistance, whereas this work prevents the agent from overriding human inputs.The cited POMDP approach can restrict scalability for ad-hoc coordination; blended assistance can modify human actions for safety.Source: §2 · p. 2
  • Human-proxy training can prepare agents for zero-shot coordination, but behavioural cloning requires collected human trajectories.The Overcooked proxy approach used paired trajectories to construct a partner whose suboptimal behaviour exposed the agent to human conventions, delays, and errors.Source: §2 · p. 2
  • Population methods generate diversity through trajectory divergence, self-play checkpoints, or other policy-distribution objectives, while the proposed approach uses procedural cognitive constraints.FCP demonstrated robust zero-shot coordination with live humans without human-in-the-loop training data, using historical self-play checkpoints; the proposed pool instead varies processing delays and perceptual limitations.Source: §2 · pp. 2–3

3 Partitioned Lunar Lander Environment

The modified Lunar Lander task divides control between partners so that landing requires concurrent coordination. The environment discretizes concurrent engine and lateral-thruster inputs and retains shared reward incentives rather than shaping partner diversity.

3 Partitioned Lunar Lander Environment

  • The modified Lunar Lander task requires landing in a target zone using a primary thruster and two lateral thrusters, with episodes ending in landing or crashing.Random top-screen spawns provide varied starting states while keeping the task computationally tractable.Source: §3 · p. 3, Figure 1 · p. 3
  • Control authority is split between the main engine and lateral thrusters, preventing either partner from completing the task alone.This decomposition requires concurrent, real-time coordination across the joint action space.Source: §3 · p. 3
  • A multi-discrete wrapper lets partners act concurrently: the main engine is off/on and lateral thrusters are off/left/right, mapped to fixed thrust magnitudes.The main engine fires at full power; lateral thrusters use 70% of maximum because full lateral thrust was excessive for the required corrections.Source: §3 · p. 3
  • Reward shaping was rejected as a way to create population diversity because it encodes designer bias; constrained agents instead optimize the standard incentives also used for people.The authors locate diversity in computational rationality under cognitive and physical limits, with observable latency and motor bottlenecks more empirically tractable than latent reward preferences.Source: §3 · pp. 3–4

4 Cognitively Constrained Partners

Cognitively constrained partners receive physiological limits on reaction speed, perception, and action frequency. The implementation combines stochastic delays and perceptual thresholds with recurrent policies to address the resulting partial observability.

4 Cognitively Constrained Partners

  • A psychological wrapper limits reaction speed, perceptual access, and action frequency to generate hesitant, temporally chunked behaviour.The design treats human-like behaviour as optimization under physiologically grounded constraints.Source: §4 · p. 4
  • Processing delay means are initialized between 4.0 and 5.5 frames, or 80–110 ms, while mechanical delay means are 3.0–4.0 frames, with a 70 ms motor processor cycle.Both delay phases are sampled using an exponentially modified Gaussian distribution; together their ranges are 140–190 ms.Source: §4 · p. 4
  • Because imposed delays make the task partially observable, the authors use recurrent PPO so hidden state can integrate sequential observations across temporal blind spots.The recurrent hidden state is described as a continuous cognitive buffer for temporally consistent input generation.Source: §4 · p. 4
  • The wrapper filters horizontal and vertical velocities and tilt angle, retaining each last-registered value until the change reaches its perceptual threshold.Comparing against the last registered value lets gradual changes accumulate until detectable, rather than allowing the agent's estimate to drift arbitrarily far.Source: §4 · p. 5
  • Perceptual thresholds are bounded below because a strict Weber-law rule at low stimulus intensities would grant superhuman acuity during landing.Tilt thresholds are also reduced near cardinal angles to reflect the reported oblique effect; the resulting agents remain less sensitive than a literal Weber-law reading would imply.Source: §4 · p. 5
  • Each agent's perceptual sensitivity is set by one sampled scalar mapped to both velocity and angle thresholds, coupling the two traits.The shared scalar models generalized visual aptitude across the reported 7–11% velocity and 4.5–8% angle sensitivity ranges.Source: §4 · p. 5

5 Teaming Agent Implementation and Baselines

The teaming agent learns to control the main engine while adapting to a fixed, cognitively constrained partner controlling lateral movement. Baselines are adapted to avoid shared conventions where possible, with matched partner-pool sizes and deliberately different observation access.

5.1 Teaming Agent

  • The teaming agent controls the main engine while a fixed cognitively constrained partner supplies lateral input, and partial weight transfer lets the agent begin with learned flight behavior.The agent has a reduced action space, Amain = {0, 1}, and recurrent architecture addresses partial observability from partner delays and unobserved forthcoming decisions.Source: §5.1 · p. 5
  • Each episode pairs the teaming agent with a uniformly drawn partner from a pool of 45 agents with distinct reaction-delay and perceptual constraints.The partner’s executed lateral action, which may differ from its requested action because of the motor pipeline, is appended to the agent’s next observation.Source: §5.1 · p. 5

5.2 Baselines

  • Because real-time, reflex-driven partitioned control is under-explored and lacks a tailored benchmark, the study compares adapted FCP, PBT, and standard MARL approaches.FCP is the primary baseline.Source: §5.2 · p. 5
  • PBT and FCP train best responses against frozen partners to avoid co-adapted conventions that novel human partners do not share, while MARL retains co-adaptation as its defining baseline.MARL uses staggered rather than simultaneous updates to control gradient instability.Source: §5.2 · p. 6
  • Training pools contain 45 partners per framework and evaluation uses 50 disjoint partners, matching pool size to isolate the source rather than quantity of partner diversity.The pools comprise 45 cognitively constrained partners, 15 FCP lineages sampled at three skill tiers, and a 46-agent PBT population in which each learner trains against the other 45.Source: §5.2 · p. 6
  • MARL partners observe teammate actions, whereas FCP, PBT, and cognitively constrained partners receive physical state; this avoids giving partners a noiseless teammate-action feed unavailable from people.Cognitively constrained partners also receive an efference copy of their own input.Source: §5.2 · p. 6

6 Preliminary Results

In preliminary behavioral validation, constrained agents were closer to human play than unconstrained agents on several measures, though the small pilot limits precision. In cross-play, they alone improved on held-out partner pools, while matching PBT and FCP on shared held-out partners.

6.1 Behavioural Validation

  • Human players were 4.12 distance units from constrained agents and 7.41 from unconstrained agents, a +3.29 difference (95% CI [+1.64, +4.86]); constrained agents were closer in 99.9% of bootstrap samples.The small human sample supports the relative ordering more firmly than precise claims about either population's absolute similarity to humans.Source: §6.1 · p. 7
  • Constrained agents were closer to human play on 15 of 22 features under each of three measures, which agreed on 17 features.The measures were difference in group means, normalized Wasserstein distance, and probability-mass overlap.Source: §6.1 · p. 7
  • Side-thruster press duration was the closest feature match: constrained agents shared 73% of its probability mass with participants, while unconstrained agents shared none.Main-engine presses were 353 ms for constrained agents and 98 ms for unconstrained agents, against 562 ms for participants; constrained agents also showed less between-controller variation.Source: §6.1 · p. 7, Figure 2 · p. 7
  • Lateral presses were 0.94 times per second for participants, 1.62 for constrained agents, and 6.12 for unconstrained agents; burst rates were 0.43, 1.39, and 9.73, respectively.All four terminal measures also favored constrained agents, with terminal tilt showing 78% overlap, although no constraint specified episode endings.Source: §6.1 · p. 7
  • Five features favored unconstrained agents under all three measures, concerning idle time, directional preference, or tilt-correction latency.For example, participants corrected tilt in 286 ms, compared with 131 ms for constrained and 146 ms for unconstrained agents.Source: §6.1 · p. 7

6.2 Cross-Play Generalisation

  • 203.4 to 266.6: the constrained agent's mean return improved on held-out versus matched pools; PBT fell 34.9 points, FCP 47.6, and MARL's 19.9-point decline was not distinguishable from zero.The comparison pairs results by replicate, cancelling run strength from the difference.Source: §6.2 · p. 8, Table 1 · p. 8
  • On the same 30 held-out partners, the constrained agent was indistinguishable from PBT (−4.5, 95% CI [−11.4, +1.9]) and FCP (+4.0, [−1.8, +10.0]), but ahead of MARL (+27.5, [+8.7, +52.7]).Averaged across all four pools, returns were 250.8 for constrained, 251.7 for PBT, 244.4 for FCP, and 227.7 for MARL.Source: §6.2 · p. 8
  • The constrained pool's partner-to-partner standard deviation exceeded 100 return points, versus 7–29 for other pools; against it, the constrained agent beat FCP and MARL but was not distinguishable from PBT.All agents performed worst against this pool, and only the constrained agent had trained against its generating procedure.Source: §6.2 · p. 8

7 Human Study Design

The planned within-subject study will compare solo play with play alongside four teaming-agent types, then test whether cross-play rankings persist with human partners.

7 Human Study Design

  • 100 participants are scheduled to play alone and with each of four agent types, with ten instances of each.The study has received institutional ethical approval.Source: §7 · p. 9
  • Scores will be analysed with a linear mixed-effects model using agent as a fixed effect and participant as a random intercept.This compares each agent relative to each participant’s own baseline; planned contrasts compare the proposed agent with each baseline.Source: §7 · p. 9
  • The study is designed to test whether cross-play rankings persist with human partners, addressing a rarely tested proxy relationship.Cross-play against synthetic partners is described as the field’s standard proxy for teaming performance.Source: §7 · p. 9

8 Limitations

The evaluation is preliminary and bounded: it targets initial cooperation, tests generalisation across artificial partners, and leaves human-team performance unverified pending the planned study.

8 Limitations

  • The work focuses on first instances of cooperation, when humans approach with relatively fixed cognitive heuristics and physiological constraints, rather than adaptation over time.The architecture is intended as a starting point for specialised human-in-the-loop training.Source: §8 · p. 9
  • Cross-play tests generalisation across held-out artificial partner types, not human partners; whether its performance predicts human teaming remains untested until the study is complete.Behavioural validation places the constrained partners in a region similar to real players, but does not settle that inference.Source: §8 · p. 9
  • The constrained pool is hardest for every agent, so its lower matched-pool baseline makes its improvement not directly comparable with PBT’s or FCP’s.On the same partners, the constrained agent shows parity with PBT and FCP rather than an advantage.Source: §8 · p. 9
  • Constrained partners see physical state but not the main engine’s exhaust plume, deliberately under-modelling player awareness and yielding a cautious policy.A more faithful treatment would add intermittent teammate-action awareness filtered through the perceptual pipeline.Source: §8 · p. 9
  • Perceptual restrictions do not cover every channel, and unfiltered angular velocity could let an agent infer the angle it was meant not to perceive.Extending restrictions to all channels is left to future work.Source: §8 · p. 9

9 Conclusion

The paper introduces procedurally generated partners constrained by human-grounded perceptual and motor limits in a partitioned ZSC environment. Preliminary findings show behavioural similarity to real players and cross-play parity with strong baselines, while human-teaming performance remains to be tested.

9 Conclusion

  • Partner diversity is generated procedurally from perceptual and motor constraints grounded in human physiological literature.The framework is applied in a partitioned environment where neither partner can complete the objective alone.Source: §9 · p. 9
  • The constrained partner population occupies a behavioural region similar to real players, but does not reproduce their full variation.Source: §9 · p. 9
  • Agents trained against constrained partners match the strongest cross-play baselines and lose nothing when the partner-generation procedure changes, while PBT and FCP degrade.Source: §9 · p. 9
  • Whether cross-play rankings persist with human partners remains an open test for the planned study.That study is intended to measure how well cross-play predicts performance in real human-agent teams.Source: §9 · p. 9

A Implementation Details

The appendices specify reward shaping, delay and perception constraints, and training procedures for the partner-generation baselines. Together, these details operationalise the constrained partners and the comparison methods.

A.1 Reward Function

  • Full-power main-engine firing incurs a 0.3 per-frame fuel penalty, while lateral firing at 0.7 incurs 0.021; landing and crash rewards are +100 and −100.The lateral penalty is roughly fourteen times cheaper per frame than the main-engine penalty, and landing penalties are deducted only from landings.Source: §A.1 · p. 13

A.2 Ex-Gaussian Delay Distribution

  • Each constrained agent has independently sampled phase means; the dashed curve convolves processing and mechanical delays into total output latency.The figure shows distributions for one agent at the midpoint of each parameter range.Source: Figure 4 · p. 13

A.3 Perceptual Thresholds

  • Perceptual thresholds are sampled from a scalar p and applied to velocity and tilt changes; velocity uses a 0.02 floor, while the tilt threshold includes a constant offset.For velocity, the Weber-fraction term governs above the floor; the velocity floor dominates through a substantial portion of a controlled descent. Tilt sensitivity is about 2.6–4.6° at typical tilts.Source: §A.3 · pp. 13–14
  • Near cardinal angles, tilt thresholds are halved within 10° and quartered within 2°; filtering evaluates changes on magnitudes and tracks each channel independently.The cardinal-angle adjustment uses the last perceived angle, making it operative during much of a controlled descent.Source: §A.3 · p. 14

A.4 Efference Copy

  • The constrained agent observes its current lateral input as an efference copy, but receives no access to internal refractory or commit-window timers.This expands observation from eight to nine dimensions; the input is not delayed or perceptually filtered.Source: §A.4 · p. 14

A.5.1 MARL

  • MARL alternates which agent learns every 20,000 timesteps while freezing the partner, and appends the partner’s prior action to each observation.The frozen policy stabilises training; decoupled environments preserve uninterrupted episodes, and the interval is intended to avoid both moving-target instability and overfitting.Source: §A.5.1 · pp. 14–15

A.5.2 PBT

  • PBT first pretrains agents on environment physics, then trains the primary agent against a randomly selected frozen partner from the population each episode.The primary agent does not know which partner controls the lateral thrusters, so training yields a generalised best response to the partner distribution.Source: §A.5.2 · p. 15

A.5.3 FCP

  • FCP generates lateral partners against frozen pretrained main agents and samples checkpoints at 33%, 66%, and 100% of peak learning.Fifteen lineages sampled at three tiers form the 45-partner training pool; extreme early incompetence could otherwise produce unrecoverable failures without actionable feedback.Source: §A.5.3 · p. 15

B Training Details

Training uses a shared recurrent PPO setup with condition-specific transfer choices, fixed-partner evaluation, and a patient early-stopping rule.

B Training Details

  • Runs used Python 3.9.6 and listed software versions, ran on CPU, and took approximately 8–12 hours per training instance.Training was unseeded, so exact numerical reproduction is not possible; sampled constraint profiles are recoverable from disk.Source: B Training Details · pp. 15–16

B.1 Hyperparameters

  • All agents use RecurrentPPO with an MlpLstmPolicy, and hyperparameters are identical across conditions except where Table 2 notes otherwise.Unlisted parameters use Stable-Baselines3 defaults.Source: §B.1 · p. 16, Table 2 · p. 16

B.2 Weight Transfer

  • PBT, FCP, and primary teaming agents inherit solo-agent representations, whereas MARL and cognitively constrained partners train from scratch.The transferred agents relearn the action head because the solo and teaming action spaces differ.Source: §B.2 · p. 16

B.3 Evaluation Protocol

  • Evaluation uses 50 fixed seeds and fixed partners; calibration found this set sufficient to distinguish typical improvements from seed variability.The calibration progressively tested larger subsets of 100 seeds with paired one-tailed t-tests.Source: §B.3 · pp. 16–17

B.4 Early Stopping

  • Training stops after 5,000 episodes without mean-reward improvement, within a 2,500,000-timestep budget for most conditions.The MARL budget is 5,000,000 timesteps split between two agents; the patience rule allows recovery from temporary destabilisation.Source: §B.4 · p. 17

C Future Analysis Plan

The planned human study compares four agent types within participants, using a mixed-effects model and prespecified secondary analyses.

C.1 Human Study Details

  • The approved within-subject study plans 100 participants, with 20 solo episodes before and after 20 episodes alongside each of four randomized agent types.Each agent type is represented by ten instances, with participant assignment balanced so each instance is played by ten participants.Source: §C.1 · p. 17

C.2 Mixed-effects Model

  • The mixed-effects model compares each condition with the constrained agent while accounting for participant proficiency and variation among trained instances.Trial index is included as a covariate; participant and agent-instance random effects represent variation across both populations.Source: §C.2 · pp. 17–18
  • With only ten instances per condition, instance-level variance may be estimated imprecisely or indistinguishable from zero, though fixed-effect estimates still account for it.The planned analysis checks random-effect normality and may model participant proficiency as a two-component normal mixture if needed.Source: §C.2 · p. 18

C.3 Secondary Analyses

  • The hypothesis requires all three baseline contrasts to be significant; secondary analyses will examine episode outcomes and differences by age and gaming experience.A participant random intercept accounts for repeated scores and substantial differences in baseline competence.Source: §C.3 · pp. 18–19

D Extended Results

The extended results describe behavioural-feature comparisons and their uncertainty estimates, followed by the planned questionnaire and participant instructions.

D Extended Results

  • Twenty-two behavioural and performance metrics are compared, using mean differences, normalized Wasserstein distance, and distribution overlap to capture complementary aspects of similarity.Mean differences are scale-dependent, Wasserstein distance can be distorted by long tails, and overlap does not distinguish separation magnitudes when distributions do not overlap.Source: D Extended Results · p. 19
  • Table 3 compares feature-level overlap and normalized Wasserstein distance, while Table 4 compares group means and standard deviations against human means.Table 4 complements Table 3 because pooled-standard-deviation normalization can distort rankings for features with long tails.Source: Table 3 · p. 20, Table 4 · p. 20
  • Section 6.2 intervals use a two-way bootstrap over trained replicates and held-out partners, treating their crossed sampling units and repeated episode pairings accordingly.The intervals are conditional on the fixed set of twenty evaluation seeds.Source: D Extended Results · pp. 21–22

E Questionnaire Items

  • The questionnaire records demographics, gaming experience, perceived cooperation, partnership rankings, score impact, support for intended play style, and solo-versus-agent preference.It also invites free-text comments.Source: E Questionnaire Items · p. 22

F Gameplay Information

  • Figure 6 shows the instructions participants see at the start of a trial.Source: Figure 6 · p. 23
Loading 2610.31726v1…