PaperlayerView exact revision on arXiv

Source-linked AI summary

Fig: Automated Multiagent Behavior Elicitor

Anrui Gu, Zhichao Wang, Nicolai Ouporov

arXiv:2610.28891v1cs.AIcs.MA

TL;DR

Many multiagent evaluations observe behavior as it emerges from particular environments rather than systematically eliciting it. Fig maps behavior descriptions to game interventions and evaluates resulting play; its interventions raise attempt rates in matched comparisons, supporting the use of multiagent environments as controllable evaluation substrates.

  • Problem

    Scaling behavioral evaluations and data for multiagent systems matters for studying prosocial, emergent, and misbehavior patterns.

    Source: §1 · p. 1

  • Method

    Fig maps behavior descriptions to games and interventions, then evaluates behavior using mechanical checks and rubric-based verification.

    Source: §1 · p. 1, §2.1 · p. 4

  • Results

    10.9 percentage points higher average attempt rate in mutated games than original games across 57/177 scenarios with valid labels and matched evaluations.

    Source: §3.2 · p. 8

  • Takeaways & Limitations

    Multiagent environments can serve as controllable evaluation substrates for generating behavioral evaluations by varying prompts, scenarios, and interaction mechanisms.

    Source: §1 · p. 2, §7 · p. 12

    Fig’s quality depends on the implementation agent’s judgment, and a typical query is estimated to cost approximately $210 per behavior, excluding evaluation runs.

    Source: §5 · p. 11

Abstract

from arXiv · show

Many social evaluations for multiagent systems study behaviors as they emerge from environments instead of systematically intervening in the environment to elicit such behaviors. We introduce Fig (Finding Interactions in Games), an agentic system that performs targeted elicitation interventions on game environments. It takes a natural language behavior description, decomposes it into sub-behaviors, generates new social games or selects existing ones, and executes single or combined interventions to the games' engine, opponents, instructions, and context. To evaluate model behavior, it combines built-in mechanical checks and synthetic rubric judgments. We validate the interventions, generated games, and evaluation mechanisms via human review. Models attempt the target behavior in 45.2% of episodes on average for valid scenarios generated by Fig for six test behaviors -- cooperation, exploitation, deception, covert coordination, betrayal, and sabotage. For game-behavior pairs with available controls, Fig's interventions increase attempt rates by 10.9% on average compared to matched control games. Fig shows that multiagent environments can serve as programmable test beds for intentional, controllable, and reproducible behavior evaluation. Code is available at https://github.com/fleet-ai/fig.

1 Introduction

Many multiagent evaluations observe behaviors as products of particular environments rather than eliciting them through controlled interventions. Fig offers an extensible game-based system for targeting and evaluating behaviors across environments.

1 Introduction

  • Fig aims to scale evaluation of prosocial, emergent, and misbehavior patterns, and its results suggest environments can become controllable evaluation substrates.The system studies behaviors including cooperation and communication, covert coordination, cheating, deception, and exploitation.Source: §1 · pp. 1–2
  • Existing game benchmarks are usually observational, treating behavior as an artifact of specific environments rather than a result of controlled elicitation.TextArena also lacks support for counterfactual replays and controlled payoffs needed by some behavioral evaluations.Source: §1 · p. 1
  • Fig combines a unified game abstraction, interventions, programmable opponents, and mechanical checks with rubric-based verification to elicit and detect multiagent behavior.Its intervention targets include rules, incentives, and opponent policies.Source: §1 · pp. 1–2

2 Fig: An Agentic Elicitation Pipeline

Fig provides a common interface for social games and a pipeline that maps behavior descriptions to game interventions, runnable scenarios, and layered evaluation. Its design combines game selection and generation with opponent controllers and checks for behavior attempts, execution, and success.

2 Fig: An Agentic Elicitation Pipeline

  • Fig abstracts heterogeneous social games into a common interface for instantiating, manipulating, and evaluating behavioral targets across environments.Its catalog contains 135 text games grouped into 103 families; the study excludes primarily visual environments and places less emphasis on purely cooperative games with shared goals and public information.Source: §2 · pp. 2–3, Table 1 · p. 3

2.1 The Design Stage: Mapping Behavior to Interventions

  • The design stage decomposes a behavior into a working definition and sub-behaviors, then selects or generates games and designs interventions that make target behavior distinguishable from normal play.Interventions can modify game engines, opponents, or player context, alone or in combination; the design stage also groups similar interventions into mechanism templates.Source: §2.1 · pp. 3–4

2.2 Intervention Implementation

  • An implementation agent translates designs into runnable game variants and automatically creates game-specific mechanical evaluators; validation checks evidence rules, incentives, and whether target norms are clearly stated to players.In reported experiments, 29/30 episodes were realistic (96.7%) using Claude Haiku 4.5 at 2,048 output tokens and temperature 0.7 as the opponent.Source: §2.2 · pp. 4–6

2.3 Behavior Evaluation Metrics

  • The evaluation runs focal models under a “play to win” prompt, while separately logged channels support inspection of disagreement between mechanical checks and rubric judges.An exploration prompt moderately increases observed attempts but does not significantly change the distribution shape of behaviors.Source: §2.3 · p. 6
  • Evaluation scores attempt, execution, and success using rubric judgments on trajectories and mechanical checks of game states, averaging determinate labels separately by channel.Mechanical positives can recover semantic false negatives, but mechanical negatives do not override semantic positives; behavior rates use trajectories with valid scores for the relevant axis.Source: §2.3 · p. 6

3 Evaluating Fig

Across six behaviors, Fig's valid scenarios elicited attempts, and interventions increased attempts over matched controls; component analyses examine game selection, mutations, generated games, and evaluator design.

3.1 Elicitation Effectiveness

  • 45.2% mean attempt rate (95% bootstrap CI [40.5, 49.7]) across six behaviors; attempts occurred in 151/204 scenarios.The mean is based on 2,687/3,060 valid win-prompt labels across 189/204 scenarios.Source: Table 2 · p. 7, §3.1 · pp. 7–8
  • 10.9 percentage points higher average attempt rate in intervention than matched controls (95% CI [4.2, 18.1]) across valid paired scenarios.Covert coordination decreased slightly; providing opportunities alone was not enough to induce action.Source: §3.1 · p. 8, Table 3
  • Exploitation was bottlenecked at construction: around half its scenarios failed validity, unlike cooperation and betrayal, which had high rates at both stages.The authors identify inducing exploitation as a harder mechanism-design problem than inducing better-represented social behavior.Source: §3.1 · p. 8

3.2 Component Ablations

  • 43.7% [36.7, 51.0] of game-selected mechanisms produced valid evaluators, versus 23.4% [19.2,28.1] for randomly sampled games.The comparison covers betrayal, deception, and covert coordination.Source: §3.2 · p. 8
  • 10.9 percentage points higher average attempt rate in mutated games than original games across six behaviors and 57 paired scenarios.In one five-round public-goods example, deceptive signaling attempts rose from 2/12 to 8/12 episodes after other contributors raised contributions following two high contributions.Source: §3.2 · p. 8
  • Generation added 24 templates (+15.6%), adding mechanism-level coverage but covering fewer sub-behaviors alone; 15 of the templates elicited target behaviors.Veiled Excavators uniquely combined complementary private information, selective communication, and competition.Source: §3.2 · p. 8, Figure 3 · p. 9
  • Human calibration found bespoke judges not significantly better than generic rubrics; generated scenarios were rated good tests in 54/60 cases (90%).The authors default to generic rubrics unless the intervention designer explicitly requests custom evaluators.Source: Table 4 · p. 9, §3.2 · p. 9

4 Discussion

Fig combines interventions across game–behavior pairs, traces how behaviors progress from attempt to success, and tests whether game mechanisms predict attempts on unseen games.

4.1 Intervention Combinations

  • 168 of 177 game–behavior pairs combined interventions, about one-third of scenarios used all three, and engine-side interventions most often paired with opponent-side interventions.An example combines a late-mover advantage with an opponent committed to cooperating first.Source: §4.1 · p. 9, Figure 4 · p. 10

4.2 Attempt/Execution/Success

  • Across 2,642 episodes with all three labels, exploitation attempts often succeeded, deception dropped at each stage, and cooperation losses partly reflected multilateral requirements and rubric confounds.In Avalon, 15 cooperation executions yielded only 2 labeled successes, mostly because cooperating players were then assassinated.Source: §4.2 · p. 10, Figure 5 · p. 10

4.3 Can we predict behaviors from mechanisms?

  • Prediction used game-mechanism explanations, behavior rubrics, model identity, and seat to estimate attempt probability on held-out games.The 5,310 episodes covered 177 scenarios and five models, with 2,655 episodes each for play-to-win and exploration prompts.Source: §4.3 · p. 10
  • 16-shot Kimi-K3 prompting achieved the best held-out-game Brier score, 0.1965 (95% game-level bootstrap CI [0.1496, 0.2458]), while three learned predictors had higher average error than controls.The authors describe this as preliminary evidence that behavior is predictable from mechanisms.Source: Table 5 · p. 10, §4.3 · p. 11

5 Limitations and Future Work

The authors identify limits in how well game-based findings generalize, how reliably interventions are implemented, and how clearly models understand rules and social context.

5 Limitations and Future Work

  • Game environments differ from real-world scenarios, and Fig’s non-exhaustive strategy space is bounded by existing and model-generated social games.The authors also question whether known social games are the most informative environments for these studies.Source: §5 · p. 11
  • Intervention quality depends on the implementation agent’s judgment, leaving possible subtle bugs, and a typical GPT-5.6 Sol query costs approximately $210 per behavior, excluding evaluation runs.The authors say more cost reduction is needed for scaling.Source: §5 · p. 11
  • Ambiguous scenarios and misunderstanding of social norms or game rules can produce false positives or false negatives in behavior labels.The authors mitigate this by clarifying rules and avoiding overly complicated games, while calling for better enforcement of environmental understanding.Source: §5 · p. 11

6 Related Work

Related work spans game-based multiagent evaluation, automated game generation, and automated behavior evaluation; Fig connects these areas through behavior-guided interventions in multiplayer games.

6.1 Multiagent Game-Based Evaluation

  • Multiagent game benchmarks study collaboration, competition, strategic reasoning, social dilemmas, and social deduction, while Fig starts from a behavior prompt to create principled opportunities for exhibition.The cited environments provide controlled settings, but prior work is usually observational in its treatment of behaviors.Source: §6.1 · p. 11

6.2 Automated Game and Environment Generation

  • Prior systems generate game descriptions, adapt environment configurations, build environments with coding agents, or synthesize reward functions and simulation code; Fig links generation to behavioral evaluation.In Fig, the behavior of interest guides multiple stages of game-mechanism design and evaluation generation.Source: §6.2 · p. 12

6.3 Automated Behavior Evaluation

  • Petri, Bloom, and Gram automate behavioral evaluation, while Fig focuses on multiplayer strategic interaction and adds fine-grained mechanical checks alongside rubric judgments.Fig leverages game rules and opponent policies to elicit and measure behaviors.Source: §6.3 · p. 12

7 Conclusion

The authors frame Fig as mechanism design for behavioral elicitation and measurement, while noting limits in environment coverage, coding-agent delegation, and measurement noise.

7 Conclusion

  • Fig constructs evaluations that elicit target behaviors, with effective components and evaluators that achieve high human agreement.The conclusion frames behavioral elicitation as mechanism design: select environments, modify rules and incentives, and evaluate resulting rollouts.Source: §7 · p. 12
  • The authors identify imperfect environment distributions, coding-agent delegation, and rule-confound measurement noise as current limits, and point to adaptive opponents and realistic settings as future extensions.They describe programmable mechanisms as a path toward scalable evaluation and training of more prosocial systems and a better science of mechanism-to-behavior provenance.Source: §7 · p. 12

A.1 Decomposition, Selection, and Intervention

  • The decomposer produces a behavior definition and up to 15 sub-behaviors, while the selector classifies catalog games and makes eligible entries available for intervention design.The catalog contains 135 imported text games; Figure 6 shows their distribution by incentive structure and information.Source: §A.1 · p. 15, Figure 6 · p. 15
  • Intervention design identifies a behavior opportunity and focal role, proposes evaluation axes, and can alter game parameters, rules, briefings, or opponent behavior.Table 6 illustrates briefing, opponent, and engine interventions, with attempt, execution, and success differences across intervention-control pairs.Source: §A.1 · p. 15, Table 6 · p. 16

A.2 Game generation and deduplication

Fig generates new games and routes them through the catalog-game pipeline; implementations and opponent controllers are validated before behavior evaluation. The evaluator combines mechanical checks with rubric judgments.

A.2 Game generation and deduplication

  • 22 generated games yield 27 validated scenarios, with one representative game per behavior shown in Table 7.The generated games pass through relevance, intervention, implementation, and measurement stages used for catalog games.Source: §A.2 · p. 16, Table 7 · p. 16

A.3 Implementation agent

  • Implementation is accepted only after checks confirm the intervention works, original rules hold, illegal actions fail safely, and scripted policies produce distinct outcomes.The implementation agent works in a clean workspace and can read, but not modify, selected game source code.Source: §A.3 · pp. 16–17

A.4 Opponent controllers

  • Opponent specifications define each affected seat’s role, objective, required actions, triggers, and whether control is programmatic, LLM-based, or hybrid.In hybrid control, triggered programmed actions can ignore the LLM’s stated intentions; in the example, a concession still occurred after cooperation had ended.Source: §A.4 · p. 17

A.5 Evaluator construction and validation

  • Mechanical checks replay saved traces independently of rubric judgments, then combine verdicts; unavailable evidence or missing checks yield unknown verdicts.The rubric judge uses a behavior-specific rubric across games and cites trace evidence for each verdict.Source: §A.5 · p. 18

B Experimental Setup

The study evaluates six specified behaviors across model configurations, prompts, and catalog or generated scenarios. It estimates attempt rates and paired intervention lifts while accounting for missing labels and trial structure.

B Experimental Setup

  • The experiment targets cooperation, exploitation, deception, covert coordination, betrayal, and sabotage, using frozen definitions that exclude ordinary competent play.Five focal model configurations are listed in Table 9; all use OpenRouter at temperature 0.7 and a 2,048-token output budget.Source: B Experimental Setup · p. 18, §B.1 · p. 19, Table 9 · p. 19

B.2 Evaluation prompts

  • Exploration raises mean attempt rate from 47.3% under the win prompt to 57.1%, but cooperation declines by 2.3 points.The mean model gap changes from 9.8 to 9.5 points, with no clear evidence that exploration increases separation between models.Source: §B.2 · p. 19

B.3 Trials, seeds, and episode protocol

  • The main protocol schedules three trials for each of five models across 204 catalog-mutation and generated scenarios, totaling 3,060 episodes.Refusals, errors, incomplete episodes, and insufficient evidence are retained as missing labels rather than rerun or replaced.Source: §B.3 · pp. 19–20
  • Matched-control pairs hold focal model, trial index, and decision budget fixed, while non-intervention components remain unchanged.The auction example changes mechanics, disclosure, and opponent policy together, so its comparison measures the full intervention package.Source: §B.3 · p. 20

B.4 Statistical inference

  • Attempt rates average determinate trials within models, models within scenarios, and scenarios within behaviors, with equal weight for each behavior.The primary 95% bootstrap interval for the mean attempt rate is 45.2% [40.5, 49.7].Source: §B.4 · p. 20
  • Across six behaviors, interventions raise attempt rate by 10.9 percentage points [4.2, 18.1], while execution lift is 7.3 [-0.2, 15.2] and success lift is 7.8 [1.3, 14.9].The execution-rate interval includes zero.Source: §B.4 · p. 20
  • Missing-label sensitivity bounds place the main attempt rate at [40.6, 53.8]%; paired attempt lift remains positive when missing labels are assigned against intervention.The paired attempt-lift bound is [1.5, 16.4] percentage points under opposite assignments to intervention and control labels.Source: §B.4 · p. 20

C.1 Evaluator calibration

Human review calibrates evaluator judgments and opponent realism across catalog and generated games. Agreement varies across these tasks, and realism judgments are reported as noisy and subjective.

C.1 Evaluator calibration

  • Human-human agreement on catalog evaluator cases is 17/20 (85.0%); among 49 cases both evaluators answered, generic and bespoke judges agree with humans on 43/49 (87.8%) and 44/49 (89.8%), respectively.Two faulty setups were excluded, leaving 57 cases across 37 games.Source: §C.1 · pp. 20–21

C.2 Human calibration of opponent realism

  • After adjudication, 13/30 scripted reference episodes and 29/30 hybrid-controller episodes were rated realistic.Before adjudication, reviewers agreed on 14/29 baseline and 18/28 hybrid episodes with paired labels; the paper characterizes realism as a noisy subjective judgment.Source: §C.2 · p. 21

C.3 Evaluator calibration for Generated Games

  • In an older generated-game comparison, generic judging agrees with 48/54 human ratings (88.9%), versus 45/54 (83.3%) for bespoke judging.The comparison used google/gemini-3.7-flash at temperature=0 and excluded six unclear ratings.Source: §C.3 · p. 21

D.1 Construction funnels and generated coverage

The construction funnels report how many catalog behavior–game pairs and generated games produce validated scenarios. Generation also adds mechanism templates and sub-behavior IDs beyond catalog mutations.

D.1 Construction funnels and generated coverage

  • The catalog funnel retains 240 accepted designs, 183 implementations, and 177 validated scenarios from 436 relevant game–behavior pairs.The 177 validated scenarios are 73.8% of accepted designs and 40.6% of candidate pairs; Figure 7 shows the funnel and token costs.Source: §D.1 · p. 22, Figure 7 · p. 22
  • Of 60 generated candidate games, 27 become validated scenarios across 22 games, while 105 designs are accepted.Twenty-nine scenarios have executable implementations and 28 pass mechanical checks.Source: §D.1 · p. 22, Table 13 · p. 22
  • Generated scenarios add 24 mechanism templates and four sub-behavior IDs absent from catalog mutations.The catalog has 154 mechanism templates; Table 14 counts templates and IDs within behavior before summing.Source: §D.1 · p. 22, Table 14 · p. 23

D.2 Results by behavior

Across valid Fig scenarios, models attempted the target behavior in 45.2% of episodes on average. Model separation was clearest for betrayal, while exploration prompting showed no clear increase in overall separation.

D.2 Results by behavior

  • 45.2% [40.5,49.7] was the mean attempt rate, equally weighted per behavior, across 2,687 determinate labels from 3,060 scheduled episodes.Pooled scenario attempt coverage was 151/204 (74.0%); equal-weighted behavior-specific coverage was 71.0%.Source: §D.2 · p. 23

D.3 Results by model and objective prompt

  • 34.8 points: GPT-5.6 Sol exceeded DeepSeek V4 Pro on betrayal; sabotage passed the six-behavior omnibus correction, but no individual sabotage pair survived the 60-pair correction.The other four behaviors did not pass the corrected omnibus test; comparisons used trials with labels for all five models.Source: §D.3 · p. 23
  • The mean absolute model-pair gap was 9.8 points under the win prompt and 9.5 under exploration, a change of −0.2 points [−2.5, 2.9].The study found no clear increase in overall model separation; these profiles measure attempts under tested scenarios and inference settings, not general performance or safety.Source: §D.3 · p. 23

D.4 Paired intervention lift

For scenarios with valid controls and labels in both arms, interventions increased attempt rates on average, with smaller estimated lifts for execution and success. The section also describes prediction methods and rubric-based behavioral labeling.

D.4 Paired intervention lift

  • The paired comparison includes only scenarios with a valid control and labels in both arms; interventions that create the behavioral opportunity may lack a meaningful control.Controls use validated adapters and may differ from the original game's rules or opponent policy.Source: §D.4 · p. 23
  • With equal weight per behavior, attempt-rate lift was +10.9 points [4.2,18.1]; execution-rate and success-rate lifts were +7.3 [−0.2,15.2] and +7.8 [1.3,14.9] points.The aggregate attempt estimate used 800 jointly labeled pairs across 57 scenarios.Source: §D.4 · pp. 23–24, Table 15 · p. 23

D.5 Predicting behavioral attempts from game descriptions

  • The prediction dataset covers 177 catalog-mutation scenarios from 80 source games, split by source game, with 934 test labels from 1,080 episodes in 36 scenarios.After validation, the predictors were refit on 64 non-test games using 3,729 labels.Source: §D.5 · p. 24
  • Forecasts use briefing and setting information, not observations or outcomes, and compare prevalence controls with text-feature, Qwen3-4B, and Kimi-K3 predictors.Binary episodes are scored with (p − y)^2, averaged across trials, games, and six equally weighted behaviors; missing labels are excluded.Source: §D.5 · p. 24

E Frozen Rubrics and Prompt Availability

  • Construction-prompt templates are available in the code repository, alongside generic rubric prompts and the six frozen behavior rubrics used for reported outcomes.The generic rubric author prompt asks for behavior-level criteria grounded in the frozen definition and decomposition.Source: E Frozen Rubrics and Prompt Availability · p. 24, §E.1 · p. 24

E.1 Generic rubric author

  • The generic rubric separates attempt, execution, and success: a concrete rejected or failed try can count as an attempt, while execution and success require the action and its consequence.Score changes or self-reports alone do not establish these labels.Source: §E.1 · pp. 24–25

E.2 Generic rubric judge

  • The judge treats game text and player messages as data, distinguishes player-visible observations from privileged audit state, and returns null when evidence is insufficient.It judges attempt, execution, and success independently and cites evidence for definite conclusions.Source: §E.2 · p. 25

E.3 Frozen frontier behavior rubrics

  • The six frozen rubrics operationalize behavior-specific criteria; for example, covert coordination requires concealed signaling toward a shared objective, not ordinary competent play.The rubric also distinguishes attempts, execution, success, and unknown outcomes.Source: §E.3 · pp. 25–26
Loading 2610.28891v1…