PaperlayerView exact revision on arXiv

Source-linked AI summary

Frontier Autolab: Organizational Memory, Adversarial Dissent and Temporal Leakage in Multi-Agent LLM Firms Across Fifty Years of Technological Change

Bravish Ghosh

arXiv:2609.36739v1cs.CEcs.AI

TL;DR

Whether LLM organizations inherit, avoid, or amplify human firms’ long-run failures is unknown because existing evaluations do not run long enough to observe them. The paper formulates temporally gated organizational decisions with memory and dissent, implements them in Frontier Autolab, and records four complete trajectories. Boards built the layer their assets could reach (including ranking in 1996 and neural-network scale in 2020).

  • Problem

    Whether LLM organizations inherit, avoid, or amplify human firms’ long-run failures is unknown because existing evaluations do not run long enough to observe them.

    Source: §1 · p. 1

  • Method

    The paper formulates temporally gated organizational decisions with memory and dissent, implements them in Frontier Autolab, and records four complete trajectories.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Boards built the layer their assets could reach (including ranking in 1996 and neural-network scale in 2020).

    Source: §1 · p. 2

  • Takeaways & Limitations

    Long-horizon evaluation reveals organizational effects of critic and memory design, while historically scored results require safeguards against leakage and self-judging.

    Source: §8 · p. 12

    The four exploratory trajectories use only two models, later runs were probably exposed to the first, and forecast eras have no ground truth.

    Source: §10 · p. 13

Abstract

from arXiv · show

Multi-agent LLM systems are increasingly structured like organizations, with roles, critics and shared memory, yet they are evaluated on tasks that last minutes. We ask how such an organization behaves when the ground it stands on keeps moving. Frontier Autolab is a long-horizon testbed in which one simulated firm, voiced by sixteen role personas and a dedicated Red Team, must re-found itself in nine technology eras from 1990 to 2040. Each era is temporally gated: the firm decides from a dated briefing, a historian-judge then reveals what happened and scores the decision on a five-dimension rubric, and lessons enter a persistent Playbook. Six eras are scored against history, one against the live market and two are open forecasts. Across four trajectories (36 era decisions, 180 subscores) we find a consistent foresight-commitment gap: in all 24 historically scored eras the judge rated the firm's recognition of the coming shift above its choice of where to build (mean gap 1.9 points on a 10-point scale), because boards chose the layer their existing assets could reach. Organizational design shaped long-run character. A Red Team armed with numeric kill gates produced fifty years of gated pilots and no product, and the rubric rated this firm highest; firms whose memory stored market-structure lessons pivoted every era, while a firm whose memory stored only validation procedure kept one method throughout. We also show why such results are hard to trust. Scores rise across eras in every run while the judge's own hindsight subscore falls (within-run r = -0.58), so apparent learning is confounded with recall of history, and we trace further distortions to self-judging, briefing selection and score aggregation. We release all records and an API harness, and specify fictional and post-cutoff eras that would turn the testbed into a benchmark.

1 Introduction

Frontier Autolab tests whether organizations of LLM agents inherit long-run failures of human firms, which short-task evaluations cannot reveal. Across four trajectories, the firm repeatedly recognized technological shifts better than it chose where to build, while its design shaped its long-run behavior and scores were confounded by hindsight.

Evaluation

  • The testbed follows one simulated company through nine technology eras, with departments proposing strategies, a Red Team challenging them, a board deciding, and a judge scoring.Six eras are scored against history, one against the live market, and two are forecasts; lessons accumulate in a Playbook that later eras must cite.Source: §1 · pp. 1–2

Findings

  • In all 24 historically scored eras, judges rated frontier accuracy above layer choice: firms named shifts but built where existing assets could reach.The introduction identifies hypertext, click-based ranking, and neural-network scale as examples of shifts named in memos.Source: §1 · p. 2
  • A Red Team using numeric kill gates produced fifty years of gated manual pilots without shipping a product, yet the rubric rated that firm highest.The rubric rewarded discipline and penalized hindsight.Source: §1 · p. 2
  • Memory content shaped company identity: market-structure lessons accompanied a pivot every era, while validation-only lessons accompanied just three identity changes in nine eras.Source: §1 · p. 2

Measurement

  • Scores rose across eras in every run while the judge’s hindsight subscore fell, confounding apparent learning with recall; self-judging, briefing selection, and score aggregation also distorted results.Source: §1 · p. 2

Contributions

  • The paper contributes a long-horizon organizational-evaluation formulation, an open testbed, four documented trajectories, behavioral findings, and an audit with a proposed benchmark design.Source: §1 · p. 2

2 Related work

Prior work studies organizational learning, multi-agent LLM systems, forecasting, and LLM judges, but typically evaluates agent organizations over short task horizons. The paper formalizes a temporally gated sequence of organizational decisions, memory updates, and judging, including pathways for hindsight leakage.

2 Related work

  • Prior LLM-agent work studies role-based software, cooperation, debate, and accumulated lessons, while this paper measures strategic decisions across five simulated decades rather than short tasks.The related-work discussion distinguishes this organizational horizon from task duration measured in working hours.Source: §2 · p. 2
  • Forecasting benchmarks use resolvable questions, whereas strategic decisions require a judge; retrospective LLM evaluations also face contamination, uncertain cutoffs, and look-ahead bias.The paper identifies briefing selection and a judge labeling foresight after the fact as additional channels specific to organizational simulation.Source: §2 · p. 3

3 Problem formulation

  • A trajectory defines admissible information as events dated before each era and its outcome window as events from that era’s start until the next era.The E6 outcome window extends to the live era and is longer than the others.Source: §3 · p. 3, Table 1 · p. 4
  • The organization comprises role personas, a Playbook memory, and company state; its structured procedure maps the briefing, memory, and state to a decision, with departments proposing, a critic attacking, and an executive deciding.Source: §3 · p. 3
  • The judge observes decisions and, in historical eras, outcomes; it assigns subscores, updates company state and Playbook lessons, and authors the next briefing.Source: §3 · p. 3
  • Hindsight leakage occurs when a model’s parametric knowledge makes decisions depend on future outcomes; briefing-selection leakage occurs when outcome knowledge shapes which otherwise admissible facts appear.Source: §3 · p. 3
  • The formulation defines the foresight–commitment gap as frontier-shift rating minus layer-choice rating, and an identity pivot as adopting a different company identity from the prior era.A positive gap means the organization saw more than it acted on; pivot rate is the fraction of transitions that are pivots.Source: §3 · p. 4

4 The Frontier Autolab testbed

Frontier Autolab applies its organizational-evaluation formulation to nine eras, using a role charter and a repeated protocol of proposals, adversarial review, board decisions, and judging. Era-specific rules and rubrics govern decisions and scoring.

4.1 Eras and mandate

  • The nine-era trajectory spans training eras scored against history, a live-market audit, and two forecasts without an answer key.The firm’s mandate is reinvention toward the frontier.Source: §4.1 · p. 4

4.2 Roles

  • The charter defines sixteen personas in five groups and two external judges, with persona-to-model-context mappings varying by run.Source: §4.2 · p. 4

4.3 Per-era protocol

  • Each era has parallel departmental memos, Red Team attacks, a CEO decision, and a judge reveal or audit that updates the Playbook, company state, and next briefing.Rules require arguments for keeping the prior business, era-realistic buildable wedges, and Playbook citations from the second era onward; rubrics differ for training versus live and forecast eras.Source: §4.3 · pp. 4–5
  • In the protocol figure, the judge both scores the current era and selects facts for the next briefing, a dual role identified as a leakage channel.Source: Figure 1 · p. 5

5 Experimental setup

The experimental section reports four trajectories with different orchestration schemes, while noting they were not a controlled sample. It describes the per-era decision and judging loop and flags implementation and recording limitations.

5 Experimental setup

  • Four trajectories ran on 28–29 September 2026 and differed in orchestration, but the authors state they are not a controlled sample.The orchestration variation was used to observe which behaviors persisted across schemes.Source: §5 · p. 5
  • Run 001 used Claude Opus 5.5 in an agent environment, while Runs 002–004 used Luna 6 through OpenAI Codex; the same model voiced personas and judges within each run.The API harness separating player and judge models was not used for these runs.Source: §5 · pp. 5–6
  • Each era runs parallel department memos, Red Team critique, a CEO decision, judge reveal or audit, and updates to the Playbook, company state, and next briefing.The memos propose two theses each, and the CEO decision records what the company kills and logs dissent.Source: §5 · p. 6
  • The released API harness supports player–judge separation and ablations, but was not used for the reported runs, and environment details were not recorded.The harness partly implements the controlled version described by the authors.Source: §5 · p. 6
  • In Run 001, an agent-launch service timed out at the E6 board step, so the operator wrote the decision and informed the judge, which penalized the missing Red Team.The authors exclude nothing from analysis and flag Run 001’s E6 wherever relevant.Source: §5 · p. 6

6 Results: how LLM firms behave over fifty years

Across four trajectories, firms showed a gap between recognizing technology shifts and choosing where to build, while critic and memory designs shaped their long-run behavior.

6.3 Critics can freeze a firm

  • Run 004’s numeric kill gates produced fifty years of gated pilots and no product, yet earned the highest training mean (66.0).The judge rated the firm highly for hindsight avoidance and discipline, although it never reached where value pooled.Source: §6.3 · pp. 7–8

6.1 Overview

  • All 36 published era totals fell between 52 and 72 (mean 62.7), and training-era scores rose by 1.4 to 2.6 points per era in every run.The last three training eras beat the first three by 4.3 to 10.0 points; totals use different rubrics across training and live/forecast modes.Source: §6.1 · p. 6, Figure 2 · p. 8

6.2 The foresight–commitment gap

  • In 24/24 training eras, frontier accuracy exceeded layer choice (mean gap 1.88); boards often chose adjacent layers despite memos naming shifts.The gap was positive in every era; in five of six Run 001 eras, a memo named the shift the judge later identified as decisive.Source: §6.2 · pp. 6–7, Table 4 · p. 7, Figure 3 · p. 8

6.4 What the organization remembers

  • Firms with market-structure lessons re-founded themselves at every transition, while procedure-only memory changed identity three times in nine eras.Run 001’s Playbook accumulated technology and market-structure lessons; Run 004’s lessons covered validation procedures but no technology or market structure.Source: §6.4 · p. 9, Table 5 · p. 9

6.5 Convergent futures

  • By 2040, all four firms converged on accountability for delegated AI work, but the shared-model runs were likely exposed to Run 001.The authors interpret the convergence as a model prior about where value pools, not independent evidence about the future.Source: §6.5 · p. 9

7 Can the scores be trusted?

Scores rise across eras even as the judge’s hindsight subscore falls, leaving learning difficult to distinguish from recall. Briefing selection, self-judging, rubric choices, inconsistent aggregation, and non-independent runs further complicate interpretation.

7.1 Learning or remembering?

  • Across runs, era totals rose as hindsight subscores fell; pooled within-run correlation was r = −0.58.The common negative direction held across four runs and three orchestration schemes, though each per-run estimate rests on six points.Source: §7.1 · p. 10, Table 7 · p. 10, Figure 4 · p. 11
  • Historical eras cannot separate learning from recall, because later eras may be easier for the model to recall and score favorably.In Run 001 E6, an operator-written tripwire described GPT-3 five months before its release; eras the model cannot remember could distinguish the readings.Source: §7.1 · p. 10

7.2 Channels of distortion

  • A pre-era briefing can leak through selective emphasis: Run 004 foregrounded CERN’s 1989 hypertext proposal, and the judge wrote later briefings after seeing outcomes.The cited fact was admissible, but selecting it was not neutral.Source: §7.2 · p. 10
  • The judge was also the player model in every run, and Run 004 used the same root context to write board decisions and score them.In Run 001, the E8 judge’s kill case became the premise of the E9 world, so forecast eras partly scored the judge’s own scenario.Source: §7.2 · pp. 10–11
  • Counting hindsight discipline in the total rewards saying little about the future and partly produces the rubric’s caution attractor.Figure 5 compares published totals with the mechanical aggregate ˜T; published totals followed different rules across runs.Source: §7.2 · p. 11, Figure 5 · p. 11
  • Published totals used different aggregation rules across runs, so cross-run comparisons mix rules.Recomputed training means ranged from 62.3 to 66.0, narrower than the published 60.0–66.0 range.Source: §7.2 · p. 11, Figure 5 · p. 11

8 Discussion

The discussion connects long-run organizational behavior to design choices in critics and memory, and argues that historical evaluation needs safeguards against leakage and scoring distortions.

8 Discussion

  • Across fifty simulated years, a critic with hard numeric gates made the firm unable to act, while memory content shaped the kind of firm it became.The paper recommends calibrating critics against inaction as well as error, and auditing what memory stores.Source: §8 · p. 12
  • LLM firms reproduced documented patterns of human firms: near-competence commitments, timid choices under loss-averse evaluation, and routines shaped by encoded experience.The authors suggest agent organizations could serve as a manipulable laboratory if leakage is controlled.Source: §8 · p. 12
  • Historically scored evaluations should report leakage, separate briefing authors from scorers, compute totals in code, and fix caution treatment in advance.Each identified distortion was large enough on its own to change a naive comparison between runs.Source: §8 · p. 12

9 Toward a benchmark

The paper specifies a controlled benchmark design intended to distinguish learning from temporal recall and test organizational components under reproducible conditions.

9 Toward a benchmark

  • The proposed splits use historical eras to measure behavior under known leakage, fictional history to measure false-positive leakage, and post-cutoff eras to separate foresight from recall.The released harness partly implements this controlled version.Source: §9 · p. 12
  • Conditions vary the full organization, Playbook, Red Team, and single-prompt baseline across five to ten runs, with separate-model judges blinded to condition.Each run is intended to use a clean workspace; the design calls for at least two player models.Source: §9 · p. 12
  • The proposed metrics include a code-computed aggregate excluding hindsight, foresight–commitment gap, pivot rate, dissent accuracy, leakage measures, and human-rater agreement.They also include training-era slope and frontier capture.Source: §9 · p. 12
  • Five preregistered hypotheses test Playbook and organization effects, fictional-era leakage, positive foresight–commitment gaps, and Red Team effects on pivoting and frontier capture.The hypotheses predict higher training-era slope with a Playbook and higher pivot rate and frontier capture without a Red Team.Source: §9 · p. 12
  • The design requires about 46 model calls per run, or 900–1,850 calls across conditions, plus human rating time.Source: §9 · p. 12

10 Limitations

The study is exploratory, with limited models and likely exposure between later runs; its scores and labels depend on a hindsight-informed judge, and several outcomes lack ground truth.

10 Limitations

  • The four exploratory trajectories use only two models, later runs were probably exposed to the first, and no run instantiated sixteen independent agents.Runs 002 and 003 also have thinner records.Source: §10 · p. 13
  • All subscores and qualitative labels are judge-assigned with hindsight, while revenue and exit outcomes are the judge’s calibrated guesses compounded across eras.Source: §10 · p. 13
  • Forecast eras have no ground truth, live-era audits reflect web sources as of September 2026, and per-run correlations rest on six points.Source: §10 · p. 13

11 Conclusion

The conclusion argues that long-horizon agent organizations can be revealing, but rising scores alone cannot establish learning when hindsight also changes. It recommends eras the model cannot remember and scores it does not assign to itself.

11 Conclusion

  • Firms saw coming shifts but built beside them; a strong critic froze one firm for half a century while it retained the highest scores.Source: §11 · p. 13
  • The content of each firm’s memory set its character.Source: §11 · p. 13
  • Scores rose as hindsight crept in, so the conclusion recommends eras the model cannot remember and scores it does not assign to itself.The paper releases the testbed and a design for conducting such evaluations.Source: §11 · p. 13

Ethics statement

  • Simulated companies, financings, and outcomes are fictional; one Run 001 board decision was written by the human operator, and three personal-planning passages were removed before release.Source: Ethics statement · p. 13

A Full score table and subscores

This section documents the score aggregates, the board’s company identities by era, and the training-era judge’s scoring and briefing procedure.

A Full score table and subscores

B Company identities by run

  • Table 9 lists the company identity chosen by the board in each era.Source: Table 9 · p. 17

C Judge instructions (excerpt, Run 001)

  • The training-era judge scored five dimensions, including hindsight leakage, assigned an overall 0–100 era score, appended Playbook lessons, and wrote the next briefing using pre-era public information.The judge was instructed to describe real-world events and provide one-line justifications; full prompts are in the repository.Source: C Judge instructions (excerpt, Run 001) · p. 17

Related papers

Loading 2609.36739v1…