OfficeTown: Generative Organizations for Evolving Agent Intelligence

1 INTRODUCTION

• Professional assignments emerge from ongoing activity: deliverables are reviewed and revised, decisions prompt follow-up requests, and project materials inform later work.Existing benchmarks and environment-synthesis methods provide valuable resources but typically treat tasks as isolated instances.Source: §1 · p. 1
• OfficeTown Bench contains 406 tasks and 4,836 evaluation criteria from 23 simulated organizations across eight domains, refined with domain-expert feedback.Training resources include 2,202 SFT demonstrations and 828 OPD task–environment pairs.Source: §1 · p. 2
• Training on OfficeTown data improves Qwen3.5-9B on external professional-work benchmarks, while frontier models retain substantial headroom on OfficeTown Bench.On 79 shared GDPval tasks, its rubric score rises from 65.75 to 73.65 after SFT and OPD.Source: §1 · p. 2
• OfficeTown models professional organizations as systems where persistent, permissioned state shapes subsequent work.Professionals coordinate, delegate bounded assignments to assistants, review deliverables, and follow up; artifacts and decisions update organizational state.Source: §1 · pp. 1–2

2.1 ORGANIZATIONS, ROLES, AND AUTHORITY

• An organization is initialized from a natural-language description that guides its objectives, professional roles, and reporting relationships.Roles specify continuing responsibilities and authority; the organization begins with a seed of public-source materials and organization-specific synthetic records.Source: §2.1 · p. 3

2.2 PERSISTENT AND PERMISSIONED ORGANIZATIONAL STATE

• Organizational state persists across work and is not globally observable: permissions, ownership, sharing, and confidentiality shape each member’s view and authority.Saving, acceptance, and explicit sharing are distinct; saved work need not be accepted or visible to others.Source: §2.2 · p. 3

2.3 COORDINATION, DELEGATION, AND REVIEW

• The simulation activates professionals asynchronously as requests, messages, meetings, follow-ups, or assistant deliveries arrive, allowing several matters to progress concurrently.Members clarify objectives and allocate responsibilities; principals specify delegated objectives, inputs, constraints, and deliverables, then accept work, request revisions, or seek more information.Source: §2.3 · pp. 3–4

2 OFFICETOWN: A GENERATIVE ORGANIZATIONAL SIMULATION FRAMEWORK

• OfficeTown’s workflow links organizational setup, retained materials and follow-up assignments, and task packaging into executable environments for training and evaluation.Revisions build on submitted artifacts and review comments; accepted results can inform authorized colleagues’ work, while unresolved questions remain available for later investigation.Source: §2 · p. 3, Figure 2 · p. 4, §2.3 · p. 4

2.4 FROM ONGOING WORK TO EXECUTABLE TASKS

• Each executable task captures the responsible professional’s available materials at assignment time, excluding later solutions and evaluator-only information.Attempts start in fresh workspaces initialized from those fixed inputs, so evaluation does not require replaying the organization’s entire history.Source: §2.4 · p. 4

3.1 BENCHMARK DESIGN

• OfficeTown Bench contains 406 tasks and 4,836 criteria from 23 simulated organizations across eight domains, with 139 distinct delegating-professional job titles.The benchmark includes 247 file-based tasks and 159 tasks involving service state.Source: §3.1 · p. 5
• Each task pairs a work request and executable environment with a task-specific rubric that agents cannot access at the start.Environments provide files, tools, and authorized organizational state.Source: §3.1 · p. 5

3.2 CONSTRUCTION AND EXPERT REVIEW

• Task environments retain relevant starting materials but exclude historical solutions and evaluator-only information; domain experts reviewed task realism, feasibility, and rubric quality.Seventy-one experts contributed feedback, and the released task and criterion sets are fixed for model comparisons.Source: §3.2 · p. 6

3.3 TRAINING DATA

• Training resources comprise 2,202 complete execution trajectories for supervised fine-tuning and 828 task–environment pairs for on-policy distillation.The trajectories include teacher reasoning, tool calls, observations, and final responses; the distillation pairs omit teacher response histories.Source: §3.3 · p. 6

4.1 EVALUATION ON OFFICETOWN BENCH

• Astra fulfills 88.8% of critical criterion weight on average, yet 20.8% of eligible tasks miss at least one critical requirement.Critical-requirement miss rates are 53.3% for Terra and 78.3% for DeepSeek-V4-Flash; Figure 5 also compares P0/P1 fulfillment and scores across overlapping input-file groups.Source: Figure 5 · p. 7, §4.1 · pp. 7–8
• Across five recorded budgets on the same 406 tasks, execution-framework scores differ: OfficeTown Executor reaches 61.9, Pi 48.6, and OpenCode 40.7 in Weighted Score.These runs use each framework’s native prompts and tools, not token-matched comparisons; Figure 6 also contrasts high-effort models and Flash effort settings.Source: Figure 6 · p. 7, §4.1 · p. 8
• GPT-6 Astra leads OfficeTown Bench at 74.4% Pass@1, followed by Grok-4.7 at 58.4% and Claude Opus 5.5 at 50.7%.Astra nevertheless leaves roughly one quarter of tasks incompletely delivered; Claude passes more tasks than Terra despite a lower Weighted Score.Source: §4.1 · p. 7

4.2 LEARNING FROM OFFICETOWN

• On 79 shared GDPval tasks, Qwen3.5-9B’s rubric score rises from 65.75 to 73.65 after SFT and OPD, a 7.90-point gain over Base.JobBench-Easy and APEX-Agents also improve across the reported stages.Source: §4.2 · p. 8, Table 3 · p. 8

4.3 HARNESS EVALUATION

• OfficeTown Harness has the highest reported point estimate in all five external-benchmark columns among OfficeTown, Codex, MetaGPT, and CAMEL.On research code, OfficeTown scores 46.40 versus MetaGPT’s 41.15 with 21.8% fewer reported tokens, but uses 3.49 times Codex’s tokens.Source: §4.3 · p. 8

Professional Work Benchmarks

• Professional-work benchmarks cover web, desktop, enterprise, cross-application, and occupation-level tasks; TheAgentCompany is closest in combining a simulated company, office applications, and model-driven colleagues.TheAgentCompany’s tasks are manually curated.Source: §5 · pp. 8–9

Environment and Task Synthesis

• Environment-synthesis systems generate executable tasks, environments, or training resources, while OfficeTown focuses on generating work through ongoing organizational activity.The paper distinguishes its focus on work generation from recursive self-improvement.Source: §5 · p. 9

Generative Agents and Organizational Simulation

• Prior agent and organizational simulations study social activity, interaction, policy-constrained tool use, or institutional learning; OfficeTown links persistent organizational work to executable environments.Roles, ownership, and sharing rules scope information and authority for assignments.Source: §5 · p. 9

6 CONCLUSION AND FUTURE WORK

• OfficeTown treats ongoing work as the source of tasks, context, and feedback; its conclusion reports a gap between partial fulfillment and complete delivery and training gains on external benchmarks.The authors identify improving fidelity and efficiency and validating simulated work against real practice as important future work.Source: §6 · pp. 9–10
• OfficeTown uses referenced public sources, adapted materials, and synthetic records rather than private customer records; simulated organizations are not authoritative occupational representations or settings for real-world decisions.The ethics statement also notes privacy, security, and operational risks from tool-using agents.Source: §6 · p. 10

A.1 CREATING AN ORGANIZATION FROM A NATURAL-LANGUAGE DESCRIPTION

• A natural-language brief initializes an organization with its scope, departments, roles, reporting relationships, and synthetic customers, grounded in occupational descriptions and supplied with retrieved, adapted, and synthetic seed materials.The runnable package initializes personal workspaces, authorized shared resources, and office services.Source: §A.1 · pp. 15–16

A.2 EXAMPLE ORGANIZATION AND CONFIGURATION

• Juniper combines professional references and an adapted accounting workbook with synthetic customer packets, while its configuration distinguishes generated properties from run settings.The customer packets include business records, control totals, source references, and unresolved assumptions.Source: §A.2 · pp. 16–17, Table 5 · p. 16

A.3 COMMUNICATION AND WORK COORDINATION

• Professionals coordinate through messages, meetings, and explicit work actions that assign outcomes, owners, materials, and deadlines; meeting notes alone do not approve or complete assignments.The illustrative liquidity meeting establishes required inputs, handoffs, forecast scenarios, and checks before execution.Source: §A.3 · p. 17, Figure 8 · p. 17

A.4 SESSIONS AND MEMORY

• Structured work records preserve commitments, discussions, and unresolved decisions outside active conversation, while optional session branches support inspection and return concise conclusions and actions to coordination.At simulated day boundaries, saved records and office state refresh the session; periodic compaction is optional and disabled by default.Source: §A.4 · pp. 17–18, Figure 9 · p. 18

A.5 WORKSPACES, TOOLS, AND ACCESS CONTROL

• Assistants work in temporary containers with writable principal workspaces and permission-filtered, read-only shared materials; committed changes persist only after runtime checks.Filesystem validation confirms what was saved, while professional review determines whether the content meets assignment requirements.Source: §A.5 · pp. 18–19, Figure 10 · p. 19

A.6 SIMULATION TIME AND RECOVERY

• The simulated clock slows or pauses as unfinished work accumulates, and recovery can restore local files and classifications but cannot reverse external actions such as sent email.Unfinished requests and incomplete reviews remain saved at shutdown rather than being marked finished merely because the target time was reached.Source: §A.6 · pp. 19–20

A.7 A SIX-MONTH OFFICE SIMULATION

• 497 files had accumulated by simulated day 181, up from 27 initial files, with 209 and 376 files recorded by days 60 and 120.Revisions and recorded renames count once, exact copies are merged at first appearance, and navigation READMEs, backups, and trash paths are excluded.Source: §A.7 · p. 20, Figure 11 · p. 21

B.1 RESOURCE MEMBERSHIP AND SEPARATION

• The benchmark contains 406 tasks and 4,836 final evaluation criteria, with separate membership records for benchmark tasks, SFT demonstrations, and OPD environments.The same fixed task and criterion sets underpin aggregate statistics, model comparisons, and case studies.Source: §B.1 · p. 22

B.2 EXPERT REVIEW

• 71 domain experts reviewed task realism, feasibility, and rubric quality; feedback is bound to 394 of 406 retained tasks, while 12 have model-based review without bound human annotation.Expert construction-stage records cover 1,051 candidate tasks and 32,274 rubric criteria.Source: §B.2 · p. 22

B.3 DOMAIN AND FILE-TYPE COVERAGE

• The eight domains cover all 406 tasks, and input coverage counts file families per task, including archive contents, with overlapping categories that do not imply every supplied file is necessary or used.Figure 12 shows the five most frequent delegating role titles per domain; Table 7’s carrier categories are not occupational or task-purpose labels.Source: §B.3 · p. 23, Table 6 · p. 23, Figure 12 · p. 24

B.4.1 FINANCIAL RECONCILIATION

• In the Riverbend reconciliation case, workpapers must trace three exception streams to records, explain unresolved differences, and keep unsupported conclusions open.The $42.50 card-clearing difference lacks settlement records and fee details; a balanced total alone does not establish resolution.Source: §B.4.1 · p. 25, Figure 13 · p. 25

B.4.2 MECHANICAL FIXTURE DESIGN

• In the Oakridge fixture-design case, two distinct concepts must preserve the workpiece, fit the equipment envelope, permit inspection access, and include reproducible geometry and calculations.Using pin holes as functional locators remains unresolved in the supplied evidence, and the recommendation is for further engineering review rather than production approval.Source: §B.4.2 · p. 26, Figure 14 · p. 26

C.1 METRICS AND SCORING

• Weighted Score averages each task’s fulfilled applicable criterion weight; tier weights preserve original shares after filtering, while unmet and unassessable criteria count zero.Pass@1 requires at least one applicable required criterion and first-attempt satisfaction of every such criterion.Source: §C.1 · p. 27

C.2 REASONING AND INTERACTION BUDGETS

• A logical turn is one model-response round; budget curves reuse recorded executions and assign zero before the recorded ending, rather than rerunning tasks at shorter budgets.Shared limits are 48 logical turns, 900 seconds, and 262,144 total output tokens; effort and framework panels vary requested effort or executor.Source: §C.2 · p. 27

C.3 COVERAGE AND RESOURCE ACCOUNTING

• Shared-task comparisons use paired observations, but repeated attempts and criteria within tasks—and tasks within organizations or projects—may be dependent.Equal turn limits or completion thresholds do not establish equal token or monetary cost.Source: §C.3 · p. 28

D.1 WEIGHTED SCORES BY DOMAIN

• Table 8 complements Pass@1 with rubric-weighted performance by domain under the Appendix C.1 scoring rules.Higher Weighted Scores are better; the table covers assurance, clinical, design, entertainment, finance, industrial, legal, and software work.Source: Table 8 · p. 28, §D.1 · p. 28

D.2 REQUIREMENT IMPORTANCE

• 88.8% of critical criterion weight is fulfilled by GPT-6 Astra on average, yet 84 of 404 eligible tasks have a critical miss.Critical-miss rates depend on applicable requirements and complement, rather than replace, benchmark scores.Source: Table 9 · p. 29, §D.2 · p. 29

D.3 INPUT-FILE GROUPS

• 40.7 on code-containing tasks and 28.0 on CAD-containing tasks are Kimi-K2.5’s Weighted Scores; model order also reverses between CAD and media groups.These are task-population differences, not isolated file-format effects: domain, requested work, and accompanying inputs vary together.Source: Table 10 · p. 29, §D.3 · p. 30

D.4 PASS RATES WITHIN A TURN BUDGET

• At 48 turns, Pass@1 is 16.5% for Low and 13.5% for High; OfficeTown Executor, Pi, and OpenCode reach 13.5%, 8.6%, and 4.9%, respectively.The binary-pass curves use the same records and ending-turn rule as Figure 6; the gap from weighted scores distinguishes partial fulfillment from complete delivery.Source: Figure 15 · p. 30, §D.4 · p. 30

E.1 TRAINING RESOURCES

• Preparation covers 6,172 cases, not a census of raw Live activity; SFT retains 2,202 complete trajectories and OPD provides 828 task–environment pairs.Service-dependent tasks fall outside the teacher-execution configuration rather than counting as failed executions.Source: §E.1 · p. 30

E.2 SFT AND ON-POLICY DISTILLATION

• Qwen3.5-9B receives SFT followed by OPD, which samples student trajectories, queries teacher token log probabilities, and uses negative log-ratio advantages for policy-gradient updates.OPD uses token-mean aggregation; task rewards and a separate reference-policy KL term are disabled.Source: §E.2 · p. 31
• OPD runs one epoch over all 828 training tasks, with validation disabled and no independent held-out set.The final batch of 100 tasks is retained.Source: §E.2 · p. 32, Table 13 · p. 32

• GDPval excludes tasks requiring live web search, and GPT-5.5 scores original rubrics as absolute scores rather than the official pairwise win rate.Source: §E.3 · p. 32
• JobBench uses 63 released Easy environments, native Grok-4.3 weighted grades, and a 7,200-second task budget without an additional turn cap.Later recovery phases exclude verified infrastructure-blocking time from active time.Source: §E.3 · p. 32
• APEX-Agents v1.1 allows 150 assistant turns and 16,200 seconds per task, while Terminal-Bench 2.1 excludes search-dependent tasks and uses native binary rewards.APEX verification uses an Azure DeepSeek-V4-Flash route rather than the published Fireworks snapshot.Source: §E.3 · p. 32

E.4 TRAINING DYNAMICS

• SFT progress on JobBench-Easy is non-monotonic, declining at step 300 and varying more at step 496; error bars describe run variation, not independent-seed uncertainty.Source: Figure 16 · p. 33, §E.4 · p. 33

E.5 BUDGET-CONDITIONED OUTCOMES

• Figures 17–20 summarize existing executions at five fixed turn budgets, using final outcomes after their recorded ending turns rather than reruns or intermediate-artifact rescoring.Their endpoints match Table 3; JobBench retains original partial grades when runs stop at the response-token limit.Source: §E.5 · p. 33, Figure 17 · p. 33, Figure 18 · p. 33
• In the full GDPval records, SFT + OPD reaches a higher final rubric score but takes more turns to reach 50.Source: Figure 18 · p. 33