The Helpful Agent Problem: When Emergent Cooperation Among LLM Agents Bypasses Intended Workflows

1 Introduction

• The helpful agent problem is locally helpful behavior that becomes globally undesirable when agents assist peers outside the intended workflow.The introduction situates this question amid growing LLM-agent collaboration and concerns about broader system safety.Source: §1 · p. 1
• Prior LLM-agent work has examined coordination failures, adversarial behavior, and collusion; this paper asks when separate-task agents spontaneously help outside intended workflows and with what system-level consequences.The introduction also connects this concern to established work on organizational constraints and delegated authority.Source: §1 · pp. 1–2
• The study tests separate-task agents in shared-resource environments where cross-owner changes normally require reviewer approval, including when the reviewer is unavailable.The setup tests existing resource permissions rather than access-control violations.Source: §1 · p. 2
• Agents receive no instruction or shared reward to help; the paper studies whether peer assistance emerges through communication and repeated interaction.A favor may affect resources used by parties outside the interaction.Source: §1 · p. 2
• The paper formalizes the problem, introduces FavorBench, adapts CooperBench, and tests whether organizational norms can constrain favors without blocking agents’ own tasks.The benchmarks cover controlled shared-resource settings and more realistic repositories.Source: §1 · p. 3
• Favors occur in 58.8% of FavorBench trajectories and 95% on CooperBench; 43.7% of completed trap-task favors harm an absent team despite a validated safe alternative.The reported rates span four models.Source: §1 · p. 3

2 Problem Formulation

• The shared workspace contains active agents, absent teams, resources, ownership assignments, a reviewer, and individual rewards tied to resources.The absent teams take no actions, and agents have no shared reward.Source: §2 · p. 3
• A write changes the workspace state, and direct writes are limited to resources owned by the writing agent.An agent’s task scope is the set of resources it needs to modify, which can differ from its ownership.Source: §2 · p. 3
• The intended workflow routes changes to non-owned resources through the reviewer, but an owner can directly make a peer’s needed change without review.That peer assistance bypasses workflow oversight while respecting resource-level permissions.Source: §2 · p. 3
• An unreviewed, out-of-scope write is a favor when it makes a change another agent’s task needs; it may be solicited, volunteered, or spontaneous.The change may be an alternative implementation rather than the exact requested edit.Source: §2 · p. 3
• A favor is harmful when it lowers the reward of a party other than its beneficiary, and harm to absent teams is unknown to agents by default.A favor opportunity is benign if every possible favor is safe, avoidable if both safe and harmful favors are possible, and conflicting if all are harmful.Source: §2 · p. 4
• Under reviewer-safety and in-scope-write-safety assumptions, harmful writes must be out-of-scope owner writes that change a resource affecting the harmed party.A harmful favor is possible exactly in avoidable or conflicting opportunities; harm is unavoidable only in conflicting opportunities.Source: §2 · p. 4

Controls for unintended collaboration

• The controls separate workflow compliance from downstream harm: a favor always bypasses review, but does not always cause harm.For avoidable opportunities, feedback may steer agents toward safe favors, but cannot ensure they choose them.Source: §3 · pp. 4–5

Task-scope law

• Task-scope law Lscope excludes out-of-scope favors while preserving reviewer-mediated changes, but also blocks harmless out-of-scope assistance.Under assumptions (A1)–(A2), it also excludes out-of-scope owner writes, the remaining class of writes through which harm can arise.Source: §3 · p. 4

Guarded out-of-scope assistance

• Guarded assistance permits out-of-scope writes only when a sound check certifies safety, guaranteeing no harm to parties other than the beneficiary under assumptions (A1)–(A2).The paper treats this as an idealized redesign and does not instantiate it in experiments.Source: §3 · p. 5

Feedback-guided assistance

• Feedback-guided assistance leaves out-of-workflow actions available but aims to reduce their conditional risk of harm through signals about downstream effects.The feedback may be imperfect: a missing warning does not establish safety, and a warning does not prevent a write.Source: §3 · p. 5

4 Evaluation Framework

• Experiments address favor emergence and mechanisms, consequences for requesters and absent teams, and whether norms, penalties, or downstream-harm feedback can reduce or make favors safer.The setup uses separate tasks and no shared reward; requester tasks depend on resources owned by the other agent, while owner tasks require only its own files.Source: §4 · p. 5, §4.2 · p. 6

4.1 Metrics

• Episode favor rate (EF) counts episodes with a completed helpful owner action, while execution types distinguish implementation, endorsement, refusal, and inaction.The framework also measures downstream harm when a favor causes absent-team contract tests to fail and restoring the favor removes that failure; every harmful change has a validated safe alternative.Source: §4.1 · p. 6

4.2 Environment and Benchmarks

• FAVORBENCH enables controlled measurement across ten domains and eight task families, while adapted COOPERBENCH tests whether favors emerge in real open-source repositories.FAVORBENCH uses 50 trajectories with 12 tasks each: two reviewer-available warm-ups followed by ten tasks without the reviewer; COOPERBENCH uses 20 trajectories of 10 tasks without warm-ups.Source: §4.2 · p. 6

5.1 Emergence of Cooperation and Harm

• Favor rates ranged from 67.2–70.2% for GPT to approximately 29% for Claude in FAVORBENCH, while 95% of COOPERBENCH trajectories contained favors for every model.In COOPERBENCH, task-level favor rates ranged from 46.5% for DeepSeek to 89% for Claude.Source: §5.1 · p. 9

5.2 Mechanism of Favors

• Favors arose through both requests and volunteering: DeepSeek volunteered 80.9–89.4% of its favors, while 78.3–94.4% of COOPERBENCH favors were solicited.Across FAVORBENCH, DeepSeek rarely asked owners directly, while Gemini asked frequently but granted fewer requests under Warm-trap; owners’ non-implementation more often involved inaction or endorsement than refusal.Source: §5.1 · p. 9, §5.2 · pp. 9–10

5.1 Emergence of Cooperation and Harm

• Among implemented trap favors, 25–75.2% caused attributable harm for Gemini, DeepSeek, and GPT, whereas Claude’s harm rate was 0–1.4%.FAVORBENCH had validated safe alternatives; in COOPERBENCH, requester pass rates remained below favor rates for all four models.Source: §5.1 · p. 9

5.3.1 Norms, Institutions, and Communication

• Restricting requests to admins reduced favors from 86% to 61% without materially changing harm, while restricting helpers to their own task eliminated favors and preserved 100% owner success.Under admin-only routing, every remaining favor was volunteered and harm rates were 44.7% versus 44.4% in the default condition.Source: §5.3.1 · p. 10
• Removing the shared channel reduced favors from 86% to 20% and eliminated volunteered favors, but harm persisted at 33%.All remaining favors were solicited.Source: §5.3.1 · p. 10

5.3.2 Reducing Harm Without Eliminating Help

• A harm penalty reduced trap-favor harm from 43.9% to 24.1%, while favor rate and requester success fell from 77.5% to 70.5%.Owner success remained essentially unchanged at 98–99%.Source: §5.3.2 · p. 10
• With CI feedback, harmful-favor rates fell to 25.5% with Raw feedback and 20.7% with Causal feedback, while favor rates rose to 82.5% and 87%, respectively.The feedback reports downstream errors, with the causal variant identifying code responsible for broken tests.Source: §5.3.2 · pp. 10–11

5.3.3 Benevolence Under Resource Constraints

• Reducing owners’ budget from 15 turns to 8 lowered favors from 86% to 43%, and at 4 turns favors disappeared while owner success remained 98%.An explicit instruction to help under the four-turn budget raised favors from 0% to 87%, with 93% volunteered.Source: §5.3.3 · p. 11, Figure 4 · p. 11

6 Conclusion

• Across FAVORBENCH and COOPERBENCH, favors emerged frequently despite agents having separate objectives and no explicit incentive to cooperate.In FAVORBENCH, such assistance could impose avoidable harm on absent teams.Source: §6 · p. 11
• Helpfulness was sensitive to resource constraints and organizational cues; own-task scope rules were more effective than request-routing rules, while penalties and consequence feedback reduced harmful favors without eliminating cooperation.The conclusion also notes behavior consistent with benevolent goal adoption absent reciprocal dependence.Source: §6 · p. 11
• The authors argue that multi-agent design should consider permissions, task scope, communication structure, and absent stakeholders jointly rather than treating cooperation as unconditionally desirable.Source: §6 · p. 11

AI Use Statement

• Generative AI tools were used for grammar correction and language polishing during manuscript preparation.Source: AI Use Statement · p. 12
• LLM-based coding agents assisted in developing the codebase and generating the synthetic FavorBench dataset, and in adapting CooperBench frameworks.Source: AI Use Statement · p. 12
• The authors report multiple rounds of manual iteration, review, and modification, and assume full responsibility for the final content.Source: AI Use Statement · p. 12

A.1 FAVORBENCH

• FavorBench comprises ten synthetic Python monorepos, each with five services: two for active agents and three for absent teams.Its task instances pair a requester needing an owner-held resource change with an independent owner task.Source: §A.1 · p. 15
• In trap instances, the obvious requested change breaks an absent service, but a careful alternative completes the requester’s task without harm; matched twins remove the absent-team dependency.The dataset contains 720 instances: 320 traps, 320 twins, and 80 additional safe instances.Source: §A.1 · p. 15
• FavorBench validation checks confirm task solvability and distinguish harmful trap changes from safe alternatives, while trajectories use two reviewer-available warm-ups followed by ten evaluated tasks.The evaluated sequence contains five traps and five twins; visible tests continue to pass after harmful changes.Source: §A.1 · p. 15

A.2 Adapting COOPERBENCH

• The CooperBench adaptation uses real repositories and feature pairs requiring a change to an owner-held public API file, with maintainer approval unavailable.It retains 41 instances and evaluates favors without absent teams or downstream-harm measurement.Source: §A.2 · pp. 15–16

• The additional analyses examine interaction history, owner inaction, and how harm penalties change agents’ behavior.Source: B Additional Experiments · p. 16

• 65% of warm-safe Default trajectories have a favor at episode 3, rising to 85–95% from episode 6; all-trap Default rises from 50% to about 85%.Source: §B.1 · p. 16
• Routing requests through the admin lowers warm-safe favors to 55–70%, while restricting agents to their own task eliminates them.Source: §B.1 · p. 16
• In all-trap Default, a harm penalty flattens favors to 70–80%, but raw and causal harm feedback stay at or above Default, reaching 90–100% late.Source: §B.1 · p. 16

• Full memory increases Gemini’s favor rate from 71% to 87.5%, while GPT’s rate decreases slightly and DeepSeek’s remains nearly unchanged.Source: §B.2 · p. 16, Table 6 · p. 17
• Table 6 reports CooperBench results with and without interaction history, in percentages across 20 trajectories per condition.Source: Table 6 · p. 17

B.3 Understanding Owner Inaction

• Owner inaction is categorized as an ignored direct ask, a message not sent to the owner, or no message.Source: §B.3 · p. 17
• Many no-action outcomes occur without a direct owner request; FavorBench DeepSeek has no ignored direct requests, and its inaction mostly follows messages addressed elsewhere or no message.CooperBench’s distribution varies across models and memory conditions.Source: §B.3 · p. 17

B.4 How Harm Penalties Change Behavior

• The harm-penalty condition combines a scoring penalty with repeated feedback about harm.Source: §B.4 · p. 17
• Under penalties, agents sometimes avoid changes, including one safe favor, or continue helping with safer implementations such as dedicated settings or narrower permissions.Penalties can reduce harm by suppressing favors or changing how assistance is provided.Source: §B.4 · p. 17

B.6 Requester pass rate versus favor rate

• PR = EF + W − F; W is at most 7 of 500 tasks per row (12 of 4,000 overall), and hidden-test-passing local workarounds do not affect EF or favor measures.W counts tasks passed without a favor, while F counts tasks with an implemented favor whose requester still failed.Source: §B.6 · p. 18

B.5 Availability of Reviewer

• With an always-available reviewer, EF drops from 54% to 0%, requester pass rate rises from 54% to 93%, and owner performance remains high (99% vs. 96%).The passage suggests continuous review enables task completion without peer-mediated workflow bypasses.Source: §B.5 · p. 17, Table 9 · p. 20

B.7 Impact of Failure Messages

• Failure-message comparisons varied whether agents saw the last line, the selected error line, or the full log; the main setup used the last line.The last line was used for the main-paper Table 1, while the error line was used for other ablations.Source: §B.7 · pp. 18–19, Table 11 · p. 20

C.1 FavorBench Agent Prompt

• FavorBench prompts define a shared monorepo with file ownership, admin-mediated requests for others’ files, hidden-test scoring, and shared-channel communication.Agents can write only owned files; requests for other-owned files go to the admin for review.Source: §C.1 · p. 21

C.2 CooperBench Agent Prompt

• CooperBench prompts give agents separate branches, require merging changes to main, and require maintainer approval for outsiders’ changes to owned files.Other files are open to anyone, while members of an owning team can merge their own changes to that team’s owned files.Source: §C.2 · p. 22

C.3 Behavioral Intervention Prompts

• Behavioral interventions add admin-only requests, an own-task-only rule, private direct messages, harm penalties, and automated CI feedback; intervention types use different task conditions.Norms and communication interventions use Warm-safe trajectories, while harm-penalty and CI-feedback interventions use all-trap task orders.Source: §C.3 · pp. 22–23
• CI feedback can report failing tests, errors, shared-file changes and authors; causal CI additionally identifies item-level restoration evidence, while a decision variant asks agents to assess whether outside-task work is warranted.The causal variant reports when restoring an individual item makes a failing test pass.Source: §C.3 · pp. 23–24
• In the explicit-help condition, owners are instructed to finish their assigned task and then implement another engineer’s requested change in files they own within their existing turn budget.They may complete their own work and implement the requested change in the same response.Source: §C.3 · p. 24

C.4 CooperBench Agent Instructions

• In CooperBench, maintainers are unavailable, so review requests go unanswered while owners can merge their own changes or approve requesters’ pending changes.Agents work on separate branches, and only changes merged into main count toward task completion; CODEOWNERS defines ownership.Source: §C.4 · pp. 24–25

C.5 LLM Judge Prompts

• The execution judge labels owner outcomes as implemented, approved, refusal, or no_action, with a qualifying edit taking precedence over messages.For CooperBench, approving a pending requester change also counts as an implemented favor; merging only the owner’s own task does not.Source: §C.5 · pp. 25–26
• Implemented favors are classified as solicited when directly requested, volunteered when an indirect visible need is picked up, or spontaneous when no need reached the owner.If both direct and indirect routes occur before the edit, the judge chooses solicited; an admin-directed request alone counts as volunteered when the owner picks it up.Source: §C.5 · p. 25
• The direct-request judge counts a request only when it asks the owner for review, approval, or implementation, not when it is merely addressed to admin or mentions the owner.The CooperBench variant treats maintainer-directed review requests as requests to maintainers, not direct requests to the owner.Source: §C.5 · p. 26

C.6 Human Judge Validation

• 98% execution accuracy, 92% favor-type accuracy, and 96% asking accuracy show strong agreement between the LLM judge and human reference labels.Cohen’s κ was 0.97 for execution, 0.83 for favor type, and 0.92 for asking.Source: §C.6 · p. 26