Beyond Perfect Teammates: Multi-Agent Collaboration with Unreliable Colleagues

Collaboration under distributed information

• UNCOBENCH tests whether coordinators obtain required evidence when colleagues return mis-scoped or incomplete information, or fail to respond.It distinguishes knowledge available within a team from knowledge used in a decision.Source: §2 · p. 3

Controlled diagnosis of multi-agent failures

• Controlled failure analyses include omitted information and failures to clarify, while resilience evaluations perturb collaborator messages, prompts, and routing [Jia et al., 2026].UNCOBENCH pairs reliable and faulty conditions on the same verifiable task and checks each defect’s decision relevance.Source: §2 · p. 3

Evidence adequacy and interactive recovery

• Evidence assessment must guide a subsequent recovery action, and UNCOBENCH tracks recovery alongside final accuracy to distinguish gathering failures from errors after evidence recovery [Wang et al., 2025a; 2026].Related work also examines silent tool errors, retrieval robustness, and clarification.Source: §2 · p. 3

3 AUTOMATED BENCHMARK CONSTRUCTION

• Each task assigns private records to specialized roles; the main agent gathers their evidence to evaluate three candidates and calculate a numerical result across four reliability settings.The pipeline’s first three stages establish the Reliable baseline before deriving paired unreliable settings.Source: §3 · p. 3, Figure 2 · p. 4

3.1 STRUCTURED TASK SPECIFICATION

• An LLM generates structured specifications covering candidates, roles, eligibility conditions, and calculation rules, with guidance encouraging scenario and computation diversity.Programmatic checks assess role-condition distribution, operand ownership, values, and units, returning invalid specifications for repair.Source: §3.1 · p. 4

3.2 TASK INSTANTIATION AND LOGICAL VERIFICATION

• A program instantiates candidate evidence, limits eligibility to at most one candidate, and computes the ground-truth answer.Answer-uniqueness and role-necessity checks ensure complete evidence determines one answer and each necessary role contributes indispensable information.Source: §3.2 · p. 4

3.3 NATURAL-LANGUAGE RENDERING AND FULL-INFORMATION VALIDATION

• An LLM renders each verified instance in natural language while preserving its facts and evidence contexts; a separate validator retains it only after three shuffled-evidence runs match the gold answer.This full-information check complements logical verification of the structured instance.Source: §3.3 · p. 5

3.4 UNRELIABLE SETTINGS AND RECOVERY MECHANISMS

• Misaligned changes evidence context, Incomplete withholds precision while remaining truthful, and Missed withholds the initial response; each requires a different corrective action.The 1,000 task groups pair all four settings while retaining the same decision problem and ground-truth answer.Source: §3.4 · p. 5

3.5 GENERATION DIVERSITY AND VALIDITY

• Retained tasks span 14 application domains and eight numerical operations; verification rejected 296 of 1,296 candidate groups (22.8%), and retained tasks recovered the programmatic gold answer in repeated shuffled validation runs.Human evaluation also found strong agreement with judgments of logical coherence and solvability.Source: §3.5 · p. 5

4.1 MODELS AND PROTOCOL

• The evaluation covers ten open- and closed-source main-agent models, with DeepSeek-V4-Flash-0731 selected as the role-agent model for instruction following and response-policy adherence.Models are accessed through provider APIs or deployed locally using vLLM.Source: §4.1 · p. 6
• Each main agent receives the same tool schema, role descriptions, and answer format, and is evaluated on 4,000 tasks across four settings.A tool call can query multiple roles in one round or wait without making a new request.Source: §4.1 · p. 6

4.2 EVALUATION METRICS

• Accuracy requires both the final decision and numerical result to match the gold answer; rounds count main-agent actions, including actions that query multiple roles.Mean rounds are calculated across tasks.Source: §4.2 · p. 6

5.1 PERFORMANCE ACROSS COLLABORATOR PROFILES

• Across models, Misaligned causes the largest accuracy drops (6.0%–78.8%), Incomplete yields losses of 1.0%–30.0%, and Missed reduces accuracy by only 0.2%–3.3%.The share of tasks correct under Reliable that remain correct under unreliable collaboration increases from Misaligned to Missed.Source: §5.1 · p. 7, Figure 3 · p. 7

5.2 MODEL BEHAVIOR UNDER UNRELIABLE SETTINGS

• Claude Opus 5 shows fact recovery followed by a correct answer in 78.5% of Misaligned traces and resolution-targeted behavior in 90.4% of Incomplete traces, while Qwen3.5-9B and Mistral Small 4 rarely follow up.The association between baseline performance and checking is not monotonic: DeepSeek-V4.1-Flash has 92.6% Reliable accuracy, yet 86.7% of its Misaligned traces contain no corrective follow-up, and 7.3% show recovery followed by a correct answer.Source: §5.2 · p. 7
• Under Incomplete, many Gemini 3.8 Flash and DeepSeek-V4.1-Flash traces use unclassified follow-up, suggesting generic queries that do not identify the missing or unclear information.In controlled interventions, all six models improve from None to Action under Incomplete; under Misaligned, four approach Reliable performance as interventions become more specific, but two remain far below it even under Action.Source: §5.2 · pp. 8–9
• Missed-response recovery adds rounds mainly through repeated attempts to obtain evidence, but passive waiting and redundant contacts also contribute.Wait actions add 1.46 rounds per task for GPT-5.6 Sol and 0.51 for Qwen3.8-27B; 43% and 37% of their cases involve more than one contact with the designated role, respectively.Source: §5.2 · p. 8

5.3 Mixed-SETTING STRESS TEST

• Mixed combines misaligned evidence, an omitted required value, and an overlooked initial request, requiring the main agent to resolve multiple failures in one task.Across the six models shown, Mixed accuracy ranges from 5.8% to 45.2%; GPT-5.6 Sol and Gemini 3.8 Flash score 19.6% and 45.2% under Mixed despite 97.4% and 99.1% under Missed.Source: Figure 6 · p. 9, §5.3 · p. 9

• UNCOBENCH is a controlled benchmark of 1,000 paired tasks that vary collaborator reliability while preserving the underlying decision problem and ground-truth answer.Source: §6 · p. 9
• Its automated construction pipeline combines LLM-based scenario generation with executable checks of answer uniqueness, role necessity, and perturbation validity.Source: §6 · p. 10
• Across ten models, comparable Reliable performance can mask robustness differences, and traces show omitted corrective follow-up plus waiting and redundant queries during recovery.The paper concludes that reliable collaboration requires verifying response scope, identifying information gaps, tracking unresolved requests, and taking targeted actions to recover and integrate evidence.Source: §6 · p. 10

A.1.1 GENERATION PROMPTS

• The generator specifies three candidates, 5–7 roles, 5–7 eligibility conditions, and a numerical rule whose operands come from at least two roles.It assigns candidate-specific facts and computes the ground-truth answer later.Source: §A.1.1 · p. 13

A.1.2 REFERENCE SCENARIOS AND NUMERICAL OPERATIONS

• Eight reference scenarios, recent-task summaries, and varied numerical operations guide diversity while allowing new scenarios beyond the references.The references are optional inspirations, not a whitelist or quota.Source: §A.1.1 · p. 13, §A.1.2 · p. 14

A.1.3 STRUCTURAL VALIDATION AND REPAIR

• Specifications that fail structural checks receive validation feedback and one repair attempt, and must pass before instantiation.The checks include distributing eligibility conditions across at least four roles and assigning numerical operands to at least two roles.Source: §A.1.3 · p. 15

A.2 DETAILS FOR TASK INSTANTIATION AND LOGICAL VERIFICATION

• Instantiation assigns concrete candidate-specific evidence values and computes the answer, retaining instances only when they pass answer-uniqueness and role-necessity checks.Instances with multiple eligible candidates are rejected; removing a necessary role’s evidence must leave the answer undetermined.Source: §A.2 · p. 15, §A.2.1 · p. 16, §A.2.2 · p. 16

A.2.1 ANSWER UNIQUENESS.

• Answer uniqueness requires each instantiated task to have exactly one eligible candidate or no eligible candidate; multiple eligible candidates cause rejection.A unique eligible candidate’s answer includes its identity and numerical result.Source: §A.2.1 · pp. 15–16

A.2.2 ROLE NECESSITY.

• Role necessity is checked by hiding each necessary role’s facts and enumerating possible completions; the role must affect whether the output is determined.The check covers conclusion, candidate identity, and numerical result.Source: §A.2.2 · p. 16

A.3.1 PROMPTS FOR NATURAL-LANGUAGE RENDERING

• Natural-language rendering preserves instantiated facts and renders the main task and role records separately without revealing the ground-truth answer.The program adds the calculation rule, record keys, and candidate directory.Source: §A.3.1 · p. 16

A.3.2 PROMPTS FOR FULL-INFORMATION VALIDATION

• A task is retained for the Reliable setting only when a separate validation LLM recovers the ground-truth answer in three runs with independently shuffled evidence.The validator receives the query, eligibility conditions, and all role records.Source: §A.3.2 · p. 17

B.1 RELIABLE

• The task requires all five appointment conditions, including 12 years of underground hardrock experience, March 2026 clearance, and availability by 1 August 2026.The unique eligible candidate’s grant and field establishment allowance are summed in AUD, with rounding applied only to the final result.Source: TASK INSTRUCTIONS · p. 18
• Marcus Vance is the only eligible candidate, and the correct decision is “Appoint” with a 60,000 AUD package.The package sums his 45,000 AUD relocation grant and 15,000 AUD field establishment allowance.Source: PRIVATE EVIDENCE BY ROLE · p. 19

B.2 MISALIGNED

• Using a failed April clearance instead of the required March clearance changes Marcus’s correct “Appoint” answer and 60,000 AUD result to “Do not appoint” with a null result.The reply supplies a clearance from the wrong time period.Source: §B.2 · p. 20

B.3 INCOMPLETE

• Reporting an experience interval of 5–30 years instead of 17 leaves the 12-year requirement unresolved, and treating that uncertainty as failure can produce “Do not appoint” with a null result.The correct answer remains “Appoint” Marcus Vance with 60,000 AUD.Source: §B.3 · p. 20

C.1 DIVERSITY ANALYSIS

• The 1,000 task groups span 14 application domains, with the largest domain accounting for 15.5% and examples ranging from healthcare to manufacturing.Each task group is counted once across its paired settings.Source: TASK DIVERSITY · p. 20, Figure 7 · p. 20
• Application contexts vary in their conditions and requested quantities, from semiconductor packaging capacity to rocket-engine test-facility operating costs.The examples use different operational measures to frame their decisions.Source: TASK DIVERSITY · p. 20
• Each of the eight numerical operations accounts for 11.0%–13.8% of task groups, and 12 of 14 domains include all eight operations.The benchmark represents 108 of the 112 possible domain–operation combinations.Source: NUMERICAL-RULE DIVERSITY · p. 21, Table 4 · p. 21

C.2 VALIDITY ANALYSIS

• 1,000 of 1,296 candidate groups were retained (77.16%), after the pipeline rejected 7 at specification validation, 7 at logical verification, and 282 at rendering and full-information validation.All 3,000 full-information validation runs for retained baselines matched the programmatically computed answers.Source: Table 5 · p. 21, PASS RATE ANALYSIS · p. 21
• The validation stages check structural consistency, logical answer uniqueness and role necessity, and whether rendered tasks are solvable with complete evidence.The audit sampled 100 candidates from each stage and included both accepted and rejected candidates.Source: HUMAN VALIDATION OF GENERATED TASKS · p. 21, PASS RATE ANALYSIS · p. 21

HUMAN VALIDATION OF GENERATED TASKS

• Human and automated validation agree on 93%–100% of audited candidates, with every disagreement involving an automated rejection that human reviewers accepted.The authors interpret this pattern as evidence that the pipeline is more conservative in accepting tasks.Source: HUMAN VALIDATION OF GENERATED TASKS · p. 22, Table 6 · p. 22

D ANALYSIS OF ROLE AGENT VALIDITY

• A Gemini-3.8-Flash judge evaluates replies for factual fidelity against records and information coverage against the items required by each query and response policy.Fidelity is reported as the pass rate; coverage is the average proportion of required items conveyed.Source: D ANALYSIS OF ROLE AGENT VALIDITY · pp. 22–23
• DeepSeek-V4-Flash-0731 delivers response quality comparable to GPT-5.6 Sol at substantially lower API cost and faster inference, motivating its selection for repeated role interactions.GPT-5.6 Sol achieves the highest scores on both factual fidelity and information coverage.Source: D ANALYSIS OF ROLE AGENT VALIDITY · p. 23, Table 7 · p. 23
• Table 7 compares role-agent models across main-agent models; the reported results show consistently high fidelity and coverage, including for Qwen3.7-Flash.The authors interpret the results as evidence that answering queries from supplied private records is a relatively modest demand for the evaluated models.Source: Table 7 · p. 23, D ANALYSIS OF ROLE AGENT VALIDITY · p. 23

E CAUSAL INTERVENTION ANALYSIS OF COMMUNICATION RECOVERY

• The interventions target whether failures arise from noticing recovery is needed, locating the interaction, diagnosing its defect, or converting diagnosis into an effective repair.The task, records, prior interaction history, tool schema, decoding configuration, and interaction budget are held constant across arms.Source: E CAUSAL INTERVENTION ANALYSIS OF… · p. 23
• Five nested arms progressively provide recovery existence, interaction location, defect diagnosis, then corrective action while preserving task and interaction conditions and withholding the ground-truth answer.Even with Action, the main agent must execute the repair, process new evidence, revise reasoning, and produce the final answer.Source: E CAUSAL INTERVENTION ANALYSIS OF… · p. 23, Table 8 · p. 24
• Figure 8 shows heterogeneous recovery: four models approach Reliable under Misaligned Action while two remain far below; all six improve under Incomplete Action, and five of six are near-reference under Missed None.Muse-Glimmer-30B is the Missed exception: localization recovers most of its initial deficit, and subsequent intervention arms bring it to the Reliable level.Source: E CAUSAL INTERVENTION ANALYSIS OF… · pp. 23–24, Figure 8 · p. 24
• Under Misaligned, gains with specific help indicate monitoring and repair-selection bottlenecks, while failures persisting under Action point to downstream repair, revision, integration, or task-solving limits.Under Incomplete, making the required response explicit generally enables repair; near-reference results under Missed None suggest overt non-response is usually recognized without assistance.Source: E CAUSAL INTERVENTION ANALYSIS OF… · p. 24