PaperlayerView exact revision on arXiv

Source-linked AI summary

Strategic Calibration in LLM Multi-Agent Systems: Audited Pivot Scoring for Truthful Elicitation

Jinhao Zhou, Zhaohan Zhang, Xijian Xu, Jun Wu

arXiv:2610.33728v1cs.GTcs.AI

TL;DR

Confidence reporting becomes strategic when it controls scarce downstream allocation, a problem not addressed by approaches that treat reports as passive predictions. The paper proposes APSM, an audited allocation-sensitive mechanism with a strict truthfulness guarantee under its assumptions. LLM agents inflate confidence under hard allocation incentives; with utility assistance, both CPS and APSM reduce distortion, with different metric advantages.

Abstract

from arXiv · show

In large language model (LLM)-based multi-agent systems, reported confidence increasingly determines the allocation of scarce downstream attention, computation, and decision influence. This makes reporting strategic: an agent may overstate the relevance of its contribution even when its underlying belief is well calibrated. Existing calibration methods largely assume passive reporting and therefore do not address strategic incentives created when confidence determines downstream allocation. We formulate strategic calibration as a reporting-layer incentive problem and propose the Audited Pivot Scoring Mechanism (APSM). APSM offsets platform-controlled allocation benefits, uses inverse-propensity-weighted sparse audits, charges the Clarke pivot externality, and applies a strictly proper score. Under the stated model assumptions, truthful reporting is the unique expected-utility best response. We also characterize approximate execution and the audit-variance trade-off. Calculator-free experiments show that LLM agents inflate confidence under Top-$K$ incentives. With a shared agent-local utility interface, LLM agents reduce distortion under both Compensated Proper Scoring (CPS) and APSM. CPS achieves higher exact allocation match, whereas APSM yields lower Brier loss and numerically fewer cutoff crossings. Experiments further show that APSM enlarges the utility margin against allocation-changing deviations by $1.93\times$ and improves robustness under approximate execution. These results support incentive-aware elicitation as a distinct layer for reliable multi-agent coordination.

Introduction

Confidence reports can become strategic when they determine scarce downstream influence, a problem distinct from intrinsic miscalibration. The paper frames strategic calibration as an incentive-design problem and proposes APSM to elicit truthful reports despite allocation incentives.

Introduction

  • Figure 1 depicts Agent 3 inflating its report and displacing Agent 2, illustrating how misreporting changes allocation.The paper separates this reporting-incentive problem from recalibrating the underlying belief.Source: Figure 1 · p. 1
  • When confidence controls scarce selection, agents may exaggerate reports for influence even if their underlying beliefs are meaningful, unlike intrinsic model miscalibration.Near hard allocation boundaries, a discrete influence gain can outweigh the smoothly vanishing proper-score loss from a small deviation.Source: Introduction · p. 1
  • APSM offsets platform-controlled allocation benefits, uses sparse audits and a Clarke pivot charge, and applies a strictly proper score to elicit truthful reports.The paper states strict expected truthfulness under its model and says the Groves term enlarges the utility margin against allocation-changing deviations.Source: Introduction · p. 2

Method

The method treats confidence as the strategic report while holding candidate content fixed. APSM combines allocation-benefit compensation, audited outcome scoring, and a pivot charge to make truthful reporting uniquely optimal under the stated design.

Problem Setup

  • The strategic action is the confidence report: candidate content is fixed before reporting, and the platform observes reports but not private relevance beliefs.The design seeks truthful reports as a unique expected-utility best response while maximizing a belief-space allocation surrogate.Source: Method · p. 2, Problem Setup · p. 2

Audited Pivot Scoring

  • APSM combines benefit compensation, audited outcome credit, a Clarke pivot charge, and a strictly proper score to internalize allocation consequences and uniquely elicit relevance probabilities.Relative to CPS, its audited Groves term adds a belief-space loss when a deviation induces an inefficient allocation.Source: Method · p. 2, Audited Pivot Scoring · p. 3
  • The pivot charge prices the welfare loss imposed on other agents, using a counterfactual allocation re-optimized after removing the reporting agent.It is not simply a realized leave-one-out reward.Source: Audited Pivot Scoring · pp. 2–3
  • Positive audit probabilities are fixed from pre-report public information, and fresh report- and outcome-independent sampling enables inverse-propensity weighting to preserve expected scoring credits.Lower audit rates increase the variance and tail magnitude of realized transfers.Source: Audited Pivot Scoring · p. 3
  • The mechanism’s expected utility cancels platform-controlled allocation benefits, while the proper-score term distinguishes reports that otherwise induce the same efficient allocation.Agent-local execution keeps private beliefs inside the trust boundary; LLM-controlled execution leaves the final report to the LLM rather than selecting the utility maximizer for it.Source: Audited Pivot Scoring · p. 3

Theoretical Analysis

The analysis shows why proper scoring alone can fail at hard selection thresholds and establishes unique expected truthfulness for APSM. It also bounds approximate execution and identifies modeled-benefit accuracy and audit variance as relevant boundaries.

Theoretical Analysis

  • The APSM guarantee assumes fixed candidate content, additive position-weighted relevance welfare, accurately offset platform-controlled benefits, report-independent audits, and risk-neutral current-round utility.The theorem’s expected-truthfulness claim is made under these stated assumptions.Source: Theoretical Analysis · p. 3, Truthful Reporting under APSM · p. 4

Why Matched-Brier Is Insufficient

  • For any finite score weight and positive selection benefit, profiles arbitrarily close to a Top-K threshold can make truthful reporting suboptimal under Matched-Brier.The selection benefit is discrete, while the Brier loss from a small reporting deviation is quadratic.Source: Why Matched-Brier Is Insufficient · p. 3

Truthful Reporting under APSM

  • APSM’s truthful report uniquely maximizes expected utility for every private belief and every profile of other agents’ reports.The pivot structure makes the allocation belief-efficient, and the strictly proper score removes indifference among reports inducing the same allocation.Source: Truthful Reporting under APSM · p. 4

Boundedly Rational Execution

  • Approximate utility maximization is analyzed with a δ_i-bounded utility gap, while inverse-propensity auditing preserves expected best responses and yields transfer variance growing as O(1/ϵ_i).With deterministic Top-K, a residual platform-controlled benefit can reintroduce threshold manipulation near the cutoff.Source: Boundedly Rational Execution · p. 4

Experiments

The experiments test whether allocation incentives distort LLM confidence and whether APSM changes reporting and allocation outcomes across controlled workflows. APSM enlarges the utility margin against allocation-changing deviations, while CPS and APSM show different strengths across reporting and allocation metrics.

Experimental Setup

  • The evaluation uses two six-agent environments, paired seeds, shared tool-assisted execution conditions, and paired statistical comparisons with query- or incident-level bootstrap procedures.AgentWebBench covers 200 web-retrieval queries; the database extension covers 50 PostgreSQL incidents.Source: Experimental Setup · pp. 4–5, Experiments · p. 4

RQ1: Do Allocation Incentives Create a Strategic-Calibration Failure?

  • Selection-only raises confidence by 0.218 relative to Neutral, while tool-assisted CPS and APSM reduce RefDev from 0.325 to 0.007 and 0.014, respectively.The calculator-free result also reports 95.8% of eligible agents crossing the Top-K cutoff; with tool assistance, cutoff crossings fall to 2.1% and 0.8%.Source: RQ1: Do Allocation Incentives Create a… · p. 5, Table 1 · p. 5, Table 2 · p. 5
  • Collaborative Calibration improves frozen-belief Brier loss from 0.414 to 0.198, while controlled APSM replay restores the induced allocation with zero RefDev and 100% exact match.The revised beliefs induce NDCG@5 of 0.364, which APSM recovers in replay.Source: RQ1: Do Allocation Incentives Create a… · p. 5, Table 3 · p. 6

RQ2: How Do the Mechanisms Behave under Controlled Task-Grounded Replay?

RQ3: What Does the Audited Groves Component Add Beyond Compensated Proper Scoring?

  • 1.93×: APSM raises the mean margin against allocation-changing reports from 0.3350 under CPS to 0.6462 and reduces allocation-changing errors at noise level 0.05.At that noise level, NDCG@5 rises from 0.299 to 0.356; the comparison is under approximate execution.Source: RQ3: What Does the Audited Groves… · p. 6, Figure 3 · p. 6
  • At ϵ = 0.1, sparse audits raise transfer variance to 15.4× and 30.1× the full-audit level on AgentWebBench and database diagnosis, respectively.Zero-audit rates are 53.7% and 52.9%, respectively.Source: RQ3: What Does the Audited Groves… · p. 6

Related Work

Prior calibration and multi-agent deliberation generally treat confidence as passive, while this work studies strategic reporting when confidence controls scarce allocation. It distinguishes APSM’s allocation-sensitive contribution from established exact-truthfulness mechanisms.

LLM calibration and strategic multi-agent behavior

  • Prior calibration and deliberation methods generally treat confidence as passive, whereas this work examines strategic reporting under allocation incentives.The paper frames this as a reporting-layer problem distinct from improving the underlying confidence quality.Source: Related Work · p. 7

Truthful elicitation and decision-dependent allocation

  • Boutilier (2011) establishes strict elicitation through decision compensation and a strictly proper score; the authors instantiate this as CPS without claiming novelty for exact truthfulness.The paper instead locates APSM’s contribution in enlarging the margin against allocation-changing deviations under approximate execution.Source: Related Work · p. 7
  • Unlike decision-market work, this setting uses LLM agents’ relevance reports for scarce deterministic Top-K allocation and adds an audited Groves term.The paper also distinguishes its setting from Doval and Smolin (2025), which studies principals with calibrated beliefs about agent types.Source: Related Work · p. 7

Conclusion and Limitations

The paper presents APSM as a reporting-layer mechanism for strategic confidence under downstream allocation incentives. Its experiments report reduced distortion with utility assistance and improved allocation-sensitive margins under approximate execution, subject to explicit model assumptions and scope limits.

Conclusion

  • APSM internalizes report-induced allocation externalities, and utility-assisted LLM agents reduce confidence distortion under both CPS and APSM.Controlled replay finds that APSM enlarges the utility margin against allocation-changing deviations and improves robustness under approximate execution.Source: Conclusion and Limitations · p. 7

Limitations

  • The guarantees assume fixed candidate content, risk-neutral myopic agents, report-independent auditing, and accurately modeled platform-controlled benefits.The reference belief is an agent-local proxy rather than observable truth.Source: Conclusion and Limitations · p. 7
  • APSM is not uniformly easier to execute than CPS, and the framework does not address external benefits, collusion, budget balance, or strategic generation.The paper identifies broader strategic settings as future work.Source: Conclusion and Limitations · p. 7

Appendix

The appendix formalizes APSM by specifying the repeated coordination protocol and information structure, then deriving the transfer and expected-utility representation used in the theoretical analysis.

Appendix

  • The formalization starts by specifying a repeated coordination protocol between a platform and a persistent population of content agents.The interaction proceeds over rounds.Source: Appendix · p. 10
  • It specifies the information structure before deriving the mechanism’s transfer and expected-utility representation.The expected-utility representation is used by the theoretical analysis.Source: Appendix · p. 10
  • The section presents the complete formalization of the Audited Pivot Scoring Mechanism.The protocol and information structure provide the setup for the subsequent derivation.Source: Appendix · p. 10

Repeated Coordination Protocol

The protocol has a platform and persistent population of agents interact over rounds, with each round beginning from a query and agent-generated candidate items. Candidates are fixed before confidence reports, which are the strategic actions studied.

Repeated Coordination Protocol

  • Across rounds, a platform interacts with a persistent population of N content agents, each of which generates a candidate item for the round’s query.Examples include a retrieved document, evidence passage, or candidate answer.Source: Repeated Coordination Protocol · p. 10
  • Candidate generation finishes before confidence elicitation, so the candidate set is fixed when reports are submitted.The strategic action is the confidence report rather than the candidate-generation policy.Source: Repeated Coordination Protocol · p. 10
  • Each candidate has a binary relevance outcome, while its agent forms a private relevance belief from the query, candidate, and local information.The platform does not observe that belief; agents submit observable confidence reports.Source: Repeated Coordination Protocol · p. 10

Allocation and Surrogate Welfare

The section defines feasible ranked allocations, report-space welfare, and a belief-space surrogate; the mechanism seeks reports that align the allocation maximizing each objective.

Allocation and Surrogate Welfare

  • Feasible allocations assign at most K agents to ranked coordination positions.The allocation set is defined for each round.Source: Allocation and Surrogate Welfare · p. 10
  • The platform evaluates allocations through additive report-space welfare and selects an allocation under that evaluation.The passage identifies the selected allocation but does not provide its formula.Source: Allocation and Surrogate Welfare · p. 10
  • The mechanism seeks reports that make the selected allocation maximize the belief-space surrogate welfare.The section distinguishes report-space welfare from its belief-space surrogate.Source: Allocation and Surrogate Welfare · p. 10

Platform-Controlled Selection Benefit

Selection can confer platform-controlled benefits beyond outcome quality, creating incentives to exaggerate confidence; the mechanism offsets such benefits under a bounded model.

Platform-Controlled Selection Benefit

Pivot Externality Charge

The Clarke pivot charge uses a counterfactual allocation without agent i to measure the report-space opportunity cost that agent imposes on others.

Pivot Externality Charge

  • The counterfactual removes agent i and re-optimizes the allocation among the remaining agents.This differs from deleting an item from an already chosen set.Source: Pivot Externality Charge · p. 10
  • The Clarke pivot charge measures the report-space opportunity cost imposed on other agents by agent i’s participation and allocation.Source: Pivot Externality Charge · p. 10

Sparse Outcome Verification

This section specifies APSM’s sparse-audit and proper-scoring components, its controlled evaluation protocols, and the assumptions and implementation choices used to assess truthful reporting, allocation alignment, and approximate execution. It characterizes audit-variance and payment-stability trade-offs and examines limits from payoff misspecification and approximate utility evaluation.

Sparse Outcome Verification; Sparse-Audit Unbiasedness and Variance

  • Independent post-report audits with positive propensity support inverse-propensity outcome credits; for bounded transfers, lower audit rates increase variance and tail magnitude without changing a risk-neutral executor’s expected best response.The variance result assumes audit sampling is independent of the outcome-dependent transfer component conditional on pre-round information.Source: Sparse Outcome Verification · p. 10, Sparse-Audit Unbiasedness and Variance · p. 12

Complete APSM Transfer; Proof of Theorem 1; Strictly Proper Score

Agent-Local Policy Optimizer; Efficient Pivot Computation; Experimental Protocol

Pivot-Only Allocation Alignment

  • With η = 0, truth is a weak best response, but any report inducing an allocation that maximizes Φ_i is utility-equivalent.The complete set of optimal reports includes reports inducing any such maximizing allocation.Source: Pivot-Only Allocation Alignment · p. 12

Compensated Proper Scoring

  • Compensated proper scoring also uniquely elicits truth under exact influence compensation and universally verifiable reports; APSM is therefore not the only truthful mechanism under those conditions.The pivot component is described as a Groves-consistent allocation and externality-pricing rule.Source: Compensated Proper Scoring · p. 13

Difference Rewards versus Pivot Charges

  • A realized leave-one-out difference reward is generally not a Clarke pivot charge, so its counterfactual and outcome-dependent construction does not support the standard Groves truthfulness argument.The reward compares realized total welfare rather than the externality imposed on other agents, and its counterfactual may not re-optimize remaining positions.Source: Difference Rewards versus Pivot Charges · p. 13

Payoff Misspecification under Hard Top-K

  • For every finite proper-score weight, residual selection benefits can make upward deviations profitable arbitrarily near deterministic Top-K cutoffs, ruling out uniform approximate truthfulness.The stated limitation concerns arbitrary residual selection benefits when the platform-controlled benefit is misspecified.Source: Payoff Misspecification under Hard Top-K · p. 13

Noisy Payoff Calculation; Finite Report Grids and Tie-Breaking

Experimental Protocol and Metrics

  • The controlled comparisons vary mechanism-specific utility while holding paired instances and reporting conditions fixed, and uncertainty estimates account for query or incident clustering.Calculator-free reports remain LLM-selected against neutral re-report, Selection-only, and Matched-Brier conditions; the tool-assisted mechanisms share instances, prompts, and tool access. Confidence intervals use query block bootstrap, seed–query hierarchical bootstrap, or incident-cluster bootstrap as appropriate.Source: Experimental Protocol · p. 15, Metrics and Statistical Analysis · p. 16

Complete AgentWebBench Results

  • APSM reduces AgentWebBench reporting deviation to 0.001, achieves 0.992 exact allocation match, and records the highest NDCG@5 (0.452).Matched-Brier has deviation 0.354 and never reproduces the reference allocation; Pivot-only reaches deviation 0.059 and match 0.702 without uniquely identifying the reference probability.Source: Complete AgentWebBench Results · p. 16, Table 5 · p. 17

Collaborative Calibration Followed by Strategic Reporting

  • With CC beliefs fixed, APSM returns reference reports, restores NDCG@5 to 0.364, and matches the CC-reference allocation in every query.CC lowers intrinsic Brier loss from 0.414 to 0.198, while NDCG@5 falls from 0.453 for original beliefs to 0.364 for CC beliefs; Matched-Brier reporting then has deviation 0.221, NDCG@5 0.301, and 2.5% allocation match.Source: Collaborative Calibration Followed by… · p. 16, Figure 6 · p. 18

Complete Database-Diagnosis Comparison

  • Under exact maximization, CPS and APSM both achieve zero reference deviation, match every reference-induced Top-2 allocation, and reach MRR of approximately 0.90.For dual-root incidents, APSM eliminates deviation and reaches MRR 1.000; for single-root incidents, it improves MRR from 0.747 to 0.806 while preserving allocation. CPS is not claimed to be non-truthful under exact optimization.Source: Complete Database-Diagnosis Comparison · pp. 16–17, Table 6 · p. 17

Mechanism Decomposition

  • In near-threshold instances, manipulation falls from 100% under Selection-only and 28% under scale-matched Brier scoring to 0% under APSM.Pivot-only yields an optimal-report-set width of 0.459; in the representative case it admits [0.01, 0.360], while APSM uniquely identifies q = 0.165. APSM’s zero manipulation result assumes correct compensation of the platform-controlled benefit.Source: Mechanism Decomposition · p. 17, Figure 8 · p. 19

Sparse-Audit Verification–Variance Trade-off

  • Sparser audits sharply increase transfer variance and payment tails; the rate of instances with no audited agent is nearly eliminated around an audit rate of 0.7.Moderate sparsity reduces verification cost while retaining relatively stable settlement, but aggressive sparsity can produce high-variance or uninformative rounds, consistent with O(1/ϵ) dependence.Source: Sparse-Audit Verification–Variance… · pp. 17–18, Figure 9 · p. 19

Sensitivity to Selection-Benefit Misspecification

  • At a ±5% benefit mismatch, APSM retains 46.5% exact allocation match versus approximately 15% for CPS.Both mechanisms recover reference reports and allocations at exact specification (c = 1); APSM degrades more slowly, with lower report error and fewer threshold crossings throughout the sweep, but this is relative resilience, not an absolute guarantee.Source: Sensitivity to Selection-Benefit… · p. 18, Figure 10 · p. 20

Approximate Agent-Local Execution

  • With utility noise of 0.05, APSM reduces allocation changes from 18.6% under CPS to 11.2% and raises exact match from 41.0% to 59.7%.The mean allocation-changing utility margin is 0.6462 for APSM versus 0.3350 for CPS, a 1.93× increase. At CPS score weight η = 1.43, allocation-error and cutoff-crossing rates are 17.2% and 14.3%, versus 11.2% and 8.3% under APSM; matching aggregate metrics requires CPS near η = 8, which scales its proper-score transfer eightfold and audit-induced variance 64-fold.Source: Approximate Agent-Local Execution · p. 19, CPS Score-Weight Sweep · p. 20

Controlled Best-Response Implementation Details

  • Selection provides unit private utility regardless of rank; position-dependent value enters through audited realized-value and Groves terms, and the implementation uses a normalized Brier score.Source: Controlled Best-Response… · p. 20
  • The exact-calculator candidate set always includes the frozen reference belief and adds reports immediately below and above competitors’ allocation thresholds.Duplicate candidate values are removed and the remaining reports are sorted.Source: Controlled Best-Response… · pp. 20–21
  • Approximate execution compares CPS and APSM using additive utility noise of 0.01 or 0.05 and Monte Carlo samples of 10, 50, or 200, with paired conditions sharing perturbations and streams.For Monte Carlo execution, only outcome Y is sampled; audit indicators are integrated analytically, while compensation, verified value, pivot charges, and inverse-propensity scaling remain analytically evaluated. Figure 11 reports report error, allocation changes, exact allocation match, NDCG@5, utility suboptimality, and comparison with the execution bound.Source: Figure 11 · p. 21, Controlled Best-Response… · pp. 21–22
  • Allocation ties go to the smaller agent index; utility ties choose the smallest maximizing report, except approximate execution chooses the highest, while Pivot-only retains the full maximizing set.Source: Controlled Best-Response… · p. 21
  • Across controlled-replay experiments, agents independently optimize against the other agents’ reference reports, then evaluate the joint profile under a shared Top-K rule.The protocol is unilateral and simultaneous, and is shared by all mechanisms within an experiment.Source: Controlled Best-Response… · p. 21
  • Under additive utility noise σ = 0.05, the CPS score-weight sweep reports allocation-changing error, cutoff-crossing rate, and NDCG@5 against APSM with η = 1.The starred CPS point matches APSM’s mean utility range; error bars are 95% seed–query hierarchical bootstrap intervals.Source: Figure 12 · p. 22

Related papers

Loading 2610.33728v1…