PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds

Caixin Kang, Ke Wang, Yuchuan Wu, Xiaoqiang Wang, Ling Li, Bingxue Qiu, Yongbin Li, Yoichi Sato

arXiv:2610.20195v1cs.CLcs.AIcs.SI

TL;DR

Social-agent evaluations often rate conversation without checking whether the resulting plan fits participants’ private conditions. HICO-Bench verifies plans and their conversational support, while PaCPO uses paired worlds to assign turn-level credit; on Qwen3-8B, it raises Verified Success from 20.4% to 28.3%.

  • Problem

    Social-agent evaluations mainly rate conversations without verifying that resulting plans satisfy participants’ private conditions, although everyday coordination often depends on such conditions.

    Source: §1 · p. 1

  • Method

    HICO-Bench evaluates paired private-information scenarios with program-checked Feasibility and judge-checked Integrity, while PaCPO uses tentative plans across paired worlds to redistribute terminal reward across turns.

    Source: §1 · p. 2

  • Results

    On Qwen3-8B, PaCPO raises Verified Success from 20.4% to 28.3%, versus 25.0% for PPO and 26.3% for GRPO trained on the same number of dialogues.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Verified outcomes complement conversation ratings for evaluating social interaction and can also provide a training signal for improving Verified Success.

    Source: §6 · p. 10

    HICO-Bench covers six coordination families with structured outcomes, not interactions such as emotional support or relationships without a well-defined final outcome.

    Source: L Limitations and Future Work · p. 29

Abstract

from arXiv · show

Effective social interaction requires more than fluent responses: agents must reach plans that fit the participants' needs and circumstances. Existing evaluations of social agents mainly rely on LLM judges that rate the conversation, without checking the resulting plan. We introduce HICO-Bench, a benchmark of 180 everyday coordination scenarios, such as resource allocation, scheduling, and authorization, in which two agents with private information converse and then independently record the plan they agreed on. A program checks Feasibility (the two records match and satisfy both participants' private conditions), and an LLM judge checks Integrity (the conversation supports the recorded plan); passing both is a Verified Success. Each scenario has two versions that share the same public context but differ in one private condition, so the correct plan differs between them. Across thirteen LLMs, 36% of conversations rated highly for goal completion fail Verified Success. To learn from outcomes that arrive only at the end of a conversation, we propose Paired-Credit Policy Optimization (PaCPO). After each turn, PaCPO asks the agent for a tentative plan, checks it in both versions of the scenario, and rewards the turns after which the tentative plans fit their respective versions. PaCPO raises Qwen3-8B's Verified Success from 20.4% to 28.3%.

1 Introduction

Everyday coordination requires plans that satisfy participants’ private conditions, yet many social-agent evaluations rate dialogue without verifying its outcome. HICO-Bench checks both plans and their conversational support, while PaCPO uses paired scenarios to assign turn-level credit from verified outcomes.

1 Introduction

  • Conversation-focused ratings can miss whether an agreed plan satisfies participants’ private conditions in everyday coordination.Acceptable plans may depend on facts such as budget, availability, or authority that are revealed through dialogue.Source: §1 · p. 1
  • Across thirteen LLMs, 36% of conversations rated Goal ≥8/10 fail Verified Success.Verified Success requires matching records that satisfy both participants’ private conditions and a conversation that supports the recorded plan.Source: §1 · p. 2
  • PaCPO raises Qwen3-8B’s Verified Success from 20.4% to 28.3%, versus 25.0% for PPO and 26.3% for GRPO trained on the same number of dialogues.The paired-world method redistributes terminal reward to turns whose tentative plans fit their respective worlds.Source: §1 · p. 2
  • HICO-Bench evaluates 180 paired scenarios across six coordination families using program-checked Feasibility and judge-checked Integrity alongside SOTOPIA ratings.A scenario’s two worlds share public context but differ in one private condition, with disjoint sets of correct plans.Source: §1 · p. 2
  • PaCPO compares tentative plans across paired worlds after the differing private condition is disclosed to assign credit to turns.It redistributes terminal reward without changing each conversation’s total return.Source: §1 · p. 2

2 Related Work

Prior work studies social reasoning, dialogue ratings, outcome verification in fixed-domain tasks, clarification, and reinforcement-learning credit assignment. HICO-Bench focuses on plans reached through everyday dialogue, while PaCPO compares verifiable tentative plans across paired worlds.

Agent reasoning and interaction

  • SOTOPIA rates goal completion, believability, and related dimensions in role-played social scenarios, while HICO-Bench checks whether the dialogue-supported plan fits private conditions.SOTOPIA ratings are retained as complementary measures of conversation quality.Source: §2 · p. 3

Task-oriented dialogue and negotiation

Clarification and information seeking

  • Prior clarification work rewards or scores questions (Andukuri et al., 2024), but HICO-Bench rewards the correctness of the resulting plan rather than questions directly.The decisive fact may be volunteered, already known, or elicited.Source: §2 · p. 3

Credit assignment in reinforcement learning

  • PaCPO compares tentative plans across paired worlds and checks them with rules, unlike methods that label reasoning steps, sample continuations, compare recurring states, or spread LLM-judge ratings across utterances (Cheng et al., 2026; Yu et al., 2025; Feng et al., 2026).The passage notes that identical states rarely recur in free-form dialogue and LLM ratings are noisy.Source: §2 · p. 3

3 HICO-Bench

HICO-Bench has two agents converse using shared public context and private role cards, then independently record their understanding of the agreed plan. Paired worlds make correct plans depend on a private condition, and outcome verification checks both plan feasibility and support in the dialogue.

3.1 Interaction Protocol

  • Each role sees public context, its own private card, and the conversation history, then independently records the plan it believes was agreed.The records test whether the roles reached a shared understanding.Source: §3.1 · p. 3

3.2 Paired Scenarios

  • Each scenario’s two worlds share public context but differ in one private condition chosen so that no plan is correct in both.The 180 scenarios are split into 96 training, 24 development, and 60 test scenarios, with both worlds kept in the same split.Source: §3.2 · p. 4

3.3 Outcome Evaluation

  • Feasibility passes only when the two records match, the outcome kind is correct for that world, and the terms satisfy the family’s rules.Any plan satisfying the rules is accepted, such as any mutually free slot in scheduling.Source: §3.3 · p. 4
  • Integrity fails when the dialogue contains an uncorrected factual error, repeated exchanges without progress, a record contradicting the spoken agreement, or leaked output-format text.Corrected proposals and truthful referrals are allowed.Source: §3.3 · p. 4
  • PaCPO checks tentative plans from rollouts in both worlds after each turn, uses the checks to assign turn-level credit, and redistributes terminal reward before the PPO update.The overview labels worlds, turns, and tokens with indices i, j, t, and k; the dashed branch carries GAE value targets.Source: Figure 2 · p. 5, §3.3 · p. 5
  • The evaluation reports SOTOPIA’s seven conversation ratings, including Goal and Believability, separately from the primary fraction of tasks passing both checks.The ratings are never used for training; they test whether high ratings coincide with verified outcomes.Source: §3.3 · pp. 4–5

4 Paired-Credit Policy Optimization

PaCPO redistributes terminal outcome rewards across conversation turns by checking tentative plans in paired worlds. It credits turns when the plans fit their respective worlds after the differing private condition is available.

4 Paired-Credit Policy Optimization

  • PaCPO redistributes a terminal reward across turns, crediting moments when tentative plans diverge to fit their respective worlds.After each turn, the agent is probed for the plan it would record if the conversation ended; these probes stay outside the conversation.Source: §4 · p. 5

4.1 Terminal Reward

  • All RL methods optimize the same terminal reward, but PPO and GRPO receive it at the final turn while PaCPO redistributes it.The reward combines Feasibility and Integrity, with weights 0.8 and 0.2, respectively.Source: §4.1 · p. 5

4.2 Tentative Plans and Cross-World Checks

  • After each turn, PaCPO checks both tentative plans against both worlds’ Feasibility rules.The tested role’s plan is probed using only what that role can see, and the probe is not added to the conversation.Source: §4.2 · pp. 5–6

4.3 Paired Credit

  • Paired credit is positive when a world’s tentative plan fits that world but the other world’s plan does not.The comparison is gated until both plans are valid, no common plan fits both worlds, and the differing condition is available in both conversations.Source: §4.3 · p. 6

4.4 Reward Redistribution

  • PaCPO combines within-world feasibility and paired credit in a turn-level potential, then redistributes reward without changing each conversation’s total return.With γ = 1, shaping terms telescope; turns that raise potential are rewarded, and the final turn receives the remaining terminal reward.Source: §4.3 · p. 6, §4.4 · p. 6

5 Experimental Results

The experiments compare outcome verification with conversation ratings, test PaCPO against RL baselines, and examine transfer and credit-design choices. PaCPO improves Verified Success over Base, with additional results across interaction families and partner settings.

5.1 Experimental Setup

  • Training uses 384 conversations per RL run, while evaluation tests 240 tasks against a frozen Qwen3-8B partner with rule-based Feasibility and judge-based Integrity.Group-based methods cover 48 training scenarios, compared with 96 for PPO and PaCPO; evaluation uses a different judge from training.Source: §5.1 · pp. 6–7

5.2 Multi-Model Benchmark

  • 35.99% of conversations rated Goal ≥8 fail Verified Success, and Goal ratings can rank models differently from verified outcomes.Across the thirteen-model benchmark, Verified Success ranges from 20.42% to 82.50%; Qwen3-8B passes Feasibility on 49.58% but reaches 20.42% Verified Success.Source: §5.2 · p. 7
  • On 12 development scenarios with hidden conditions, dialogue raises Base Feasibility by 50 points over submitting records directly.Source: §5.2 · p. 7

5.3 Learning from Verified Outcomes

  • 28.33% Verified Success: PaCPO improves Base by 7.92 points and exceeds GRPO through higher Integrity despite equal Feasibility.The methods use the same terminal reward and number of dialogues.Source: §5.3 · p. 8
  • Outcome-learning gains vary by family: the largest improvements over Base are 22.50 points in bounded quantities and 10.00 points in authorization.Resource allocation remains at 17.5% Verified Success for Base, PPO, and PaCPO, and both scheduling families remain difficult.Source: §5.3 · p. 8, Figure 3 · p. 8
  • In the shared-table example, both runs submit the same feasible records, but only PaCPO’s dialogue supports them.PaCPO identifies a four-minute gap between nonoverlapping use windows; Base incorrectly calls them overlapping, and its partner accepts the error.Source: §5.3 · p. 8, Figure 5 · p. 9

5.4 Self-Play and Interactive Transfer

  • PaCPO reaches 28.33% mean Verified Success in self-play, exceeding Base by 6.39 points and PPO by 1.11 points, and leads in both SOTOPIA partner settings.On MultiWOZ it gives a small communication gain; on CaSiNo it leads agreement and normalized Nash surplus, with larger gains in bargaining than constraint communication.Source: Figure 4 · p. 8, §5.4 · p. 9

5.5 Credit Ablations

  • 26.25% Verified Success results when paired-world plans are replaced with same-world plans or credit is shuffled across turns; opening the gate before disclosure yields 25.83%.During training, paired credit is nonzero at 55.6% of gate-open turns.Source: §5.5 · p. 10

6 Conclusion

HICO-Bench evaluates social interaction through both conversation and resulting plans, while PaCPO uses paired worlds to turn final outcomes into turn-level training credit. The benchmark covers six coordination families, and the authors identify structured outcomes and judge-based Integrity as current scope boundaries.

6 Conclusion

  • Verified Success requires both matching records that satisfy private conditions and a conversation that supports the recorded plan.PaCPO uses paired worlds to assign turn-level credit; the conclusion reports improved Verified Success over the base model and RL baselines.Source: §6 · p. 10
  • The benchmark covers six coordination families with structured outcomes, and Integrity relies on an LLM judge.The authors identify extending verification to less structured interactions and expanding human validation as next steps.Source: §6 · p. 10

AI use statement

  • Generative AI assisted research design and analysis, data and code development, and manuscript preparation, including figures.The authors state that they take full responsibility for the final work.Source: AI use statement · p. 10

Ethics statement

  • The scenarios use fictional or adapted role-play material and do not establish judgments about real individuals.The statement distinguishes broader social quality from feasibility under the benchmark’s formal conditions.Source: Ethics statement · p. 10

Reproducibility statement

  • The intended release includes scenario lineage, role cards, verifier tests, prompts, rubrics, training settings, and checkpoint IDs.The reproducibility statement says tables and figures derive from recorded results.Source: Reproducibility statement · p. 10

A Scenario Families

  • Each family has 30 scenarios split into 16 training, 4 development, and 10 test scenarios, with two worlds per scenario.Shared-resource scheduling checks non-overlapping reservations; set-based consent checks required and allowed item sets.Source: A Scenario Families · p. 17

B Evaluation Metrics

The evaluation reports exploratory information-management, understanding, and action diagnostics alongside standard social-quality ratings. In the SFT initialization study, PaCPO reaches the highest reported Verified Success, while social ratings and verified outcomes show differing profiles.

B Evaluation Metrics

  • Three exploratory diagnostics—Information-gap Management, Grounded Understanding Accuracy, and Information-informed Action Appropriateness—are reported separately from primary metrics and training reward.The diagnostic ratings use frozen task-specific rubrics; Information-gap Management does not require asking a question when the information is already available.Source: B Evaluation Metrics · p. 17

C.1 SFT Initialization

  • 31.67% Verified Success for SFT-initialized PaCPO versus 25.83% for SFT alone, the highest result reported in the experiments.Its Feasibility is 48.75%, slightly below Base, and its Goal is lower than Base-initialized PaCPO.Source: §C.1 · p. 18, Table 8 · p. 19

C.2 Complete Social-Quality Scores

  • Goal scores are high for most API models, while outcome rates are more widely separated; PaCPO improves Goal and Believability over Base.SFT-initialized PaCPO obtains the highest Believability; Figure 7 compares Feasibility, Integrity, and Verified Success across models.Source: §C.2 · pp. 18–20, Figure 7 · p. 19, Table 9 · p. 19

E Training Details

PaCPO redistributes terminal outcome reward across turns using tentative plans checked across paired worlds, with disclosure and validity conditions governing paired credit. The training details also specify implementation boundaries, baseline comparisons, training cost, and an illustrative authorization scenario.

E Training Details

  • PaCPO generates a tentative plan after each tested-role turn and compares plans across paired worlds to redistribute terminal reward across turns.Matched turns with valid plans receive paired credit only when the differing condition is known in both worlds; unmatched or invalid plans receive zero paired credit, while within-world feasibility remains available.Source: E Training Details · pp. 20–21
  • The value head predicts return for the PPO baseline, while proposal verification supplies process rewards and the rule verifier checks tentative-plan feasibility.The value head uses detached policy hidden states, so value regression updates its own parameters without backpropagating through the policy backbone.Source: E Training Details · pp. 21–22
  • Alternate-world state and undisclosed partner-private facts remain training-side and are never injected into actor or reference-policy prompts.When the differing condition is on the tested role’s card, it is available from the first turn; otherwise, a local judge annotates the earliest partner disclosure.Source: E Training Details · p. 22
  • Main methods share the backbone, partner, terminal reward, generation limits, and test protocol, but comparisons match dialogue counts rather than scenario coverage or total compute.Main comparisons use one training seed and one test generation seed.Source: E Training Details · p. 22

E.1 GiGPO and GraphGPO Adaptation

  • GiGPO compares discounted returns at repeated exact-history anchors, whereas GraphGPO uses shortest-path distances to success; unmatched anchors retain only the episode-level component.The reported results characterize an exact-history HICO adaptation, so grouping and state representation are part of the comparison alongside the advantage estimator.Source: §E.1 · p. 23

E.2 Training Cost

  • PaCPO generates 1,869 tentative plans, and its generated output is 12.51% higher than PPO’s.The plans contribute 57,405 output tokens and 2,024,795 input tokens; 216 disclosure annotations complement 168 dialogues where the differing condition is on the tested role’s card.Source: §E.2 · p. 24

F A Paired Scenario

  • The paired authorization scenario changes only Blair’s private authorization limit, determining whether approval or truthful referral is feasible.Both records retain the requested one-hour parameter and approve_extra_review operation, while Integrity checks conversational support.Source: F A Paired Scenario · p. 24, Table 13 · p. 24

G Outcome Reliability and Information Diagnostics

High conversation-goal ratings do not ensure Verified Success, which requires a feasible plan supported by the conversation. A diagnostic separates information access from dialogue, while the supplied transfer and self-play passages specify evaluation protocols rather than their outcome values.

G Outcome Reliability and Information Diagnostics

H Interactive Transfer and Self-Play

  • The transfer protocol evaluates HICO checkpoints without further training on 70 environment–character pairs per partner setting, while self-play uses the same checkpoint for both participants.Self-play reports Feasibility, Integrity, and Verified Success across three generation seeds, with 120 dialogues and 240 scored role views per seed.Source: H Interactive Transfer and Self-Play · p. 26, Table 15 · p. 26

I External-Task Training

The external-task experiments test whether agents can elicit private constraints for database recommendations and negotiate allocations under private preferences. Paired worlds preserve public context while changing private conditions relevant to a valid outcome.

I External-Task Training

  • MultiWOZ tasks use hotel and restaurant goals with explicit user constraints, retaining original splits and removing repeated constraint profiles across splits.Goals require a database match; booking requests and preselected named entities are omitted.Source: I External-Task Training · p. 26
  • For recommendation, paired worlds keep the public request and database fixed while changing one private constraint, producing disjoint valid entity sets.Success requires satisfying every private constraint and unambiguously naming the recommendation; the test set has 77 paired scenarios and 154 dialogues.Source: I External-Task Training · p. 26
  • CaSiNo allocation worlds keep the tested participant’s card fixed and replace the partner’s preference-and-reason card with one from the same original split.The campsite setting has three units each of food, water, and firewood, with private priorities valued at 5, 4, and 3 points per unit.Source: I External-Task Training · p. 26
  • Negotiation evaluation measures both agreement rate and normalized Nash surplus, distinguishing reaching agreement from finding a mutually beneficial allocation.Surplus uses the product of both participants’ nonnegative utility gains above the five-point walk-away value, divided by the maximum attainable product; no-agreement episodes receive zero. The test set has 43 paired scenarios and 86 dialogues.Source: I External-Task Training · pp. 26–27

J Family-Level Analysis

Family-level results show that PaCPO’s gains vary by interaction family: it improves Verified Success in some families, while coordination tasks remain difficult. The analysis also reports paired-credit activation during training.

J Family-Level Analysis

  • Figure 9 reports Feasibility and Verified Success on 40 tasks per family, and all 60 test scenarios have disjoint valid outcome sets across paired worlds.Each family has ten scenarios, two worlds, and two roles; scheduling remains difficult across methods.Source: Figure 9 · p. 27, J Family-Level Analysis · p. 27
  • 30.0% to 52.5%: PaCPO’s largest Verified Success gain over Base is for bounded quantities.For authorization, Verified Success rises from 37.5% to 47.5%; on bounded quantities, PPO and PaCPO have identical Feasibility at 75.0%, but PaCPO’s Verified Success is 52.5% versus 42.5%.Source: J Family-Level Analysis · p. 27
  • 17.5%: resource-allocation Verified Success remains the same for PaCPO, Base, and PPO despite PaCPO’s 77.5% Feasibility.Joint scheduling and shared-resource scheduling are PaCPO’s lowest-scoring families, at 15.0% and 7.5% Verified Success; low Feasibility also indicates coordination itself remains difficult.Source: J Family-Level Analysis · p. 27
  • Table 16 reports paired-credit activation; for scheduling, activation is lowest because both tentative plans are frequently infeasible, producing a tie.The supplied result passage ends before giving the tie notation’s complete value.Source: J Family-Level Analysis · p. 27

K Robustness and Human Validation

The robustness and validation analyses examine variation across partners and generation seeds, agreement between Integrity judges, and human assessments of plans and helpfulness. The paper also limits HICO-Bench’s scope to interactions with structured, checkable outcomes.

K Robustness and Human Validation

  • The self-play robustness experiment varies both participants, repeats generation with three seeds, and holds the scenario set fixed.Figure 4 reports Verified Success, while Table 15 gives corresponding Feasibility and Integrity means and standard deviations; Table 16 describes paired-credit activation during training over 384 dialogues and 24 rounds.Source: K Robustness and Human Validation · pp. 27–28, Table 16 · p. 28

K.1 Cross-Judge Agreement

  • Qwen3.8-Max and Kimi-K3 agree on 216 of 240 Integrity decisions, with κ = 0.783; Kimi-K3 accepts 24 additional cases and rejects none accepted by Qwen3.8-Max.The fixed-output comparison reuses the same 240 model–task positions per method and judge; positions are selected by family and task identifiers, not Integrity or Feasibility scores.Source: §K.1 · p. 28, Table 17 · p. 28

K.2 Human Review

  • Human Feasibility judgments match executable checks on 100% of dialogues, and human Integrity judgments match Qwen3.8-Max on 82.22%.Across 24 matched Base–PaCPO pairs, annotators prefer PaCPO in 8, Base in 5, find them similar in 3, and judge both inadequate in 8.Source: §K.2 · p. 29

L Limitations and Future Work

  • HICO-Bench covers six coordination families with structured records and explicit conditions, not the full range of social interaction.Emotional support and relationships without well-defined outcomes are outside its scope; Feasibility is success under specified conditions, not a general judgment of social appropriateness.Source: L Limitations and Future Work · p. 29

Related papers

Loading 2610.20195v1…