PaperlayerView exact revision on arXiv

Source-linked AI summary

ProMediConv: Benchmarking Proactive Conversational Agents in Legal Dispute Mediation

Zesheng Wei, Mengfan Li, Wenhao Liu, Yixin Zhang, Zilei Wang, Yang Deng

arXiv:2609.11101v1cs.CL

TL;DR

Legal mediation research lacks realistic multi-party task formulations, high-fidelity data, and metrics that capture within-dialogue change. ProMediConv addresses these gaps with a proactive, staged, party-aware benchmark, a 972-case dataset, and MAD; its evaluations reveal persistent limitations in current models while ProMediAgent improves benchmark performance.

  • Problem

    Existing LLM-based mediation research is constrained by unrealistic task formulations, low-fidelity datasets, and outcome-focused metrics that miss within-dialogue party-state changes.

  • Method

    ProMediConv models proactive multi-party mediation across stages using 11 strategies and four Behavior Pattern states, with a 972-case dataset, MAD metric, and benchmark models.

  • Results

    ProMediAgent improves over the best-performing baseline, increasing SR@t by 5.61% and SSR by 2.99%, while decreasing AT by 1.34% and achieving 8.86% lower MAD.

  • Takeaways & Limitations

    ProMediConv provides a quantitative foundation for analyzing behavioral change and advancing AI-assisted conflict resolution in complex mediation.

  • Takeaways & Limitations

    The framework relies exclusively on text-based dialogue in a specifically Chinese legal context, abstracting away non-verbal dynamics and broader cultural nuances.

Abstract

from arXiv · show

Dispute mediation is essential for maintaining social harmony and resilience, yet developing skilled mediators is costly and time-consuming. Existing LLM-based mediation research remains limited by unrealistic task formulations, low-fidelity datasets, and coarse evaluation metrics that obscure turn-by-turn dynamics. To address these gaps, we introduce ProMediConv, a novel benchmarking framework that models mediation as a proactive, multi-stage, and party-aware dialogue process incorporating 11 mediation strategies and four party behavior pattern (BP) states. Using 972 complete real-world cases, we construct a high-fidelity mediation dataset with utterance-level annotations of strategies and BP states. Furthermore, to better assess agent impact, we propose MAD (Mean Attribute Difference), a fine-grained metric that captures BP shifts throughout the dialogue. Leveraging this framework, we establish a comprehensive benchmark by evaluating diverse models alongside our tailored baseline ProMediAgent. Extensive empirical analyses reveal critical behavioral phenomena and underscore the persistent challenges current models face in dynamic, multi-party mediation. Ultimately, ProMediConv provides a rigorous foundation and a vital quantitative standard for advancing AI-assisted conflict resolution. Our dataset and codebase are accessible at https://github.com/ZsWei66/ProMediConv_repo.

1 Introduction

ProMediConv addresses gaps in legal mediation research by modeling proactive, multi-party dialogue with party-state tracking and by introducing finer-grained evaluation. It contributes a 972-case dataset, the MAD metric, and a benchmark revealing persistent model limitations.

  • Research gaps: Existing mediation frameworks often simplify the task, lack high-fidelity interactive data, and use outcome-focused metrics that miss evolving party states.Prior work may treat mediation as generic dialogue, rely on static legal documents or simplified synthetic conversations, and emphasize efficiency or final resolution.
  • ProMediConv framework: ProMediConv models mediation as a proactive, strategy-driven, multi-party dialogue process with 11 mediation strategies and four Behavior Pattern states.The framework explicitly tracks the evolving psychological stances of disputing parties.
  • Dataset: The authors construct a high-fidelity dataset of 972 realistic cases while preserving authentic interactive dynamics despite privacy barriers.An automated reconstruction pipeline is constrained by immutable factual and legal records.
  • Evaluation: MAD measures within-dialogue qualitative shifts in parties’ Behavior Patterns, addressing blind spots in traditional efficiency- and outcome-focused metrics.The metric is intended to quantify the mediator agent’s fine-grained influence on disputing parties.
  • Benchmark: The benchmark evaluates diverse general and legal-specific LLMs alongside ProMediAgent and exposes persistent limitations in real-world legal mediation.The contribution includes analysis of behavioral phenomena in mediation dialogue.

2 Related Works

Prior dialogue research covers negotiation and proactive agents, but existing work remains poorly aligned with complex legal mediation. In particular, it generally omits multi-party interaction, explicit party states, or proactive mediation behavior.

  • Mediation and negotiation dialogues: Legal mediation and negotiation research has expanded beyond casual bargaining, but existing frameworks remain overly simplified for professional scenarios.The cited limitations include two-party restrictions, missing party-state modeling, and neglected proactive mediation behavior.
  • Proactive conversational agents: Proactive conversational agents aim to anticipate impacts and steer dialogue rather than merely react to user inputs.Prior approaches use chain-of-thought, planning plugins, self-play, and learning from AI feedback to improve proactivity or strategy.
  • Proactive conversational agents: Existing proactive-agent approaches mainly target two-party tasks, leaving complex multi-party mediation scenarios unexplored.This gap motivates evaluation of proactive conversational agents in legal mediation settings.

3 ProMediConv

ProMediConv formalizes mediation as a dynamic, multi-stage dialogue that tracks mediation strategies and parties’ evolving behavior patterns. It also introduces MAD and constructs a high-fidelity dataset from authentic mediation cases with structured annotations.

  • 3.1 Mediation Workflow & BP States of Party: ProMediConv models mediation through Understanding, Intercession, and Reconciliation stages, while allowing returns or skips for context-dependent trajectories.Mediators may revisit Understanding for reassurance or bypass Intercession in straightforward disputes.
  • 3.1 Mediation Workflow & BP States of Party: The framework tracks four developmental behavior-pattern states to represent how disputing parties’ perspectives evolve during strategic mediation.The classification draws on the Transtheoretical Model’s view of behavioral change as gradual and multi-stage.
  • 3.3 Mean Attribute Difference: MAD measures progressive changes in client attributes by comparing initial and final states across clients and cases.The evaluated attribute must be ordinal so that the quantified progress is meaningful.
  • 3.4 Dataset Construction: The dataset contains 972 high-quality structured dialogues reconstructed from authentic mediation records while preserving immutable facts, trajectories, and legal clauses as constraints.The source collection began with 981 cases from 15 published mediation books and was filtered for quality.
  • 3.4 Dataset Construction: The resulting corpus includes comprehensive behavior-pattern and strategy annotations across 13 dispute types, with realistic long-tail category frequencies and phase-based strategy evolution.Strategy distributions change across dialogue turns in alignment with modeled strategy-stage correlations.

4 Overall Evaluations

ProMediConv evaluates proactive mediation agents with conventional outcome metrics and MAD, using diverse baselines and a simulated multi-party environment. Results show unstable or modest gains for general and legal-specific LLMs, while ProMediAgent improves substantially across metrics.

  • 4.1 Evaluation Metrics: The benchmark combines AT, SR@t, and SSR with MAD to measure both dialogue efficiency, resolution, and fine-grained influence on disputing parties.MAD complements conventional metrics by tracking mediator effects on party states during the dialogue.
  • 4.1 Implementation Details: Evaluation uses general and legal-specific LLM baselines, prompt-based planning methods, and an LLM-mediated environment with role-play, speaker selection, and outcome rewards.Overall results are reported in Table 2.
  • Analysis of General LLMs: Policy-planning methods produce unstable effects for general LLMs while doubling or tripling mediator-response token consumption.Some methods are counterproductive across ChatGPT settings, indicating that larger models and additional planning tokens do not reliably solve the task.
  • Analysis of Legal-Specific LLMs: Legal-specific LLMs show only minimal improvement from legal-clause references, with Fuzi gaining 6.91% SSR, 7.64% MAD, and 2.11% SR@t.The models can retrieve relevant legal provisions, but improvements remain insufficient for successful mediation.
  • Analysis of ProMediAgent: ProMediAgent improves over the best baseline by 5.61% SR@t, 2.99% SSR, 1.34% AT, and 8.86% MAD.SFT and RL training both improve performance, with RL producing the strongest overall gains.

5 Further Analysis

Further analyses show that constrained strategy taxonomies, MAD, and party-aware evaluation reveal dynamics missed by coarse efficiency metrics. Unrestricted strategies can degrade performance, shortcut resolutions obscure incomplete mediation, and increasing party numbers make improvement harder.

  • 5.1 Mediation Strategy Analysis: Predefined strategy constraints outperform an open-ended strategy space, indicating that unrestricted strategy generation can degrade mediation performance.The ablation compares Qwen2.5-14B-Instruct with unrestricted strategy generation against the Proactive setting.
  • 5.2 Strategy Selection: The strategy-selection comparison evaluates random selection against a policy planner while holding the response generator constant.Five independent random rollouts per case are averaged for robustness, and the comparison is reported in Table 4.
  • 5.3 Case Study on the Necessity of MAD: Short-cut Resolution occurs when agents reach superficial agreement without addressing all parties’ claims, which MAD exposes through party-level BP tracking.In the cited case study, fewer turns correspond to better AT but lower MAD when claims remain unresolved.
  • 5.4 Impact of Party Numbers: As party numbers increase, MAD and SSR decrease while AT increases, indicating that more complex cases require more turns and make party-state improvement harder.The relationship is reported from Qwen2.5-14B-Instruct samples and summarized in Figure 6.
  • 5.5 Correlation Analysis: Human mediators achieved 87% pairwise agreement, and alignment with Human Success Rate supports MAD’s validity for identifying high-quality mediation.The human annotations were used to compare MAD with existing metrics and establish a reference completion measure.

6 Conclusion

ProMediConv contributes a benchmark, high-fidelity dataset, and MAD metric for proactive multi-party mediation. Its analyses expose persistent limitations in current LLMs and establish a quantitative foundation for future AI-assisted conflict resolution.

  • 6 Conclusion: ProMediConv provides a 972-case high-fidelity dataset, a benchmark for proactive multi-party mediation, and MAD for measuring shifts in user states.The framework is designed to address evaluation blind spots and support analysis of complex mediation behavior.
  • 6 Conclusion: Benchmarking reveals critical behavioral phenomena and persistent limitations of current LLMs in complex real-world mediation.The conclusion frames the resources as a scalable foundation for future AI-assisted mediation and social resilience.

Limitations

ProMediConv is limited to text-based dialogue modeling in a specifically Chinese legal context, leaving non-verbal and cross-cultural dynamics outside its scope.

  • Limitations: ProMediConv exclusively models text-based dialogue within a specifically Chinese legal context.This setting abstracts away non-verbal dynamics and may not capture dispute-resolution nuances in other languages and cultures.

Ethical Considerations

ProMediConv formalizes mediation as a structured, multi-party dialogue involving parties and a mediator whose behavior, strategies, state updates, and rewards are recorded throughout the process.

  • Task Formalization: The framework models each round as a speaking turn for one party or the mediator within a structured mediation process.The participant set includes all disputing parties and mediator M.
  • Party Modeling: Party utterances depend on personal circumstances, the current mediation state, and the speaker’s behavior pattern, which an automated classifier marks.The current state is referenced as C_t−1, while party behavior is tracked at turn t.
  • Mediator Modeling: On mediator turns, the mediator selects a mediation strategy before speaking, and the mediation state is updated after each successive utterance.The strategy set is represented by S, while C evolves throughout the dialogue.
  • Outcome Modeling: A reward model evaluates the current mediation state and determines whether the mediation goal reaches completion.The reward contributes to the completion-state assessment described in the formalization.
  • Evaluation Records: The framework records dialogue turns, completion state, final-turn reward, and parties’ initial and final behavior patterns for dataset-level metrics.These records support AT, SR@t, SSR, and MAD computation.

A.2 Complete Mediation Strategy Set in ProMediConv

ProMediConv defines 11 mediation strategies to represent proactive, directive mediation across the full three-stage workflow.

  • A.2 Complete Mediation Strategy Set in ProMediConv: The strategy taxonomy contains 11 mediation strategies designed to capture the proactive and directive nature of human mediators.The complete specification is presented in Table 6.
  • A.2 Complete Mediation Strategy Set in ProMediConv: Each strategy is mapped to applicable stages, illustrated with an authentic-case utterance, and explained through its theoretical objective.The stages are Understanding, Intercession, and Reconciliation, covering the mediation lifecycle.
  • A.2 Complete Mediation Strategy Set in ProMediConv: The automated reconstruction pipeline ultimately yields the structured ProMediConv dataset from unstructured authentic cases under factual and legal constraints.This overview connects the strategy framework to the dataset’s broader construction pipeline.

B.1 Dataset Construction

The dataset construction pipeline converts authentic but disorganized mediation materials into constrained, annotated dialogues, then evaluates agents for behavioral-state classification and legal-clause coverage.

  • B.1 Dataset Construction: 972 high-quality authentic cases are reconstructed into dialogues using extracted facts, trajectories, and legal clauses as hard constraints.The pipeline is designed to preserve real-world dynamics while preventing hallucinated content.
  • B.1 Dataset Construction: The source cases contain disorganized formats, insufficient legal references, lengthy contexts, and missing annotations for party behavior and mediator strategies.These challenges motivate the pipeline-filter architecture.
  • B.1 Dataset Construction: A Legal Counsel Agent retrieves applicable laws, while a Secretary extracts case-critical information into a structured mediation note used as a reconstruction constraint.The Rebuilder Agent conditions dialogue generation on this note to maintain factual alignment.
  • B.1 Dataset Construction: 8 cases were manually re-transformed after infinite repetition issues occurred during reconstruction.This postprocessing step was used to ensure data quality.
  • B.1 Dataset Construction: Qwen2.5-14B-Instruct achieved 82.3% macro-averaged F1 in zero-shot BP classification, the highest among the tested models, and was selected as the classifier backbone.Table 7 covers Qwen2.5-14B-Instruct, Llama3.1-8B-Instruct, and GLM-4-9B-0414 under zero-shot and one-shot settings.
  • B.1 Dataset Construction: Adding the Legal Clause agent reduced cases lacking explicit legal clauses and increased their overall dialogue frequency.Figure 9 compares legal-clause counts before and after enhancement.

B.3 Intrinsic Evaluation of Data Quality

The intrinsic evaluation tests reconstruction quality across full and ablated pipelines. Results show specialized agents improve distinct quality dimensions, while the complete pipeline preserves information with only a marginal CP decline.

  • +14.0% automatic and +18.2% human CS improvement shows that the Text Purifier strengthens mediator-persona consistency.The Mediation Secretary improves Fidelity, while the Legal Counsel enriches legal provisions and supports Completeness.
  • The Mediation Secretary enhances Fidelity by extracting and preserving key information from original records.
  • The Mediation Secretary and Legal Counsel improve Authority and Completeness by integrating critical information and enriching legal provisions.
  • The complete pipeline incurs only a marginal decline in CP, indicating minimal information loss during dialogue rewriting.

C Training Details of ProMediAgent

ProMediAgent combines a decoupled strategy planner with a response generator and trains the planner through supervised strategy prediction followed by interactive reinforcement learning. Its simulated environment, AI-based rewards, bounded interaction flow, and auxiliary prompt templates support online mediation evaluation.

  • Architecture: ProMediAgent decouples a tunable policy planner π(σ_t|C_t) from a response generator to select mediation strategies from dialogue history and produce corresponding utterances.The planner can be implemented as a small language model, while the generator conditions on the mapped strategy instruction and dialogue history.
  • Interactive environment: The simulated environment role-plays dispute parties from annotated identity, situation, and self-claim profiles, while an LLM judge selects the next speaker.The environment follows the stated mediation workflow and supports online learning and evaluation.
  • Reward design: An LLM reward model classifies whether mediation has worsened, remained unchanged, improved, or been explicitly resolved, then supplies the planner’s scalar learning signal.Multiple decoded reward-model outputs are mapped to scalar values and averaged; the reward controls completion within a 20-turn limit.
  • Training procedure: Supervised fine-tuning minimizes cross-entropy between the planner’s strategy distribution and mediator-turn ground-truth labels restricted to the predefined strategy set S.Inputs contain the current mediation history prefixed with case background information.
  • Training procedure: After supervised fine-tuning, REINFORCE updates the planner from AI-feedback rewards using discounted returns, after which each predicted strategy directs context-aware response generation.The discounted return G_t reduces the weight of future delayed rewards through γ.
  • Evaluation and resources: The interaction flow records turns, completion, final reward, trajectories, and metrics including AT, SR@t, SSR, and MAD, alongside strategy and prompt resources for mediation processing and annotation.The supplementary materials include strategy overviews and prompt templates for text purification, secretary extraction, dialogue rebuilding, behavioral-pattern classification, and evaluation.
Loading 2609.11101v1…