Robots in the Middle: Evaluating LLMs in Dispute Resolution

1. Introduction

• Intermediaries can calm emotions, clarify facts, identify key issues, and propose settlements, but their cost and limited availability constrain where they can be used.Technology-supported mediation is presented as a way to increase scalability and extend facilitated dispute resolution to new contexts.Source: §1 · pp. 1–2
• LLMs’ performance on complex, integrated dispute-resolution tasks remains under-explored despite strengths in discrete language tasks.The tasks require contextual understanding, emotion perception, and balanced, contextually appropriate solutions.Source: §1 · p. 2
• The study tests LLM selection of intervention types and drafting of intervention messages using a novel corpus of 50 hypothetical disputes.Its research questions also ask whether LLM-generated messages are safe and free of hallucinations.Source: §1 · p. 2

2. Related Work

• Earlier computational dispute-resolution systems used preference mechanisms, repeated offers, or potential court-outcome ranges to support settlement.The paper presents these approaches as groundwork for applying technology to mediation.Source: §2 · p. 2
• Prior NLP work considered language models for dispute analysis and facilitator recommendations, including an example based on Utah’s online dispute resolution system.Source: §2 · p. 2
• LLMs’ language and reasoning capabilities have been applied across legal tasks, including legal information, fiduciary roles, empirical research, text analysis, and expert systems.The paper notes that LLMs can be adapted through fine-tuning or prompt engineering.Source: §2 · p. 2
• This paper explores how LLMs select intervention types and generate messages, rather than evaluating direct practical deployment.Source: §2 · p. 3

3. Proposed Framework

• The LLMediator framework models two disputing parties communicating by text and a mediator intervening to help them reach resolution.The framework permits human, LLM-assisted human, or potentially fully automated mediation; this paper compares human-written and LLM-generated messages.Source: §3 · p. 3
• A screenshot depicts a dispute before the mediator intervenes.Source: Figure 1 · p. 3, §3 · p. 3
• The mediator first chooses an intervention type suited to the discussion, such as calming the exchange or encouraging information sharing.The type list is adopted from, and selection requires understanding context and empathy.Source: §3 · p. 3
• The mediator then drafts specific wording that communicates the chosen intervention clearly and empathically while avoiding mistakes and ambiguity.Source: §3 · p. 3
• The two-step framework makes it possible to compare human and LLM performance separately on intervention choice and message drafting.Source: §3 · p. 3

4. Experimental Design

• The study manually constructed 50 dispute scenarios and had humans and LLMs intervene under the same instructions, followed by blind evaluation on the same scenarios.Source: §4 · p. 4

4.1. Constructing disputes

• Each scenario contains two dialogues between Party A and Party B, totaling four messages, and the scenarios vary in dispute characteristics and communication styles.The scenarios span varied legal areas, including parcel delivery, property rights, and noise complaints.Source: §4.1 · p. 4

4.2. Human interventions

• Human annotators selected one to three intervention types and drafted one to two sentences per type; the annotators were co-authors with varying legal knowledge.The messages maintained a one-to-one correspondence between selected types and drafted text.Source: §4.2 · p. 4

4.3. LLM Interventions

• The LLM process used GPT-4o to select one to three types from a 13-type guide and generate messages; human-selected types were also supplied to the LLM for the message-quality comparison.This design compared type selection and message drafting as separate tasks.Source: §4.3 · p. 5

4.4. Evaluation

• Evaluators blindly compared intervention-type choices and messages, rating message preference and rubric criteria including contextualization, impartiality, empathy, and resolution quality.A separate check assessed hallucinations and safety; evaluators also considered whether particular interventions fit emotional exchanges or deadlocks.Source: §4.4 · pp. 5–6

5.1. E1 - Evaluation of intervention types

• Evaluators generally preferred LLM-selected intervention types, though preferences varied substantially across cases.The blind evaluation compared preferences using a five-point Likert scale.Source: §5.1 · p. 6, Table 4 · p. 7, Figure 2 · p. 7

5.2. E2 - Evaluation of intervention messages

• 84% of evaluators rated LLM-generated messages superior or equivalent to human messages, with LLMs outperforming humans in 60% of cases.LLM messages were equal to or better than human messages in 80%–96% of cases across categories.Source: §5.2 · p. 7, Figure 3 · p. 8

5.3. E3 - Safety evaluation of LLM-generated intervention

• Manual review found no harmful messages or hallucinations in this experimental scenario, but this does not guarantee safety at larger scale or in real-world applications.Figure 3 presents the distribution of evaluations across five metrics.Source: §5.3 · p. 7, Figure 3 · p. 8

6.1. RQ1 - To what extent can LLMs comprehend dispute scenarios and select appropriate intervention types?

• 22 cases favoured LLM-selected intervention types, 9 were ambivalent, and 19 favoured human-selected types.LLMs most often chose helping parties understand each other, exchanging information, and evaluating alternatives; humans most often chose exchanging information, productive expression, and mutual understanding.Source: §6.1 · p. 8

6.2. RQ2 - How do LLMs compare to humans in drafting intervention messages?

• 84% of scenarios had LLM-generated intervention messages rated better than or equal to human-written messages.Evaluators often found LLM messages smoother and clearer; they also noted fewer judgment errors than in human messages and lower ratings for human proposals that specified solutions or assigned fault.Source: §6.2 · pp. 8–9

6.3. RQ3 - To what degree are messages generated by LLMs safe and free of hallucinations?

• No unsafe messages or hallucinated information were observed in the generated messages.The authors note that this does not rule out such issues and suggest human review before messages are sent as a possible safeguard.Source: §6.3 · p. 9

6.4. On the use of gold-standard data

• The study compared human and LLM messages directly rather than treating human outputs as a reliable gold standard.The authors note that rating complex, nuanced tasks without obvious defects is difficult and subjective, and suggest annotations may represent individual views rather than one truth.Source: §6.4 · p. 9

7. Limitations

• Selecting an intervention type before drafting, then organizing message drafting by type, may impose artificial constraints on how mediators work and what interventions they write.The authors caution that this structured evaluation may not correspond to real mediator practice.Source: §7 · p. 10
• Disputes and messages were drafted and evaluated by people without mediation training, and none were native English speakers.The authors say this may advantage LLMs; the study cannot establish how trained mediators would approach the tasks.Source: §7 · p. 10
• Without real-world outcomes, determining which intervention type or message is better may remain subjective.Grammar errors and expectations about mediation tone may have mattered more in the comparison than they would in real contexts.Source: §7 · p. 10
• The design assumes four prior messages and that the mediator intervenes next, omitting later messages and the choice of when to intervene.The authors say these choices limit general applicability and recommend evaluating tools in real-world contexts with expert mediators.Source: §7 · p. 10

• LLMs performed on par with or surpassed human annotators in selecting appropriate intervention types and crafting effective intervention messages.Source: §8 · p. 10
• The authors highlight LLMs’ understanding of complex interactions, empathy, and neutrality, and suggest possible access-to-justice gains where mediation is inaccessible due to cost or availability.Source: §8 · p. 10
• Future work should incorporate multimodal data and pilot studies in actual ODR systems to assess practical effectiveness.Source: §8 · p. 10