Source-linked AI summary
TalkMatrix: Generating Character Dialogue that is Both Consistent and Diverse
Ayuto Tsutsumi, Yuu Jinnai
TL;DR
Independent prompt-level selection may not meet collection-wide requirements when completion quality depends on relationships among outputs, as it does in character dialogue. TalkMatrix jointly selects lines across a character–situation matrix using consistency and diversity objectives. The reported evaluations find advantages over random and independent cell-level selection baselines, within role-playing scenarios.
Problem
Many applications need collections of completions whose quality depends on cross-prompt consistency or diversity, requirements that independent prompt-level selection may miss.
Source: §1 · p. 1Method
Results
Across 75 scenarios, coordinate ascent attains higher objective values than random selection, and on 25 curated scenarios a language-model judge rates selected matrices significantly better than random.
Source: §5 · p. 7Takeaways & Limitations
The reported main gain comes from optimizing a matrix-level objective over multiple cells rather than independently reranking each cell.
Source: §1 · p. 1Coordinate ascent is not guaranteed to find a global maximum, and the objectives and empirical evidence concern character dialogue.
Source: §1 · p. 1, §3.3 · p. 4
Abstract
from arXiv · showhide
Candidate-based decoding typically selects a completion for each prompt independently, but many applications require a collection of outputs that satisfies global, non-decomposable requirements. We formulate this setting as structured multi-prompt, multi-completion selection: given a candidate pool for every prompt, select one completion per prompt to optimize a collection-level objective. We instantiate the problem in character dialogue, where each character should remain consistent across situations, each line should fit its situation, and characters and situations should remain distinguishable. Our method, TalkMatrix, generates multiple candidates for every character--situation pair and jointly selects a complete matrix using four embedding-based consistency and diversity objectives. Because a weighted sum can improve some dimensions by sacrificing another, TalkMatrix maximizes the worst-performing objective through a two-level minimax formulation. We approximately optimize the resulting discrete objective with multi-start coordinate ascent, and compare it with local, partial-matrix, and generic combinatorial search baselines. We run experiments on $50$ synthetic role-playing scenarios and $25$ curated board game scenarios where multiple characters interact in predefined situations. An LLM-as-a-judge rates matrix-level selection higher than random and independent cell-level selection baselines. These results show the value of structured selection for globally controlled dialogue generation, while our empirical validation remains specific to role-playing scenarios.
1 Introduction
Independent prompt-level selection can miss collection-wide requirements because dialogue lines must work together across characters and situations. TalkMatrix addresses this with joint selection over a character–situation matrix and evaluates the resulting approach against selection baselines.
1 Introduction
- Dialogue lines must fit their local situations, preserve a coherent persona across a character’s lines, and keep characters and situations distinguishable.These requirements interact across the complete set of lines, creating a coupled selection problem.Source: §1 · p. 1
- Manual dialogue writing across many characters and situations is costly, while generation alone does not ensure a balanced collection.The paper focuses on scripted settings with fixed characters appearing across predefined situations.Source: §1 · p. 1
- TalkMatrix formulates dialogue construction as joint selection over a character–situation matrix, using four embedding-based objectives and a two-level minimax formulation.Its evaluation reports that the main gain comes from optimizing a matrix-level objective rather than independently reranking cells.Source: §1 · p. 1
- Across 50 role-playing scenarios and 25 curated scenarios, the reported main gain comes from matrix-level optimization rather than independent cell-level reranking.The evaluation uses a language-model judge and additional selection-policy baselines on the curated scenarios.Source: §1 · p. 1
2 Background
Standard generation selects a completion for one prompt, but collection-level quality can depend on choices across related prompts. TalkMatrix applies this structured selection perspective to character dialogue, where consistency and diversity must hold across a character–situation matrix.
2 Background
- For a single prompt, generation selects a completion from possible completions, commonly by sampling from the language model or using beam search to find a high-probability output.The selection goal is to maximize a scoring function measuring completion quality.Source: §2 · pp. 1–2
- Figure 1 organizes dialogue as characters by situations, with persona consistency and character distinction among its row- and column-level goals.Its stated requirements also include fitting each situation and varying appropriately across situations.Source: Figure 1 · p. 2
- Structured multi-prompt selection chooses one candidate per prompt while accounting for cross-prompt terms that make collection quality depend on the other selected candidates.This is post-hoc selection from independently generated pools, rather than joint generation by the language model.Source: §2 · p. 2
- Persona and role-playing work conditions generation on character profiles or traits, while TalkMatrix jointly selects lines across a whole character–situation matrix.The cited persona work includes Zhang et al. (2018), Zheng et al. (2019), and Chen et al. (2023); role-playing work includes Wang et al. (2024b), Tao et al. (2024), and Huang et al. (2025).Source: §2 · p. 2
- Role-playing-agent benchmarks assess dimensions including persona consistency and conversational ability, and this paper evaluates dialogue matrices with an LLM-as-a-judge protocol.The cited benchmark work includes Wang et al. (2024b), Yuan et al. (2025), Wu et al. (2025), He et al. (2025), and Ngokpol et al. (2025).Source: §2 · pp. 2–3
- Prior dialogue-diversity work modifies decoding or training objectives, whereas this paper targets variation across characters and situations within a fixed candidate set.The cited decoding approaches include Vijayakumar et al. (2016), Ippolito et al. (2019), Holtzman et al. (2020), Jinnai et al. (2024), and Li et al. (2024).Source: §2 · p. 3
3 TalkMatrix: Structured Multi-Prompt Selection for Character Dialogue
TalkMatrix generates candidate dialogue independently for each character–situation pair, then jointly selects a complete matrix. It evaluates the matrix with embedding-based consistency and diversity objectives combined through a minimax formulation.
3 TalkMatrix: Structured Multi-Prompt Selection for Character Dialogue
- TalkMatrix generates multiple candidate lines for each character–situation pair, then jointly selects one per pair to balance consistency and diversity across the matrix.The method focuses on the joint-selection stage after candidate generation.Source: §3 · p. 3
- The matrix has characters as rows and situations as columns, with each cell selected from an LLM-generated candidate pool.Selections interact across rows and columns, making the objective non-decomposable; the search is not guaranteed to find its global maximum.Source: §3 · p. 3
3.1 Evaluating Dialogue with Embeddings
- Content and style embeddings support four matrix-level requirements: situation diversity, character diversity, character consistency, and situation consistency.Diversity compares dialogues across situations or characters; consistency compares dialogue with character or situation references.Source: §3.1 · p. 3, §3.2 · p. 3
3.2 Objectives
- The minimax objective takes minima within requirements and across the four dimensions, so no weak dimension is compensated for by a stronger one.A local representativeness term discourages selecting outlier generations that score well globally but poorly represent their cell.Source: §3.2 · pp. 3–4
3.3 Optimization
- Multi-start coordinate ascent samples an initial matrix for each restart, updates cells to maximize the full objective, then returns the highest-scoring assignment.Each update holds other cells fixed, and the best assignment across restarts is returned after a fixed number of sweeps.Source: §3.3 · p. 4
- Coordinate ascent can end at a coordinate-wise local optimum or stop before reaching one, and neither case guarantees the global maximum.Random initialization makes the algorithm stochastic; fixing the seed makes runs reproducible under the stated cell order and tie-breaking.Source: §3.3 · p. 4
- The JAX implementation batches candidate evaluations and vectorizes restarts on one GPU, changing throughput but not the coordinate-ascent search rule.Source: §3.3 · p. 4
- The central modeling claim is joint selection for a non-decomposable matrix-level objective, not that coordinate ascent is the uniquely best optimizer.The evaluation compares coordinate ascent with independent cell-level selection, partial row/column greedy variants, simulated annealing, and TPE sampling.Source: §3.3 · p. 4
4 Experiment
Across synthetic role-playing and curated board-game scenarios, the experiments test TalkMatrix’s objective, search strategy, component choices, and lexical diversity. On curated scenarios, the minimax/minimax configuration improves the GLM judge’s overall score over random selection, while matrix-level search outperforms independent cell selection.
4 Experiment
- The evaluation covers 50 synthetic role-playing and 25 curated board-game scenarios, using the GLM judge’s overall score as its primary metric.Curated scenarios contain six characters and six situations; synthetic scenarios contain five of each.Source: §4 · pp. 4–5
4.1 Does Coordinate Ascent Improve the Objective over Random Selection?
- Coordinate ascent reaches a mean objective value of 0.967 ± 0.008 versus 0.935 for random, winning in all 75 scenarios.Optimization takes an average of 5.2 seconds per scenario on a single GPU; this comparison does not establish proximity to the global optimum.Source: §4.1 · p. 5, Table 2 · p. 5
4.2 Ablation Study on Multi-Objective Formulation
- The minimax/minimax configuration improves the GLM judge’s overall score by +0.98 over random (6.44 vs. 5.46) on 25 curated scenarios.It wins in 17 scenarios, ties in 3, and loses in 5.Source: §4.2 · p. 5, Table 3 · p. 5
4.3 Selection-Policy and Search Baselines
- Cell-level consistency reranking is insufficient (situation −0.62; character or combined +0.02), while TalkMatrix (+0.98) and column greedy (+0.86) outperform random.Cell MBR alone is stronger than the consistency-based cell methods (+0.58).Source: Table 4 · p. 5, §4.3 · p. 6
4.4 Comparison with a Black-Box Optimizer
- On 50 synthetic scenarios, Optuna TPE is indistinguishable from random with minimax (6.50 vs. 6.46; p = 0.7407), while its weighted-sum score is lower (6.13 vs. 6.46; p = 0.0204).The wall-clock comparison with coordinate ascent is not direct because Optuna uses many CPU operations while coordinate ascent is GPU-parallelized.Source: §4.4 · p. 6
4.5 Which Requirements Matter? Objective-Component Ablation
- Dropping CD alone falls below random (−0.10) and is 1.08 below the full objective; single SC comes closest in mean gain (−0.24 gap) and win rate (0.676 vs. 0.688).Removing SD or CC also degrades performance, but both variants remain above random.Source: §4.5 · p. 6, Table 5 · p. 6
4.6 Lexical Diversity
- TalkMatrix improves Distinct-1 and Distinct-2 over random, with positive differences in 21/25 and 20/25 scenarios, respectively.Distinct-1 and Distinct-2 are proportions of unique unigrams and bigrams within each character’s Japanese dialogues; n-grams do not cross dialogue boundaries.Source: §4.6 · p. 6, Table 6 · p. 7
5 Conclusions
TalkMatrix frames dialogue generation as joint selection across a character–situation matrix, balancing consistency and diversity with a minimax objective. Experiments report higher judged quality than random and independent cell-level selection, while validation remains specific to character dialogue.
5 Conclusions
- On 25 curated role-playing scenarios, minimax/minimax TalkMatrix matrices were judged significantly better than random; coupled matrix search also outperformed independent cell-level consistency reranking.The conclusion also reports higher objective values than random across 75 scenarios and identifies coordinate ascent as an effective default solver.Source: §5 · p. 7
- TalkMatrix selects one candidate per prompt under a non-decomposable collection-level objective, using four consistency and diversity requirements and a two-level minimax formulation.It approximately optimizes the objective with GPU-parallel, multi-start coordinate ascent.Source: §5 · p. 7
- The experiments validate the general formulation only in character dialogue; application to other multi-output tasks remains future work.The authors also propose tuning objective weights and operators against human judgments and integrating objectives directly into generation rather than post-hoc selection.Source: §5 · p. 7
A Prompts
The appendix gives the prompts used to generate dialogue candidates and judge complete matrices, followed by a translated example comparing selected matrices.
A Prompts
- The experiment prompts are in Japanese, with English translations provided for accessibility; braced expressions mark implementation-inserted values.Candidate-generation messages are sampled repeatedly for each character–situation cell.Source: A Prompts · p. 10, §A.1 · p. 10
A.1 Dialogue Candidate Generation Prompt
- Candidate-generation prompts frame the model as a role-playing scriptwriter and request one line for a named character in a described situation.The prompt includes separate character and situation specifications.Source: §A.1 · p. 10
A.2 Full-Matrix Judge Prompt
- The matrix judge receives a professional reference matrix defined as score 5, and its temperature is zero..Source: §A.2 · p. 10
- The evaluation criteria include character speech consistency and inter-character diversity, alongside situation consistency and inter-situation diversity.The criteria assess consistent character speech, differentiated personalities, situation-appropriate lines, and changes in each character’s dialogue across situations.Source: §A.2 · p. 10
- The judge scores from 0 to 10, analyzes strengths and areas for improvement, then assigns criterion-level and overall scores.The analysis provides the basis for the scores.Source: §A.2 · pp. 10–11
- The illustrated instance compares a TalkMatrix matrix with a uniformly sampled random matrix for six Three Kingdoms figures across six Go situations.Tables 8–13 show every cell of both matrices.Source: §A.2 · p. 11
- In the example, random selection includes identity confusions and battlefield narration that replaces the Go situation; TalkMatrix avoids two conspicuous identity confusions but retains martial metaphors.The selected matrices are shown in Tables 8–13.Source: §A.2 · p. 11
- Table 7 presents the character specifications for the Three Kingdoms–Go example, translated from the generation input.The example characters are described as having distinct personalities and backgrounds.Source: Table 7 · p. 12