PaperlayerView exact revision on arXiv

Source-linked AI summary

Stranger, Fan, or Peer? A Systematic Study on the Role of Interlocutor in Persona-Based Dialogue Generation

Daniela Occhipinti, Malvina Nissim, Marco Guerini

arXiv:2608.28467v1cs.CL

TL;DR

The paper addresses how interlocutor-biography visibility across training, inference, and evaluation affects persona-based dialogue generation and identifiability. It systematically varies these visibility conditions and finds that training-time visibility primarily determines copying, while asymmetric disclosure can leak target biography content into interlocutor turns.

  • Problem

    Prior work largely treated biography visibility as a single factor, leaving unclear how interlocutor information affects persona preservation and copying across training, inference, and evaluation.

  • Method

    The study varies target and interlocutor biography visibility across training and inference, evaluates generated dialogues with LLM-based author identification, and measures rare-word overlap as copying.

  • Results

    Training-time visibility primarily determines copying: models trained with interlocutor-biography visibility copy less target-biographical text, while inference-time changes have less consistent effects.

  • Takeaways & Limitations

    Separating training, inference, and evaluation visibility is necessary for understanding biography leakage and whether models adapt to interlocutors or reproduce biographical text.

  • Takeaways & Limitations

    The study uses a single movie-dialogue dataset, whose fictional characters and stereotyped roles limit generality across domains, profile formats, and conversational settings.

Abstract

from arXiv · show

Persona-based dialogue systems are usually conditioned on speaker biography, but dialogues involve at least two participants, and who has access to whose biography can vary across training, inference, and evaluation. Prior work often neglected these aspects, obscuring mechanisms that only appear when biography visibility is toggled separately across training, inference, and evaluation, a three-stage factorisation that prior work has largely treated as a single factor. We study this factorisation on a dataset of dialogues paired with speaker's biographies, varying whether the target and interlocutor speakers see each other's biographies during training and inference, and using an LLM as a judge to perform author identification. We find that (i) training-time visibility, more than inference-time visibility, determines whether models express persona traits through dialogue or fall back on copying biographical text (a known problem/phenomenon in persona-based generation); (ii) models trained with interlocutor-biography visibility copy less target-biographical text than models trained without it, while changing visibility only at inference time has a less consistent effect; and (iii) under asymmetric disclosure, where only the interlocutor sees the target biography, target content leaks into interlocutor turns more often, and dialogues containing such traces are easier for the judge to identify, especially when interlocutor turns are visible. These results suggest that biography leakage into generated turns is an artefact of how interlocutor visibility is configured across training and inference, and separating the three stages is necessary.

1 Introduction

Persona dialogue generation must account for what each participant sees across training, inference, and evaluation. This study separates those stages to examine persona expression, copying, and target identification under asymmetric and mutual biography visibility.

  • Persona systems condition models on speaker biographies to make utterances more consistent and recognisable.
  • The study compares Stranger, Fan, and Peer visibility configurations, from no disclosure to asymmetric and mutual biography access.
  • The central challenge is adapting to interlocutor characteristics while preserving the target persona instead of copying biographical text.
  • Prior evaluation varied interlocutor information available to the judge, but did not separate visibility across training, inference, and evaluation.
  • Training-time and inference-time visibility are tested separately, including whether aligning them produces different behaviour from mismatching them.
  • Under asymmetric disclosure, the study asks whether target biography details enter interlocutor turns and make the target easier to identify.
  • The study reports that training-time visibility primarily shapes copying, while asymmetric leakage increases target identifiability, especially across unrelated narrative worlds.

2 Related Work

Related work establishes persona conditioning, partner-aware generation, biography copying, and LLM-based identification as relevant foundations. The paper extends these lines by measuring how interlocutor information shapes generated dialogue across a three-stage visibility pipeline.

  • Persona dialogue research conditions models on target biographies, while partner-aware work additionally conditions generation on interlocutor personas.
  • Existing partner-aware studies show helpfulness but do not measure how interlocutor information changes the generated output.
  • Biography leakage commonly appears as copied biographical text rather than persona expressed through dialogue, especially in zero-shot language models.
  • Prior leakage work focuses on a speaker’s biography appearing in that speaker’s own turns, leaving cross-speaker propagation less studied.
  • The paper uses an LLM judge because humans perform poorly on persona authorship identification and related personality prediction tasks.

3 Experimental Design

The experiments vary biography visibility across training, inference, and evaluation using paired LLM agents, familiar and unfamiliar speaker pairings, and an LLM author-identification judge.

  • Dataset and Dialogue Setting: The setup uses two LLM agents, each conditioned on its own biography, while independently varying cross-biography visibility across training, inference, and evaluation.
  • Dataset and Dialogue Setting: PRODIGy is a movie-dialogue corpus with biographies, communication styles, personalities, and genders; the study focuses on biographies.
  • Dataset and Dialogue Setting: The experiments use 5,660 dialogues with biographies for both speakers, an 80:10:10 split, five-sentence biography truncation, and topic conditioning.
  • Dataset and Dialogue Setting: Agents alternate for 8 turns, with one target and one interlocutor, and training subset size fixed across configurations.
  • Familiar vs. Unfamiliar Pairings: Familiar pairings share a narrative background or generation source, whereas unfamiliar pairings violate the target’s established narrative context.
  • Models and Training Objectives: Generators use LLaMA 3.1 8B Instruct with separate LoRA adapters and masked loss computed only on the final target turn.
  • Interlocutor Biography Visibility: Strangers provide no cross-biography access, Fan gives only the interlocutor the target biography, and Peers provide mutual access.
  • Interlocutor Biography Visibility: Training compares TrnStrangers and TrnPeers, while inference evaluates InfStrangers, InfFan, and InfPeers; the reverse asymmetric condition is omitted from the main analysis.

4 Results

Across the experiments, training-time interlocutor visibility more strongly shapes copying and adaptation than inference-time visibility. Asymmetric disclosure can leak target-biographical information into interlocutor turns, making target identification easier, while lower copying need not reduce dialogue quality.

  • Training-time and inference-time visibility: Fine-tuning reduces BioTrg→TurnTrg overlap from 41–42% to below 5% while preserving recognisability at 0.672–0.854.Zero-shot accuracy is 0.936–0.971 but copies heavily from the target biography.
  • Training-time and inference-time visibility: TrnPeers lowers copying relative to TrnStrangers under both inference settings, from 3.82%→2.95% and 4.00%→2.65%.The reduction costs 2–4 accuracy points, whereas changing inference visibility shifts copying inconsistently.
  • Training-time and inference-time visibility: Training-time interlocutor visibility has larger effects on unfamiliar pairings, including a 5.7-point accuracy gap and copying changes from 3.32%→4.28%.Familiar pairings provide alternative identification routes through shared narrative background or a common generation source.
  • Training–inference interaction: Lower-copying TrnPeers dialogues are more lexically diverse, close 42% of the distinct-1 gap to gold, and contradict biographies no more often.Repetition and coherence shift only slightly, suggesting reduced copying changes reused content rather than dialogue amount.
  • Information leakage under asymmetric disclosure: Fan increases leakage more on unfamiliar pairings, by 1.1 points versus 0.6 on familiar pairings, with accuracy rising 0.76→0.81 versus 0.87→0.88.Rare-word traces in interlocutor turns are used by the judge as identification cues.

5 Conclusion

The study shows that biography visibility across training, inference, and evaluation shapes persona expression, copying, and identifiability. Training-time visibility is the strongest driver of copying, while asymmetric disclosure can leak target information through interlocutor turns.

  • Training-time interlocutor visibility determines how models use that information at inference.Models trained with mutual disclosure learn role separation and contextual adaptation, whereas models trained without it may compensate by copying when interlocutor information is introduced later.
  • Training with interlocutor biographies reduces target-biography copying more reliably than changing visibility only at inference.Withholding biographies at deployment does not reliably undo copying learned during fine-tuning.
  • Masked-loss training favors persona expression over biography reproduction while preserving recognisability.The conclusion recommends interlocutor-biography visibility during training together with masked loss for this objective.
  • Under asymmetric disclosure, target biographical content can propagate into interlocutor turns and affect target identifiability.This identifies a privacy-adjacent leakage channel created by unequal biography access.

Limitations

The study’s conclusions are bounded by its dataset, model family, and automatic leakage and authorship measures. These choices limit how directly the reported effects and effect sizes generalize beyond the evaluated settings.

  • Using a single movie-dialogue dataset limits generality across domains, profile formats, and conversational settings.The authors leave cross-domain validation for future work because movie dialogues may contain stereotyped roles and inherent biases.
  • The experiments use one backbone, LLaMA 3.1 8B Instruct, so exact effect sizes may not transfer to other model families or scales.The authors report that patterns appear across zero-shot and fine-tuned generators, but do not claim unchanged effect sizes elsewhere.
  • Rare-word overlap measures exact lexical transfer and is therefore a lower bound on leakage.It does not capture paraphrase or other semantic uptake.
  • The LLM judge is an automatic proxy for human judgement despite validation and complementary judge-free metrics.Validation uses held-out human-written dialogues and deliberately hard distractors.

Ethics Statement

Role-play agents pose established risks including toxicity, bias, hallucinations, and privacy violations. This study adds a privacy-adjacent concern: asymmetric biography disclosure can transmit target information through interlocutor turns.

  • Role-play agents can produce toxicity, bias, hallucinations, and privacy violations, creating safety concerns.Prior studies also report harmful-content generation and role-based biases.
  • Under asymmetric disclosure, target biographical content can propagate into interlocutor turns as an indirect persona-information leak.This concern arises when only the interlocutor has access to the target biography.
  • The study supplements PRODIGy with newly generated non-PRODIGy characters to control for contamination from public movie scripts.
  • The generated biographies follow the PRODIGy format and cover structured persona attributes from Persona Hub entries.GPT-4o-mini generates ten-sentence first-person biographies using assigned gender and MBTI attributes.

C Loss Masking: Objective and Ablation

The ablation compares standard next-token prediction with MaskLoss, which trains only on the final target turn while retaining biographies and dialogue history as context. MaskLoss reduces biographical copying while preserving turn-based recognisability in the selected setting.

  • C.1 Training Objectives: MaskLoss restricts the training objective to the final target turn, whereas NoMaskLoss predicts over all sequence tokens.Biographies and dialogue history remain available as context under MaskLoss but are excluded as prediction targets.
  • C.1 Training Objectives: Under MaskLoss, tokens outside the final target turn are masked from the loss.The objective is evaluated alongside NoMaskLoss and a zero-shot baseline under Peers training and inference.
  • C.2 Overall Comparison: The zero-shot model has the highest identification accuracy but also much higher rare-word overlap with biographies.Its recognisability is therefore strongly supported by surface reproduction in this ablation.
  • C.2 Overall Comparison: Among fine-tuned models, MaskLoss achieves the same TurnsDisc accuracy while reducing several forms of rare-word overlap.NoMaskLoss is slightly more accurate when the judge sees the interlocutor biography, but MaskLoss is selected for the main experiments.
  • C.3 Speaker Familiarity: For unfamiliar speakers, NoMaskLoss reaches 0.741 accuracy versus 0.711 for MaskLoss, alongside 3.200% versus 2.520% target-turn copying.MaskLoss trades some accuracy for lower surface-form reproduction when memorised or source-specific cues are least available.

E Asymmetric Training Regimes

The asymmetric regimes isolate training-time from inference-time disclosure. Leakage changes when disclosure occurs at inference for models trained without interlocutor biographies, but not for models trained with them.

  • Asymmetric training regimes: Asymmetric training regimes separate training-time and inference-time effects that the main text cannot distinguish.The appendix includes TrnFan, where only the interlocutor sees the target biography during training, and TrnI+I−, its converse.
  • Training effects: −0.09 pp leakage and +0.75 pp identification accuracy show no significant effect from training under Fan rather than Strangers.The paired tests report p = 0.85 for leakage and p = 0.21 for identification accuracy.
  • Inference effects: Under TrnStrangers, switching inference from Strangers to Fan raises leakage by 0.91 pp (p = 0.007).The same inference change is not significant under TrnFan, TrnPeers, or TrnI+I−, with all p > 0.3.
  • Interaction: Inference-time disclosure matters only for models trained without interlocutor biographies.The results suggest that training with interlocutor biographies removes the channel opened by inference-time disclosure.
  • Measurements: The full grid reports METEOR biography–turn overlap and judge accuracy, while rare-word leakage percentages come from the main text and Appendix G.Higher METEOR values indicate greater lexical similarity.

F.1 Gold Dialogue Baseline

Gold dialogues provide a low-copying reference, while fine-tuned models trade recognisability against biography overlap. Speaker familiarity consistently matters more than topic familiarity, and zero-shot recognition is strongly copy-based.

  • Gold dialogue baseline: Gold dialogues combine high judge accuracy with uniformly low biography–turn overlap.This establishes that human-written persona signals can be recognisable without heavy lexical reuse.
  • MaskLoss: TrnStrangers: MaskLoss TrnStrangers achieves strongest accuracy when inference supplies interlocutor-side information, while overlap stays near the gold range.Familiar speaker pairs score 0.864–0.879, whereas unfamiliar pairs range from 0.731 to 0.804.
  • MaskLoss: TrnPeers: MaskLoss TrnPeers lowers METEOR overlap in all three directions, especially target-biography overlap in target turns.The model incurs a small accuracy drop, most visible when evaluated without interlocutor context seen during training.
  • NoMaskLoss: TrnStrangers: NoMaskLoss TrnStrangers shows accuracy close to its MaskLoss counterpart, with similar overlap across inference configurations.The dominant factor is the absence of interlocutor-biography visibility during training rather than the loss objective alone.
  • NoMaskLoss: TrnPeers: NoMaskLoss TrnPeers recovers some accuracy but produces slightly higher biography–turn overlap than MaskLoss TrnPeers.This trade-off motivates using MaskLoss, because greater recognisability can come from making biography text more predictable.
  • Zero-shot baseline: Zero-shot accuracy is near ceiling but biography overlap is much higher than in gold dialogues and fine-tuned systems.Its high recognisability is accompanied by strong surface similarity, and topic familiarity contributes little additional diagnostic structure.

G Rare-Word Leakage Subsets

Exact rare-word overlap is a powerful identification cue, but its absence does not eliminate broader leakage through paraphrase, topic steering, or semantic uptake.

  • Rare-word leakage: 0.968–1.000 accuracy on overlap-positive subsets confirms that exact rare-word overlap strongly cues identification.These subsets are near ceiling in every configuration.
  • Rare-word leakage: Under TrnStrangers, no-overlap accuracy is higher with Fan than Strangers inference: 0.839 vs. 0.785 for TurnsDisc and 0.869 vs. 0.827 for BothDisc.The pattern suggests that exact rare-word overlap is a lower bound on broader leakage.

H Training–Inference Dialogue Examples

The examples contrast residual lexical leakage, absent adaptation, generic output, and thematic adaptation across training–inference configurations using the same target and interlocutor biographies.

  • Setup: The examples reuse Walter Sobchak as target and Peter Parker / Spider-Man as interlocutor for direct configuration comparisons.The biographies come from PRODIGy and the examples are illustrative rather than aggregate evidence.
  • Concordant no disclosure: Concordant TrnStrangers/InfStrangers shows Spider-Man-related terms surfacing despite masking.The setting illustrates residual lexical leakage.
  • Discordant disclosure: Discordant TrnStrangers/InfPeers does not visibly integrate the newly supplied interlocutor biography.Inference-time disclosure alone does not produce visible contextual use in this example.
  • Discordant masking: Discordant TrnPeers/InfStrangers produces generic output when expected interlocutor context is absent at inference.The model was trained to expect interlocutor context but receives none at inference.
  • Concordant mutual disclosure: Concordant TrnPeers/InfPeers produces thematic adaptation to Spider-Man’s photographer context.Aligned training and inference make contextual adaptation visible in the example.
Loading 2608.28467v1…