PaperlayerView exact revision on arXiv

Source-linked AI summary

PragAlign: Feedback-Guided Pragmatic Alignment for Controlled Synthetic Dialogue Generation

Smitha Muthya Sudheendra, Jaideep Srivastava

arXiv:2609.02480v1cs.CL

TL;DR

Synthetic service dialogues need to preserve communicative intent, affective meaning, and natural flow when real conversations are private or difficult to share. PragAlign uses feedback-guided generate–evaluate–revise control and achieves 99.50% evaluator-defined acceptance on 800 specifications, while exposing limits in emotion realization and independent human-quality validation.

  • Problem

    Synthetic dialogue must preserve service intent, affective meaning, coherence, and fluency when real customer–agent conversations are private, costly, or difficult to share.

  • Method

    PragAlign generates controlled dialogues from service context, target intent and emotion, and trait-style controls, then uses criterion-specific LLM feedback for up to three revisions.

  • Results

    99.50% evaluator-defined acceptance on 800 matched specifications exceeds 72.25% for one-shot generation and 95.88% for repeated generation without structured feedback.

  • Takeaways & Limitations

    Structured feedback primarily improves complete evaluator-defined constraint satisfaction beyond repeated generation, with refinement gains concentrated in emotion alignment.

  • Takeaways & Limitations

    Automatic acceptance depends on an evaluator that also guides refinement, and the human evaluation is not a paired comparison of experimental conditions.

Abstract

from arXiv · show

Synthetic dialogue generation can support research in privacy-restricted service settings, but generated conversations must preserve communicative intent, affective meaning, and natural dialogue flow. We introduce PragAlign, a feedback-guided framework for controlled synthetic dialogue generation conditioned on service context, target intent, and target emotion, with auxiliary trait-style controls. PragAlign uses a generate--evaluate--revise loop in which an LLM-based evaluator scores intent alignment, emotion alignment, coherence, fluency, and aggregate quality, then provides criterion-specific feedback for up to three refinement rounds. On 800 matched dialogue specifications, PragAlign achieves 99.50\% evaluator-defined acceptance, compared with 72.25\% for one-shot generation and 95.88\% for repeated generation without structured feedback. This indicates that repeated attempts account for much of the gain over one-shot generation, while structured feedback primarily improves last-mile multi-constraint satisfaction rather than broad average quality. Refinement gains are concentrated in emotion alignment, which is also the dominant failure mode in ablations. A separate human evaluation of 1,200 generated dialogues shows that intent expression and dialogue flow are highly recognizable to annotators, while emotion appropriateness is less stable and more subjective. These results support PragAlign as a quality-control framework for improving evaluator-defined communicative constraint satisfaction, while showing that affective realization and independent human-perceived quality remain open challenges.

Introduction

PragAlign frames synthetic service-dialogue generation as pragmatic constraint satisfaction, combining controlled inputs with evaluator-guided revision. Across complementary automatic and human analyses, it improves evaluator-defined acceptance, especially for emotion alignment, while distinguishing that measure from independent human-quality evidence.

  • The framework uses a generate–evaluate–revise loop in which criterion-specific evaluator feedback guides up to three refinement rounds.
  • 99.50% automatic acceptance on 800 matched specifications exceeds 72.25% for one-shot generation and 95.88% for repeated generation without structured feedback.
  • PragAlign generates controlled service dialogues under explicit intent, emotion, coherence, and fluency constraints.
  • Structured evaluator feedback improves complete constraint satisfaction beyond repeated generation alone, with strongest refinement effects in emotion alignment.
  • The study combines ablations and a separate human evaluation while treating automatic acceptance as evaluator-defined rather than independent human-quality evidence.

Related Work

PragAlign connects iterative feedback and model-based evaluation with attribute-conditioned dialogue generation. Its distinctive framing treats service-dialogue synthesis as communicative-goal alignment rather than general text-quality optimization.

  • Iterative refinement and model-based evaluation: Prior work includes iterative feedback, verbal reflection, deliberative search, and model-based evaluation for improving or assessing generated text.
  • Attribute-conditioned dialogue generation: Related dialogue-generation research conditions outputs on emotion, empathy, persona, user needs, knowledge, personality, and multiple attributes.
  • Positioning of PragAlign: PragAlign formulates synthetic service-dialogue generation as a communicative-goal alignment problem rather than optimizing general text quality alone.
  • Positioning of PragAlign: Its acceptance decision is evaluator-defined pragmatic constraint satisfaction, not an independent measure of human-perceived dialogue quality.

Model Framework

PragAlign represents each service-dialogue goal structurally, generates a candidate, evaluates multiple communicative dimensions, and revises failures while preserving the original specification. The closed loop therefore targets better realization of fixed intent, emotion, style, and service-context constraints.

  • Framework: PragAlign uses an LLM generator and evaluator in a closed loop for controlled multi-turn service-dialogue generation.
  • Goal Representation: Each goal representation includes service context, an auxiliary trait-style vector, target intent and emotion sets, and generation hyperparameters.
  • Goal Representation: The domain and intent inputs ground dialogues in service settings and functional goals such as billing clarification, refunds, account access, complaints, and follow-up requests.
  • Goal Representation: Target emotions may appear directly through emotion words or indirectly through tone, urgency, concern, frustration, relief, or reassurance.
  • Evaluation and Revision: The evaluator scores intent alignment, emotion alignment, coherence, and fluency, then returns criterion-specific feedback when thresholds are not met.
  • Evaluation and Revision: The original domain, intent, emotion, and personality specification remains fixed throughout refinement, so revision strengthens realization rather than changing the target.
  • Evaluation and Revision: Accepted dialogues exit the refinement loop without further revision, while failed candidates are regenerated using the original representation and feedback.
  • Evaluation and Revision: Automatic acceptance measures evaluator-defined constraint satisfaction, while human evaluation separately checks recognizable intent, flow, and emotion cues.

Experimental Setup

The experiments compare controlled dialogue generation conditions using matched specifications, automatic evaluator criteria, and separate human validation. PragAlign permits up to three feedback-guided refinements, while the no-feedback baseline uses fresh generation attempts without criterion-specific feedback.

  • Experimental Design: 800 matched dialogue specifications are used across automatic experimental conditions for paired comparison.Each specification includes a service-domain scenario, target intent set, target emotion set, and auxiliary trait-style control vector.
  • Experimental Design: The specifications cover 10 approximately balanced customer-facing service domains.Each domain contributes between 72 and 89 specifications.
  • Generation Conditions: PragAlign allows up to three refinement rounds, whereas the no-feedback baseline uses the same maximum attempts but regenerates from the original specification.The baseline does not pass the previous dialogue, failed criteria, missing-intent or missing-emotion lists, or evaluator rationale to the next prompt.
  • Evaluation Protocol: Automatic evaluation scores intent alignment, emotion alignment, coherence, fluency, and aggregate quality.Aggregate quality is computed from these component measures, while acceptance also requires individual component thresholds.
  • Evaluation Protocol: A dialogue is accepted only when aggregate quality and every component-level threshold are satisfied.The reported thresholds are Q(d) ≥ 0.80, SI(d) ≥ 0.80, SE(d) ≥ 0.70, C(d) ≥ 0.80, and F(d) ≥ 0.80.
  • Human Validation: The study supplements the 800-dialogue automatic evaluation with a separate human evaluation of 1,200 generated dialogues.The human-evaluated set is treated as dataset-level validation rather than matched automatic comparison.

Results

PragAlign’s main benefit is higher evaluator-defined complete multi-constraint satisfaction, with structured feedback improving on repeated sampling mainly through targeted repairs, especially for emotion alignment. Refinement yields diminishing returns, while automatic gains do not fully establish human-perceived affective quality.

  • 99.50% of 800 specifications were accepted by full PragAlign, versus 72.25% for one-shot generation and 95.88% for no-feedback repeated generation.Full PragAlign accepted 796 specifications; no-feedback generation accepted 767.
  • 3.62 percentage points separated full PragAlign from no-feedback generation, showing structured feedback improved last-mile complete threshold satisfaction beyond repeated sampling.The continuous aggregate-quality difference from no-feedback generation was only 0.0024 and was not significant.
  • 73.75% of dialogues were accepted initially, rising to 95.25% after one refinement, with later rounds adding 3.38 and 0.87 percentage points.Among 210 refined dialogues, 172 were accepted after the first refinement, 27 after the second, and seven after the third.
  • 0.340 was the mean improvement in emotion alignment among refined dialogues, exceeding the 0.025 improvement in intent alignment while coherence changed little and fluency decreased marginally.Aggregate quality increased by 0.059 on average.
  • Explicit emotion conditioning had the largest ablation impact: removing it reduced acceptance to 79.75% and increased mean refinement rounds to 1.221 from 0.324.Removing structured feedback reduced acceptance to 95.88%, while removing personality produced 99.75% acceptance without improving evaluated controls.
  • Emotion misalignment dominated failures, yet automatic emotion scores exceeded human judgments: 0.998 mean automatic alignment versus 79.1% positive human appropriateness.Only four full-method dialogues failed the automatic emotion threshold, whereas human emotion appropriateness had 75.8% raw agreement and AC1 of 0.639.

Limitations

The study’s automatic results depend on an evaluator that also guides refinement, while human evaluation is unpaired and affective and personality controls are not fully validated.

  • Evaluator dependence: 99.50% automatic acceptance should be interpreted as evaluator-defined constraint satisfaction, not an independent measure of human-perceived dialogue quality.The generator and evaluator both come from OpenAI model families, motivating reevaluation with independent model families.
  • Human evaluation scope: The human evaluation samples 1,200 dialogues separately from the 800 matched automatic-evaluation specifications, preventing paired preference comparisons across conditions.It provides dataset-level validation but does not establish that humans prefer PragAlign outputs for the same goals.
  • Affective evaluation: Human emotion appropriateness is positive in 79.1% of cases and less consistent than intent expression or dialogue flow, revealing a calibration gap.The automatic evaluator may reward explicit affective markers without fully capturing contextual appropriateness.
  • Personality validation: Personality conditioning is not directly validated because neither automatic nor human evaluation measures whether the traits are recognizable in generated dialogues.Personality is therefore treated as an input conditioning variable rather than a validated output attribute.
  • Metric and threshold scope: Additional limitations include ceiling effects in intent, coherence, and fluency metrics and dependence on hand-selected acceptance thresholds.These constraints limit how broadly the reported automatic metrics should be interpreted.

Conclusion

PragAlign uses feedback-guided refinement to generate service dialogues under explicit communicative constraints and improves evaluator-defined acceptance beyond one-shot generation and repeated generation without structured feedback. The results also identify emotion alignment and independent human validation as continuing challenges.

  • Conclusion: PragAlign is a feedback-guided framework for generating synthetic service dialogues under explicit communicative constraints.The constraints concern communicative properties such as intent, emotion, coherence, and fluency.
  • Conclusion: Across 800 matched specifications, PragAlign improves evaluator-defined acceptance over one-shot generation and further improves complete constraint satisfaction beyond repeated generation without structured feedback.The framework’s strongest refinement effects concern emotion-alignment failures.
  • Conclusion: Reliable affective control requires calibration against human judgment rather than evaluator acceptance alone.Human annotators found emotion appropriateness less stable than intent expression or dialogue flow.

Prompting and Evaluation Setup

PragAlign combines structured prompting for generation and revision with an evaluator that scores four component dimensions and aggregate quality, accepting dialogues only when all thresholds are met.

  • Prompting: PragAlign uses initial generation, evaluator-guided revision, and automatic evaluation stages.Revision preserves the original specification while adding feedback about failed criteria, missing targets, and revision advice.
  • Evaluation: The evaluator scores intent alignment, emotion alignment, coherence, and fluency on a [0, 1] scale and returns criterion-specific feedback.Intent and emotion scores use detected target coverage, whereas coherence and fluency are scored directly.
  • Evaluation: Aggregate quality Q(d) combines A(d), C(d), and F(d), with A(d) defined as the mean of intent and emotion alignment.C(d) denotes coherence and F(d) denotes fluency.
  • Acceptance rule: A dialogue is accepted only when Q(d) ≥ 0.80, SI(d) ≥ 0.80, SE(d) ≥ 0.70, C(d) ≥ 0.80, and F(d) ≥ 0.80.This prevents high aggregate quality from masking a component-level failure.

Dataset and Goal Specification

Each dialogue goal specifies service context, auxiliary trait-style controls, target intents, target emotions, and generation hyperparameters, while the automatic study uses matched specifications with bounded generation settings.

  • Goal specification: Each dialogue goal is represented by service-domain context D, trait-style vector p_i, target intent set I*_i, target emotion set E*_i, and hyperparameters Ω.These variables define the controlled generation specification.
  • Dataset: The 800 matched specifications contain service scenarios, two or three target intents, one to three target emotions, and an auxiliary Big Five-style control vector.The trait-style vector is used only as a generation-side control and is not evaluated as an output attribute.
  • Generation settings: Generated dialogues are constrained to six to eight turns, with temperature 0.7 and a maximum output length of 1024 tokens.Unless otherwise stated, GPT-4.1 generates dialogues and GPT-4o-mini evaluates them.
  • Generation settings: The full PragAlign condition allows up to three refinement rounds after the initial generation attempt.This bounds the iterative refinement process used in the main setup.

Evaluator Prompt and Output Schema

PragAlign uses structured generator and evaluator outputs to score controlled dialogues and guide refinement. The supplied materials also document human validation, threshold sensitivity, generator transfer, and reproducibility artifacts.

  • Evaluator Prompt and Output Schema: The evaluator separately assesses intent alignment, emotion alignment, coherence, and fluency while labeling detected intents and emotions per utterance.It returns scalar scores in [0, 1], evidence, weaknesses, and revision advice.
  • Evaluator Prompt and Output Schema: The generator returns dialogues as ordered speaker-text turns, which the evaluator processes as structured dialogue objects.
  • Evaluator Prompt and Output Schema: Scalar scores determine acceptance, whereas weaknesses and revision advice are used only when a dialogue enters refinement.This separates evaluation from the guidance supplied to later revision rounds.
  • Human Evaluation Protocol: Human validation covered 1,200 final-dataset dialogues, separately from the 800 automatic-evaluation specifications, so it was not a paired condition comparison.Two English-speaking annotators judged intent expression, dialogue flow, and emotion appropriateness without seeing automatic scores or refinement histories.
  • Human Evaluation Protocol: Gwet’s AC1 was reported because highly positive-skewed intent-expression and dialogue-flow labels can make Cohen’s κ underestimate agreement.The paper reports positive rate, raw agreement, and Gwet’s AC1.
  • Threshold Sensitivity and Ceiling Effects: Quality-only acceptance creates near-ceiling results across methods, showing that aggregate quality alone can obscure target intent or emotion failures.Full PragAlign remains above the comparison conditions under the stricter all-criteria threshold.
  • Generator Transfer: With Llama as generator and GPT-4o-mini as evaluator, PragAlign accepted 250 of 300 dialogues, an automatic acceptance rate of 83.33%.Mean quality was 0.907; intent alignment was 0.971, emotion alignment 0.956, coherence 0.850, and fluency 0.909.
  • Reproducibility Artifacts: Recorded outputs, iteration histories, scores, decisions, failure reasons, improvement summaries, and human-evaluation exports support inspection of refinement trajectories and condition comparisons.
Loading 2609.02480v1…