PaperlayerView exact revision on arXiv

Source-linked AI summary

Individual and Domain Adaptation in Sentence Planning for Dialogue

F. Mairesse, R. Prasad, A. Stent, M. A. Walker

arXiv:1111.0048v1cs.CL

TL;DR

Spoken-language generation must adapt to domain, context, and users, yet individual linguistic preferences have been underrepresented in generation research. This paper presents SPaRKy, a trainable sentence planner that generates and ranks alternatives for restaurant information in MATCH. Its outputs approach template-based quality, individualized models generally outperform group-trained models, and n-gram features can match higher-level representations.

  • Problem

    Spoken-language generation must adapt to domain, context, and users, while prior individual adaptation mainly addressed content rather than sentence-planning form.

  • Method

    SPaRKy generates multiple sentence-plan realizations and ranks them with a statistical model trained on human feedback and rhetorical structure.

  • Results

    Individualized models generally outperform group-trained models; SPaRKy is comparable to MATCH templates, and its learned plans average less than 10% below human-selected plans.

  • Takeaways & Limitations

    Trainable generation can model individual sentence-planning preferences affecting content order, discourse structure, sentence structure, and sentence scope.

  • Takeaways & Limitations

    Training requires user ratings, and the proof of concept uses only two individuals.

Abstract

from arXiv · show

One of the biggest challenges in the development and deployment of spoken dialogue systems is the design of the spoken language generation module. This challenge arises from the need for the generator to adapt to many features of the dialogue domain, user population, and dialogue context. A promising approach is trainable generation, which uses general-purpose linguistic knowledge that is automatically adapted to the features of interest, such as the application domain, individual user, or user group. In this paper we present and evaluate a trainable sentence planner for providing restaurant information in the MATCH dialogue system. We show that trainable sentence planning can produce complex information presentations whose quality is comparable to the output of a template-based generator tuned to this domain. We also show that our method easily supports adapting the sentence planner to individuals, and that the individualized sentence planners generally perform better than models trained and tested on a population of individuals. Previous work has documented and utilized individual preferences for content selection, but to our knowledge, these results provide the first demonstration of individual preferences for sentence planning operations, affecting the content order, discourse structure and sentence structure of system responses. Finally, we evaluate the contribution of different feature sets, and show that, in our application, n-gram features often do as well as features based on higher-level linguistic representations.

1. Introduction

Spoken-language generation must adapt to domains, contexts, and individual users, but hand-crafted and single-output statistical methods struggle with linguistic variation. SPaRKy addresses this by generating and ranking sentence plans, with results showing domain-comparable quality and benefits from individualized models.

  • Human dialogue adapts content, syntax, wording, referring expressions, and prosody to conversational partners.
  • Individual adaptation remains largely limited to content planning rather than linguistic choices in sentence planning or surface realization.
  • Hand-crafted generators cannot feasibly encode individual sentence-planning rules and struggle to satisfy interacting domain and linguistic constraints.
  • SPaRKy generates multiple alternative sentence plans and ranks them using a model trained on human feedback, adapting to individual preferences.
  • SPaRKy learns plans significantly better than random choices and averages less than 10% below human-selected plans; n-gram features perform as well as higher-level representations.
  • Individualized models outperform group-trained models, while SPaRKy realizations are comparable to MATCH templates and learn rules difficult to hand-craft.

2. Related Work

Related work has adapted dialogue content to users and domains, while sentence-planning and realization research has addressed broader linguistic or stylistic variation. The paper extends this line toward individual differences in sentence planning.

  • Prior dialogue research used user models and context to adapt information content, but not presentation strategies to particular individuals.
  • SPoT showed that a trainable planner could match a travel-domain template generator, but its information-gathering utterances were simpler than information presentations.
  • Surface-realization work modeled lexical and syntactic adaptation, including user-specific n-gram models, but generally did not model individual text preferences.
  • Ranking from user feedback had been applied to multimedia planning and joint syntactic-realizer/text-to-speech optimization without studying individual differences.
  • Other research examined stylistic variation linked to personality, politeness, emotion, and domain-specific style.

3. Overview of MATCH’s Spoken Language Generator

MATCH generates restaurant recommendations and comparisons from SPUR content plans using either templates or SPaRKy. SPUR adapts selected content to the user, while SPaRKy generates and ranks alternative realizations.

  • MATCH is a multimodal dialogue system that provides restaurant and entertainment information, including route descriptions, recommendations, and comparisons.
  • SPUR: SPUR converts communicative goals into user-tailored recommendation or comparison content plans using conciseness and a decision-theoretic user model.Its user model selects restaurants and attributes predicted to interest the user.
  • SPUR: A content plan specifies assertions and rhetorical relations, but leaves assertion ordering and rhetorical realization unspecified.These plans form the input to both MATCH generators.
  • Template-Based Generator: The template-based generator produces one high-quality output per content plan, including sentence planning, cue insertion, clause combining, and referring expression generation.
  • SPaRKy: SPaRKy uses a sentence plan generator to create many realizations and a sentence plan ranker trained on user ratings to select among them.Because it is trained from feedback rather than handcrafted, SPaRKy supports individualized generation.

4. Sentence Plan Generation

SPaRKy generates alternative discourse and sentence plans from SPUR content plans, then represents each realization through rhetorical, syntactic, and referring-expression operations. Randomized generation supplies candidates for ranking while preserving rhetorical constraints.

  • The SPG takes a SPUR content plan and first performs discourse planning, followed by sentence planning.Content plans specify assertions and rhetorical relations, but not their order or rhetorical expression.
  • Discourse Planning: Bottom-up discourse planning groups content items by shared discourse center, while Centering constraints restrict allowable orderings.For example, alternatives that repeatedly change the discourse center are not generated.
  • Discourse Planning: The discourse stage produces tp-trees whose leaves are assertions with speech acts and whose interior nodes encode rhetorical relations, including infer when no relation is explicit.
  • Sentence Planning: During sentence planning, the SPG assigns assertions to sentences, orders them, inserts discourse cues, generates referring expressions, and incrementally builds sp-trees from d-trees.
  • Sentence Planning: Clause-combining operations are applied bottom-up and left-to-right, with the rhetorical relation constraining which operations may combine assertions.Operations can insert cue words whose choice depends on the rhetorical relation.
  • Sentence Planning: The SPG randomly samples alternative sp-trees using a hand-crafted distribution that favors preferred operations such as merge, relative-clause, and with-reduction.
  • Sentence Planning: A single tp-tree can yield substantially different realizations: Alternative 2 received an average rating of 4, whereas Alternative 6 received 2.5.

5. Feature Generation

The sentence plan ranker represents each SPG realization with automatically generated lexical, conceptual, and structural features. These features capture both surface sequences and the organization of sp-trees and d-trees.

  • Each candidate realization is encoded as real-valued features derived from its realization, sp-tree, and d-tree.Binary features are represented with values 0 and 1.
  • N-Gram Features: N-gram features capture lexical selection and ordering by counting unigram, bigram, and trigram patterns after named entities are replaced with types.
  • Concept Features: Concept features represent selected concepts and their linear order using concept n-grams extracted after restaurant entities are tagged.
  • Tree Features: Tree features encode structural configurations in sp-trees and d-trees, including how merge, infer, and cue-word operations are applied.
  • Tree Features: Local tree templates record traversal, sister, ancestor, and leaf configurations, while global templates summarize rule-specific leaf counts across an sp-tree.Features occurring fewer than 10 times are discarded to avoid content-plan-specific patterns.

6. Training the Sentence Plan Ranker

The sentence plan ranker learns to rank alternative realizations from user feedback, using automatically generated features and RankBoost. SPaRKy generates candidate sentence plans, while the learned ranker orders them for each content plan and user setting.

  • System and data: SPaRKy’s sentence plan generator produces alternative sentence plans, and its ranker learns to order them from human feedback.The system supports recommendation, compare-2, and compare-3 dialogue strategies.
  • Feature representation: Features are automatically generated from surface realizations and sentence plans, representing each alternative with real-valued feature values.The feature representation is used as input to the learned ranking function.
  • Training data: Training feedback consists of user ratings, with each realization represented as an ordered preference pair when one alternative is preferred to another.The training corpus can use averaged feedback from users A and B or separate feedback for individualized rankers.
  • Learning algorithm: RankBoost learns a function from realization features to feedback and produces a ranking over competing alternatives rather than selecting only one output.Its learned rule-based models can be inspected to compare individual and group preferences.
  • Objective and evaluation: The ranker score orders competing sentence plans for the same content plan, while training adjusts feature parameters to reduce misordered pairs.RankLoss is the percentage of pairs whose predicted order disagrees with the human preference ordering.
  • Objective and evaluation: Evaluation uses RankLoss and TopRank, where TopRank compares the human rating of the best realization with the rating of the realization predicted as best by the ranker.These metrics assess ranking errors and the quality gap between the predicted and human-selected top alternatives.

7. Quantitative Results

Across feature, template, and individualized-planner experiments, SPaRKy learned useful sentence-planning preferences but did not uniformly surpass the template generator. Individualized models generally outperformed population-level models, while performance depended on the dialogue strategy and user.

  • Feature Sets for Trainable Sentence Planning: N-gram features sometimes perform as well as the complete feature set, while concept features generally perform worse, especially for recommendations.All features always produced the best RankLoss, but differences were often not significant; lexical items associated with combination operators may explain n-gram performance.
  • Individualized Sentence Planners: Individualized planners were better than or statistically indistinguishable from the template generator in many cases, but preferences differed sharply by user and comparison task.User B preferred SPaRKy for compare-2, whereas User A preferred the template for compare-3.
  • Individualized Sentence Planners: The individualized results support modeling preferences for information ordering, sentence aggregation, and discourse-cue use in trainable sentence planning.The authors characterize these as specific individual preferences that can be modeled by the system.
  • Feature Sets for Trainable Sentence Planning: SPaRKy significantly outperformed random sentence-plan selection but remained significantly worse than the human-selected top plan.This pattern held across all three presentation types, with df = 59 and p < .001 for both comparisons.
  • Comparison with Template-Based Generation: The template-based generator performed better overall than the best trainable sentence planner, although SPaRKy showed a trend toward better performance for compare-2.Template was significantly better for recommend and compare-3, while the compare-2 difference favored SPaRKy at p = .055.
  • Comparison with Template-Based Generation: Template performance had wider variability than Human or SPaRKy, indicating poor performance on some inputs despite strong overall results.The authors relate this variability to the wide range of content selected by SPUR’s user model and conciseness settings.
  • Individualized Sentence Planners: Training an individualized ranker from approximately 120 examples could produce ranking error rates around 0.20, reducing the labeling burden relative to the 600-example corpus.This estimate comes from the training-data learning-curve analysis.
  • Individualized Sentence Planners: Individualized sentence planners generally outperformed models trained and tested across multiple individuals, with individual RankLoss ranging from 0.11 to 0.17 versus cross-tested ranges of 0.13 to 0.52.Average models performed worse for both users than individually tailored models.

8. Qualitative Analysis

The qualitative analysis shows that users prefer different sentence-planning operations, content orders, and discourse structures, while some preferences are shared or interact with content. These interpretable differences explain why individualized models can rank realizations differently.

  • Differences in Individual Models: User B prefers the claim first, whereas User A prefers food-quality information followed by cuisine information.These ordering preferences contribute to opposite rankings for alternative realizations.
  • Differences in Individual Models: User A and User B differ in their preferences for combination operators, although both dislike combining neighborhood information with subsequent content using with-ns.User A generally demotes with-ns, while User B generally favors it except in this interaction.
  • Differences in Individual Models: User B strongly penalizes relative-clause-infer after cuisine and food-quality information, illustrating preferences for specific content-operation interactions.The same cuisine and food-quality combination can therefore receive different ratings depending on how later information is integrated.
  • Differences in Individual Models: User A prefers merge-infer and food quality before cuisine, while User B reduces ratings for plans beginning with price or decor.These differences show that individual preferences concern both operations and the ordering of content items.
  • Average Model Differences: The interpretable model differences suggest that users’ differing perceptions of utterance quality reflect individual differences rather than random noise.The analysis connects learned rules to concrete differences in users’ ratings of alternative realizations.
  • Average Model Differences: The average model combines preferences resembling both users’ models, including conflicting preferences for claim-first and food-quality-first ordering.The authors note that accounting for contradictory preferences may reduce performance.

9. Conclusions

SPaRKy combines alternative realization generation with statistical ranking and supports sentence planning tailored to domains and individual users. Its outputs can approach a highly tuned template-based generator, while feedback collection, handcrafted resources, and limited participants remain important constraints.

  • Approach: SPaRKy generates alternative realizations and ranks them with a statistical model trained on human feedback.The approach also extends to rhetorical structures for complex information presentations.
  • Findings: The training technique generalizes to new domains and supports more complex rhetorical structures.These capabilities are demonstrated within trainable sentence planning for information presentation.
  • Findings: Individualized sentence planners generally perform comparably to MATCH’s tuned template-based generator, while human-selected outputs are as good or better.Exceptions depend on the comparison type and user, including User B preferring SPaRKy for compare-2 and User A preferring the template generator for compare-3.
  • Feature analysis: N-gram features can perform as well as linguistically structured tree features in some cases, although combined feature sets perform better for recommend and compare-3 than earlier tree features.The authors suggest lexical items may correlate with particular combination operators, but leave the explanation for future work.
  • Limitations: Training requires substantial user feedback, and SPaRKy still relies on a handcrafted dictionary of syntactic realizations.The paper discusses automatic behavioral feedback and learned generation dictionaries as possible directions.
  • Limitations: The individualized evaluation uses only two individuals, so broader populations are needed to characterize the range of preferences.The authors describe the results as a proof of concept and call for evaluation with more individuals.

SPaRKy

The supplied passages identify text-plan trees, dependency trees, sentence plans, and ranked sentence-plan outputs as representations in SPaRKy’s planning pipeline.

  • Representations: Text-plan trees are identified as “tp-trees” in the supplied representation labels.
  • Representations: Dependency trees are identified as “d-trees,” while sentence plans are identified as “sp-trees.”
  • Outputs: The pipeline includes a ranked list of sentence-plan-tree and DSyntS pairs.
Loading 1111.0048v1…