An Information Retrieval Approach to Short Text Conversation

Zongcheng Ji; Zhengdong Lu; Hang Li

Source-linked AI summary

An Information Retrieval Approach to Short Text Conversation

Zongcheng Ji, Zhengdong Lu, Hang Li

arXiv:1408.6988v1Aug 29, 2014cs.IRcs.CL

TL;DR

The paper addresses short text conversation, a simplified human-computer conversation problem for which large real-conversation datasets have been limited. It formalizes STC as retrieval, combines learning-to-rank with matching models over social-media data, and finds precision at position one of 0.64 while documenting important limitations.

Problem
Human-computer conversation remains difficult, partly because large volumes of real conversation data have been limited; the paper focuses on one-round short text conversation.
Method
The paper retrieves existing responses from a large social-media conversation repository and ranks them using learned combinations of multiple matching models.
Results
Precision at position one is 0.64, and all matching models significantly improve retrieval performance in the reported experiments.
Takeaways & Limitations
A retrieval-based model can perform reasonably well when paired with carefully designed matching features and a huge repository of conversation data.
Takeaways & Limitations
Lexical gaps can rank suitable responses very low, and entity association is only partially addressed by treating entity names as words.

Abstract

from arXiv · show

Human computer conversation is regarded as one of the most difficult problems in artificial intelligence. In this paper, we address one of its key sub-problems, referred to as short text conversation, in which given a message from human, the computer returns a reasonable response to the message. We leverage the vast amount of short conversation data available on social media to study the issue. We propose formalizing short text conversation as a search problem at the first step, and employing state-of-the-art information retrieval (IR) techniques to carry out the task. We investigate the significance as well as the limitation of the IR approach. Our experiments demonstrate that the retrieval-based model can make the system behave rather "intelligently", when combined with a huge repository of conversation data from social media.

1. Introduction

The paper simplifies human-computer conversation to one-round short text conversation and studies it as an information retrieval problem using large-scale social-media data. It proposes a retrieval framework with multiple matching models and reports fairly good performance while examining the approach’s significance and limitations.

Short text conversation consists of a human message followed by a computer response in one conversational round.
The paper motivates social-media conversation data as a large resource for advancing research on short text conversation.Twitter and Weibo provide extremely large amounts of short conversation data.
Retrieval-based STC retrieves related responses from a repository and selects an existing response instead of generating a new one.
The proposed framework combines learning to rank with cosine, translation, latent-space, deep-matching, and topic-word models as matching features.The topic-word model is devised specifically for short text conversation.
Precision at position one reaches 0.64, and all matching models significantly improve retrieval performance in large-scale Weibo experiments.The study also uses case analysis to examine the significance and limitation of the IR approach.
The paper contributes an IR approach, empirical evaluation, a topic-word model, and a publicly released dataset for STC research.

2. Related Work

Earlier dialogue systems relied on rules, limited-data learning, or small question-answer repositories, while newer work exploits large social-media conversation collections and IR-related methods. The paper situates retrieval-based STC among these approaches and distinguishes it from related tasks such as community question answering.

Rule-based and learning-based dialogue systems required substantial manual modeling effort and offered unsatisfactory coverage.
Knowledge-base dialogue systems select responses from question-answer pairs, but creating the knowledge base is a major bottleneck.
Social media made very large amounts of naturally occurring short conversation data available for dialogue research.
The paper conducts STC using state-of-the-art IR technologies and large-scale social-media conversation data.
Learning to rank and semantic matching are established techniques for retrieving and matching documents against queries.

3. Conversation on Social Media

Weibo post-comment pairs provide short, flexible, and diverse conversational exchanges that can serve as data for studying STC. Many pairs are self-contained, although some require contextual or background information to understand fully.

Weibo limits posts and comments to 140 Chinese characters, creating short-text exchanges between users.
Post-comment pairs contain flexible forms and diverse topics, while comments can remain appropriate responses to a post.
Table 1 presents a translated example of a Weibo post with associated comments.
A single post can receive comments that ask questions, make requests, offer greetings, or address other conversational purposes.
Many post-comment pairs are self-contained and require no background knowledge or contextual information for understanding.
Some exchanges require extra context, such as knowing that a Hawaii trip concerns an international conference and talk.

4. Retrieval-based Short Text Conversation

The paper formalizes STC as retrieving the highest-scoring response from a repository of post-comment pairs. Its three-stage architecture retrieves candidates, computes matching features, and ranks responses with a learned linear function.

4.1. Problem Definition: STC is formalized as an IR task in which a query retrieves related responses from a large repository.
4.1. Problem Definition: For query q, the system selects the response r from post-comment pairs (p, r) with the highest ranking score.
4.1. Problem Definition: The ranking score is an ensemble of individual matching features, whose weights combine their contributions.
Stage I: retrieval: Stage I uses three fast basic linear matching models to retrieve a reduced candidate set.
Stage II: matching: Stage II applies additional learned matching models to evaluate candidates and produce a matching-feature set for each pair.
Stage III: ranking: Stage III assigns candidate ranking scores with a learned linear function, ranks the responses, and returns the highest-scoring comment.
Learning to rank: RankingSVM is trained offline from pairwise preferences derived from labeled positive and negative post-comment instances.

5. Short Text Conversation Dataset

The dataset contains a large retrieval repository of Weibo post-comment pairs and a separately labeled subset for training and testing. It is constructed through focused sampling, filtering, cleaning, and relevance-based manual labeling.

Original Post-Comment Pairs: The original pairs are sampled from Weibo posts and the comments they received, which need not come from the crawled community.
Original Post-Comment Pairs: The dataset is created by crawling a loosely connected user community, collecting posts and comments, and cleaning the resulting data.The user community comprises NLP and ML researchers and students in China, and the posts were collected over two months.
Processing, Filtering, and Data Cleaning: Filtering removes very short pairs, comments addressing other comments, likely advertisements, and comments beyond the first 100 per post.The first-100 restriction reflects observations that later comments increasingly address earlier comments rather than the original post.
Original Post-Comment Pairs: The retrieval repository contains 38,016 Weibo posts and 618,104 corresponding comments, forming 618,104 original post-comment pairs.These pairs are used as the retrieval repository in all experiments.
Labeled Post-Comment Pairs: Comment suitability is judged by semantic relevance, entity association, logical consistency, and speech-act agreement.The criteria distinguish topical relevance and entity alignment from logical consistency and whether the response matches the query’s communicative act.

6. Matching Features

The retrieval-based system combines fast basic matching models with learned semantic and topic-sensitive features. These features support candidate retrieval and later ranking, while also addressing lexical gaps between queries and responses.

Advanced Matching Models: Translation, deep matching, and topic-word models extend the feature set to address lexical gaps, nonlinear matching, and topic alignment.
Basic Matching Models: Three basic linear matching models support fast retrieval in Stage I and generate matching features for Stage II.
Query-Response Similarity: Query-response cosine similarity compares TF-IDF vectors and provides a simple, learning-free signal for finding relevant responses.It can capture relations that semantic matching features miss and remains useful for infrequent words.
Query-Post Similarity: Query-post similarity retrieves comments associated with posts that resemble the query, but can fail when a response addresses information absent from the query.Combining this measure with other features can alleviate the failure mode.
Latent Space Model: A latent-space model learns mappings for posts and responses so their inner product captures semantic matching beyond word-to-word overlap.The mappings are learned from many query-response pairs using a large-margin method.
Latent Space Model: The learned linear model captures meaningful semantic patterns from massive data, such as associating Italy with Sicily, the Mediterranean Sea, and travel.

6.2. Translation-based Language Model

The retrieval-based STC model addresses lexical gaps with a translation-based language model, then extends matching with DeepMatch to capture hierarchical semantic relations and rank responses.

Motivation: Basic matching models can perform fairly well, but lexical gaps can rank suitable responses below the top 30 candidates.In the example, suitable responses share no words with the query, while their original posts share “晚安(Good Night)” with it.
Translation-based Language Model: TransLM bridges lexical gaps by assigning non-zero probability to translations between semantically related words in queries and candidate post-comment pairs.It combines unigram language models for posts, responses, and the collection with word-to-word translation probabilities, smoothing, and interpolation.
Translation-based Language Model: Unlike the CQA version, this TransLM adds a response-side translation component because STC retrieves suitable responses rather than similar questions.The design also reflects that STC responses tend to be shorter than posts, whereas CQA answers tend to be longer than questions.
Translation-based Language Model: Translation probabilities are learned from original post-comment pairs using GIZA++, with posts and responses serving as alternative source-target configurations.The passage describes two source-target settings for constructing the parallel corpus.
Deep Matching Model: DeepMatch replaces purely linear matching with local bilinear models followed by a deep neural network that produces the final matching score.Its design targets complicated matching relations and combines local decisions through a multi-layer architecture.
Model Training: DeepMatch is trained in two phases: bilingual topic modeling builds matched word-pair subsets and the architecture, followed by parameter training with a ranking-based large-margin objective.The objective uses triples where a query matches a positive response better than a negative response; m controls the margin and is set to 2 experimentally.

6.4. Topic-Word Model

The topic-word model addresses topic mismatch by identifying words that represent a short text’s main theme and weighting them in query-response matching. It uses logistic regression trained on labeled short texts and incorporates lexical, positional, entity, and part-of-speech features.

6.4.1. Motivation: Topic-word matching addresses cases where frequent but irrelevant words cause unsuitable responses to outrank responses sharing the query’s main topic.In the example, “菜鸟(rookie)” dominates basic matching, while “代码控制工具(code control tool) SVN GIT” identifies the relevant topic.
6.4.1. Motivation: The model identifies topic words as words that capture a short text’s specific theme.Topic words are operationally distinguished from other words in the text.
6.4.2. Learning Topic Words: Logistic regression estimates P(topic|w), the probability that word w is a topic word.The binary topic variable indicates whether w is a topic word.
6.4.2. Learning Topic Words: Features include TF, IDF, sentence frequency, first or last position, named-entity indicators, and part of speech.The feature design reflects observed tendencies for topic words to occur in early or final sentences and often be nouns or verbs.
6.4.2. Learning Topic Words: Training labels 2,008 words from 200 short texts, and the resulting classifier reaches 81.57% accuracy.The model uses ICTCLAS for linguistic information and LIBLINEAR for logistic regression.
6.4.3. Model Description: Predicted topic-word probabilities become term weights for vector-space similarities between a query and a candidate response or original post.The weighted similarities are used as matching features in retrieval-based STC.

6.5. Other Simple Matching Features

The framework supplements semantic matching with simple features that capture shared strings and word co-occurrence between queries and candidate responses or posts.

6.5. Other Simple Matching Features: Longest Common String measures the longest shared string between a query and candidate response.It captures quoted text and is relatively robust to Chinese word-segmentation errors.
6.5. Other Simple Matching Features: Co-occurrence features summarize the size, rate, sum, and average IDF of words shared by the query and candidate response or original post.These features represent both the amount and informativeness of lexical overlap.

7. Experiments

Experiments evaluate retrieval models with MAP and P@1 on 422 queries using five-fold cross-validation. Adding TransLM, DeepMatch, and TopicWord to the baseline significantly improves retrieval, with the full combination performing best.

7. Experiments: MAP measures ranked retrieval quality, while P@1 measures the fraction of suitable responses among the top-ranked responses.Results are based on five-fold cross-validation over 422 queries.
7.2. Results of Basic Linear Matching Models: The baseline combines three basic linear matching models with simple matching features in a learning-to-rank framework.The three models are Q2R, Q2P, and LatentMatch.
7.2. Results of Basic Linear Matching Models: LatentMatch slightly improves overall P@1 when added to the basic linear matching models and simple features.This baseline configuration is used for subsequent comparisons.
7.3. Results of Combining all the Features: 3.3 percent MAP and 6.3 percent P@1 improvements are achieved over Baseline when TransLM, DeepMatch, and TopicWord are combined.The full model achieves the best performance in the comparison.
7.3. Results of Combining all the Features: TransLM contributes at least 1.3 percent MAP and 3.1 percent P@1 improvements.The feature is reported as the largest contributor among the three new features.
7.3. Results of Combining all the Features: DeepMatch contributes at least 1.3 percent P@1 improvement but little MAP improvement.Its contribution is reported as the smallest among the three new features.
7.3. Results of Combining all the Features: TopicWord contributes at least 1.1 percent MAP and 1.2 percent P@1 improvements.Its contribution is reported as the second largest among the three new features.

8. Case Study

The case studies show that specialized matching features improve retrieval-based short text conversation by handling semantic similarity beyond shared words. They also expose unresolved failures in entity association and logic consistency.

8.1. Basic Linear Matching: Basic linear matching features capture semantic relevance and can retrieve suitable responses even when queries and responses share no common words.The case studies attribute suitable retrieval mainly to common words, while also showing examples without lexical overlap.
8.2. Translation-based Language Model: TransLM improves ranking by filling lexical gaps and elevating lexically dissimilar but semantically similar responses.Its word translation probabilities help rank suitable responses higher than models without TransLM.
8.3. Deep Matching Model: DeepMatch ranks suitable responses higher even when they share almost no common words with the query.The comparison is between models with and without DeepMatch as a feature.
8.4. Topic-Word Model: TopicWord lowers unsuitable responses lacking shared topic words and raises suitable responses containing highly weighted shared topic words.The model assigns greater weight to shared topic words such as “控制(control)” and “工具(tool)”.
8.5. Some Failed Issues: The current model does not adequately address entity association, allowing responses with incorrect entity names when other parts match well.Entity names are treated as words, which is insufficient to prevent the illustrated mistakes.
8.5. Some Failed Issues: The current model does not directly maintain logic consistency because doing so requires deeper text analysis that is difficult to implement.The failed examples are semantically relevant and correct with respect to speech act but logically inappropriate.

9. Conclusions

The paper proposes a retrieval-based model for short text conversation using massive social-media data and carefully designed matching features. Experiments show that this combination performs reasonably well, while several matching factors remain open challenges for improving accuracy.

The paper proposes a retrieval-based short text conversation model that leverages massive data collected from social media.
The model performs reasonably well when combined with carefully designed matching features and a huge repository of conversation data.
Future work must address semantic relevance, speech act, sentiment, entity association, logic consistency, and discourse structure to enhance short text conversation accuracy.