Improved Deep Learning Baselines for Ubuntu Corpus Dialogs

1 Introduction

• The Ubuntu Dialogue Corpus contains almost one million two-person conversations derived from Ubuntu-related chat logs, making it well-suited to deep-learning exploration.The average dialogue is eight turns long, with a minimum of three turns.Source: §1 · p. 1
• The paper introduces experiments on the corpus, evaluates models and training-data sizes, and concludes with plans for future work.The paper reports state-of-the-art results.Source: §1 · p. 1

2 Data

• The data are preprocessed to replace named entities with tags, then represented as context-response-flag tuples whose Boolean flag marks response correctness.The tags include name, location, organization, URL, and path.Source: §2 · p. 1
• Training uses each utterance from the third onward as a response, pairing it with preceding context and a randomly sampled negative response; examples are shuffled.A dialogue of length n yields n − 2 positive and n − 2 negative examples.Source: §2 · p. 2

2.1 Evaluation Metric

• Evaluation ranks n response candidates and counts a result as correct when the true response appears among the first k, reported as Recall@k.Two percent of conversations form the test set; baselines use (n, k) settings of (2, 1), (10, 1), (10, 2), and (10, 5).Source: §2.1 · p. 2

3 Approaches

• Ranking approaches include pointwise, pairwise, and listwise methods.Source: §3 · p. 2
• Although pairwise and listwise methods are reported as empirically superior, the preliminary experiments use pointwise ranking for simplicity, as did the original baselines [1].Source: §3 · p. 2

3.1 Pointwise Ranking

• Pointwise ranking scores each context-response pair with a learned neural network and sorts candidates by their probabilities.The scoring function is denoted g(context, response).Source: §3.1 · p. 2

4 Previous Work

• Previous pointwise models for the task included TF-IDF, RNN, and LSTM approaches [1].Source: §4 · p. 2

4.1 TF-IDF

• The TF-IDF baseline ranks responses by cosine similarity between context and candidate-response vectors, reflecting the assumption that correct responses share more words with context.The vectors weight word frequency by inverse document frequency over the corpus.Source: §4.1 · pp. 2–3

4.2 Neural Network Embeddings

• Neural-network embedding models score context and response embeddings through a learned matrix and sigmoid; the embeddings can be produced by different architectures.Prior experiments found LSTM significantly outperformed vanilla RNN, and the approach can be viewed as predicting one embedding from the other and comparing it with the actual embedding.Source: §4.2 · p. 3

5 Our Architectures

• Input words are projected into embedding vectors, which the three architectures use to represent the context and candidate response.Source: §5 · p. 3

5.1 CNN

• CNN filters of varying widths are followed by max-pooling, whose output forms an embedding with one dimension per filter.The paper describes filter widths as n-gram-like features.Source: §5.1 · p. 3, Figure 2 · p. 4

5.2 LSTM

• The LSTM embedding is its final cell state, so its dimension equals the number of LSTM units.Source: §5.2 · pp. 3–4, Figure 2 · p. 4

5.3 Bi-Directional LSTM

• The Bi-LSTM reads the sequence in forward and reverse directions and concatenates the corresponding recurrent states into an embedding.Source: §5.3 · p. 4, Figure 2 · p. 4

6.1 Method

• The best classifier averages predictions from 11 LSTMs, 7 Bi-LSTMs, and 10 CNNs trained with different meta-parameters.Experiments use the original one-million-example training data and GloVe vectors; shared-parameter models consistently achieved higher accuracy.Source: §6.1 · pp. 4–5

6.2 Results

• The results section reports that best Recall metrics were usually recorded in the second training epoch; Table 1 compares experiments with prior results, and Table 2 gives a ranked-response example.Source: §6.2 · p. 5, Table 1 · p. 5, Table 2 · p. 5

6.3 Discussion

• The ensemble sets a new state of the art for Ubuntu response ranking; Recall@1 is 68.3 with CNNs versus 66.8 without the 10 CNNs.The results are presented as baselines for more complicated models.Source: §6.3 · pp. 5–6
• LSTMs and Bi-LSTMs achieve almost the same accuracy, while recurrent models outperform CNNs individually; CNNs nevertheless improve the ensemble.The authors hypothesize that early context utterances may be less important or that LSTMs capture the important sequence parts; they describe CNN representations as complementary to recurrent models.Source: §6.3 · p. 5

6.4 Varying Training Data Size

• With less training data, CNNs outperform recurrent models; with enough data, recurrent models perform better, and their accuracy had not peaked at one million examples.The authors suggest that the CNN max operation aids generalization on small datasets, while its simplicity limits handling of dependencies such as n-gram order.Source: §6.4 · p. 6, Figure 3 · p. 6

7 Future Work

• Future work will test neural architectures extended with memory and investigate bootstrapping the system with external information such as manuals or man pages.Source: §7 · p. 6
• Attention over context sentences is proposed as a way to identify facts important to predictions and make them more interpretable to humans.Source: §7 · p. 6
• Alternative preprocessing may improve accuracy; the current replacement of named entities with generic tags could possibly harm performance.Source: §7 · p. 6

• The paper reports a new state-of-the-art result on the next utterance ranking problem.Source: §8 · p. 6
• The best-performing system is an ensemble of multiple diverse neural networks.Source: §8 · p. 6
• The authors plan to use the system as a base for more complex models beyond the standard neural-network paradigm.Source: §8 · p. 6