PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval

Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, Rabab Ward

arXiv:1502.06922v3cs.CLcs.IRcs.LGcs.NE

TL;DR

Sentence embedding must capture semantic meaning and context for comparing text. This paper trains an LSTM-RNN on click-through data to encode sentences, significantly outperforming existing state-of-the-art methods on web document retrieval.

  • Problem

    Sentence embedding needs representations that encode semantic meaning and context for measuring similarities between text strings.

  • Method

    The paper trains an LSTM-RNN with weak supervision from commercial search click-through data to encode each sentence as a contextual semantic vector.

  • Results

    The proposed embedding significantly outperforms existing state-of-the-art methods on web document retrieval.

  • Takeaways & Limitations

    The embedding is especially useful for web document retrieval because it attenuates unimportant words and represents keywords and topics in sentence vectors.

Abstract

from arXiv · show

This paper develops a model that addresses sentence embedding, a hot topic in current natural language processing research, using recurrent neural networks with Long Short-Term Memory (LSTM) cells. Due to its ability to capture long term memory, the LSTM-RNN accumulates increasingly richer information as it goes through the sentence, and when it reaches the last word, the hidden layer of the network provides a semantic representation of the whole sentence. In this paper, the LSTM-RNN is trained in a weakly supervised manner on user click-through data logged by a commercial web search engine. Visualization and analysis are performed to understand how the embedding process works. The model is found to automatically attenuate the unimportant words and detects the salient keywords in the sentence. Furthermore, these detected keywords are found to automatically activate different cells of the LSTM-RNN, where words belonging to a similar topic activate the same cell. As a semantic representation of the sentence, the embedding vector can be used in many different applications. These automatic keyword detection and topic allocation abilities enabled by the LSTM-RNN allow the network to perform document retrieval, a difficult language processing task, where the similarity between the query and documents can be measured by the distance between their corresponding sentence embedding vectors computed by the LSTM-RNN. On a web search task, the LSTM-RNN embedding is shown to significantly outperform several existing state of the art methods. We emphasize that the proposed model generates sentence embedding vectors that are specially useful for web document retrieval tasks. A comparison with a well known general sentence embedding method, the Paragraph Vector, is performed. The results show that the proposed method in this paper significantly outperforms it for web document retrieval task.

I. INTRODUCTION … A. The basic version of RNN

The paper learns sentence embeddings by recurrently encoding words with an LSTM-RNN trained on weak supervision from search-engine click-through data. It analyzes the resulting representations, relates them to prior methods, and introduces the basic RNN architecture in which the final hidden activation represents the sentence.

  • I. INTRODUCTION: Sentence embedding transforms a sentence into a vector encoding its semantic meaning, with learning defined over sentence pairs rather than word pairs.
  • I. INTRODUCTION: The proposed LSTM-RNN sequentially maps words and historical information into a latent space, using the final hidden activations as the sentence embedding.LSTM cells address the difficulty of learning long-term memory in recurrent networks.
  • I. INTRODUCTION: Weak supervision from commercial search-engine click-through logs supplies limited semantic-similarity feedback between queries and clicked documents.
  • I. INTRODUCTION: The learned LSTM-RNN automatically attenuates less important words, detects keywords, and associates different cells with different topics.These behaviors are examined by visualizing internal activation responses to text inputs.
  • II. RELATED WORK: DSSM models sentences as bags of words, whereas CLSM uses fixed-window n-grams and max pooling, limiting explicit or global dependency modeling.
  • III. SENTENCE EMBEDDING USING RNNS WITH AND WITHOUT LSTM CELLS: LSTM-RNN incrementally updates a sentence’s semantic vector with each word and its dependencies until the vector represents the complete sentence.This sequential encoding differs from DSSM and CLSM and supports global contextual information without an additional max-pooling layer.
  • II. RELATED WORK: Unlike prior sentence-embedding studies, this method trains paraphrase sentences to be close in semantic embedding space and targets supervised web-document retrieval.The LSTM-RNN is also described as robust to noise and avoids CLSM’s fixed-window limitation.
  • A. The basic version of RNN: The basic RNN processes each sentence word recurrently, mapping it into a dense low-dimensional vector whose global contextual features are represented in the hidden state.Its sentence embedding is the hidden activation corresponding to the last word.

B. The RNN with LSTM cells · IV. LEARNING METHOD

The paper uses an LSTM-RNN to address long-term dependencies in sentence-to-vector transformation and trains it from click-through supervision to bring similar sentences closer than dissimilar ones. Training uses cosine similarity, pairwise logistic loss, BPTT, minibatches, gradient control, and Nesterov-style updates.

  • B. The RNN with LSTM cells: LSTM memory cells, forget gates, and peephole connections address the vanishing-gradient difficulty of learning long-term sequence dependencies.The architecture includes input, forget, and output gates, a cell-state vector, peephole connections, recurrent connections, and biases.
  • B. The RNN with LSTM cells: The model computes a representation for each word and uses the vector corresponding to the sentence’s last word as its sentence embedding.The embeddings are generated by the RNN or LSTM-RNN described in Section III.
  • IV. LEARNING METHOD: The learning objective brings embeddings of semantically similar sentences together while separating embeddings of sentences with different meanings.The method uses click-through data because manually labeled semantic-similarity data are difficult to collect at scale.
  • IV. LEARNING METHOD: Cosine similarity measures sentence-vector similarity, with clicked documents treated as positive samples and unclicked documents as negative candidates.The query and document embeddings are compared using the vectors corresponding to their final words.
  • IV. LEARNING METHOD: Training maximizes the likelihood of the clicked document for each query through a logistic loss over the query–positive-document versus query–negative-document similarity difference.The loss upper-bounds pairwise accuracy, while γ scales the similarity difference to penalize prediction errors more strongly.
  • IV. LEARNING METHOD: The RNN and LSTM-RNN are trained with Back Propagation Through Time, using minibatch updates and gradient re-normalization to accelerate training and control gradient explosion.The method also uses Nesterov updates to accelerate convergence.
  • IV. LEARNING METHOD: Algorithm 1 summarizes training with separate query-side and document-side parameter sets, negative sampling, truncated BPTT, gradient clipping, and iterative parameter updates.Its inputs include a gradient-clip threshold, maximum epochs, query/clicked-document pairs, negative documents, and maximum sequence length.
  • IV. LEARNING METHOD: Momentum is scheduled at µk = 0.9 during the first and last 2% of updates and µk = 0.995 during the remaining 96%.Fixed step sizes are used for both RNN and LSTM-RNN training.

V. ANALYSIS OF THE SENTENCE EMBEDDING PROCESS AND PERFORMANCE EVALUATION

The section analyzes how LSTM-RNNs capture context, attenuate unimportant information, detect keywords, and identify global topics through visualization. It also evaluates models trained with click-through data and specifies task-specific architectural simplifications.

  • Analysis: Visualization examines how word dependencies and context information are captured in the LSTM-RNN sentence embedding.The analysis targets input gates, output gates, cell states, and semantic vectors.
  • Analysis: The analysis investigates how the model attenuates unimportant information, detects critical keywords, and embeds them into semantic vectors.It also examines how LSTM-RNN identifies global topics and extracts useful information according to topic.
  • Experimental setup: Training uses 200,000 positive query/document pairs from commercial search-engine click-through logs, with clicked signals providing weak supervision.The experiments train RNNs with and without LSTM cells on this click-through dataset.
  • Experimental setup: The current LSTM-RNN task removes peephole connections and forget gates because sequence lengths are known and each cell state starts at zero.The paper states that forget gates are therefore not a great help for these sequences.

A. Analysis · 1) Attenuating Unimportant Information:

The analysis examines how the LSTM-RNN sequentially extracts and embeds sentence information over time. It finds that the model progressively enriches the semantic representation while attenuating unimportant words through input-gate dynamics.

  • A. Analysis: The analysis examines how information is sequentially extracted and embedded into the semantic vector over time by the LSTM-RNN model.
  • 1) Attenuating Unimportant Information:: The experiment feeds test-dataset query and document sentences into the trained LSTM-RNN to examine semantic-vector evolution and attenuation of unimportant words.
  • 1) Attenuating Unimportant Information:: Semantic representation y(t) and cell states c(t) evolve over time as valuable context is absorbed, progressively enriching the vectors.The entire sentence’s semantic information is embedded into y(t) by applying output gates to c(t).
  • 1) Attenuating Unimportant Information:: Input gates attenuate unimportant information while detecting important information from the input sentence.
  • 1) Attenuating Unimportant Information:: The document sentence ends at the ninth word, so all values to its right are zero.
  • 1) Attenuating Unimportant Information:: Input gates reduce the effects of “accommodation,” “discount,” and “reservation” in the final semantic representation y(t).The passage states that this prevents these words from affecting semantic similarity between the query and document sides.

2) Keywords Extraction: … 1) Web Document Retrieval Task:

The trained LSTM-RNN extracts keywords by tracking activation changes and allocates them to topic-specific cells. For web document retrieval, sentence embeddings are compared by cosine similarity, and the LSTM-RNN significantly outperforms the evaluated baselines, including CLSM and doc2vec.

  • 2) Keywords Extraction:: The model detects keywords when activation changes occur in the 10 most active cells of the final semantic representation.Keyword detection is illustrated by backtracking semantic representations y(t) over the sentence.
  • 2) Keywords Extraction:: A word is labelled a keyword when more than 40% of the top 10 active cells in both reading directions identify it.The analysis uses a Bidirectional LSTM-RNN, with boldface indicating more than four assigned cells.
  • 3) Topic Allocation:: The LSTM-RNN allocates detected keywords to different cells according to their topics.This finding comes from searching the test dataset for keywords detected by specific cells.
  • 3) Topic Allocation:: Individual cells collect topic-specific keywords, including food-related terms in cell 26 and health-related terms in cells 2 and 6.The examples are drawn from queries covering the topics “food” and “health”.
  • 1) Web Document Retrieval Task:: The retrieval system embeds query and document sentences into semantic vectors and measures their similarity using cosine similarity.The evaluation uses a commercial web search engine and compares candidate-document rankings.
  • B. Performance Evaluation: NDCG is used to evaluate ranking performance for the RNN and LSTM-RNN on a standalone human-rated test dataset, alongside DSSM and CLSM baselines.The metric is mean Normalized Discounted Cumulative Gain, where higher values indicate better performance.
  • 1) Web Document Retrieval Task:: The proposed method significantly outperforms doc2vec and skip-thought for document retrieval, supporting task-specific embedding training.Doc2vec and skip-thought are evaluated by calculating cosine similarities between query-document embedding vectors and then NDCG.
  • B. Performance Evaluation: 1.3%: the LSTM-RNN exceeds CLSM in NDCG@1, a statistically significant improvement, and significantly outperforms all evaluated models.The paper attributes this improvement to embedding contextual and semantic sentence information in a finite-dimensional vector.

VI. CONCLUSIONS AND FUTURE WORK

The paper proposes an LSTM-based sentence embedding that retains useful long-range information in a single semantic vector and demonstrates robustness to noise and topic-specific cell allocation. It significantly outperforms existing state-of-the-art methods on web document retrieval, with future work targeting additional language-processing tasks such as question answering.

  • Contributions: The LSTM model embeds long-range contextual information and useful sentence content in a single semantic vector.Input gates detect and attenuate useless information as the semantic vector evolves over time.
  • Model analysis: The model is robust to noise, primarily embedding keywords in the final semantic representation of the sentence.The conclusion reports this finding as one outcome of detailed model analysis.
  • Model analysis: LSTM cells are usually allocated to keywords from specific topics, supported by extensive examples.This topic-specific allocation is presented alongside the model’s keyword-focused behavior.
  • Application: The proposed sentence embedding significantly outperforms all existing state-of-the-art methods on web document retrieval.Web document retrieval is presented as a concrete application of the method.
  • Future work: Future work will apply the sentence embedding method to other language-processing tasks, including question answering.The authors identify question answering as a task where sentence embedding may play a key role.

APPENDIX A EXPRESSIONS FOR THE GRADIENTS … 2) Input Gate:

The appendix gives gradient expressions for training the proposed RNN and LSTM-RNN models, completing the parameter-gradient calculation used with Nesterov updates. It then organizes LSTM-RNN derivatives by gate connections, including recurrent, input, peephole, and bias parameters.

  • APPENDIX A EXPRESSIONS FOR THE GRADIENTS: The appendix presents final gradient expressions needed to train the proposed models, with full derivations provided in supplementary materials.The derivations are located in section III of the supplementary materials.
  • A. RNN: For the RNN, recurrent parameters use Λ = Wrec, while candidate-document indexing and document-side notation specify corresponding gradient terms.The recurrent-parameter notation omits the r subscript for simplicity.
  • A. RNN: The RNN appendix also defines normalization factors b and c from the final query- and document-side hidden representations.The expressions are b = 1 ∥yQ(t = TQ)∥ and c = 1 ∥yD(t = TD)∥.
  • B. LSTM-RNN: For LSTM-RNN training, Λ ranges over weight matrices and bias vectors, and ∇L(Λ) follows the general cost-gradient format used for parameter updates.The parameters include input, recurrent, peephole, and bias terms for the four LSTM cells.
  • B. LSTM-RNN: The LSTM-RNN gradient calculation is completed by deriving ∂Λ for each cell parameter and then applying the stated update rule, with vQ and vD defined subsequently.The full truncated-BPTT derivation appears in section III of the supplementary materials.
  • 1) Output Gate:: For the output gate, the appendix specifies derivatives for recurrent connections, input connections through W1, peephole connections through Wp1, and output-gate biases.The document-side output is denoted yD(t).
  • 2) Input Gate:: For the input gate, the appendix gives recurrent-connection derivatives involving Wrec3, then specifies corresponding input, peephole, and bias derivatives.The document-side expressions are stated to match the query-side expressions with the D subscript.

3) Forget Gate: … APPENDIX D DERIVATION OF BPTT FOR RNN AND LSTM-RNN

The merged sections detail LSTM-RNN gate-gradient derivations and error backpropagation, illustrate input-gate behavior on a document example, compare RNN and LSTM-RNN retrieval scores, and present the full BPTT gradient derivation.

  • 3) Forget Gate:: The forget-gate derivation specifies gradients for recurrent, input, peephole, and bias connections.The section separately introduces each connection type and its corresponding derivative expression.
  • 3) Forget Gate:: The section also gives recurrent, input, and bias derivatives for the ungated input pathway.These equations are presented under “Input without Gating (yg(t)).”
  • 5) Error signal backpropagation:: Error signals are back propagated through time using a set of stated equations.The supplied passage introduces the equations without reproducing their full contents.
  • A CLOSER LOOK AT RNNS WITH AND WITHOUT LSTM CELLS IN WEB DOCUMENT RETRIEVAL TASK: For the “hotels in shanghai” query-document pair, RNN assigns score “0.8165” while LSTM-RNN assigns “0.9161” for a human-rated “Good” match.Scores range from 0 to 1, and the passage states that the LSTM-RNN score is more correspondent with the human-generated label.
  • APPENDIX D DERIVATION OF BPTT FOR RNN AND LSTM-RNN: Appendix D presents the full derivation of gradients for RNN and LSTM-RNN.It frames the derivation as backpropagation through time for both recurrent architectures.

A. Derivation of BPTT for RNN · 1) Recurrent Weights:

The section derives backpropagation through time for the recurrent weights by decomposing the query-document representation and propagating gradients across unfolded time steps. It also references comparisons involving RNNs and LSTM-RNNs for documents and queries.

  • A. Derivation of BPTT for RNN: The surrounding section includes RNN-versus-LSTM-RNN comparisons for a document, a query, and another document example.The listed examples concern Shanghai hotel accommodation and bathtub-related query and document text.
  • 1) Recurrent Weights:: The derivation begins by dividing R(D, Q) into three components.The components are labeled a, D, and E in the supplied equations.
  • 1) Recurrent Weights:: The method computes component E using a basic rule and component F similarly to equation (55).These intermediate calculations are then combined with equations (50), (53), (55), and (56).
  • 1) Recurrent Weights:: The endpoint errors for query and document embeddings are expressed using elementwise products involving 1 − y and 1 + y.Separate expressions are given for yQ(t = TQ) and yD(t = TD).
  • 1) Recurrent Weights:: Equation (58) unfolds the network for one time step before backpropagation extends the calculation across the remaining time steps.The unfolding proceeds over τ time steps.
  • 1) Recurrent Weights:: The unfolding range runs from 0 to TQ for queries and from 0 to TD for documents.The passage defines τ as the number of time steps used for unfolding the network over time.
  • 1) Recurrent Weights:: The final gradient is obtained by folding the network back over time and applying equation (45).The derivation explicitly describes this as the final value of the gradient.

2) Input Weights: … 2) Input Gate:

The merged sections outline backpropagation-through-time derivations for LSTM-RNN parameters, covering input weights, general parameter derivatives, output-gate terms, and input-gate updates. The input-gate section derives updates for recurrent, input, peephole, and bias parameters, including Q and D components.

  • 2) Input Weights:: Input-weight derivatives are obtained using a procedure analogous to the preceding parameter derivations.
  • B. Derivation of BPTT for LSTM-RNN: For every parameter Λ in the LSTM-RNN architecture, the derivation follows from equation (50).
  • B. Derivation of BPTT for LSTM-RNN: The derivation uses the identity α◦β = diag(α)β = diag(β)α for vector-element products.
  • 1) Output Gate:: Output-gate parameter derivatives are obtained by substituting the preceding result, with analogous derivations for W1 and Wp1.
  • 2) Input Gate:: The resulting input-gate expression includes factors involving o(t), c(t), yg(k), and i(k), while Wrec3 is identified as zero.
  • 2) Input Gate:: Update equations for Wrec3 use the corresponding Q and D expressions together with equation (6), while analogous equations are derived for W3.The W3 updates are specified by equations (89), (90) for Q and D and equation (6).
  • 2) Input Gate:: Peephole and bias updates are likewise given for Q and D, using equations (91), (92) and (93), (94), respectively, together with equation (6).

3) Forget Gate:

The forget-gate section derives gradient-based update equations for its recurrent, input, peephole, and bias parameters. It identifies equation sets for the corresponding Q and D updates together with equation (6).

  • 3) Forget Gate:: The forget gate’s recurrent-weight updates, Wrec2, are given by equations (97) and (98) for Q and D, together with equation (6).The derivation substitutes the preceding expression involving yQ(t), the cell-state activation, output gate, and cell value.
  • 3) Forget Gate:: The input-weight updates, W2, for the forget gate are specified by equations (99) and (100) for Q and D, together with equation (6).The section separately introduces the forget gate’s input weights before stating their update equations.
  • 3) Forget Gate:: Update equations for the forget-gate bias values, b2, are provided in the following equations together with equation (6).The passage introduces a separate update-equation set for the bias parameters.

A. LSTM-RNN Semantic Vectors: Another Example … APPENDIX G DIAGRAM OF THE PROPOSED MODEL

The examples visualize how LSTM-RNN representations evolve, suppress less important words, and support keyword extraction, while Doc2Vec similarity tests and a training-architecture diagram provide supplementary analysis.

  • A. LSTM-RNN Semantic Vectors: Another Example: Semantic representation y(t) and cell states c(t) evolve over time in the LSTM-RNN example.The query is “how to fix bath tub wont turn off,” and the document concerns painting a bathtub.
  • A. LSTM-RNN Semantic Vectors: Another Example: Input gates reduce the influence of “you,” “paint,” “and,” and “paint” on the document’s final representation y(t).These words correspond to small input-gate values in the visualization.
  • APPENDIX E LSTM-RNN VISUALIZATION: The semantic representation y(t) is obtained after applying the output gate to cell states c(t), which store valuable context information.
  • B. Key Word Extraction: Another Example: The second example tracks the 10 most active cells over time for both query and document, with word assignments reported in Tables IX and X.
  • APPENDIX F DOC2VEC SIMILARITY TEST: The resulting words are presented as evidence that the trained Doc2Vec model is meaningful and can recognize semantic similarity.
  • APPENDIX G DIAGRAM OF THE PROPOSED MODEL: Figure 13 illustrates the proposed model’s training procedure and distinguishes it from general sentence embedding methods.The diagram defines n as the number of negative (unclicked) documents; other parameters match Figures 2 and 3.
Loading 1502.06922v3…