PaperlayerView exact revision on arXiv

Source-linked AI summary

End to End Binarized Neural Networks for Text Classification

Harshil Jain, Akshat Agarwal, Kumar Shridhar, Denis Kleyko

arXiv:2010.05223v1cs.LGcs.CL

TL;DR

Deep networks can be costly to train and run, limiting their use on resource-constrained devices. The paper combines binary token-statistic embeddings with a binarized classifier for intent classification. It reports comparable state-of-the-art results on three standard intent datasets, alongside reduced memory use, but lower accuracy on the larger 20NewsGroups dataset.

  • Problem

    Deep networks’ computational and memory costs raise concerns for training and limit their use on resource-constrained platforms, while an end-to-end trainable binarized architecture for text and intent classification had not been demonstrated.

    Source: §1 · p. 1

  • Method

    The architecture embeds token statistics as high-dimensional vectors and binarizes both the input representations and classifier in an end-to-end network.

    Source: §1 · p. 2, §3.2 · pp. 4–5

  • Results

    The architecture achieves results comparable to the state of the art on three standard intent datasets but is less accurate on 20NewsGroups; HD BNN uses around 20–40% less memory than Text-LeNet with GloVe embeddings on intent datasets.

    Source: §1 · p. 2, §4.3 · pp. 8–9

  • Takeaways & Limitations

    The authors present binary representations as a way to reduce memory footprint and increase inference energy efficiency, and frame the work as a step toward NLP functionality on resource-constrained devices.

    Source: §5 · p. 10

    On 20NewsGroups, binarized HD vectors perform worse than non-HD vectors, and simple Linear SVC fails with only binarized values.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

Deep neural networks have demonstrated their superior performance in almost every Natural Language Processing task, however, their increasing complexity raises concerns. In particular, these networks require high expenses on computational hardware, and training budget is a concern for many. Even for a trained network, the inference phase can be too demanding for resource-constrained devices, thus limiting its applicability. The state-of-the-art transformer models are a vivid example. Simplifying the computations performed by a network is one way of relaxing the complexity requirements. In this paper, we propose an end to end binarized neural network architecture for the intent classification task. In order to fully utilize the potential of end to end binarization, both input representations (vector embeddings of tokens statistics) and the classifier are binarized. We demonstrate the efficiency of such architecture on the intent classification of short texts over three datasets and for text classification with a larger dataset. The proposed architecture achieves comparable to the state-of-the-art results on standard intent classification datasets while utilizing ~ 20-40% lesser memory and training time. Furthermore, the individual components of the architecture, such as binarized vector embeddings of documents or binarized classifiers, can be used separately with not necessarily fully binary architectures.

1 Introduction

The paper presents an end-to-end binarized architecture for intent classification to address the computational and memory costs of deep networks. It reports comparable state-of-the-art results on three standard intent datasets, with lower accuracy on a larger dataset, and improved efficiency.

1 Introduction

  • Deep networks’ growing computational expense limits their use on resource- and power-constrained platforms, motivating more efficient architectures.The introduction describes binarized networks as restricting weights and activations to {+1, −1}, reducing computation and memory footprint.Source: §1 · p. 1
  • The proposed intent classifier combines binary input representations with a binarized neural network classifier in an end-to-end architecture.The input representations are tokenized and embedded in binary high-dimensional vectors using hyperdimensional computing.Source: §1 · pp. 1–2
  • The architecture achieves results comparable to the state of the art on three standard intent datasets but is less accurate on the larger 20NewsGroups dataset.It reduces memory consumption by about 40% compared with a non-binarized neural network using GloVe embeddings.Source: §1 · p. 2
  • The architecture’s components, including binarized document embeddings and classifiers, can also be used separately in other NLP solutions.The paper presents these components as usable without requiring the whole architecture to be binary.Source: §1 · p. 2

2 Related Work

The related work covers tokenization, token representations, and classification models. It distinguishes this work by combining binary embeddings with a trainable architecture.

2.1 Tokenization

  • Tokenization approaches range from word-based dictionaries, which have out-of-dictionary problems, to n-gram and subword methods, including SemHash, BPE, unigram segmentation, and BERT-based tokenization.BPE merges frequent tokens, while BERT tokenization is based on its masking objective.Source: §2.1 · p. 2

2.2 Representation of tokens

  • Sparse n-gram representations grow with dictionary size and exponentially with n, whereas hyperdimensional embeddings map them to fixed-size vectors.Earlier HD embedding work did not combine the embeddings with a trainable architecture; here, binary embeddings form the input to a BNN.Source: §2.2 · pp. 2–3

2.3 Classification models

  • CNNs have been used for sentence-level NLP classification, while BNN applications in NLP have been limited; this work applies BNNs to intent classification with binarized embeddings and classifiers.The cited related work includes a binarized recurrent network for language modeling.Source: §2.3 · p. 3

3 Methods

The method preprocesses and tokenizes text, vectorizes token counts, embeds those statistics in a hyperdimensional vector, and passes the representation to a classifier. Several tokenizers provide alternative ways to form the input tokens.

3 Methods

  • The pipeline tokenizes preprocessed text, forms a count-based vector, embeds it as a distributed HD representation, optionally binarizes it, and feeds it to a classifier.The main classifier studied is a BNN, with other classifiers included for benchmarking.Source: §3 · p. 3

3.1.1 Word tokenizer

  • The word tokenizer splits preprocessed text on spaces and discards tokens such as punctuation marks, unidentified tokens, and hyphen symbols.For example, “hello! how are you?” becomes “hello”, “how”, “are”, and “you”.Source: §3.1.1 · p. 3

3.1.2 Semantic Hashing tokenizer

  • SemHash splits text into words, adds # markers around each word, and extracts its n-gram subwords.For n = 3, “hello” yields “#he”, “hel”, “ell”, “llo”, and “lo#”.Source: §3.1.2 · p. 3

3.1.3 Byte Pair Encoding tokenizer

  • BPE merges frequent tokens after splitting text into words, using a special end-of-word token to represent inputs with frequent token units.The example “hello how are you” is tokenized as “hello˙g”, “how˙g”, “are˙g”, and “you”.Source: §3.1.3 · pp. 3–4
  • Figure 2 compares t-SNE projections of GloVe embeddings and HD vectors derived directly from n-gram statistics of the plotted words.The caption states that point colors have no associated meaning.Source: Figure 2 · p. 4

3.1.4 Char level Byte Pair Encoding tokenizer

  • The char-level tokenizer uses unigram-language-model subword segmentation and emits character tokens with ˙g marking word endings.For example, “hello how are you” is split into characters, with ˙g attached to each word’s final character.Source: §3.1.4 · p. 4

3.1.5 BERT-based tokenizer

  • The BERT-based tokenizer learns token splits from a masking objective on an unsupervised dataset and adds [cls] and [sep] markers.The example places [cls] at the sentence start and [sep] at the end.Source: §3.1.5 · p. 4

3.2 High-Dimensional embedding of vectorized representations

The section embeds sparse, high-dimensional token-statistic vectors in distributed high-dimensional vectors. It forms position-sensitive n-gram vectors and bundles them into document representations, optionally binarizing the result.

3.2 High-Dimensional embedding of vectorized representations

  • Sparse, high-dimensional token-statistic vectors are embedded with hyperdimensional computing to reduce representation dimensionality.Each unique token receives a random d-dimensional bipolar vector stored in item memory.Source: §3.2 · p. 4
  • For each n-gram, token vectors are permuted according to position and consecutively bound to form its high-dimensional vector.A word-count vectorizer is the special case of n-gram statistics with n = 1.Source: §3.2 · p. 5
  • Document representations bundle all observed n-gram vectors, weighted by their frequencies, into a single vector h.The representation can be binarized with the sign function; without binarization, cosine similarity normalizes differences in vector amplitudes.Source: §3.2 · p. 5

3.3 Binarized Neural Networks

The classifier constrains weights and activations to binary values, using bitwise operations in place of multiply-accumulate operations. The supplied table passages identify F1 comparisons, but provide no table values.

3.3 Binarized Neural Networks

  • Binary weights and activations allow bitwise operations, such as XNOR multiplication, instead of multiply-accumulate operations.The input HD representations are also binarized using the sign function.Source: §3.3 · p. 5
  • A 1D convolutional layer applies a kernel across the input HD vector to produce an output tensor.The straight-through estimator and gradient clipping are used to make gradient-descent training possible with binary weights.Source: §3.3 · pp. 5–6
  • The supplied table captions describe F1 comparisons of binarized Text-LeNet against non-binarized Text-LeNet and of three binarized vectorizers using the same classifier.The captions identify comparison scope but do not provide scores.Source: Table 1 · p. 6, Table 2 · p. 6

4 Empirical Analysis

The evaluation spans three short-text intent datasets and 20NewsGroups, using multiple tokenizers and classifier configurations. HD BNN achieves strong intent-classification results with lower memory and training-time costs, though outcomes vary by dataset and comparison method.

4.1 Datasets

  • Experiments used Chatbot, AskUbuntu, and WebApplication for short-text intent classification, and also used 20NewsGroups, which contains 20 categories with exactly 18,846 text samples in each category.Chatbot had 206 questions; AskUbuntu had 53 training and 109 test samples; WebApplication had 88 samples. For 20NewsGroups, 11,314 samples were used for training and 7,532 for testing.Source: §4.1 · p. 6

4.2 Experimental Details

  • The evaluation used six tokenizers, nine sklearn classifiers and Text-LeNet variants, with five-fold cross-validation and results averaged over five simulations.CNN experiments ran on NVIDIA Tesla GPUs, while sklearn classifier experiments ran on CPUs.Source: §4.2 · pp. 7–8

4.3 Results and Discussions

  • HD BNN used 20–40% less memory than non-binarized neural networks and reduced training time by up to 50% versus non-binarized classifiers.The memory comparison also includes GloVe Text-LeNet and GloVe BNN; the training-time-per-epoch comparison is between BNN and Text-LeNet.Source: Figure 3 · p. 7, §4.3 · p. 8
  • HD BNN achieved state-of-the-art F1 on Chatbot; AskUbuntu favored binarized HD with the best classifier, while WebApplication and average results were comparable to leading methods.WebApplication scores were 0.84 for binarized HD with the best classifier and 0.83 for HD BNN, compared with 0.87 for SemHash; average scores were 0.92 and 0.91 versus 0.92 for SemHash.Source: Table 3 · p. 9, §4.3 · p. 9
  • Binarized HD vectors outperformed binarized GloVe by 4–25% and binarized SemHash by 2–8% on two smaller datasets, and achieved 5–7% better F1 on 20NewsGroups.On AskUbuntu, results were comparable; all binarized representations used the same BNN classifier.Source: §4.3 · p. 9
  • HD-vector MLP and Linear SVC results were comparable to state-of-the-art; binarized HD vectors beat non-HD baselines by 2% on AskUbuntu and Chatbot and 5% on WebApplication.On 20NewsGroups, binarized HD vectors scored lower than non-HD vectors; the authors attribute this mainly to dataset size and simple Linear SVC’s difficulty with binarized values.Source: Figure 4 · p. 8, §4.3 · p. 9

5 Conclusion

The conclusion presents an end-to-end binarized architecture for intent classification and reports results comparable to the state of the art. It frames binary representations as a way to reduce memory footprint and improve inference energy efficiency, with resource-constrained devices as the intended scope.

5 Conclusion

  • The paper presents an end-to-end binarized architecture for intent classification using binarized representations throughout the NLP classification architecture.The conclusion describes the results as comparable to the state of the art.Source: §5 · p. 10
  • Binary representations are presented as a way to reduce architecture memory footprint and improve inference energy efficiency through binary operations.Source: §5 · p. 10
  • The work takes a step toward enabling NLP functionality on resource-constrained devices.Source: §5 · p. 10

A.1 Datasets

The appendix describes the four evaluation corpora: three intent-classification datasets and the 20NewsGroups corpus. It notes their sources and directs readers to tables for sample distributions.

A.1 Datasets

  • The Chatbot Corpus consists of public-transport questions posted through a Telegram chatbot, with its train/test split shown in Table 1.Source: §A.1 · p. 12, Table 1 · p. 12
  • AskUbuntu and WebApplication comprise questions and answers from StackExchange, with their sample distributions detailed in Tables 2 and 3.Source: §A.1 · p. 12, Table 2 · p. 12, Table 3 · p. 12
  • 20NewsGroups contains news posts labeled into categories, with its sample distribution detailed in Table 4.Source: §A.1 · p. 12

A.2 Experimentation Details

The appendix specifies the Text-LeNet layer sequence and selected classifier hyperparameters. It provides the associated hyperparameter table, while the stated text describes the main architecture and tuning choices.

A.2 Experimentation Details

  • Text-LeNet uses convolutional filters of 128, 256, and 512, followed by max pooling and batch normalization, then flattening, a 128-unit dense layer, dropout, and linear classification.The classification layer has dimensions (128, number of classes).Source: §A.2 · p. 12
  • MLP hyperparameters were selected by grid search across three hidden-layer configurations, with a maximum of 200 iterations.The configurations were (100, 50), (300, 100), and (300, 200, 100).Source: §A.2 · p. 13
  • The experiments used 50 Random Forest estimators with minimum leaf size 1 and three KNN neighbors; other classifiers used sklearn defaults.Source: §A.2 · p. 13

A.3 Experiments

Across four datasets, SemHash generally achieved better results than other tokenizers, followed by BERT; the experiments compare F1 scores across classifiers and tokenization variants.

A.3 Experiments

  • SemHash generally achieved better results than other tokenizers, followed by BERT, across all four datasets.The comparisons cover AskUbuntu, Chatbot, WebApplication, and 20NewsGroups.Source: §A.3 · p. 14, Table 6 · p. 14, Table 7 · p. 14
  • Tables 6–8 compare F1 scores for nine classifiers across three small datasets, using HD tokenization variants and selected non-HD variants.The non-HD variants include SemHash and BPE for the small datasets; for 20NewsGroups, the comparison uses non-HD SemHash.Source: §A.3 · p. 14, Table 6 · p. 14, Table 7 · p. 14
  • Table 9 compares tokenizer results on 20NewsGroups using MLP, SGD Classifier, Linear SVC, and Random Forest.The table reports F1 scores.Source: §A.3 · p. 14, Table 9 · p. 14

Related papers

Loading 2010.05223v1…