PaperlayerView exact revision on arXiv

Source-linked AI summary

TextConvoNet:A Convolutional Neural Network based Architecture for Text Classification

Sanskar Soni, Satyendra Singh Chouhan, Santosh Singh Rathore

arXiv:2203.05173v1cs.CLcs.AIcs.LGcs.NE

TL;DR

Existing CNN text-classification models primarily extract intra-sentence n-gram features from sentence matrices, leaving inter-sentence relationships insufficiently represented. TextConvoNet uses an alternative paragraph representation with two-dimensional multi-scale convolution to capture both feature types, and experiments on five datasets report performance improvements over other machine-learning and deep-learning models.

  • Problem

    Existing CNN-based text models use one-dimensional convolution on sentence matrices, primarily extracting intra-sentence n-gram features rather than inter-sentence n-gram features.

  • Method

    TextConvoNet represents paragraphs as embedding matrices and applies two-dimensional multi-scale convolution through parallel pathways to extract intra- and inter-sentence features.

  • Results

    TextConvoNet outperforms other machine-learning and deep-learning models across the evaluated text-classification datasets and performance metrics, with reported average gains of 1.5%–31% in accuracy, 4.1%–70.8% in precision, and 7.4%–about 30% in MCC.

  • Takeaways & Limitations

    The evaluation indicates that extracting inter-sentence relationships improves text classification for both binary and multi-class problems.

Abstract

from arXiv · show

In recent years, deep learning-based models have significantly improved the Natural Language Processing (NLP) tasks. Specifically, the Convolutional Neural Network (CNN), initially used for computer vision, has shown remarkable performance for text data in various NLP problems. Most of the existing CNN-based models use 1-dimensional convolving filters n-gram detectors), where each filter specialises in extracting n-grams features of a particular input word embedding. The input word embeddings, also called sentence matrix, is treated as a matrix where each row is a word vector. Thus, it allows the model to apply one-dimensional convolution and only extract n-gram based features from a sentence matrix. These features can be termed as intra-sentence n-gram features. To the extent of our knowledge, all the existing CNN models are based on the aforementioned concept. In this paper, we present a CNN-based architecture TextConvoNet that not only extracts the intra-sentence n-gram features but also captures the inter-sentence n-gram features in input text data. It uses an alternative approach for input matrix representation and applies a two-dimensional multi-scale convolutional operation on the input. To evaluate the performance of TextConvoNet, we perform an experimental study on five text classification datasets. The results are evaluated by using various performance metrics. The experimental results show that the presented TextConvoNet outperforms state-of-the-art machine learning and deep learning models for text classification purposes.

I. INTRODUCTION

Text classification motivates automated NLP methods, while existing CNNs primarily extract intra-sentence n-gram features with one-dimensional convolution. TextConvoNet addresses this gap through multidimensional representations and two-dimensional convolution, evaluating multiple variants on five benchmark datasets.

  • Motivation: Automated text classification is important because textual data has grown exponentially across industry and the Internet.
  • Existing approaches: Existing CNN-based text classifiers generally use one-dimensional convolution and pooling to extract n-gram features from input word embeddings.
  • Proposed approach: TextConvoNet represents text as a multidimensional embedding structure and applies two-dimensional convolution to extract intra-sentence and inter-sentence n-gram features.
  • Evaluation: The study evaluates TextConvoNet and its variants on five benchmark datasets using multiple performance metrics and comparisons with existing models.
  • Evaluation: The paper includes ablation studies focused on optimizing hyperparameters across different layers of the proposed model.

II. LITERATURE REVIEW

Prior CNN text-classification models represent each sentence as a word-vector matrix and use one-dimensional convolution to capture n-gram features. The literature motivates TextConvoNet’s alternative paragraph-level representation and multidimensional convolution.

  • Background: Text classification assigns labels to textual content, and CNNs are among the neural architectures used for this task.
  • Related neural models: LSTM models capture sequential information but can require high training time, while attention mechanisms add computation for word-level alignment.
  • Research gap: The paper identifies inter-sentence n-gram extraction as an unresolved limitation of previously discussed CNN-based networks.
  • CNN text classification: Kim’s CNN uses a single convolution layer with kernel sizes 3, 4, and 5, followed by max-pooling and a fully connected classification layer.
  • CNN text classification: In the standard CNN setup, each sentence is a matrix whose rows are word vectors, and one-dimensional convolution extracts local features before classification.

B. Proposed TextConvoNet Architecture

TextConvoNet represents paragraphs as three-dimensional embedding matrices and applies parallel two-dimensional convolutions to capture both intra- and inter-sentence n-gram features. Its convolution and activation layers transform the input into feature maps for classification.

  • Paragraph Modelling: TextConvoNet represents a paragraph with m sentences and n words per sentence as an embedding matrix W ∈ R^m×n×d.Each row represents a sentence, each cell contains a word, and the third dimension contains word embeddings.
  • Proposed TextConvoNet Architecture: The model uses an alternative paragraph input structure and two-dimensional convolution with differing kernel sizes.
  • Proposed TextConvoNet Architecture: Four parallel convolutional pathways process the input matrix, including intra-sentence layers with 32 filters and kernel sizes of 1 × 2 and 1 × 3.The first two intra-sentence layers are concatenated to extract features.
  • Convolutional Operation: The convolutional layer applies a g × h filter and bias b(g, h) to an input paragraph F(m, n) to produce an output feature map.
  • Activation: A ReLU layer follows each convolution layer to normalize outputs and support complex learning with reduced vanishing-gradient risk and low computation cost.The activation function takes γ as its input.

3) Concatenation Layer:

The concatenation layer combines outputs from preceding pathways before classification. The subsequent network includes fully connected, dropout, classification, and task-specific loss components.

  • The concatenation layer joins multiple input blobs in a continuous manner.
  • The fully connected layer is a multilayer perceptron connected to all activations from previous layers.Its output is computed by matrix multiplication of weights and an offset value.
  • The layer maps an input β of size u to r neurons, producing an output matrix Fu×r through Equation 5.The activation function is denoted by ϑ.
  • Dropout randomly activates or deactivates hidden-unit outgoing edges during training to help reduce overfitting.
  • The final classification layer uses attributes extracted by previous layers with softmax or sigmoid activation.
  • Binary classification minimizes binary cross-entropy with sigmoid, whereas multiclass classification minimizes categorical cross-entropy with softmax.The loss definitions use final-layer outputs and ground-truth labels for each training instance and class.

C. Analysis of TextConvoNet

TextConvoNet analyzes paragraph-level sentence embeddings through parallel convolutional pathways designed to capture intra- and inter-sentence n-gram features. Its variants and experiments assess performance across five binary and multiclass datasets.

  • Paragraphs are converted into paragraph-level sentence embeddings without text preprocessing to preserve their semantics.The embedding matrix is sent into four lateral pathways.
  • TextConvoNet uses four parallel pathways, separating intra-sentence and inter-sentence convolutional feature extraction.
  • TextConvoNet 4 is the base model with four convolution layers: two for intra-sentence and two for inter-sentence n-gram features.
  • TextConvoNet 6 extends the framework to six convolutional pathways, evenly divided between intra-sentence and inter-sentence n-gram features.The study also varies filters, dropout rate, kernel sizes, fully connected nodes, and optimizers.
  • Experiments evaluate binary and multiclass text classification using performance metrics, baseline models, statistical analysis, and ablations.
  • The study uses five datasets: SST-2, Amazon reviews, R8, Twitter Airline Sentiment, and Coronavirus Tagged Datasets.The datasets include two binary and three multiclass classification tasks.

B. Performance Metrics

The evaluation uses multiple classification metrics and statistical analyses to compare TextConvoNet with other models. Effect-size analysis further characterizes the magnitude of performance differences.

  • Evaluation considers Accuracy, Precision, Recall, F1-score, Specificity, G-means, and MCC.The metric set is intended to support a broad assessment across NLP applications.
  • The Wilcoxon Signed-Rank paired sample test compares TextConvoNet 4 and 6 with other machine learning and deep learning techniques.Tests use a 95% significance level, α=0.05, without assuming normally distributed paired differences.
  • The null hypothesis is rejected when the experimental p-value is less than α, indicating a significant difference between paired groups.
  • The experimental section includes dataset details and performance-metric descriptions in Tables I and II.
  • The study defines an exceeding ratio as samples longer than the average length divided by all training samples.
  • Pearson effect size r measures the magnitude of performance differences between groups.The paper states that larger effect sizes indicate stronger relationships between variables.
  • Effect size is classified as low at r≈0.1, medium at r≈0.3, and large at r≈0.5.

C. Machine learning and Deep learning techniques used for comparison

The comparison includes seven classical machine learning techniques and several deep learning architectures. These baselines represent convolutional, recurrent, attention-based, and pretrained-transformer approaches.

  • The comparison includes Multinomial Naive Bayes, Decision Tree, Random Forest, SVC, Gradient Boosting, KNN, and XGBoost.Using multiple techniques supports a comprehensive evaluation of TextConvoNet’s usability and generalization.
  • Kim’s CNN uses pretrained word embeddings, parallel convolution layers with kernel sizes 3, 4, and 5, a 64-neuron dense layer, and classification.
  • The LSTM comparison uses a single LSTM layer for text classification tasks.
  • VDCNN uses multiple convolution and max-pooling layers, producing feature-map tiers of 64, 128, 256, and 512.Its classifier has three fully connected layers with sizes 4096, 2048, and 2048.
  • The bidirectional recurrent-attention model represents tokens with concatenated forward and backward LSTM states and forms sentence representations using attention-weighted sums.
  • The BERT baseline fine-tunes pretrained bert-base-uncased embeddings followed by a dense layer and classification layer.
  • The GRU-based comparison uses 300-dimensional word embeddings, a GRU dimension of 50, and 32-piece mini-batches.Forward and backward GRUs provide 100-dimensional word or sentence annotations.

D. Implementation Details

TextConvoNet was evaluated across multiple datasets, metrics, model variants, and statistical comparisons. Results generally favored the proposed variants, while ablations identified optimizer, dropout, kernel-size, and paragraph-length effects.

  • Evaluation setup: TextConvoNet was evaluated on five datasets using multiple classification metrics and compared with machine-learning, deep-learning, attention-based, and transformer-based models.The evaluation included accuracy, precision, recall, F1-score, specificity, G-mean, and MCC.
  • Performance comparison: 31% was the maximum average accuracy gain over all datasets, while the minimum average gain was 1.5%.All deep-learning models exceeded 70% accuracy, and the highest accuracy was reported for dataset-3 using TextConvoNet 6.
  • Performance comparison: TextConvoNet achieved higher precision than other models on all datasets except dataset-2, with average gains ranging from 4.1% to 70.8%.Recall gains ranged from 3.7% to 65.8%, with the best recall on dataset-5.
  • Performance comparison: Specificity gains ranged from 1.1% to 18.6%, while MCC gains ranged from 7.4% to about 30%.The reported Gmean1 and Gmean2 gains ranged from 3.9% to 68% and from 2.7% to 65.66%, respectively.
  • Comparative results: On multi-class datasets, TextConvoNet variants outperformed the other models across all performance metrics, with TextConvoNet 4 or 6 leading depending on the dataset.The comparison included Attention+BiLSTM, HAN, and BERT; TextConvoNet 4 performed best on datasets 1 and 3, while TextConvoNet 6 led on datasets 2, 4, and 5.
  • Ablation study: The ablation study found Adam preferable to other optimizers, dropout rate 0.4 optimal, and larger-kernel versions better for longer texts.Versions 1 and 2 performed better on comparatively smaller paragraphs, whereas versions 3 and 4 performed slightly better on longer texts.

G. Fewshot Learning

TextConvoNet performs better than baseline models in few-shot settings, achieving lower test error with fewer training examples. Its inter-sentence n-gram features provide additional information beyond 1-D CNN features.

  • G. Fewshot Learning: Few-shot learning evaluates whether models generalize effectively when only a small amount of labeled data is available for each category.The test dataset size remains constant across training proportions.
  • G. Fewshot Learning: TextConvoNet achieves lower test error rates than all baseline models with fewer training examples.This performance is achieved without changing the model’s parameter space.
  • G. Fewshot Learning: TextConvoNet extracts inter-sentence n-gram features in addition to same-sentence features extracted by 1-D CNNs.The additional features support performance with fewer training examples.

V. CONCLUSION AND FUTURE WORK

The paper concludes that TextConvoNet captures both intra-sentence and inter-sentence n-gram features through an alternative input representation. Its evaluation indicates effectiveness for binary and multi-class text classification, while future work will explore higher-dimensional representations.

  • V. CONCLUSION AND FUTURE WORK: TextConvoNet captures intra-sentence and inter-sentence n-gram features using an alternative text input representation.The architecture is presented as a CNN-based deep learning model for text classification.
  • V. CONCLUSION AND FUTURE WORK: Extensive evaluation shows that TextConvoNet is effective for binary and multi-class classification problems.The conclusion attributes the improvement to extracting inter-sentence relationships.
  • V. CONCLUSION AND FUTURE WORK: Future work will explore higher-dimensional input representations so convolution can capture additional textual features.

APPENDIX A

The supplementary file provides implementation details and additional experimental results omitted from the paper because of space constraints.

  • APPENDIX A: The supplementary file documents model implementations, control-parameter values, and additional results not reported in the paper.It is available in the GitHub repository and uploaded with the paper.
Loading 2203.05173v1…