PaperlayerView exact revision on arXiv

Source-linked AI summary

R-DEIM Net: An Efficient Rationale-Augmented Dual-Expert Interaction Model for Paraphrase Detection

Pushp, Vaibhav Prajapati, Himangshu Sarma

arXiv:2609.30100v1cs.CLcs.AI

TL;DR

Paraphrase detection faces a trade-off between the computational cost of high-performing large models and the limited rationale transparency of efficient systems. R-DEIM Net combines token interaction modeling with a rationale-generating decoder whose hidden states contribute classification features. It reports 90.07% accuracy and 90.16% F1-score, while its rationale labels for QQP are synthetically generated.

  • Problem

    Paraphrase detection faces a trade-off between high performance with substantial computational cost and efficient systems with limited transparent rationale generation.

    Source: §1 · p. 1

  • Method

    R-DEIM Net combines an interaction expert with a Flan-T5-small reasoning expert, using decoder hidden states as complementary classification features.

    Source: §1 · p. 2

  • Results

    90.07% accuracy and 90.16% F1-Score are reported for R-DEIM Net, a 76M parameter architecture.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Architectural analyses provide empirical support that model decisions are driven by semantic content rather than superficial lexical overlap.

    Source: §5 · p. 11

    QQP rationales are synthetically generated because manually annotating more than 400k pairs was infeasible, rather than human-written ground truth.

    Source: Limitations · p. 11

Abstract

from arXiv · show

Recent advances in paraphrase detection reveal a fundamental trade-off: large language models achieve high accuracy but require high computation, while efficient Siamese-BERT variants offer practical scalability with reduced transparency in rationale generation. We present R-DEIM Net, a 76M-parameter dual-expert architecture exploring whether moderate-scale models can achieve competitive accuracy on paraphrase detection while enabling human-readable rationale generation. The architecture combines two specialized components: an Interaction Expert that captures token-level similarity patterns through multi-scale 2D convolutions and attention head allowing variable input length, and a Reasoning Expert that uses a Flan-T5-small decoder to generate rationales as auxiliary supervision. Rather than re-encoding generated text, we extract and pool decoder hidden states as complementary features for classification. On the Quora Question Pairs dataset, R-DEIM Net achieves 90.07\% accuracy and 90.16\% F1-score via 10-fold cross-validation. This represents competitive performance with strong transformer-based baselines (e.g., MFAE BERT: 90.54\% accuracy) and recent large language model based approaches (LLaMA-70B) while using a substantially smaller parameter budget. The model generates rationales alongside predictions, providing potential for auxiliary human-readable descriptions.

1 Introduction

Paraphrase detection balances accuracy and computational cost against transparent rationale generation. R-DEIM Net addresses this trade-off with interaction and reasoning experts, using generated rationales and decoder features alongside token-level interactions.

1 Introduction

  • R-DEIM Net combines token-level interaction modeling with generative reasoning signals for paraphrase classification.The model outputs a binary classification alongside generated text rationales.Source: §1 · p. 2
  • The Interaction Expert uses multi-scale 2D convolutions and attention-based pooling to capture token alignment patterns in variable-length sentence pairs.Its design targets token-level similarity patterns across inputs of varying length.Source: §1 · p. 2
  • Decoder hidden states from rationale generation serve as complementary classification features without re-encoding the generated text.The paper presents this as avoiding additional encoding layers and computational overhead.Source: §1 · p. 2
  • The paper analyzes structural alignment, CNN kernel activations, and token- and POS-level perturbations to examine the experts’ contributions to semantic decisions.These analyses include structural alignment, kernel activation behavior, and POS-based perturbation.Source: §1 · p. 2

2 Related Work

Related work traces paraphrase detection from lexical features through neural sequence and interaction models to transformer and large-language-model approaches. The section identifies gaps in F1 reporting, input-length handling, and human-readable explanations.

2 Related Work

  • Prior methods progressed from manual lexical features and vectorization to sequence models, convolutional approaches, dense interaction systems, transformers, and large language models.The review describes lexical methods as computationally light but limited in deep semantic context and word-order dependencies, while later architectures target richer interactions.Source: §2 · pp. 2–3
  • High-interaction and transformer-based systems reach the 88% to 90% accuracy range, while the review also notes that existing architectures provide binary labels.Examples include DIIN, MwAN, ensemble BERT systems, and LLaMA-70B.Source: §2 · p. 3
  • Existing systems often omit F1-score, impose fixed max_tokens limits, and provide binary labels without transparent rationales.Fixed limits force a compromise between truncating sentences and adding padding, according to the review.Source: §2 · p. 3

3 Method

R-DEIM Net treats question matching as binary classification with rationale generation, combining token-level interaction patterns with decoder-derived reasoning features. Its experts produce pooled representations that support joint classification and generation training.

3 Method

  • Question matching is framed as binary duplicate-versus-non-duplicate classification, with the approach also providing a rationale.Source: §3 · p. 3

3.1 Branch A: Interaction Expert

  • Multi-scale 2D convolutions analyze token-pair similarities, while attention pooling compresses variable-length interaction patterns into a fixed-size vector.Parallel kernels capture local multi-token similarity patterns that a one-to-one dot product could miss.Source: §3.1 · pp. 3, 5, Figure 1 · p. 4

3.2 Branch B: Differentiable Reasoning Expert

  • Flan-T5-small generates rationales from shared-encoder embeddings, and its decoder hidden states are attention-pooled into a reasoning vector without re-encoding generated text.A linear projection bridges the encoder and decoder dimensions.Source: §3.2 · p. 5

3.3 Fusion and Multi-Task Training Objective

  • The experts’ vectors feed an MLP, optimized with weighted classification and generation losses; classification uses weighted per-sample BCE to reflect false-positive costs.Source: §3.3 · p. 5

4 Experiments

Across QQP and seven transfer datasets, R-DEIM Net is evaluated through benchmark comparisons and architectural analyses. Results show competitive QQP performance, gains after fine-tuning, and sensitivity to semantic content.

4.1 Experimental Setup

  • The QQP dataset began with 404,290 pairs and 400,520 annotated pairs after preprocessing; synthetic LLM rationales augmented training data.The model combines a 384-dimensional MiniLM encoder with a Flan-T5-small decoder and task-specific heads, totaling roughly 76M parameters.Source: §4.1 · p. 6

4.2 Experimental Results

  • 90.16% F1-Score and 90.07% accuracy on QQP; R-DEIM Net improves on listed efficient baselines and is within reported large-LLM ranges while ≈900x smaller.Table 2 presents the 10-fold cross-validation results; the split preceded preprocessing to prevent validation data from informing training.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 Transfer Learning Analysis

  • Zero-shot transfer yielded 98.15% F1 on SprintFAQ and 63.01% F1 on SoDD, while the paper reports a performance gap on adversarial PAWS and MRPC.The authors hypothesize that localized spatial alignments may be vulnerable to adversarial word scrambling, noting that further analysis is needed.Source: §4.3 · pp. 7–8
  • Fine-tuning for 2–4 epochs improved PAWS from 53.54% to 67.97% F1 and CQADupStack from 39.24% to 80.32% F1.The authors report gains across all seven datasets.Source: §4.3 · p. 8

4.4 Architectural Analysis

  • KS = 0.224 (p < 10^-4) distinguishes duplicate from non-duplicate diagonal-interaction distributions, but ρ = 0.335 links interaction only weakly to predicted duplicate probability.The authors connect this modest association to the need for complementary branches to handle semantically equivalent but non-aligned pairs.Source: §4.4 · pp. 8–9, Figure 2 · p. 9
  • When diagonal interaction was weak, 5 × 5 and 7 × 7 kernels showed higher relative activation than the 3 × 3 kernel.This pattern indicates that the Interaction Expert uses broader spatial contexts when matches are not diagonally aligned.Source: Table 4 · p. 9, §4.4 · p. 9
  • Masking nouns, verbs, and adjectives caused the largest local accuracy drops and flip rates; masking content-word groups degraded F1, while removing function words caused marginal loss.The token- and group-level perturbations used 1,000 question pairs.Source: §4.4 · p. 9, Figure 3 · p. 10, Figure 4 · p. 10

5 Conclusion

R-DEIM Net combines interaction and reasoning signals in a 76M-parameter model, reporting strong QQP results and analysis supporting its design.

5 Conclusion

  • 90.07% accuracy and 90.16% F1-Score are reported for the 76M-parameter R-DEIM Net.The conclusion presents the model as balancing performance with a smaller parameter budget than massive models.Source: §5 · p. 10
  • Generative reasoning features consistently improve on an interaction-only baseline, while architectural analyses examine alignment, CNN activations, and token- and POS-level perturbations.The authors interpret these analyses as support that decisions reflect semantic content rather than superficial lexical overlap.Source: §5 · pp. 10–11
  • Future work proposes applying R-DEIM to NLI, testing zero-shot transfer and parameter-efficient fine-tuning, and exploring distilled decoders.These directions target broader sentence-pair tasks and the model's efficiency-to-performance ratio.Source: §5 · p. 11

Limitations

The study identifies synthetic rationale supervision, unverified strict faithfulness, pair-level evaluation, and base-model hardware choices as limitations or scope boundaries.

Limitations

  • QQP rationales were synthetically generated with a large language model because manually annotating more than 400k pairs was infeasible.The authors note that this is less ideal than human-written ground truth, although an Appendix D validation study found strong alignment with human reasoning.Source: Limitations · p. 11
  • The rationale text is auxiliary and descriptive, not a guaranteed causal post-hoc explanation, and strict faithfulness is not claimed.The authors identify stronger faithfulness validation as future work.Source: Limitations · p. 11
  • Evaluation uses pair-level generalization on unique question-pair combinations, reflecting retrieval settings despite recurring questions across pairs.The paper frames this design as supporting direct comparison with existing literature and high-precision duplicate detection.Source: Limitations · p. 11
  • Experiments use base models because larger models require substantially more VRAM and training time than the available hardware scope allowed.The authors state that larger models might yield incremental performance gains.Source: Limitations · p. 11

A Rationale Generation Details

Synthetic rationale prompts structure each question-pair analysis around intent, entities, and scope before a synthesized comment.

A Rationale Generation Details

  • The QQP dataset was augmented with synthetic rationales generated by gemini-2.5-flash-lite to train the Reasoning Expert.The listing identifies the prompt used for that rationale generation.Source: A Rationale Generation Details · p. 16
  • The rationale-generation prompt asks for core intent, key entities, scope, and a final comment of 30–40 words while forbidding several similarity-label terms.The instructions aim to keep reasoning structured and consistent across the dataset.Source: A Rationale Generation Details · p. 16
  • The prompt requests a single valid JSON object with fields for pair ID, core intent, key entities, scope, and final comment.The output format specifies analysis content rather than a duplicate/non-duplicate label in the final comment.Source: A Rationale Generation Details · p. 16
  • The example contrasts a general airport-directions query with one asking for the fastest route, describing their shared topic and differing specificity.Its final comment frames both as inquiries about directions while distinguishing the time-efficiency constraint.Source: A Rationale Generation Details · p. 16

B Dataset Composition

The dataset composition section notes variation in dataset scale and label balance and specifies the QQP class meanings.

B Dataset Composition

  • Class 0 denotes Non-Duplicates and Class 1 denotes Duplicates in the reported dataset statistics.Table 5 is identified as presenting dataset statistics and class ratios.Source: B Dataset Composition · p. 17, Table 5 · p. 17
  • The evaluated datasets vary substantially in scale and label balance.The section states that evaluation includes balanced and highly skewed distributions.Source: B Dataset Composition · p. 17

C Hyperparameters

Table 6 documents the hyperparameter configurations used to train R-DEIM Net’s components. The configurations are presented to support reproducibility across the experimental setups in Section 4.

C Hyperparameters

D Semantic Consistency and Human Evaluation

Generated rationales aligned more closely with their corresponding question pairs than with randomly selected pairs. A human evaluation also found strong alignment between model-generated and human-written rationales.

D Semantic Consistency and Human Evaluation

  • 0.5670 mean cosine similarity for correctly matched question pairs and rationales fell to 0.0662 for randomly paired examples.The comparison used embeddings of concatenated question pairs and their corresponding LLM-generated “Final Comment” rationales.Source: D Semantic Consistency and Human… · pp. 17–18
  • 0.814 Spearman Correlation and 0.763 Cosine Similarity measured alignment between human-written and model-generated rationales in a study with 12 annotators and over 170 pairs.Average rationale lengths were 24.50 tokens for human-written text and 28.02 for Gemini-generated text.Source: D Semantic Consistency and Human… · p. 18
  • Examples A and B paired human-written and generated rationales with correlations of 0.891 and 0.883, respectively.The examples contrast unrelated topics in A and describe shared Harry Potter-related intent in B.Source: D Semantic Consistency and Human… · p. 18

E Qualitative Analysis

Figure 5 presents duplicate probabilities alongside rationales describing semantic identity or distinction. The examples show the model articulating shared intent for duplicates and contrasting differing constraints for non-duplicates.

E Qualitative Analysis

  • Figure 5 pairs duplicate predictions with rationales; its examples describe shared intent for true duplicates and differing constraints, such as salary versus skills, for non-duplicates.The paper reports that the examples correctly identify the distinction alongside the prediction.Source: Figure 5 · p. 18, E Qualitative Analysis · p. 18

Related papers

Loading 2609.30100v1…