PaperlayerView exact revision on arXiv

Source-linked AI summary

Event Interaction in Low-Rank Bottlenecks for Temporal Relation Extraction

Wei Sun, Tingyu Qu, Jesse Davis, Marie-Francine Moens

arXiv:2609.06731v1cs.CLcs.AI

TL;DR

PEFT bottlenecks restrict the interaction information needed for temporal relation extraction, where common concatenation and shallow fusion strategies are insufficient. CBI addresses this by combining depthwise convolution with element-wise multiplication inside the bottleneck, achieving consistent gains across datasets and backbones with minimal cost.

  • Problem

    PEFT routes task-specific information through low-rank bottlenecks, limiting the detailed cross-event interaction patterns needed for temporal relation extraction.

  • Method

    CBI applies lightweight depthwise convolution to event representations and then uses element-wise multiplication to model explicit event interaction inside the PEFT bottleneck.

  • Results

    Across five datasets and seven backbone models under Adapter and LoRA settings, CBI delivers consistent improvements of up to +31.7 micro-F1 with minimal computational and memory cost.

  • Takeaways & Limitations

    Explicitly modeling event interaction inside low-rank bottlenecks is effective and necessary for temporal relation extraction in PEFT settings.

  • Takeaways & Limitations

    The method assumes an absolute-difference treatment to avoid bias caused by the textual position of events.

Abstract

from arXiv · show

Temporal relation extraction determines whether an event occurs before, after, or simultaneously with another event, and therefore relies on accurately modeling how the two events interact. Mainstream systems achieve this by concatenating event spans or using shallow fusion, which works well when all model parameters are trainable. However, in parameter-efficient fine-tuning, low-rank bottlenecks restrict information flow and prevent these interaction signals from passing through, leading to clear performance drops. To address this limitation, we propose a theoretically grounded architecture, Convolutional Bottleneck Interaction (CBI), which first applies lightweight depthwise convolution to enhance event representations and then uses element-wise multiplication to capture effective event-event interactions inside the bottleneck. Across five datasets and seven backbone models in the Adapter and LoRA settings, CBI provides consistent and substantial gains, up to +31.7 micro F1, while adding minimal computational cost, showing that explicit interaction inside low-rank spaces is crucial for temporal relation extraction. The code is available at https://github.com/VRCMF/CIF.git.

1 Introduction

Temporal relation extraction depends on modeling fine-grained event interactions, but PEFT low-rank bottlenecks restrict how temporal cues and task-specific interaction patterns pass through the model. CBI addresses this with explicit bottleneck interaction, combining convolution and multiplication, and improves results across datasets and backbones with minimal computation.

  • Motivation: Temporal relation extraction identifies whether one event occurs before, after, or simultaneously with another, supporting narrative understanding, story generation, and temporal question answering.Inaccurate predictions can propagate into incoherent or misleading downstream outputs.
  • Motivation: PEFT reduces training cost but restricts task-specific learning to low-rank parameters, making temporal relation extraction especially sensitive to bottleneck information loss.The task requires fine-grained interactions between events, while all task-specific information must pass through a restricted subspace.
  • Motivation: Text Concatenation and Embedding Fusion fail to transmit temporal cues effectively through PEFT bottlenecks, whereas full fine-tuning captures them at substantially higher computational cost.Figure 1 shows attention diffused over irrelevant tokens for the PEFT approaches; the TDD-Man example illustrates the resulting loss of crucial temporal evidence.
  • Approach: CBI directly models event-event interactions inside the bottleneck by applying lightweight convolution before element-wise multiplication.The convolution enlarges token receptive fields and smooths local noise, while multiplication captures channel-wise co-activation.
  • Results: +31.7 micro-F1 is the maximum reported gain for CBI across five datasets and seven backbone models under Adapter and LoRA configurations.The method adds minimal computational and memory cost while producing consistent improvements.

2 Related Work

Prior temporal relation extraction systems typically encode paired event contexts jointly and rely on implicit interaction through self-attention. The paper argues that this paradigm is poorly matched to PEFT bottlenecks and introduces explicit interaction functions, with convolution-enhanced multiplication as its proposed solution.

  • Existing approaches: Pretrained contextual encoders support sentence-pair classification by jointly encoding two event segments and modeling their interactions.This formulation became a mainstream architecture for temporal relation extraction.
  • PEFT limitation: PEFT freezes most parameters and routes information through low-rank bottlenecks, making concatenation and other shallow fusion strategies insufficient for detailed temporal cues.The applicability of conventional architectures is therefore unclear in parameter-efficient temporal extraction.
  • Existing approaches: Most existing systems use implicit event interaction through concatenation and transformer self-attention, a strategy that assumes sufficient model capacity.Input-level element-wise fusion has also been proposed to strengthen pairwise signals.
  • Related inspiration: The paper draws on bottleneck feature-routing work that improved cross-modal alignment and low-resource cross-lingual transfer over naïve concatenation.This motivates directing interaction information explicitly through PEFT bottlenecks.
  • Proposed interaction modeling: The proposed PEFT design explicitly tests multiplication, addition, and subtraction, with multiplication performing best because it captures channel-wise co-activation patterns.Depthwise convolution is added to address multiplication’s sensitivity to representation misalignment and noise.

3 Method

CBI augments low-rank PEFT bottlenecks with local context enrichment and explicit cross-event interaction. It down-projects each event, applies depthwise convolution, multiplies enriched representations element-wise, and returns the interaction through the PEFT residual path.

  • CBI overview: CBI augments existing LoRA or adapter bottlenecks with explicit cross-event interaction while preserving parameter efficiency.The CBI adapter is inserted inside the low-rank bottleneck and is designed for two event representations.
  • PEFT foundations: LoRA trains low-rank updates to key and value projections, whereas adapters insert a small learnable bottleneck module inside each transformer block.Both strategies keep the pretrained backbone largely fixed and restrict adaptation to low-rank parameters.
  • Dual-Tower Local Enrichment: CBI down-projects each event into rank-r bottleneck space and applies depthwise convolution independently across channels to aggregate local temporal context.The convolution uses a one-dimensional kernel along the token axis and produces enriched bottleneck features.
  • Explicit Cross-Event Interaction: CBI forms cross-event interaction features by element-wise multiplication of the enriched event representations at each token position and channel.The resulting products are second-order interactions and recover diagonal bilinear terms over corresponding dimensions.
  • Explicit Cross-Event Interaction: The interaction field explicitly represents diagonal bilinear cross-event interactions that are difficult to learn implicitly under rank-r PEFT updates.Local convolution also lets near-aligned tokens interact despite positional differences by providing soft alignment and denoising.
  • Output integration: CBI converts the interaction field back to hidden size with the PEFT up-projection and injects it through a residual path.The resulting representation is passed to subsequent transformer blocks.

4 Experiments

The experiments evaluate CBI across five temporal relation extraction benchmarks spanning local intra-sentential reasoning and long-range discourse-level reasoning. They are designed to test generalization across dataset granularity, backbone capacity, and PEFT configurations.

  • Evaluation goals: The experiments assess whether CBI generalizes across varying backbone capacities and PEFT configurations while examining the effects of design choices.The evaluation is described as extensive and spans diverse datasets and model families.
  • Evaluation setup: CBI is evaluated on five widely used temporal relation extraction benchmarks with differing temporal-reasoning requirements.The benchmark suite includes datasets emphasizing intra-sentential event pairs and datasets requiring inter-sentential discourse context.
  • Intra-sentential benchmarks: The intra-sentential benchmarks TB-Dense, MATRES, and TCR emphasize local or fine-grained reasoning over mostly sentence-internal event pairs.TCR contributes temporal labels while also containing causal annotations that are not used here.
  • Inter-sentential benchmarks: The inter-sentential benchmarks TDD-Man and TDD-Auto target discourse-level temporal reasoning involving longer-range context.TDD-Man includes event pairs that can be multiple sentences apart and uses interval relations including BEFORE, AFTER, INCLUDES, IS_INCLUDED, and SIMULTANEOUS.

5 Implementation Details

The implementation evaluates CBI across diverse backbones, PEFT configurations, datasets, baselines, and interaction variants. Results consistently favor CBI, including under limited data and visualization-based analyses of cue localization.

  • Experimental setup: Experiments span seven encoder-only and decoder-style backbones using both Adapter and LoRA PEFT configurations.The evaluated models include BERT-large, RoBERTa-large, and five instruction-tuned LLMs.
  • Baseline comparison: CBI consistently achieves the strongest micro-F1 across PEFT configurations, outperforming Text Concatenation and Embedding Fusion.On RoBERTa-large with Adapter, CBI reaches 76.59% versus 44.83% for Text Concatenation and 49.94% for Embedding Fusion.
  • Baseline comparison: Up to +31.76 percentage points separates CBI from Text Concatenation on RoBERTa-large under Adapter.CBI improves RoBERTa-large from 44.83% to 76.59% micro-F1 in that comparison.
  • Interaction ablations: CBI remains stronger than simpler interaction functions, reaching 76.59% F1 versus 73.35% for Multiplication and 62.87% for Subtraction on RoBERTa-large with Adapter.On Qwen-2.5-7B with LoRA, CBI reaches 65.66% F1 compared with 59.64% for Multiplication and 61.53% for Subtraction.
  • Data efficiency: With 20% of the training data, CBI achieves 62.60% micro-F1 in Adapter experiments and degrades more gracefully than Text Concatenation as data decreases.With full supervision, CBI reaches 72.89%; in LoRA, it reaches 77.88% with complete data and 67.56% at 60% data.
  • Cue localization: Interaction maps indicate that multiplication highlights nearly all tokens in one sentence, whereas convolution filters noise and strengthens relevant temporal cues.The analysis connects broad token importance under multiplication with noise amplification under limited supervision.

Appendix 7.

The appendix examines computational efficiency and convolutional sensitivity, finding that CBI preserves the low cost of multiplication while kernel size 5 provides the most stable performance.

  • Kernel sensitivity: Kernel size 5 provides reliable performance across all five datasets, while size 1 underperforms and size 7 shows greater variance.The study therefore adopts kernel size 5 as CBI’s default configuration.
  • Computational efficiency: Multiplication and CBI reduce training time by more than 40% compared with Text Concatenation while leaving GPU memory nearly unchanged.Text Concatenation requires over 370 seconds per epoch, whereas lightweight interaction variants have similar computational cost.
  • Computational efficiency: CBI adds fewer than 0.1% parameters over Multiplication while matching its efficiency and consistently achieving better performance.The added structure consists of two depthwise convolutional layers with small kernels.

6 Conclusion

The paper attributes PEFT’s weak temporal relation extraction performance to low-rank bottlenecks that block cross-event cues, and proposes CBI to inject interaction within the bottleneck. Across five datasets and seven backbones, CBI yields consistent gains with negligible overhead, while motivating extensions to other relational tasks.

  • Low-rank bottlenecks block cross-event interactions and prevent key temporal cues such as discourse markers from being properly encoded.
  • CBI combines lightweight depthwise convolution with element-wise multiplication to model event interaction directly inside the bottleneck.The convolution enlarges each event’s receptive field and reduces local noise before multiplicative channel-wise interaction.
  • Up to +31.7 micro-F1 is achieved across five datasets and seven LLM backbones with adapters and LoRA, while adding negligible overhead.
  • The findings suggest that interaction-aware bottleneck designs may benefit relational tasks beyond temporal relation extraction.Future applications named by the paper include causal relation extraction, discourse relation classification, and document-level reasoning.

7 Theoretical Analysis

The theoretical analysis explains why PEFT struggles to represent high-rank cross-event interactions and shows how CBI restores useful second-order expressivity without increasing PEFT rank. Depthwise convolution stabilizes the interaction under misalignment and improves its signal-to-noise ratio.

  • CBI uses depthwise convolution for local temporal alignment and noise suppression, followed by element-wise multiplication for explicit second-order event interaction.These inductive biases are designed to restore relational expressivity without increasing PEFT rank.
  • Element-wise multiplication produces an explicit diagonal bilinear interaction field that later linear aggregation can directly exploit.The feature field represents channel-wise products of the two event sequences.
  • Depthwise convolution preserves diagonal interactions under controlled error and improves their signal-to-noise ratio.The stated lower bound for the improvement is 1/α^2 under the noise-reduction assumption.
  • PEFT restricts updates to rank-r forms, limiting the backbone’s ability to create high-rank bilinear interactions between events.The restriction arises from down- and up-projections with r much smaller than the hidden dimension.
  • Together, the lemmas show that CBI restores second-order relational expressivity while keeping PEFT rank unchanged and adding negligible parameters.

8 Dataset statistics

The experiments use five widely used temporal relation extraction datasets, whose statistics are reported in Table 6.

  • Five widely used temporal relation extraction datasets are included in the experimental evaluation.Dataset statistics are provided in Table 6.

9 Experimental Analysis

CBI remains stronger than baseline interaction methods across bottleneck capacities, error analyses, and a GNN-augmented setting. Its gains reflect more effective interaction modeling rather than simply increased rank, although it introduces some containment confusions and all models struggle with simultaneity.

  • 9.1 Results of Varying Bottleneck Capacity: Increasing bottleneck rank from 128 to 512 improves Text Concatenation by about 3 points, yet both baselines remain much worse than CBI.CBI performs better without increasing rank, indicating that its gains come from interaction effectiveness rather than larger parameter capacity.
  • 9.2 Error Analysis: In LoRA, CBI reduces gold IS_INCLUDED cases misclassified as BEFORE/AFTER to 28.7%, versus 53.7% for Embedding Fusion and 43.5% for Text Concatenation.
  • 9.2 Error Analysis: In LoRA, direction confusion accounts for 9.3% of CBI’s containment errors, compared with 24.1% for Embedding Fusion and 27.8% for Text Concatenation.The same analysis reports a bias toward INCLUDES for Text Concatenation in the adapter setting.
  • 9.2 Error Analysis: CBI predicts containment for 17.2% of gold BEFORE/AFTER cases, compared with 13.1% for Embedding Fusion.The paper interprets this as stronger interaction modeling sometimes treating semantic relatedness as temporal containment.
  • 9.2 Error Analysis: CBI correctly predicts 20.0% of LoRA SIMULTANEOUS cases, while both baselines achieve 0%.The result is based on 3 of 15 simultaneity cases.
  • CBI preserves more distributed temporal cues than lightweight baselines while avoiding the computational cost and longer training times of full fine-tuning.The cue analysis highlights “soon after,” “after,” “been killed,” and “defection” across adapter and LoRA settings.
Loading 2609.06731v1…