Source-linked AI summary
Evaluating Multi-Dimensional Generalization of Large Language Models in Temporal Extraction Tasks
Fahmid Shahriar Iqbal, Ritam Dutt, Soumitra Das, Arnav Verma, Sagnik Ray Choudhury
TL;DR
Temporal extraction is difficult, and in-domain evaluation offers limited evidence about reliability under distribution shifts. This study evaluates prompted LLMs across extraction tasks and generalization dimensions, finding that base-task performance is informative but not a reliable proxy across all dimensions. The results motivate evaluation beyond in-domain performance and cross-dimensional reasoning strategies.
Problem
Annotation agreement is low and LLM extraction behavior can be inconsistent, while in-domain evaluation alone is insufficient to assess reliability under distribution shifts.
Source: §1 · p. 1Method
The paper systematically evaluates prompted LLMs on time and event expression extraction across generalization dimensions, reasoning strategies, architectures, and model sizes.
Source: §1 · p. 2Results
Takeaways & Limitations
Reliable temporal extraction requires evaluation beyond in-domain performance and models or prompting strategies that generalize more consistently across dimensions.
Source: §1 · p. 3, §5 · p. 9The study covers TIMEX and EVENTX extraction rather than higher-level temporal reasoning tasks, and benchmark pretraining contamination is possible.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Time and event expression extraction are fundamental temporal reasoning tasks, but the problem remains difficult due to annotation ambiguity, domain sensitivity, and unstable model behavior. Existing evaluations focus on in-domain performance, offering limited insight into reliability under distribution shifts. We evaluate multiple model configurations across families, architectures, and reasoning strategies over four dimensions of generalization, examining transfer from base performance, cross-dimensional correlations, and the effects of scale, architecture, and prompting. This provides a systematic study of how prompted LLMs generalize in time and event expression extraction tasks. We find that strong base-task performance generally predicts better generalization. However, this relationship weakens under substantial distribution shifts. Inductive prompting performs most consistently across domain shift, adversarial perturbations, compositionality, and length increase, while gains from scale, architecture, and deductive and abductive prompting strategies are uneven and dimension-specific. We conclude that LLM generalization in temporal extraction tasks cannot be predicted from any single dimension alone and cannot be reliably inferred from in-domain or single-dimension evaluations, highlighting the need for reasoning strategies that generalize across dimensions.
1 Introduction
Temporal extraction remains difficult amid annotation disagreement and unreliable model behavior, while in-domain evaluation alone cannot assess reliability under distribution shifts. This study evaluates prompted LLM generalization across four dimensions and finds that base-task performance is informative but not sufficient, with inductive prompting comparatively consistent across dimensions.
Motivation
- In-domain evaluation alone is insufficient to assess extraction reliability because annotation agreement is low and LLM behavior can be inconsistent.The paper frames robust extraction as a bottleneck for reliable temporal reasoning.Source: §1 · p. 1
Study scope
- The study examines domain, robustness, compositional, and structural generalization in time and event expression extraction.These dimensions cover cross-domain shifts, meaning-preserving perturbations, joint time-and-event extraction, and sentence-to-document context changes.Source: §1 · p. 2
- The evaluation varies generalization dimensions alongside deductive, inductive, abductive, and combined prompting strategies.The paper treats these as complementary axes because performance under one reasoning strategy need not transfer to another.Source: §1 · p. 2
Research questions
- The study asks whether base-task performance transfers across dimensions, whether dimensions correlate, and how prompting and model factors affect generalization.The examined model factors include Dense versus MoE architecture and model size.Source: §1 · p. 2
Key findings
- Strong base-task performance generally predicts multi-dimensional generalization, but this relationship weakens under substantial distribution shifts.Inductive reasoning strategies show strong multi-dimensional generalization and greater cross-dimensional consistency; scale, architecture, and other prompting strategies have uneven effects.Source: §1 · p. 2
Implication
- Reliable temporal extraction requires evaluation beyond in-domain performance and reasoning strategies that generalize across dimensions.The conclusion cautions against assuming robust generalization follows from strong base-task performance.Source: §1 · p. 3
2 Experimental Setup
The experimental setup constructs datasets for base extraction and four kinds of generalization, evaluates model and prompting variations, and scores predicted spans using word-level matching.
2.1 Tasks & Datasets
- Base TIMEX and EVENTX datasets use sentence-level TimeBank instances, while compositional and structural variants test joint extraction and longer document inputs.Base instances retain sentences containing at least one target expression; outputs are textual spans.Source: §2.1 · p. 3
- Adversarial variants use lexical or character perturbations, while domain-shift datasets cover clinical text and controlled vocabulary replacement.The prompt-based Adv_P variant adds lexical, character-level, and structural changes while validating that target expressions remain unchanged.Source: §2.1 · p. 3
2.2 Models
- Models are organized into dense and MoE computational tiers, with a separate Mistral-family study examining increased parameter counts.The scaling comparison uses Mistral 3B, Mistral Small 3.2 (24B), and Mistral Large (123B).Source: §2.2 · pp. 3–4
2.3 Reasoning strategy
- Seven reasoning strategies combine inductive few-shot examples, deductive instructions, abductive explanations, and hybrid prompts.The number of few-shot exemplars is selected from k ∈3, 5, 8 and fixed across settings.Source: §2.3 · p. 4
2.4 Evaluation Metrics
- Predicted expressions are evaluated with micro F1 using word-level LCS similarity and optimal one-to-one matching between predicted and gold spans.The matching prevents multiple predictions from matching the same gold expression.Source: §2.4 · p. 4
3 Results
The results test whether base-task rankings transfer to different generalization dimensions, whether performance rankings correlate across dimensions, and how prompting, architecture, and model size affect generalization. Base-task performance is generally predictive, but transfer is weaker under substantial shifts, and the effects of model choices vary across dimensions.
3.1 RQ1: Transferability of Base Task Performance Across Dimensions
- On TIMEX, base-task pairwise advantages usually persist on adversarial, length, and compositional data, but are much less likely to persist under strong T2C shifts; EVENTX drops further.Source: Figure 3 · p. 5, §3.1 · p. 5
- For EVENTX, inductive+ strategies improve the likelihood that base-task advantages carry to generalization datasets, including the difficult T2C shift.Source: Figure 4 · p. 5, §3.1 · p. 5
3.2 RQ2: Correlation Across Generalization Dimensions
- TIMEX rankings are moderately to strongly consistent across most dimensions, but EVENTX T2C rankings are negatively correlated with other dimensions, and inductive+ correlations are higher than non-inductive ones.Source: §3.2 · pp. 5–6, Figure 5 · p. 6
- Inductive+ strategies improve generalization and maintain higher cross-dimensional correlations than non-inductive strategies.Source: §3.2 · p. 6
- T2C errors include missing and boundary-mismatched TIMEX spans, while 66.7–79.5% of gold EVENTX spans are missed; embeddings show greater T2C–Base than Voc–Base separation.Source: §3.2 · p. 6, Figure 6 · p. 7, Figure 7 · p. 7
3.3 RQ3: Effects of Reasoning Strategies, Model Architecture, and Scale
- For TIMEX, explanations improve on instructions while instructions hamper generalization; for EVENTX, plain induction outperforms the alternatives.Source: §3.3 · p. 6, Figure 8 · p. 7
- MoE models outperform Dense models on base tasks, but their advantage weakens under larger shifts and they perform worse on EVENTX, despite larger active parameter counts.Source: §3.3 · pp. 7–8, Figure 9 · p. 8
- Scaling Mistral models improves base, adversarial, and length performance, but gains are uneven: 3B-to-8B differences are non-significant for T2C and compositional tasks, and EVENTX domain generalization is unaffected.Source: Figure 10 · p. 8, §3.3 · p. 8
4 Related Work
Prior work studies generalization across several dimensions and proposes varied improvement methods, while this study focuses on how existing models generalize on foundational temporal extraction tasks.
4 Related Work
- Prior work spans domain, robustness, compositional, and structural generalization, while systematic dataset construction for composition and length generalization have received less attention.The review also notes that robustness methods can produce unrealistic inputs that miss shortcut-learning tendencies.Source: §4 · p. 9
- Proposed improvement methods range from domain adaptation and adversarial training to neuro-symbolic, meta-learning, and structured-prompting approaches.Related work also explores integrating inductive, deductive, and abductive reasoning.Source: §4 · p. 9
- Unlike work improving generalization methods or evaluating broader temporal reasoning, this study assesses existing models on time and event expression extraction.The review frames these extraction tasks as foundational.Source: §4 · p. 9
5 Conclusion & Future Work
The study evaluates prompted LLMs across four temporal-extraction generalization dimensions and finds that generalization cannot be reliably inferred from in-domain performance alone.
5 Conclusion & Future Work
- The study covers domain, robustness, compositional, and structural shifts in prompted LLMs for time and event expression extraction.These tasks target time (TIMEX) and event (EVENTX) expressions.Source: §5 · p. 9
- Base-task performance is not a reliable proxy for generalization across dimensions, and model size and architecture are not consistently predictive.Fine-tuned PLMs show more stable relative behavior across dimensions, though not necessarily stronger absolute performance.Source: §5 · p. 9
- Inductive reasoning strategies improve both base and generalization performance and yield stronger cross-dimensional correlations.The study motivates future work on models and prompting strategies that generalize more consistently across diverse distribution shifts.Source: §5 · p. 9
Limitations
The study's scope is limited to expression extraction, and its model comparisons do not isolate the causal effects of architecture or reasoning strategy.
Limitations
- The evaluation covers TIMEX and EVENTX extraction, not higher-level temporal relation inference, timeline construction, or temporal question answering.Possible pretraining contamination is also noted, although the authors say relative degradation across shifts remains informative.Source: Limitations · p. 9
- Dense and MoE models differ in active parameter count, pretraining data, and generation, so their comparison does not isolate architecture's causal effect.Prompting comparisons also vary reasoning style, wording, and output format, limiting attribution to reasoning strategy alone.Source: Limitations · p. 10
A.1.1 Base Task Performance
Figure 12 compares average base-task Micro-F1 for inductive+ and non-inductive setups across models on TIMEX and EVENTX.
A.1.1 Base Task Performance
- Inductive+ configurations achieve higher average Base Micro-F1 than non-inductive configurations across nearly all models in both TIMEX and EVENTX.Figure 12 compares the two setups across models.Source: §A.1.1 · p. 13
- Figure 12's comparison uses average Base performance measured by Micro-F1 for inductive+ and non-inductive setups.The passage reports no specific score values.Source: §A.1.1 · p. 13
- The authors suggest that incorporating inductive reasoning through in-context examples improves extraction performance on the base dataset.Source: §A.1.1 · p. 13
A.1.2 Correlation Across Generalization Dimensions
Configuration rankings transfer strongly among several event-extraction dimensions under inductive+ strategies, but compositional and T2C alignment is weaker. Non-inductive strategies show lower, more variable correlations, especially for T2C.
A.1.2 Correlation Across Generalization Dimensions
- Under inductive+, Base, Adversarial, Vocabulary, and Length rankings correlate strongly, while Compositional and T2C correlations are notably weaker.For non-inductive strategies, Base, Adversarial, and Vocabulary remain moderately correlated, but overall correlations are lower and more variable.Source: §A.1.2 · p. 13
- T2C correlations with most other dimensions are near-zero or negative under non-inductive strategies, but much higher under inductive+.Source: §A.1.2 · p. 13
A.1.3 Effect of Reasoning Strategy
Reasoning strategies usually perform similarly: inductive variants bring small, consistent gains for time extraction but less consistent benefits for event extraction, with greater variability in compositional and T2C settings.
A.1.3 Effect of Reasoning Strategy
- Most strategy pairs perform similarly; inductive methods bring small, consistent gains for time extraction but less consistently benefit event extraction, while compositional and T2C results vary.Combining all three reasoning types does not consistently improve time-extraction performance.Source: §A.1.3 · p. 13
A.2 Model Training Details
- RoBERTa, T5, and GPT-2 are fine-tuned independently for timex and event token classification, using BIO labels, full parameter updates, AdamW, and entity-level F1 model selection.Source: §A.2 · pp. 13–14
A.3 Prompting Templates for Reasoning Strategies
- The same overall prompt structure is used across both tasks and all generalization settings, so evaluation conditions vary without changing prompt design.Source: §A.3 · p. 15
- Inductive prompts use annotated examples, deductive prompts apply explicit definitions, abductive prompts generate and explain candidate spans, and combined prompts integrate examples, hypotheses, and span selection.Source: §A.3.1 · p. 15, §A.3.2 · p. 15, §A.3.3 · p. 15
A.4 Datasets
- Sentence-level TIMEX and EVENTX sets contain 22.7K and 23.2K training instances, respectively, and 400 test instances each; adversarial and vocabulary-shift sets derive from these test sets.Source: §A.4 · p. 16
A.4.1 Label and Vocabulary Shift in T2C
- Structural evaluation uses 80 long TimeBank documents, compositional evaluation uses 400 joint-expression instances, and clinical T2C data reflects domain-specific narratives.Source: §A.4 · pp. 16–17, §A.4.1 · p. 17
A.4.2 Time Expression extraction
- Time and event extraction are evaluated in six settings: base, adversarial, domain, domain-vocabulary, compositional, and structural.The settings cover perturbations, clinical and vocabulary shifts, joint time-and-event extraction, and longer contextual inputs.Source: §A.4.2 · p. 17, §A.4.3 · p. 18
A.5 Detailed Results
- Detailed tables cover both tasks, dense and MoE models, compositional extraction, model scale, and fine-tuned comparisons, exposing consistency trends and setting-specific weaknesses.Source: §A.5 · p. 18