Source-linked AI summary
Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models
Yang Liu, Hongming Li, Melissa Xiaohui Qin, Qiankun Liu, Chao Huang
TL;DR
Language-model benchmarks largely under-examine fine-grained phrasal semantics and often conflate distinct semantic operations. SemanticQA unifies phrase resources into operation-aligned classification, extraction, interpretation, and sequential evaluations, finding substantial challenges and variation in semantic phrase understanding. The paper concludes that phrase-level competence requires multidimensional assessment rather than a single task or metric.
Problem
Existing evaluations overlook fine-grained semantic reasoning over phrasal semantics and often isolate or conflate operations, making stable phrase-level understanding difficult to assess.
Method
SemanticQA consolidates semantic-phrase resources into an operation-aligned benchmark spanning classification, extraction, interpretation, and structurally constrained task compositions.
Results
Language models show substantial challenges and performance variation in semantic phrase processing, despite strong performance in general benchmarks.
Takeaways & Limitations
Phrase-level semantic competence cannot be captured by a single task or metric because extraction, categorization, and interpretation probe distinct aspects and constraints.
Takeaways & Limitations
SemanticQA is restricted to English, omits several long-tail phrase types, and calls for more complex sequential compositions and broader coverage.
Abstract
from arXiv · showhide
We present SemanticQA, an evaluation suite designed to assess language models (LMs) in semantic phrase processing tasks. The benchmark consolidates existing multiword expression (MwE) resources and reorganizes them into a unified testbed. It covers both general lexical phenomena, such as lexical collocations, and three fine-grained categories: idiomatic expressions, noun compounds, and verbal constructions. Through SemanticQA, we assess LMs of diverse architectures and scales in extraction, classification, and interpretation tasks, as well as sequential task compositions. We reveal substantial performance variation, particularly on tasks requiring semantic reasoning, highlighting differences in reasoning efficacy and semantic understanding of LMs, providing insights for pushing LMs with stronger comprehension on non-trivial semantic phrases. The evaluation harness and data of SemanticQA are available at https://github.com/jacklanda/SemanticQA.
1 Introduction
SemanticQA targets an under-explored form of reasoning over sub-sentential phrasal semantics, where existing evaluations often conflate semantic operations. It aligns classification, extraction, and interpretation tasks to test whether phrasal understanding transfers across operations and remains consistent in sequential settings.
- Semantic phrases have meanings or usages that may not be fully derived from their components and vary in compositionality, idiomaticity, and fixedness.
- Existing LM benchmarks emphasize mathematical, code, and logical reasoning while largely overlooking fine-grained semantic reasoning over sub-sentential units.
- SEMANTICQA introduces an operation-aligned benchmark that evaluates phrasal semantics across classification, extraction, and interpretation operations.
- The benchmark assesses semantic reasoning through compatible behavioral patterns across tasks, rather than performance on a single isolated task.
- Fixed prompt templates hold prompt structure constant while semantic operations vary, supporting fair comparisons across language models.
- Strong language models often fail to maintain semantic consistency across dependent operations, exposing phrase-level limitations hidden by single-task evaluations.
2 Related Work
Prior work studies semantic reasoning and diverse phrase resources, but evaluations remain fragmented by phrase type, task format, and phenomenon. SemanticQA responds by unifying these resources around the atomic operations underlying phrase comprehension.
- Existing reasoning benchmarks focus on explicit symbolic or compositional procedures and overlook fine-grained semantic operations in context.
- Semantic phrase research covers idiom identification, contextual paraphrasing, noun compound interpretation, and resources for idioms, collocations, and verbal constructions.
- Existing evaluations typically isolate phrase types, task formats, and semantic phenomena without modeling the atomic operations underlying phrase comprehension.
- This fragmentation limits cross-task, cross-phenomenon, and generalization analyses.
- SemanticQA is motivated as a unified, operation-aligned benchmark for phrase-level semantic processing in language models.
3 SEMANTICQA
SemanticQA consolidates varied semantic-phrase resources into controlled tasks spanning phrase types and semantic operations. Its formulation maps prompts and contextual phrases to operation-specific outputs, evaluated with task-appropriate metrics.
- Phrase coverage: The benchmark covers lexical collocations, idiomatic expressions, noun compounds, and verbal constructions with differing compositional and contextual properties.
- Benchmark construction: SemanticQA builds on prior resources while preserving source variation and emphasizes within-task trends and operation-induced changes rather than absolute phrase-type comparisons.
- Task organization: Tasks are organized by phrase types and atomic operations, allowing related tasks to share underlying phrase meaning while differing in output structure and constraints.
- Task organization: Idiomatic-expression tasks include detection, exact span extraction, and contextual paraphrase interpretation, adapted from existing annotated resources.
- Task organization: Lexical-collocation tasks include taxonomy-based categorization, base-and-collocate extraction, and context-conditioned interpretation, with balanced semantic-relation categories.
- Task organization: Verbal multiword-expression extraction identifies verbal constructions in context, covering verb–particle, light-verb, and verbal-idiom types.
- Task formulation: Given a prompt template P and contextualized phrase S, the language model generates an operation-specific output O under task constraints.
- Evaluation: Classification uses accuracy, extraction uses sequence-level exact-match accuracy, and interpretation primarily uses METEOR with ROUGE-L and BERTScore as complementary metrics.
4 Results
SEMANTICQA evaluates diverse language models across phrase-processing operations, revealing substantial variation across tasks and phrase types. Results also show that prompting and sequential composition expose distinct strengths, bottlenecks, and failure modes.
- Overall performance patterns: Models show substantial performance variation across operations and phrase types, with no model performing uniformly well across all setups.Different tasks impose distinct structural and semantic constraints, indicating operation-specific strengths and weaknesses.
- In-context learning: Three- or five-shot prompting consistently benefits interpretation tasks, although gains in MTR can reflect exemplar-guided reconstruction rather than strict semantic grounding.Improvements appear in both R-L and B-S, while embedding similarity may remain high despite limited lexical overlap.
- In-context learning: ICL has hybrid effects on classification and is highly task-dependent for extraction.Weaker classifiers may improve while others plateau or regress, and extraction gains depend on whether demonstrations clearly illustrate span structure.
- Semantic category scaling with ICL: 81.7% to 35.4%: DeepSeek-R1 accuracy decreases as lexical-relation category size grows from one to sixteen.Supervised baselines remain relatively stable, whereas frontier LMs show sharper declines at larger category scales.
- Sequential task compositions: Sequential interpretation performs better when phrases are correctly extracted than end-to-end, making extraction a primary downstream bottleneck.Conditional interpretation scores exceed overall scores, while few-shot prompting yields limited gains in these sequential settings.
- Prompting strategies: 51.6% to 64.1%: ORACLE SCHEMA improves DeepSeek-R1's VMWE extraction accuracy by supplying the target type and definition.The strategy consistently improves performance across the evaluated models.
5 Discussion and Takeaways
Discussion of SEMANTICQA argues that phrase-level semantic competence requires multidimensional, structurally grounded evaluation. Interpretation metrics and sequential workflows can otherwise obscure brittleness and error propagation.
- Phrasal semantics requires multi-dimensional evaluations: No single task or metric captures phrase-level semantic competence because interpretation, extraction, and categorization probe distinct aspects of processing.Extraction and categorization impose stronger structural or relational constraints than open-ended interpretation.
- Metric sensitivity shapes apparent model strengths: High interpretation scores can reflect paraphrasing and instruction following rather than conclusive semantic correctness.Flexible similarity-based metrics may reward fluent reconstruction, while strict span-based evaluations expose structural brittleness.
- Workflow robustness remains limited: Sequential evaluations reveal limited workflow robustness because upstream extraction or categorization errors propagate and are not reliably compensated downstream.This error accumulation is largely obscured by atomic benchmarks.
6 Conclusion
The paper introduces SEMANTICQA to evaluate language models' semantic phrase processing across diverse tasks and models. Its results show persistent challenges in phrase understanding and motivate more robust, structurally grounded processing.
- Conclusion: SEMANTICQA evaluates semantic phrase processing across a wide range of language models and includes targeted human comparisons across ten tasks.The benchmark is presented as a unified evaluation of phrase-level semantic competence.
- Conclusion: Despite strong general-benchmark performance, language models continue to face substantial challenges in SEMANTICQA.The analyses characterize behavior across task types and identify persistent limitations in semantic phrase understanding.
- Conclusion: The findings highlight directions for more robust and structurally grounded semantic processing.This conclusion follows the paper's analyses of model behavior across task types.
Limitations
The benchmark’s limitations concern coverage, task complexity, and model and language scope. The authors propose broader, more complex, multilingual, and continually updated resources as future directions.
- Although SEMANTICQA covers four common phrase phenomena, it is restricted to English and omits long-tail types such as multiword named entities and complex function words.
- Future benchmarks should include more complex sequential task compositions and additional evaluation paradigms, including semantic retrieval.
- Rapid progress in language-model architectures requires continual benchmark updates and broader model coverage.
- The authors encourage extending SEMANTICQA toward more comprehensive and multilingual resources.
Ethical Considerations
The research uses licensed public data without private or personally identifiable information and frames the benchmark as a research and diagnostic resource. It also documents limitations, considers misuse risks, and reports fair compensation for human annotators.
- The study uses publicly available datasets under their original licenses and excludes private, sensitive, and personally identifiable information.
- The benchmark is intended solely for research and diagnostic purposes, with known limitations documented to avoid overgeneralization.
- The authors state that computational resources were used responsibly and potential data-misuse and model-evaluation risks were considered.
- Human annotators were recruited under fair labor practices and received appropriate compensation.
B Additional Details of Datasets
The dataset suite combines existing resources for idioms and lexical collocations, with task-specific filtering, deduplication, sampling, and split reuse. It includes examples for interpretation, extraction, and categorization tasks.
- Idiomatic expressions: The idiom interpretation dataset collates 273 examples after adding “None of the above” where needed and deduplicating unique idiom-choice pairs.
- Idiomatic expressions: The idiom extraction test set contains 447 examples, each with a unique existing idiomatic expression after filtering and deduplication.
- Idiomatic expressions: The idiom interpretation resources combine 916 instances from PIE and another benchmark after deduplication by occurred idiom items.
- Lexical collocations: Lexical collocation categorization samples 50 test examples per semantic category from LEXFUNC while retaining the original training and validation sets.
- Lexical collocations: Lexical collocation extraction uses 50 randomly sampled English instances per semantic category, reusing the original training and validation sets and omitting identification.
B.6 Lexical Collocation Interpretation
The benchmark extends dataset construction across noun compounds and verbal multiword expressions while standardizing prompts across tasks. Noun-compound resources use filtered or sampled examples, generated context, and manual quality checks; VMWE extraction uses PARSEME data.
- Noun compounds: The noun compound compositionality dataset contains 237 examples after filtering out compounds without reference context and splitting the remainder into train, validation, and test sets.
- Noun compounds: The noun compound extraction test set contains 720 randomly sampled examples, while the original training and validation data are retained for compositional learning.
- Noun compounds: For noun compound interpretation, ChatGPT-generated reference sentences were manually inspected, achieving acc > 98%.
- Verbal constructions: Verbal multiword-expression extraction uses the English portion of PARSEME v1.3, described as the largest existing annotated VMWE corpus.
- Prompting: A unified prompt template is manually created for all tasks and adapted with task-specific filling arguments.
E Additional Experiment Details
The experiments evaluate diverse language models across SemanticQA tasks using controlled prompting, fine-tuning, and comparative baselines. Results are reported with task-specific metrics and supporting experimental references, including confusion-matrix analysis.
- Evaluation setup: API-accessed models are evaluated in zero-shot, three-shot, and five-shot settings with deterministic decoding at τ = 0 and top-p p = 1.0.Inference is accelerated with vLLM.
- Evaluation setup: Non-API models use BERT-base/large baselines for sequence classification and BIO-based sequence labeling baselines for extraction.Fine-tuning is run on an NVIDIA A100-40GB using Google Colab.
- Results presentation: Figure 22 presents a confusion matrix for GPT-5 in the five-shot setting across eight semantic relations, with predictions on the x-axis and gold standards on the y-axis.The figure describes the best-performing model with in-context learning for this classification analysis.
- Results presentation: Figure 21 plots GPT-3.5-Turbo, GPT-4, OpenAI o3, and GPT-5 across twelve tasks using task-specific metrics.Absolute values should not be compared across different tasks.
- Dataset construction: Collocation interpretation pairs contexts and collocations with base and collocate words, five narratives, and a rationale, requiring consistent meanings across narratives.The objective is to interpret each collocation in five distinct narratives according to context.