PaperlayerView exact revision on arXiv

Source-linked AI summary

The LALA Benchmark: Evaluating Literary and Linguistic Analysis in LLMs

Michael Foster, Barbara McGillivray, Gard B. Jenset, Jakub Mácha, Jason Harding, Marcus Perlman, Devyani Sharma, Patrick Skeels, Jana Willer-Gold, Jeannette Littlemore

arXiv:2610.34080v1cs.CL

TL;DR

Evidence about how models apply linguistic competencies, and how these competencies vary with post-training, remains limited. LALA evaluates seventeen models using expert-written questions and human and automated assessment. Post-trained models outperform base models, but results remain uneven across domains and do not show a general linguistic intelligence.

  • Problem

    The benchmark targets how models apply linguistic competencies and how those competencies differ between base and post-trained models.

    Source: §1 · p. 2

  • Method

    LALA evaluates 17 models on expert-written questions using ideal answers and human and automated assessments.

    Source: §1 · p. 1

  • Results

    Base models showed little assessed competence while post-trained relatives performed markedly better; parameter count had no significant association with partial-credit responses, and competencies remained uneven.

    Source: §9 · p. 28

  • Takeaways & Limitations

    The findings help identify linguistic competencies that post-training has not yet injected into models.

    Source: §1 · p. 2

    Uneven coverage of contributors and topics limits how broadly the results apply to language use.

    Source: §7 · p. 27

Abstract

from arXiv · show

We introduce the Literature and Linguistics Analysis Benchmark (LALA), comprising 236 open-ended questions written by experts in linguistics and literary studies, together with expert-provided ideal answers. By comparing models within families and at similar training scales using human and LLM judges, LALA identifies differences in linguistic competence associated with post-training, offering a framework for evaluating future models and guiding targeted improvements. We administered questions to 17 language models and evaluated their responses using both human expert assessment and two LLM evaluators, Sol and Qwen. Base models displayed little of the linguistic competence assessed by the benchmark, whereas their post-trained versions performed markedly better, suggesting that post-training plays a central role in making linguistic knowledge accessible and usable. Parameter count, by contrast, showed no statistically significant association with partial-credit performance, and competencies remained unevenly distributed across linguistic domains. We interpret this pattern as evidence that post-training adds linguistic competencies piecemeal, without producing a general linguistic intelligence capable of extending them beyond the learned policies that support them. Models performed best on logical relations between statements and selected discourse-level tasks, while even the strongest post-trained models continued to struggle with ambiguity, literary style, reference, and misleading assumptions. Human and Sol evaluations produced nearly identical model rankings in both the original and ReliableEval runs, supporting the replicability of the comparative results.

1 Introduction

LALA evaluates whether language models can analyze linguistic and literary material at a level resembling expert analysts. Across seventeen models, post-trained models substantially outperformed base models, but their competencies remained uneven.

1 Introduction

  • LALA comprises 236 expert-written questions with author-supplied ideal answers, assessing linguistic analysis, textual interpretation, and logical inference and entailment.The paper describes benchmark design and a quantitative evaluation of seventeen models as its two main contributions.Source: §1 · p. 1
  • The benchmark aims to test whether models apply subtle semantic, syntactic, and pragmatic distinctions and how competency differs between base and post-trained models.The evaluation compares models within families and at different parameter counts, using human and automated judges.Source: §1 · p. 2
  • Post-trained models substantially outperformed base models, but gains did not reliably generalize across linguistic tasks.The authors interpret this pattern as piecemeal competency rather than general linguistic intelligence.Source: §1 · p. 2
  • Models showed relative strengths in logical inference and conversational structure but continued to struggle with literary framing, stylistics, reference, and misleading assumptions.Human and Sol evaluations produced nearly identical rankings and identified several of the same strengths and weaknesses.Source: §1 · p. 2

2 Related work

LALA joins work on linguistic, pragmatic, and literary benchmarks while emphasizing open-ended analysis of meaning and form. The paper also frames human–automated judge comparisons as evidence about evaluation consistency, not direct same-response agreement.

2 Related work

  • Unlike minimal-pair benchmarks such as BLiMP, LALA asks models to explain the linguistic distinctions underlying their judgments.The paper contrasts explanation-based assessment with sensitivity tests that do not require explanations.Source: §2 · p. 2
  • LALA’s scope overlaps with benchmarks for rule induction, linguistic knowledge, literary interpretation, pragmatic explanation, and resisting misleading premises.It distinguishes its focus on complex, opaque prompts and linguistic and literary analysis from related approaches.Source: §2 · pp. 2–3
  • Human–automated judge comparisons assess consistency across evaluation procedures, but cannot establish agreement on identical responses.The judges assessed separate response sets; direct agreement would require both to score shared answers under the same rubric.Source: §2 · p. 4

3 Benchmark construction

LALA uses expert-written questions and ideal answers to assess linguistic, logical, and textual interpretation. Its prompts require models to explain context-sensitive meaning, preserve ambiguity, and analyze stylistic or logical complications.

3.1 Questions and reference answers

  • Seven linguists, two philosophers, and two literary scholars contributed 236 English questions, each with an author-supplied ideal answer and keywords for the tested skills.The ideal answers served as the evaluation gold standard.Source: §3.1 · p. 4

3.2 Scope of competencies assessed

  • Questions test contextual meaning, pragmatic and social interpretation, genre, rhetorical form, voice, and whether models recognize ambiguity.When wording permits multiple readings, a complete answer must acknowledge the ambiguity.Source: §3.2 · p. 4, §3.3 · p. 5
  • The benchmark groups competencies into Linguistics, Logic, and Textuality, with 298 keywords organized into 27 categories that overlap and are unevenly represented.Category scores therefore reflect whole-question performance, including other skills needed for an answer.Source: §3.2 · pp. 4–5

3.3 What the questions require

  • Prompts include ambiguous expressions, stylistic transformations, and arguments with misleading tense shifts, requiring explanations of the relevant readings or reasoning.For “Drop the lore man,” full credit requires both opposed interpretations; another prompt asks how chatbot style changes Hamlet’s soliloquy.Source: §3.3 · p. 5

4 Evaluation and assessment

The evaluation compares base and post-trained models alongside a broader panel, using original and reworded prompts. Human and automated judges apply a shared rubric, with scores and comparisons calculated across questions and models.

4.1 Model selection

  • The panel included two open-weight base models, twelve open-weight post-trained models, and three proprietary models, with open models spanning roughly 3 billion to 671 billion parameters.Within-family comparisons included 14-billion-parameter Ministral models and DeepSeek models at a larger scale.Source: §4.1 · pp. 6–7

4.2 The original run and the ReliableEval run

  • In the original run, models answered questions individually without gold answers or rubrics; ReliableEval used reworded versions and two automated judges.The ReliableEval run covered 222 questions with up to ten versions each, yielding 2,131 prompts per model; 14 questions lacked acceptable rewordings.Source: §4.2 · p. 7

4.3 Assessment rubric

  • Six expert assessors scored responses against ideal answers for completeness, accuracy, and relevance, using a 0–2 scale and additional keyword-level scores for overall scores of 1.Qwen and Sol also applied the rubric as automated judges; neither was fine-tuned for the task.Source: §4.3 · p. 8

4.4 Scoring and comparisons

  • Normalized rubric attainment is the mean score divided by two and expressed as a percentage; category scores average whole-answer scores for questions tagged in each category.Question difficulty was grouped into four equal sets using mean human-assessed scores across the 17 models.Source: §4.4 · p. 8
  • Bootstrap intervals measured question-selection variation in selected family comparisons and mean-score uncertainty for Figure 2.Both used 5,000 resamples of questions.Source: §4.4 · pp. 8–9
  • Judge comparisons used correlations and rank statistics on shared questions, with Monte Carlo permutation tests; model comparisons remain conditioned on training and evaluation differences.Human–automated comparisons paired original human scores with automated scores on ReliableEval versions of the same questions.Source: §4.4 · p. 9

5 Results

The results characterize model performance across training groups and linguistic categories, including differences between post-trained and base models and uneven strengths across domains. They also examine parameter count, model-ranking consistency, and the stability of results when missing or refused answers are excluded.

5.1 Overall performance

  • Claude Fable 5 ranked first at 72.3% under humans and 62.0% under Sol, while Gemini and GPT-5.6 Sol switched second and third across judges.Gemma 4 31B ranked fourth under both assessments, ahead of the much larger DeepSeek and Qwen models.Source: §5.1 · p. 9, Figure 2 · p. 11, Figure 3 · p. 12

5.2 Base models and post-trained models

  • Post-trained models substantially outscored their same-size base versions, with Mistral gaps of 32.4 points under humans and 20.4 under Sol.DeepSeek V3’s gains over V3 Base were 37.8 points under humans and 26.6 under Sol; both base models returned answers but rarely supplied the required linguistic analysis.Source: §5.2 · pp. 12–13, Table 1 · p. 13

5.3 Linguistic strengths and weaknesses

  • Logic led broad categories at 45.3% under humans and 38.7% under Sol, while models struggled more with literary passages and particular expressions.Propositional Logic and Discourse Analysis were relative strengths; Literary Analysis, Lexical Semantics, and AI Evaluation scored lower, and the chatbot-style Hamlet question received no full-credit answers.Source: §5.3 · pp. 14, 16, Table 3 · p. 15

5.4 Agreement between human and GPT-5.6 Sol assessment

  • Human and Sol model rankings were nearly identical: Pearson’s r = 0.979, Spearman’s ρ = 0.988, and Kendall’s W = 0.994 across 17 models.They agreed on which model scored higher in 132 of 136 pairings; Sol generally assigned lower scores, but the judges assessed separate answer sets.Source: Table 5 · p. 18, §5.4 · p. 18

5.5 Qwen3 14B as a second automated judge

  • Qwen’s mean score was 41.5% versus the human mean of 39.7%, but its model rankings aligned less closely with humans than Sol’s did.Qwen and Sol gave identical scores to 66.5% of shared responses; Qwen sometimes credited an invented interpretation that Sol rejected for missing the passage’s key meaning.Source: §5.5 · pp. 18–20

5.6 Missing outputs and refusals; 5.7 Human scores after excluding missing responses

  • Generation failures and refusals affected some model scores, but removing incomplete cases or refusal-triggering questions left Fable, Gemini, and Sol in the top three.The ReliableEval run had 860 failures among 36,227 responses, including 757 from Ministral Reasoning and 47 Fable safety blocks; excluding missing responses changed no model score by more than 1.3 points.Source: §5.6 · pp. 20–21, §5.7 · p. 21

5.8 Parameter count and answer quality

  • Across all models, parameter count was not significantly associated with partial-credit counts; among 14 open-weight models, the rank correlation was ρ = 0.51 (p = 0.0644).The broader size pattern depended on including proprietary models and their estimated sizes.Source: §5.8 · pp. 21–22, Figure 8 · p. 23

5.9 Keyword scores across model groups and broad linguistic categories

  • Base models received zero on 97.4% of pooled keyword judgments, compared with 46.7% for open post-trained models and 23.5% for proprietary models.Across groups, zero was most frequent in Linguistics and Textuality, while full credit was most frequent in Logic; proprietary models’ full-credit share declined from 73.3% in Logic to 52.5% in Textuality.Source: §5.9 · p. 23

5.9 Keyword scores across model groups and broad categories

  • The correspondence analysis’s first dimension accounted for 95.0% of inertia, separating zero from partial and full credit; base-model profiles clustered with zero scores.Its second dimension accounted for 5.0% of inertia. Proprietary models had a high proportion of full-credit scores, especially in Logic.Source: §5.9 · p. 24, Figure 10 · p. 25
  • Open post-trained models scored better in Logic than Textuality, while their Linguistics score mix was close to the pooled distribution.In Textuality, zero scores were more common; the open post-trained group’s Logic strength coexisted with substantially weaker Textuality performance.Source: §5.9 · pp. 24–25, Figure 9 · p. 24
  • χ²(16) = 2,999.68 and Cohen's w = 0.444 show substantial score-distribution differences across model groups and categories, revealing diagnostic patterns a single score would obscure.The benchmark particularly exposes open post-trained models’ relative strength in Logic alongside much weaker Textuality performance.Source: §5.9 · pp. 24–25, Figure 9 · p. 24

6 Discussion

Post-training substantially improves linguistic analysis, but competencies remain uneven and do not generalize beyond learned policies. Models handle some logical and discourse-level tasks better than ambiguity, reference, literary style, and misleading premises; human–Sol rankings remain highly replicable, though judge choice matters.

6.1 Post-training and linguistic competency

  • Post-trained models were substantially more capable in both families, while competencies appeared to accrue piecemeal without generalizing beyond learned behavioral policies.The authors interpret this pattern as evidence against a general linguistic intelligence emerging through post-training.Source: §6.1 · p. 25

6.2 Linguistic strengths and persistent errors

  • Recurring errors included overlooking tense or wording changes and choosing one referent where several were possible; gold answers gave assessors concrete criteria for detecting omissions.LALA’s intentionally distorted prompts test whether models recover intended meaning while explaining remaining ambiguity.Source: §6.2 · p. 26
  • Models handled explicit logical relationships and implicit conversational structure better than literary analysis, reference, and questions with misleading premises.In an ambiguous exchange, both DeepSeek R1 and Gemini settled on an intention the text did not establish, although either reading remained possible.Source: §6.2 · p. 26

6.3 Objectivity and replicability

  • Human–Sol model rankings were nearly identical (W = 0.994), and Sol reproduced nearly the same ordering after questions were reworded and the assessor changed.This supports replicability of model comparisons, not question-by-question agreement.Source: §6.3 · p. 26
  • Qwen broadly agreed with human and Sol rankings, but moved some models substantially; Sol agreed most closely with humans.The authors therefore note that automated-judge choice matters.Source: §6.3 · p. 26

7 Limitations

The benchmark’s conclusions are constrained by uneven question coverage, judgment choices, and differences between the original and ReliableEval runs. These limits make some score interpretations and training comparisons less definitive.

7 Limitations

  • Uneven topic and contributor coverage limits how broadly results apply, while literary and historical questions may also depend on background knowledge.For skills represented by only a few questions, broader testing is needed before drawing general conclusions about model competency.Source: §7 · p. 27
  • Scores depend on contributor-supplied ideal answers, and roughly 50 were examples rather than the only correct responses.Assessors judged those answers against the question; shared gold answers could affect both human and LLM assessments, and one human score per answer prevented measuring assessor agreement.Source: §7 · p. 27
  • Only two base models were included, and their shorter output limits and differing instruction, reasoning, and temperature settings weaken attribution of score differences to training alone.These differences may also have affected the answers.Source: §7 · p. 27
  • Some accepted question rewrites changed the task while retaining the original ideal answer, and judging also varied between Sol and Qwen.The authors therefore caution that differences between the original and ReliableEval runs cannot be attributed to judging alone.Source: §7 · p. 27
  • Human assessors and Sol could see model identities, which may have influenced judgments; the authors recommend hiding identities and adding independent human ratings.Sol also assessed its own responses, raising the possibility of self-preference.Source: §7 · p. 27

8 Future work

The paper proposes using LALA’s model-grouping and mixed-judge approach to study post-training effects and guide future evaluations. It also outlines expansions to question sets and judge comparisons.

8 Future work

  • Grouping models by training scale and family, with human and LLM judges, is proposed as a way to assess future models and help make post-training more efficient.The approach aims to map where different training approaches affect observed linguistic competencies.Source: §8 · p. 27
  • Future question sets could cover each category and subcategory and extend evaluation to more recent models.Questions should give sufficient context and precise instructions, with ideal responses specifying acceptable answers and competing readings when ambiguity is the task.Source: §8 · p. 28
  • Human-authored templates could seed larger LLM-generated sets testing specific features, with controlled variations across passages and contexts.Each generated item would need review to preserve its intended task and avoid unintended ambiguity.Source: §8 · p. 28
  • Future evaluations could compare more judge models on the same responses and rubric, report ideal-response coverage, accuracy, and relevance separately, and compare with human assessments.The authors also propose removing model identities from all judge materials, including record identifiers, and comparing those assessments with the present ones.Source: §8 · p. 28

9 Conclusion

Across 17 models, the conclusion highlights stronger performance by post-trained models than base models, alongside uneven competencies and persistent task-specific difficulties. It also describes the appendix’s question-level score table and its averaging conventions.

9 Conclusion

  • Across 17 models, post-trained models performed substantially better than base models, while parameter count showed no statistically significant association with partial-credit responses.The authors interpret the uneven competencies as piecemeal additions from post-training, not a general linguistic intelligence extending beyond learned policies.Source: §9 · p. 28
  • Human judges and Sol produced nearly identical rankings across the original and ReliableEval runs, while models did best on logical relations and some discourse tasks.Even the strongest post-trained models struggled with ambiguity, literary style, reference, and misleading assumptions.Source: §9 · p. 28

Appendix A Scores for every question

  • Appendix A lists all 236 questions by ID and reports mean scores across 17 models as percentages of maximum rubric scores.Human scores are from the original run; Sol and Qwen scores are from ReliableEval, with N/A marking 14 questions lacking ReliableEval results.Source: Appendix A · p. 31

Appendix B Questions with missing outputs

The Fable questions had no returned text, and error records show empty responses after retries.

Fable (6 questions)

  • For the six Fable questions, responses returned no text, with error records showing empty responses after retries.Source: Fable (6 questions) · p. 37

GPT-OSS (7 questions)

The reported failures include empty or unreadable responses, plus requests that failed because an expected answer field was missing. For Ministral Base, the records do not explain why final answers were blank.

GPT-OSS (7 questions)

Ministral Base (22 questions)

Fable (3 questions)

Failures differed across models: safety filtering blocked Fable's answers, while other records describe missing or unreadable answers. GPT-OSS, by contrast, had at least one usable answer for every question.

Fable (3 questions)

  • Safety guardrails blocked every version of the questions, accounting for all 47 of Fable's failed responses.Source: Fable (3 questions) · p. 38
  • Gemma 26B returned no answer to any version of one question, and the records do not explain the cause.Source: Fable (3 questions) · p. 38

Ministral Reasoning (23 questions)

GPT-OSS

  • GPT-OSS returned at least one usable answer for every question.Source: GPT-OSS · p. 38

Appendix C Question categories and subcategories

Appendix C catalogs the benchmark’s 236 questions by category and subcategory, listing question IDs and their assigned tags across linguistics, logic, and textuality. Categories may overlap, and the tags span areas such as discourse, social pragmatics, formal logic, and phonology.

Appendix C Question categories and subcategories

  • Counts cover 236 distinct questions and overlap because questions may have multiple category or subcategory assignments.Source: Appendix C · p. 39

C.3 Categories and subcategories for each question

  • The per-question listings assign questions across Linguistics, Logic, and Textuality, sometimes combining categories from multiple domains.For example, Q-000018 combines Linguistics and Logic, while Q-000031 spans Linguistics, Logic, and Textuality.Source: §C.3 · p. 40
  • Several listed questions combine discourse analysis, social pragmatics, and sociolinguistics in their category assignments.Some entries also include linguistic pragmatics or dynamic semantics.Source: §C.3 · p. 45
  • Logic-related entries include modal, propositional, predicate, syllogistic, and epistemic logic, sometimes alongside formal semantics.Some questions also carry textuality tags such as AI Evaluation.Source: §C.3 · pp. 46–47
  • In the later listed questions, phonology is repeatedly paired with Philology, while other entries combine morphosyntax with Philology or literary analysis.The inventory also includes questions tagged for General Linguistics and Phonology.Source: §C.3 · pp. 47–49

Related papers

Loading 2610.34080v1…