PaperlayerView exact revision on arXiv

Source-linked AI summary

From annotation to reasoning: Culture in language models

Daniel Hershcovich, Alexander Conroy, Jens Bjerring-Hansen

arXiv:2609.30897v1cs.CL

TL;DR

Cultural benchmarks often test knowledge, survey agreement, or predefined meanings, leaving open whether models can interpret references in context, support readings, and revise them after criticism. The paper proposes literary interpretation as a setting for evidence-centered benchmarks, disagreement-preserving evaluation, and model-development experiments. It argues that scholars can disagree about readings while assessing the quality of their support, while noting limits around interpretive communities and cross-language comparisons.

Abstract

from arXiv · show

How should we evaluate language models when more than one interpretation can be right? Cultural benchmarks often test factual knowledge, agreement with survey responses, or recognition of a predefined meaning. These tasks leave open whether a model can explain how a cultural reference works in a particular text, support a reading with evidence, or revise it after criticism. This is a question of interpretive depth, complementary to the breadth of cultural coverage. We argue that literary interpretation offers a useful setting for studying these capabilities. We focus on cultural referencing and reuse: how texts invoke, repeat, and transform earlier expressions across historical and linguistic contexts. Our central claim is that literary scholars can disagree about an interpretation while recognizing the quality of its support. We propose linking evidence-centered benchmarks, evaluation that preserves scholarly disagreement, and model-development experiments on literary data, contextual resources, and scholarly feedback. Danish literature provides a concrete starting point, with implications for other languages and domains. The aim is to develop alternative evaluation strategies that go beyond conventional benchmark metrics and guide model development toward cultural robustness in AI systems.

What would count as cultural reasoning?

Cultural reasoning calls for evaluating interpretive depth alongside the breadth of cultural coverage. The paper proposes assessing how models explain contextual meaning, support readings, consider alternatives, and respond to criticism.

What would count as cultural reasoning?

  • An allusion draws on an earlier source and gains meaning through its use in a new context, so evaluation must assess both the connection and its significance.The questions of how to assess culturally grounded tasks, preserve ambiguity and disagreement, and adapt models without reinforcing stereotypes motivate CuRe, a collaboration between NLP and literary studies.Source: What would count as cultural reasoning? · p. 1
  • Cultural coverage and interpretive depth are complementary: breadth concerns represented languages, communities, and knowledge, while depth concerns the quality of interpretation within contexts.The passage distinguishes cultural knowledge, population-level alignment, and interpretation as separate evaluation questions.Source: What would count as cultural reasoning? · p. 1
  • Interpretive depth concerns explaining an expression’s significance in context, supporting a reading with evidence, considering alternatives, and responding to criticism.The authors evaluate the reasons a model offers and the support for them, since fluent explanations may diverge from the computation behind an answer.Source: What would count as cultural reasoning? · p. 1
  • Establishing its reliability as a basis for model evaluation requires empirical testing [11].Source: What would count as cultural reasoning? · p. 2

Culture through references and reuse

The paper treats literature as a setting for studying cultural meaning as texts reuse and transform expressions across contexts. It illustrates how a reference can sustain distinct, evidence-based readings.

Culture through references and reuse

  • Literature circulates expressions, revisits histories, and questions conventions; the same phrase can acquire different referentiality and meaning in different textual contexts.This makes literature a useful setting for culturally grounded reasoning.Source: Culture through references and reuse · p. 2
  • Cultural patterns are learned, shared, and contested, and population-level claims require different evidence from claims about an individual speaker or literary work.People participate in overlapping traditions that change through exchange, so national labels alone are insufficient descriptions [39].Source: Culture through references and reuse · p. 2
  • Cultural referencing spans direct quotations to subtle allusions across national and historical boundaries, while cultural reuse repeats and transforms situated expressions.Their recurrence can trace how cultural knowledge and expression develop as texts alter or contest earlier formulations.Source: Culture through references and reuse · p. 2
  • In Blixen’s “The Supper at Elsinore,” the “new Jerusalem” allusion supports readings connecting Madam Bæk’s journey to approaching death or to gentle comedy.Both readings must account for her acceptance of the city and the narrator’s role; their relative plausibility depends on the biblical imagery and narrative and historical context.Source: Culture through references and reuse · p. 2
  • A model could retrieve the biblical imagery and passages on Madam Bæk’s health and journey, then explain how they support its reading and where alternatives remain plausible.Scholars can probe the model’s response to further textual evidence or commentary; available Danish and Norwegian historical fiction and contemporary writing also support studying literary exchanges, with survival, digitization, and canon formation shaping the corpus scope.Source: Culture through references and reuse · p. 2

Benchmarks that require interpretation

The proposed benchmarks use open-ended, scholarship-grounded tasks to assess how models interpret references and revise accounts. They also test whether performance reflects contextual reasoning rather than shortcuts.

Benchmarks that require interpretation

  • Benchmark questions grounded in literary scholarship, such as explaining irony or changes from reuse, can elicit answers whose grounds are open to assessment.Task design should specify the capability, eliciting task, and evidence needed to justify an assessment; close-reading tasks have also been applied to literary reasoning [29].Source: Benchmarks that require interpretation · p. 2
  • Assignments can combine categorical annotations of allusions or ambiguous expressions with multiple human-written responses while leaving room to assess unanticipated model readings.Table 1 illustrates tasks for identifying a reference, explaining its contribution, and revising an account after criticism.Source: Benchmarks that require interpretation · p. 2
  • Less familiar material, documented source access, and held-out works can help test transfer beyond reproducing commentary about canonical works encountered during training.Comparisons of recurring expressions across speakers or works can preserve source texts and documented narrative and historical differences.Source: Benchmarks that require interpretation · p. 2
  • Comparing passage-only, full-text, and retrieval-assisted conditions can test what additional context enables and whether models overlook or misapply relevant material.Source: Benchmarks that require interpretation · p. 2
  • Reports should pair coverage across languages, periods, and genres with evidence-use and revision quality in each setting.This enables testing whether broader coverage also improves interpretive depth and where each requires further development.Source: Benchmarks that require interpretation · p. 2

Judging quality without requiring agreement

The proposed evaluation separates response accuracy, evidence-based support, and expert judgments of plausibility, without requiring scholars to agree on one reading. Its protocol preserves individual assessments and makes disagreements and their grounds inspectable.

Judging quality without requiring agreement

  • A frequent position may be poorly argued while an uncommon reading is carefully supported, so evaluation must assess the grounds for a position as well as its prevalence.Source: Judging quality without requiring… · p. 3
  • Evaluation separates identifiable errors, the quality of evidence and context used to support an interpretation, and experts’ graded plausibility judgments.Several accounts may receive substantial support, and evaluators can differ in the weight they give them.Source: Judging quality without requiring… · p. 3
  • An evaluation record can document the question, materials, reading, cited passages, assumptions, evidence connections, and what could change the account.Experts can then identify divergent judgments across response formats, though a rigid template risks rewarding compliance and verbosity.Source: Judging quality without requiring… · p. 3
  • Retaining individual expert judgments and using probabilistic soft labels for mutually exclusive alternatives can preserve variation without conflating disagreement with uncertainty.Coexisting interpretations require separate graded assessments, and reports should give both judgment distributions and reasons.Source: Judging quality without requiring… · p. 3
  • Professional judgments reflect interpretive communities whose conventions can exclude voices and naturalize assumptions [12].The proposed process documents specialist expertise and familiarity with the works, while treating consensus as provisional and criteria as open to challenge.Source: Judging quality without requiring… · p. 3
  • Factual accuracy and relevance constrain defensible interpretations, while translated questions or rating terms can change what they measure.Rubrics need local adaptation and evidence of comparability before scores are pooled; separate profiles can preserve differences between evaluative traditions.Source: Judging quality without requiring… · p. 3

What should models learn from literature?

The paper proposes experiments on literary training data, contextual resources, and scholarly feedback to study evidence use and interpretive alternatives. It also stresses testing whether models revise weak readings without abandoning well-supported ones.

What should models learn from literature?

  • In a Norwegian training-data study, adding fiction reduced aggregate benchmark performance while improving textual variation and readability, leaving its contribution to cultural interpretation unresolved [10].The authors argue that this illustrates why training-data choices should be evaluated against the capabilities of interest.Source: What should models learn from… · p. 3
  • Training-data comparisons should control data amounts and document sources while testing familiar expressions and new uses to distinguish memorization from transfer.The proposed comparisons include historical and contemporary fiction, general-language material, and scholarly commentary.Source: What should models learn from… · p. 3
  • Contextual resources can be retrieved or incorporated during training, but experiments should test whether retrieved material is relevant and improves analysis.The authors propose comparing the same tasks with and without contextual resources.Source: What should models learn from… · p. 3
  • Feedback experiments should compare levels of granularity and test revision on held-out works, weighing the potential value of localized feedback against its cost.The paper proposes feedback that explains the strengths and weaknesses of several defensible accounts.Source: What should models learn from… · p. 3
  • Models may imitate scholarly caution yet make unsupported claims, or become so compliant that they abandon well-grounded readings when challenged.Evaluation should test whether models can justify retaining supported claims as well as revising weak ones, including across periods or genres.Source: What should models learn from… · pp. 3–4
  • Cultural robustness means sustaining contextually grounded reasoning across variations in language, period, genre, and interpretive standpoint.The proposed experiments examine how data, context access, and learning signals affect evidence use while retaining alternatives, and whether adaptation reinforces stereotypes or suppresses supported alternatives.Source: What should models learn from… · p. 4

Expert scrutiny of cultural reasoning

The paper frames expert scrutiny as a way to probe cultural reasoning, examine disagreement, and assess whether criticism leads to substantive revision. Related examples in teaching and summarization show how experts can assess the evidence and assumptions behind model outputs.

Expert scrutiny of cultural reasoning

  • Expert readers can probe whether a model recognizes connections, locates sources, and reasons about cultural alternatives, then challenge claims and assumptions.The framework tests whether the model can sustain a culturally grounded argument when challenged.Source: Expert scrutiny of cultural reasoning · p. 4
  • In a teaching example, students compared sentiment annotations with model predictions and discussed disagreement as a way to examine narrative perspective and textual support.The passage connects this practice to asking which textual features support a model's output and how it accounts for them.Source: Expert scrutiny of cultural reasoning · p. 4
  • Automated assessment of summaries of 19th-century Scandinavian novels was less dependable for factuality and setting-related criteria.Expert evaluation can also examine which events and characters summaries foreground and the passages and narrative contexts supporting those choices.Source: Expert scrutiny of cultural reasoning · p. 4
  • A controlled writing study found that AI suggestions shifted Indian participants’ writing toward Western styles [1], prompting a separate question about whether models privilege familiar or dominant readings.Expert evaluation can examine a system's alternatives, invoked cultural assumptions, and overlooked interpretations.Source: Expert scrutiny of cultural reasoning · p. 4
  • Evaluation should track which interpretations a model represents, how it supports them, and whether scholarly criticism leads to substantive revision.The assessments can guide choices about training data, retrieval, and feedback.Source: Expert scrutiny of cultural reasoning · p. 4

A broader role for interpretive evaluation

The paper presents evidence-centered interpretive evaluation as relevant beyond literature, while stressing that each application needs its own rubric and validation. It concludes that literary scholarship and computational experiments can help assess whether systems substantiate cultural claims, preserve plausible alternatives, and respond to informed criticism.

A broader role for interpretive evaluation

  • Visual cultural interpretation and law also require examining reasons against particular objects, settings, sources, doctrines, or procedures [38].In a small study of European Court of Human Rights cases, more comprehensive model analysis did not improve outcome accuracy [25].Source: A broader role for interpretive… · p. 4
  • Applying interpretive evaluation in another domain requires meaningful tasks, accessible evidence and assumptions, criteria for support and plausibility, and tests of revision.Each application requires its own rubric and validation.Source: A broader role for interpretive… · p. 4
  • The paper argues that literary scholarship should shape tasks and learning objectives, while computational experiments test which evidence and feedback improve interpretation.Progress can be assessed by how well systems substantiate cultural claims, sustain plausible alternatives, and respond to informed criticism.Source: A broader role for interpretive… · p. 4
Loading 2609.30897v1…