PaperlayerView exact revision on arXiv

Source-linked AI summary

Scoring Both Directions: LLMs realize the MRS they cannot reliably parse

Soham Dan

arXiv:2609.30071v1cs.CL

TL;DR

Prompted language models are tested on both directions of the ERG’s MRS benchmark, addressing whether success at realization also extends to parsing. The models do well at generating text from MRS but poorly recover MRS from text: Opus reaches 76.34 BLEU in realization, while both models rarely produce exact parses. The results caution against treating generation scores alone as evidence that models know the representation.

  • Problem

    The same benchmark can test realization and parsing, but evidence about prompted models on both directions with the grammar as reference is limited.

    Source: §1 · p. 2

  • Method

    The paper evaluates Claude Sonnet 4.5 and Opus 5 with three exemplars on realization and parsing tasks using the reconstructed test split, and compares them with trained systems and ACE.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    Prompted models are strong at MRS-to-text but weak at text-to-MRS: Opus scores 76.34 BLEU in realization, while both models almost never exact-match the gold MRS.

    Source: §1 · p. 2, §5 · p. 4

  • Takeaways & Limitations

    A high generation score alone does not show that a model knows the formal semantic representation.

    Source: §5 · p. 4

    The evaluation tests two Anthropic models under one three-example prompting setup; stronger setups such as retrieval or constrained decoding might narrow the parsing gap.

    Source: Limitations · p. 4

Abstract

from arXiv · show

The English Resource Grammar (ERG) is a hand-written computational grammar of English. Given a sentence, its processor, ACE, produces a formal meaning representation called Minimal Recursion Semantics (MRS): a graph of the sentence's predicates and their arguments. The grammar is bidirectional and can also turn an MRS back into an English sentence. \citet{hajdik2019} used the ERG's treebank to build a benchmark for that generation task, MRS to text, and trained sequence-to-sequence models to solve it. The parsing task, text to MRS, can be tested on the same sentences. We reconstruct their 10K-sentence test split, and score two large language models, Claude Sonnet~4.5 and Claude Opus~5, in both directions against their trained systems and against ACE, with no task-specific training. Given an MRS and three examples, Opus writes the sentence at 76.3 BLEU, ten points above their system trained on 72k pairs (66.1 BLEU), and comparable to their system trained on a million extra pairs (77.2 BLEU). Sonnet scores 65.7 BLEU, and letting it choose among ACE's own candidate sentences lifts it to 69.6, while a pooled judge that keeps Opus's own sentence among the candidates adds 0.6 points (77.0 BLEU). In the parsing direction, however, the models fall far behind ACE: asked for the MRS of the same sentences, they reach 57.2 (Sonnet) and 65.5 (Opus) F$_1$ on the graph's predicates and arguments against 91.0 for ACE, and exact-match the gold on about 1\% of sentences. We characterize the failure modes for the parsing tasks, and conclude that a generation score alone does not show that models understand formal semantic representations.

1 Introduction

The ERG provides a bidirectional setting for testing whether prompted language models can realize and analyze the same formal meanings. On realization they perform strongly, but their reverse parses fall well short of ACE, exposing an asymmetry between generating text from MRS and recovering MRS from text.

1 Introduction

  • The ERG is a hand-written, broad-coverage English grammar whose ACE processor maps sentences to MRS graphs and generates sentences from them.MRS represents predicates and argument roles, leaving quantifier scope underspecified.Source: §1 · p. 1
  • Opus 5 scores 76.34 BLEU, about 10 points above the 72k-pair system and on par with the million-pair system; Sonnet 4.5 matches the 72k-pair system.The models’ sentences also recover the input MRS at 85.4 (Sonnet) and 89.8 (Opus) spans-off EDM F1, versus 91.0 for the gold sentences.Source: §1 · p. 2
  • ACE candidate selection raises Sonnet to 69.64 BLEU, four points above Sonnet alone, while a pooled judge adds about 0.6 BLEU for Opus.The pooled judge retains Opus’s own sentence among its candidates.Source: §1 · p. 2
  • For text-to-MRS analysis, Sonnet reaches 57.2 and Opus 65.5 span-free EDM F1, versus 91.0 for ACE; exact matches are only 0.6% and 1.4%, versus 50.5% for ACE.The outputs are generally well-formed, but diverge in predicate inventory and argument frames.Source: §1 · p. 2
  • The benchmark makes the producing/understanding gap measurable on the same sentences, with the grammar serving as reference in both directions.The Redwoods treebank pairs sentences from several genres with hand-selected gold MRS analyses.Source: §1 · pp. 1–2

2 Setup

The setup reconstructs the released generation benchmark and evaluates realization, reranking, and parsing with prompted Claude models. ACE provides a bidirectional comparison, although its parser and ranker are trained on the benchmark profiles.

2 Setup

  • The reconstructed ERG-1214 split contains 10,201 test items across 37 Redwoods profiles and uses Hajdik et al.’s released conversion unchanged.This preserves the inputs and references in their released files.Source: §2 · p. 2
  • The study evaluates three tasks: linearized MRS-to-sentence realization, reranking ACE candidates, and sentence-to-SimpleMRS analysis on a stratified 1,000-item sample.The parsing sample is drawn from newswire, and its gold MRS passes through the evaluator unchanged.Source: §2 · p. 2
  • Claude Sonnet 4.5 (20250929) and Claude Opus 5 were prompted with three fixed exemplars from outside scored items; Sonnet used greedy decoding with a 3,000-token budget, while Opus used default thinking.Source: §2 · p. 2
  • ACE 0.9.31 with ERG 1214 serves as realizer and parser, but is in-sample because its ranking model was trained on Redwoods, including these profiles.Realization uses an n-best list of 10 and a 10-second per-item limit; parsing uses top-1.Source: §2 · p. 2
  • Realization is scored with SacreBLEU, while parsing uses EDS-based EDM F1, exact match, and Smatch measures.EDM compares predicates and argument triples, either with character spans or with spans removed; exact match requires isomorphic graphs ignoring spans.Source: §2 · p. 2

3 Realization

Prompted models perform strongly on MRS-to-text realization, and ACE candidates can improve the models’ selections. Reparse results indicate that their sentences largely preserve the input meaning, while controls find memorization does not by itself explain realization performance.

3 Realization

  • Sonnet matches Hajdik et al.’s 72k-pair system, while Opus is 8–10 BLEU points above it and essentially tied with the million-pair system; ACE reaches 61.81 BLEU on 79.2% of the split.The comparison uses the same linearized MRS test set as the trained systems.Source: §3 · pp. 2–3, Table 1 · p. 3
  • Opus’s sentences score 89.8 spans-off EDM F1 on reparse, within 1.2 points of the gold sentences’ 91.0, indicating that they largely preserve input predicates and arguments.Sonnet’s sentences score 85.4; ACE’s own realization and the judge’s pick score 94.5 and 94.3 on the 762 items ACE generates for.Source: §3 · p. 3
  • On the subset where ACE generates candidates, Sonnet’s fluency judge scores 71.25 BLEU, 9.4 points above ACE’s first choice; the full-split hybrid reaches 69.64.The top-10 often contains the correct sentence, while ACE’s ranking is poor; ACE’s ranker was trained for parse selection, not realization.Source: §3 · p. 3, Table 2 · p. 3
  • With Opus’s sentence added to ACE’s candidate pool, the pooled judge reaches 76.95 BLEU on the full split, essentially tied with the gold-plus-silver trained system.The judge selects Opus’s sentence on 66.9% of items and an ACE candidate on 33.1%.Source: §3 · p. 3
  • The substitution control finds a 3.17-point BLEU advantage for original over noun-swapped pairs, but the authors report that memorization does not by itself explain realization results.The verbatim-continuation probe recalls 33% of one essay’s sentences and 0% on three control profiles, with recall unrelated to realization quality.Source: §3 · p. 3

4 Parsing

On the reverse task, the LLMs produce decodable MRS but rarely match the gold graph, with predicate naming and argument attachment among the errors. Calibration results show that graph structure alone can inflate Smatch even when predicate names are wrong.

4 Parsing

  • Opus exactly matches 1.4% of gold MRS and Sonnet 0.6%, versus 50.5% for ACE; spans-off EDM trails ACE by 26 and 34 points, respectively.Outputs decode as SimpleMRS for 96.3% of Opus and 99.4% of Sonnet sentences.Source: §4 · p. 3, Table 3 · p. 4
  • Smatch scores 79.2 for gold graphs with every predicate name replaced by nonsense, while spans-off EDM and exact match both fall to zero.A constant MRS still earns 33.1 Smatch, showing that structural agreement can score despite incorrect predicate names.Source: §4 · pp. 3–4
  • Sonnet names 65.48% and Opus 74% of gold predicates exactly, and recover 74% and 84% of gold edges between correctly named nodes, respectively.ACE reaches 93% exact predicate naming and 95% edge recovery; ignoring role labels raises model relation F1 only modestly.Source: §4 · p. 4

5 Conclusion

The paper evaluates prompted models on both MRS-to-text generation and text-to-MRS parsing using the same sentences. They perform strongly in generation but poorly in parsing, so generation scores alone do not establish that they understand the formal representations.

5 Conclusion

  • On the same sentences, prompted models are strong at MRS-to-text generation but fail to reliably produce the gold MRS from text.Letting them choose among grammar-generated candidates helps Sonnet by four points and Opus slightly.Source: §5 · p. 4
  • Going from MRS to text, with three examples in the prompt, Sonnet matches the system trained on 72k gold pairs, while Opus scores about ten BLEU points higher and matches the system that also used silver data.Source: §5 · p. 4
  • A high generation score alone should not be treated as evidence that a model knows the underlying representation.The conclusion follows from the contrast between generation quality and the models’ poor reverse-task results.Source: §5 · p. 4

Limitations

The study measures two Anthropic models under one three-example prompting setup. More capable prompting or constrained decoding could narrow the parsing gap, but those approaches are outside the study’s evaluation.

Limitations

  • The evaluation covers two Anthropic models under one prompting setup with three examples.The authors aim to measure what prompted models can do on their own.Source: Limitations · p. 4
  • Retrieval or constrained decoding might narrow the parsing gap, but combining models with constrained decoding is left for future work.The authors state that adding examples and enabling extended thinking did not close the gap.Source: Limitations · p. 4
  • The reported findings do not evaluate stronger prompting setups such as retrieval or constrained decoding.These approaches are described as possibilities rather than tested methods in this study.Source: Limitations · p. 4

A Appendix

The appendix details the benchmark handling, evaluation measures, and controls used to interpret the generation and parsing results. It also explains why reference-based candidate selection is an oracle rather than a deployable system.

A Appendix

  • The memorization controls test verbatim continuation from test-document openings and compare BLEU after substituting a same-length content noun in both input and reference.The substitution control compares original and altered MRS-reference pairs.Source: A Appendix · p. 5
  • EDM scores shared graph facts—predicates, argument links, and features—using precision, recall, and F1; spans-off matching keys nodes by predicate name alone.Span-anchored matching additionally requires predicted and gold nodes to cover the same sentence span.Source: A Appendix · p. 5
  • Exact match tests whether the predicted graph equals the gold graph up to variable labels, ignoring spans; Smatch aligns nodes to maximize agreeing facts.Smatch’s randomized alignment search can make repeated scores differ by about 0.3.Source: A Appendix · p. 5
  • Predicate-lemma overlap ignores edges and sense labels, while edge recall measures recovered gold links between correctly named predicate nodes.For example, _find_v_1 and _find_v_mental both count as find for lemma overlap.Source: A Appendix · p. 5
  • The reference selector chooses the candidate closest to the reference by BLEU, so it measures candidate-list potential rather than a usable system.A corpus-BLEU selector iteratively swaps candidates when doing so increases corpus BLEU.Source: A Appendix · pp. 5–6
Loading 2609.30071v1…