Source-linked AI summary
How Useful are LLMs for Grammar Engineering? Cantonese ParGram Resources and Controlled Experimental Evaluation with English Baselines
Chit-Fung Lam
TL;DR
It remains unclear whether current LLMs can generate linguistically adequate, formally accurate, and computationally usable grammars for symbolic grammar-engineering pipelines. This study evaluates two models under controlled prompting conditions and finds that GPT-5.4 and target f-structure prompts generally produce better grammars.
Problem
It remains unclear whether current LLMs can generate grammars that are linguistically adequate, formally accurate, and computationally usable within existing symbolic grammar-engineering pipelines.
Method
The study systematically varies grammatical-information source, grammar-generation scope, and language model while holding other experimental conditions constant, evaluating rule accuracy, parse-level structure, and error patterns.
Results
GPT-5.4 outperformed gpt-oss-120b across both experiments, while prompts containing target f-structures generally produced better grammars than sentence-only prompts.
Takeaways & Limitations
LLMs may support intermediate grammar-development stages, with engineers verifying analyses and refining grammars generated from corrected f-structures.
Takeaways & Limitations
The findings may not generalize beyond the two evaluated models, the first 50 ParGram sentences, and the relatively small multi-construction sample, and fine-tuning was not investigated.
Abstract
from arXiv · showhide
This paper presents new Cantonese ParGram resources and evaluates LLMs for knowledge-driven grammar engineering within a controlled experimental paradigm. Using Cantonese ParGram resources as gold standards, with corresponding English baselines, we investigate whether OpenAI's gpt-oss-120b and GPT-5.4 can generate machine-processable grammars from sentences and target formal structures under systematically varied prompting conditions. GPT-5.4 outperformed gpt-oss-120b, while grammars generated from target formal structures generally outperformed those generated from sentences. Although both models could generate locally plausible phrase-structure rules, lexical entries, and templates, they often struggled to coordinate interacting formal constraints, especially in multi-construction settings. The results characterize both the capabilities and limitations of current LLMs for potential integration into AI-assisted expert workflows: LLMs may support intermediate stages of grammar development, but human linguistic expertise remains central to analysis, validation, and refinement. The study also contributes new Cantonese symbolic grammatical resources.
1 Introduction
The paper introduces new Cantonese ParGram resources and evaluates how current LLMs support knowledge-driven grammar engineering under controlled prompting conditions. It operationalizes usefulness through formal accuracy, structural quality, and error patterns rather than end-to-end development efficiency.
- The paper contributes new Cantonese treebank resources and a controlled evaluation of LLM capabilities and limitations in grammar engineering.Cantonese ParGram resources serve as gold standards alongside corresponding English baseline conditions.
- The evaluation asks whether LLMs can generate XLE-style phrase-structure rules, functional annotations, templates, and lexical entries.These components are required for XLE grammars.
- Current LLMs remain difficult to assess for linguistically adequate, formally accurate, and computationally usable grammar generation within symbolic pipelines.Knowledge-driven grammar engineering requires substantial manual effort and expertise in linguistic formalism and computational implementation.
- The experiments compare gpt-oss-120b and GPT-5.4 using one-shot in-context learning and evaluate structural validity, linguistic adequacy, and formal accuracy.The models are tested under systematically varied experimental conditions.
- Usefulness is defined through rule-level formal accuracy, parse-level structural quality, and error patterns under controlled conditions.The study varies information source, grammar-generation scope, and language model while holding other aspects constant.
2 Background and Related Work
The background situates grammar engineering within computational linguistic theory and practice, emphasizing LFG, ParGram, and XLE. It also identifies limitations in prior LLM grammar-generation studies that motivate the paper’s controlled evaluation.
- Grammar engineering computationally implements formal grammars for linguistic analysis and NLP applications across communities including LFG and HPSG.Implemented grammars also support applications such as grammar checking, dialogue systems, language learning, and annotated-data generation.
- ParGram develops parallel LFG grammars and automatically generated parsebanks containing c-structures and f-structures.Its treebanks are generated by parsing sentences with LFG/XLE grammars without manual annotation or ad hoc corrections.
- LFG relates c-structure and f-structure through mathematically defined functions, with c-structure encoding constituency and f-structure representing grammatical functions and morphosyntactic features.XLE provides parsing, generation, grammar-writing, testing, and debugging support for LFG grammars.
- Prior Moklen work used prompt-based in-context learning but focused mainly on translation, reporting limited quantitative grammar-rule evaluation.Lexical entries received an 86/100 accuracy score, while formal well-formedness was not systematically evaluated.
- Prior Cantonese and Irish work found that gpt-oss-120b captured key predicate–argument relations in around 70% of generated f-structures.The study also reported that LLMs may suggest good alternative analyses.
- The present study addresses these limitations by evaluating Cantonese XLE grammar generation for rules, annotations, templates, and lexical entries against symbolic ParGram resources.It assesses formal and notational accuracy together with structural consistency and linguistic adequacy.
3 New Cantonese ParGram Resources
The paper presents a handcrafted Cantonese XLE grammar and its automatically generated ParGram treebank, illustrating how selected constructions are represented in c- and f-structures. The resources encode theoretical analyses through computational rules and functional constraints.
- 3 New Cantonese ParGram Resources: The Cantonese ParGram grammar is handcrafted in XLE, and its treebank is automatically generated by parsing ParGramBank sentences.These resources address Cantonese’s limited availability of formal computational grammatical resources.
- 3.1 Selected phenomena: theoretical insights and computational implementation: The treebank discussion focuses on f-structures because they represent deeper syntactic relations and support cross-linguistic comparison in ParGram.ParGram emphasizes parallelism in core grammatical functions despite variation in c-structure.
- 3.1 Selected phenomena: theoretical insights and computational implementation: Functional control represents structure sharing between an embedded SUBJ and a matrix controller through an open XCOMP.The XLE implementation uses the equation (^SUBJ)=(^XCOMP SUBJ).
- 3.1 Selected phenomena: theoretical insights and computational implementation: Cantonese object fronting uses 將 zoeng to prepose the OBJ within an SVO language, with Figure 2 showing the corresponding c- and f-structures.The grammar uses rules that account for the alternation between canonical SVO and OBJ-fronting structures.
- 3.1 Selected phenomena: theoretical insights and computational implementation: A lexical rule models Cantonese GF alternation, while additional c-structure rules enable parsing of related constructions.The rule applies to transitive-verb subcategorization frames.
- 3.1 Selected phenomena: theoretical insights and computational implementation: The Cantonese passive marker 俾 is analyzed as a higher verb that selects SUBJ, OBJ, and XCOMP and requires an overt agent.This analysis omits an embedded TOPIC function used in related analyses.
4 New Case Study: Grammar Engineering and LLMs
The study uses Cantonese ParGram resources and English baselines to test whether two LLMs can generate XLE-style grammars under controlled prompting conditions. GPT-5.4 and target f-structure prompts generally performed better, but multi-construction grammar generation exposed difficulties coordinating interacting constraints.
- Methodology: Two controlled experiments varied model, input source, and construction scope while evaluating generated XLE-style grammar components against Cantonese ParGram resources.The study compared sentence-to-grammar and f-structure-to-grammar prompts in single- and multi-construction settings, with English baseline conditions.
- Experiment 1: single-construction grammar generation: GPT-5.4 outperformed gpt-oss-120b in Experiment 1 across both prompting conditions, with significant differences for QR and AR.For sentence-to-grammar, GPT-5.4 achieved QR 2.26 and AR 84%, versus 1.70 and 72%; for f-structure-to-grammar, it achieved QR 2.88 and AR 90%, versus 2.02 and 75%.
- Experiment 1: single-construction grammar generation: Target f-structures improved Experiment 1 performance, especially for GPT-5.4, whose QR rose from 2.26 to 2.88 and AR from 84% to 90%.For gpt-oss-120b, QR rose significantly from 1.70 to 2.02, while its AR increase from 72% to 75% was not significant.
- Experiment 1: single-construction grammar generation: English baselines showed similar QR patterns, suggesting that the benefit of target f-structure input may extend beyond Cantonese, pending evaluation across more ParGram languages.The authors note that broader cross-linguistic generalization would require additional ParGram languages.
- Experiment 2: multi-construction grammar generation: Multi-construction generation produced lower scores, particularly for gpt-oss-120b, while GPT-5.4 reached QR 2.50 and AR 88% with f-structure prompts.In the multi-sentence-to-grammar condition, gpt-oss-120b had QR 1.10 and 90% Bad outputs, compared with GPT-5.4’s QR 2.00 and AR 86%.
- Error analysis: The main difficulty was coordinating phrase-structure rules, lexical entries, templates, and functional constraints rather than producing XLE-like notation locally.Errors included incorrect subcategorization frames, misplaced information, missing functional annotations, incorrect grammatical functions, and missing templates.
- Implications for grammar engineering and human–AI collaboration: The findings support using LLMs for preliminary analyses and candidate grammar components, while experts retain responsibility for theoretical analysis, formal validation, and cross-constructional reasoning.One proposed workflow uses LLM-generated preliminary f-structures, expert correction, and corrected f-structures to prompt grammar generation before refinement.
5 Conclusion
The paper contributes Cantonese ParGram resources and evaluates LLMs for grammar engineering. GPT-5.4 generally performed better than gpt-oss-120b, but both models struggled with interacting constraints in multi-construction settings.
- The study presents new Cantonese ParGram resources and a controlled evaluation of LLM capabilities for knowledge-driven grammar engineering.
- GPT-5.4 outperformed gpt-oss-120b, while target formal structures generally yielded better grammars than sentence-only prompts.
- Both models struggled to coordinate phrase-structure rules, lexical entries, templates, and functional constraints, especially in multi-construction settings.
- LLMs may support grammar development under expert supervision, but human linguistic expertise remains central to robust grammar development.
Limitations
The study's findings are bounded by its models, data, prompting method, evaluation focus, and relatively small experimental grammars.
- The experiments used only gpt-oss-120b and GPT-5.4, so findings may not generalize to other open-weight or proprietary LLMs.
- The experiments used the first 50 ParGram sentences and relatively few multi-construction sets, limiting Experiment 2's statistical power.
- The study examined one-shot in-context learning but did not investigate fine-tuning approaches.
- Evaluation focused mainly on formal well-formedness rather than computational efficiency or broader downstream NLP performance.
- The experimental grammars were substantially smaller than mature broad-coverage ParGram grammars.
A ParGramBank Sentences
This appendix section identifies the first two English ParGram sentences used in the resource list.
- The English ParGram sentence list is openly accessible through the INESS Treebanks interface.
- Sentence 1 is “The driver starts the tractor.”
- Sentence 2 is “The tractor is red.”
4. Did the farmer sell his tractor?
This section lists English ParGram sentences spanning basic clauses, argument structures, embedding, modification, control, passives, and other constructions, alongside Cantonese resources.
- The list includes basic declarative, imperative, negative, intransitive, passive, and ditransitive sentences.
- The examples also cover causative, permissive, benefactive, reciprocal, reflexive, locative, existential, comparative, superlative, and adjunct-related constructions.
- The resource list contains additional examples involving aspectual, nominal, predicative, weather, cleft-like, coordination, and complex modification patterns.
- The section includes further examples of idiomatic, comparative, modifier, complementation, and raising-related patterns, with Cantonese translations corresponding to the English sentences.
B Mapping Algorithm: C-to-f Structure Mapping
This section illustrates how LFG maps a c-structure into an f-structure and specifies the prompt conditions used to elicit Cantonese and English XLE grammars from sentences or target f-structures.
- C-to-f Structure Mapping: C-structures encode constituent structure, word order, and part-of-speech information, while f-structures represent grammatical functions and morphosyntactic features.
- C-to-f Structure Mapping: The mapping algorithm resolves functional equations annotated on the c-structure to form the corresponding f-structure.
- Prompt Design: Prompts use a system prompt, user prompt, English toy grammar, and final instruction to simulate introductory XLE training before language-specific grammar development.
- Grammar Requirements: The requested grammars contain phrase-structure rules with functional annotations and language-specific lexicons, with templates optionally included.
- Sentence-to-Grammar Prompts: Sentence-to-grammar prompts ask models to create Cantonese or English XLE grammars that parse a specified sentence.
- F-Structure-to-Grammar Prompts: F-structure-to-grammar prompts instead require grammars to parse a sentence and produce the exact target f-structure.
G Quality Rating (QR): Four-point Scale
The study rates f-structures produced by LLM-generated grammars on a four-point scale based on grammatical functions, predicate-argument structure, and non-GF features.
- Evaluation Scheme: The scheme evaluates the quality of f-structures produced by LLM-generated grammars using a four-point scale adopted from Lam and Uí Dhonnchadha (2026).
- Excellent: Excellent means all grammatical functions, the subcategorization frame, and relevant non-GF features are correct.
- Good: Good means all grammatical functions are correct and consistent with the predicate-argument structure, but some features are incorrect.
- Fair: Fair means the core construction is recognizable but contains at least one grammatical-function error or subcategorization mismatch.
- Bad: Bad means major grammatical-function errors make the core construction unrecognizable or render the f-structure incompatible with the intended predicate-argument structure.