Grammar Concept Annotation at Scale: Deployed Fine-Tuned Small Language Models Outperform Prompted Frontier Models

1 Introduction

• The tracker processes lesson transcripts into overall and per-concept mastery scores, with each evidence item traceable to a highlighted learner-sentence span.Evidence is weighted by concept frequency, CEFR level, and a forgetting curve that discounts older evidence.Source: §1 · p. 1, Figure 1 · p. 2
• Grammar concept annotation identifies attempted curricular concepts, extracts evidence spans, judges uses as correct or incorrect, and explains incorrect uses.The model returns these predictions in a single structured pass per learner turn.Source: §1 · p. 1
• Fine-tuning on filtered and rebalanced teacher-generated supervision addresses the cost of repeatedly prompting GPT-5.4 with the full annotation contract.The initial pilot was limited to a small fraction of learners because production-scale prompting exceeded the feature’s serving budget.Source: §1 · pp. 1–2
• On two human-curated benchmarks, the deployed 0.8B model and 4B comparator outperform prompted GPT-5.4 and GPT-5.6 Sol in precision and recall at every nested matching level.The levels require matching concept, exact evidence span, and correctness.Source: §1 · p. 2
• A data-quality pipeline improves strict-match recall by 36.9–40.7 percentage points over matched raw-supervision models.The pipeline combines evidence-grounded review, deterministic validation, filtering, and synthetic enrichment.Source: §1 · p. 2
• The deployed 0.8B model reduces serving cost by approximately 16× relative to GPT-5.4.The production architecture pairs the model with prompt compaction, native vLLM, and autoscaling.Source: §1 · p. 2

2 Grammar Concept Annotation Task

• The model receives the original learner utterance and its correction, rather than the full transcript, and produces a set of annotations.One utterance may yield several concepts or none.Source: §2 · p. 2
• Each annotation identifies one of 34 grammar concepts, an exact contiguous span from the learner utterance, and whether that use is correct or incorrect.An explanation is non-empty if and only if the use is incorrect.Source: §2 · p. 2
• Matching concept, concept-plus-span, and concept-plus-span-plus-correctness yields progressively stricter evaluation criteria; explanations are assessed separately as free text.Exact spans can be highlighted in the learner’s sentence and compared with gold spans by string identity.Source: §2 · p. 2

3.1 Training Data Collection, Cleaning, and Enrichment

• The curation pipeline combines evidence-grounded review, validation, and synthetic enrichment, then consolidates records while removing benchmark utterances.The resulting corpus contains approximately 187k utterance groups from reviewed production and synthetic records, after deduplication.Source: §3.1 · pp. 2–3

3.2 Golden Evaluation Benchmarks

• The initial evaluation covers 34 concepts, while the architecture is intended to scale beyond the next iteration’s more than 300 fine-grained concepts.The two benchmarks share the 34-concept evaluation scope.Source: §3.1 · p. 2, Table 1 · p. 4

3.1 Training Data Collection, Cleaning, and Enrichment

• Production records are sampled from both typical traffic and suspected annotation anomalies, re-annotated with linguistic and correction cues, and checked against contract invariants.The deterministic validator rejects non-substring spans, prunes invalid labels, and enforces explanation consistency.Source: §3.1 · p. 3
• Synthetic examples use production-derived concept and correctness backbones, stratified sampling, and planned pedagogical errors to target rare concepts and mistakes.A correct slot receives an injected error type with p = 0.05; deterministic validation checks span and label alignment.Source: §3.1 · p. 3

3.2 Golden Evaluation Benchmarks

• Golden contains 689 utterances reflecting production traffic, whereas Golden2 contains 2,977 error-enriched utterances for diagnostic testing.Error-bearing utterances make up 50.1% of Golden and 95.6% of Golden2; two annotators worked independently and a third adjudicated disagreements.Source: §3.2 · p. 3
• Learner transcripts, annotations, and internal prompts are not released because they are proprietary and privacy-sensitive.The paper reports aggregate statistics and methodological details instead.Source: §3.2 · p. 3

Fine-tuning setup

• The Qwen3.5 and Qwen3 base models are adapted with completion-only LoRA, and checkpoints are selected by teacher-forced validation loss with early stopping.Training packs five utterance–correction pairs per conversation and uses a 90/10 train–validation split.Source: §4 · p. 4

Prompt internalization and model sizing

• Fine-tuning lets the 0.8B model absorb concept definitions and span conventions into adapter weights, replacing a verbose instruction prompt with a compact matched prompt.The compact prompt uses 375 tokens, a 90.2% reduction from the 3,810-token GPT-5.4 prompt.Source: §4 · p. 4
• The 0.8B deployment target is designed for high-concurrency serving on single-GPU commodity instances, while the 4B model serves as a reference comparator.The 4B comparator retains the detailed prompt, so model-size differences are not a controlled parameter ablation.Source: §4 · p. 4

Annotation evaluation

• Systems are scored on nested L1 concept, L2 concept-plus-span, and L3 concept-plus-span-plus-correctness criteria using micro precision and recall.The study also reports key-macro averages in the appendix.Source: §5 · p. 4
• Uncertainty is estimated with paired cluster-bootstrap confidence intervals, and differences from GPT-5.4 are tested with paired randomization.Each procedure uses 20,000 resamples or permutations, respectively.Source: §5 · p. 4

Explanation evaluation

• Explanation acceptability is judged by GPT-5.5 using the utterance, span, correction, and concept definition, with human review and disagreement reconciliation.The approach is chosen because surface n-gram metrics correlate poorly with pedagogical feedback quality.Source: §5 · p. 4

Pipeline ablation

• Raw-supervision controls use unreviewed GPT-5.4 annotations and matched prompts, splits, hyperparameters, and optimizer-update budgets; each model size is compared only with its own control.The evaluation also tracks L3 recall across all 34 concepts, on incorrect uses, and on the nine least frequent concepts.Source: §5 · pp. 4–5

Annotation evaluation

• Table 2 reports micro precision and recall on Golden and Golden2, with full-pipeline fine-tuned models compared against prompted GPT-5.4 and GPT-5.6 Sol.The fine-tuned Qwen–GPT-5.4 differences are significant under paired randomization tests (p < 0.0001).Source: Table 2 · p. 5

Annotation accuracy and specialization

• Both fine-tuned models outperform GPT-5.6 Sol across all criteria on the two benchmarks, with the largest uplift under strict L3 matching.Table 2 reports precision and recall.Source: §6 · p. 5

Data-pipeline ablation

• 36.91 and 39.95 points: the full pipeline raises Golden2 L3 recall for 4B and 0.8B; Golden L3 recall rises 37.33 and 40.66 points.The Golden2 L3 precision gains are 28.37 points for 4B and 29.93 points for 0.8B.Source: Table 3 · p. 5, §6 · p. 5
• On Golden2, raw-supervision recall drops 23.64–23.78 points from L1 to L2, versus 6.28–6.31 points for the full pipeline.L2 requires an exact evidence-span match.Source: §6 · p. 5

Explanation quality

• After human audit and reconciliation, GPT-5.4, 4B, and 0.8B explanations have similar acceptable rates of approximately 95%, with no significant differences.GPT-5.5 initially judged 99.06% of GPT-5.4 explanations acceptable, versus 94.91% for 4B and 93.88% for 0.8B.Source: §6 · p. 5

Online product impact

• The complete feature significantly improved learner engagement and all three business metrics in the fixed pre-scaling experiment snapshot.The experiment randomized learners with 50% exposure; its gains apply to the end-to-end feature, not the SLM in isolation.Source: §6 · p. 6, Table 4 · p. 6

Production serving architecture

• Qwen3.5-0.8B runs on autoscaled single-GPU instances, with utterance–correction pairs packed into five-pair requests and concurrent requests batched by vLLM.Validation or service failures route work to GPT-5.4.Source: §7 · p. 6

Serving cost and performance

• Recurring detection cost per lesson is 6.2% of the GPT-5.4 baseline, an approximately 16× reduction.In a 31-day production snapshot, p95 latency was 4.3 seconds and request error rate was 0.0015%.Source: §7 · p. 6

Task formulation

• Unlike work that localizes or repairs errors, labels edits, or generates feedback, this task identifies concepts from unconstrained speech and jointly predicts spans, correctness, and explanations.Prior work includes grammatical error detection and correction and learner feedback generation (Bryant et al., 2017; Nagata, 2019; Liang et al., 2023; Song et al., 2024).Source: §8 · p. 6

Teacher-generated and synthetic supervision

• Rather than generate synthetic data through tagged corruption or LLM prompting, the pipeline filters production annotations and rebalances concept–correctness backbones.Prior synthetic-data methods include tagged corruption and LLM prompting.Source: §8 · p. 6

SLM specialization and deployment

• The deployed sub-1B model combines prompt internalization, a compact schema prompt, and completion-only LoRA while evaluating accuracy, serving cost, and online impact.Related work includes prompt internalization, lightweight-model distillation, and LLM explanations for SLM GEC systems (Li et al., 2025).Source: §8 · p. 6

Conclusion

• Fine-tuned Qwen3.5 models outperform GPT-5.4 and GPT-5.6 Sol across all matching levels on two human-curated benchmarks.The matching criteria cover concepts, exact evidence spans, and correctness.Source: §9 · p. 6
• The data pipeline adds 36.9–40.7 points of strict L3 recall over raw teacher supervision.The deployed 0.8B model reduces serving cost by approximately 16×.Source: §9 · p. 6
• A feature-level online experiment showed significant engagement and business gains for the deployed system.The conclusion attributes viability at scale to data quality, prompt internalization, and efficient serving.Source: §9 · p. 6

Modeling and training controls

• The pipeline ablation compares the full combined process with raw supervision, but does not isolate component contributions; each configuration also has only one training run and no across-seed variance estimate.Fine-grained metric differences should therefore not be over-interpreted.Source: Limitations · p. 7
• Differences across model sizes are not isolated parameter effects: the 4B and 0.8B models use independently tuned recipes and prompts, and the 0.6B comparison uses an earlier Qwen generation.Source: Limitations · p. 7

Operational and evaluation scope

• The online A/B experiment evaluates the end-to-end feature, while the serving-cost estimate covers recurring grammar-detection compute and excludes upstream transcription, correction, and fallback API calls.Source: Limitations · p. 7
• Explanation quality was evaluated against gold rubrics rather than through direct teacher or learner studies, and the uncalibrated LLM judge's leniency gap points to a need for domain-specific calibration.Source: Limitations · p. 7

Data scope, speech, and supervision biases

• Deployed checkpoints target an English curriculum of 34 concepts; ASR errors may cascade into tracking, robustness to ASR noise is not isolated, and GPT-5.4-derived supervision may retain biases benchmarks cannot fully reveal.The pipeline is described as framework-agnostic and designed to scale to a broader curriculum.Source: Limitations · p. 7

• Analysis is limited to sessions with a lawful basis; learner utterances, annotations, and internal prompts are withheld, and Figure 2 uses a synthetic example.The paper reports aggregate evaluation and online-experiment results only.Source: Ethical Considerations · p. 7

A Training Configurations and Convergence

• 57.4% of 0.6B outputs failed schema validation, compared with 0.03% for 0.8B; the 0.6B run also had 28.5% higher minimum validation loss.The models shared prompt and optimization hyperparameters but belong to different Qwen generations, so parameter count alone is not isolated as the cause.Source: Table 6 · p. 10, A Training Configurations and… · p. 10

B Data Pipeline Details

• The 90k synthetic corpus used production backbones sampled uniformly, by error weighting, and by inverse concept frequency, with pedagogical errors injected into correct slots.GPT-5.4 generated sentence variations; post-processing restored intent labels, enforced span validity, and removed out-of-scope labels.Source: B Data Pipeline Details · p. 11

C.1 Confidence Intervals for Annotation Results

• 20,000 paired cluster-bootstrap resamples produced percentile 95% confidence intervals, while paired randomization tests compared predictions with GPT-5.4 at the five-utterance batch level.Tests used 20,000 Monte Carlo permutations; zero exceedances were reported as p < 0.0001.Source: §C.1 · p. 12, Table 7 · p. 12, §C.4 · p. 13

C.2 Key-Macro Annotation Results

• Key-macro precision and recall weight distinct target keys equally; the pre-fixed Rare-9 subset comprised the nine least-annotated concepts in the raw training split.Rare-9 selection used no benchmark outcomes; Golden2 and Golden supports were reserved for evaluation.Source: §C.2 · p. 12, Table 8 · p. 12, §C.3 · p. 12

D Serving Capacity, Autoscaling, and Cost

• 4,993.5 pairs per minute across eight g6.2xlarge instances measured serving capacity, equivalent to 2.08 five-pair requests per second per instance.Five-pair requests were conservative relative to the production average of 4.27 pairs per request.Source: D Serving Capacity, Autoscaling, and… · p. 13

D.1 Autoscaling Policy and Load Test

• 17,994 requests completed in the autoscaling test with a 0.011% error rate; at the 6.4 RPS peak, five instances had zero errors and 7.64-second p99 latency.The endpoint scaled from one to five instances during the peak and remained within target concurrency thresholds.Source: §D.1 · p. 13, Table 10 · p. 13

D.2 Cost Projection

• 6.2% of baseline per lesson was the projected SLM serving cost, an approximately 16× reduction versus measured May 2026 GPT-5.4 production costs.The projection covers recurring detection serving and excludes common upstream work, fallback API calls, and one-time training and deployment costs.Source: §D.2 · pp. 13–14

E.2 Prompt Contracts: Verbose Baseline and Compact Inference

• The deployed 0.8B model used a 375-token compact prompt, 90.2% shorter than the 3,810-token verbose baseline, because fine-tuning internalized concept definitions and span boundaries.Runtime instructions retain allowed concept names, the schema contract, and concise formatting constraints.Source: §E.2 · p. 14
• The compact contract maps numbered original/corrected sentence pairs to pipe-delimited annotations across 34 grammar concepts, with correctness, exact original spans, and explanations.Concept classification follows the corrected sentence; explanations are empty for correct uses and brief fixes for errors.Source: §E.2 · p. 15

E.3 Explanation Judgment Rubric

• Approximately 95% of explanations received aligned human acceptability judgments across all three Golden2 systems, with no significant differences (χ2 test, p > 0.30).Accept or pass counted as acceptable feedback; reject covered misleading or erroneous feedback.Source: §E.3 · p. 15

E.4 Synthetic Distribution-Shift Probe

• Exact-span F1 fell by 24.69 points on synthetic data, while concept and partial-span F1 remained stable and all outputs parsed.The probe used 200 learner-like utterances authored and annotated by GPT-5.5, so it is diagnostic rather than a third golden benchmark; the span drop is consistent with differing annotation protocols.Source: §E.4 · p. 15, Table 12 · p. 15