PaperlayerView exact revision on arXiv

Source-linked AI summary

SWIM: Student Writing Simulation via Proficiency-Conditioned Generation

Heejin Do, Jakub Kontak, Mrinmaya Sachan

arXiv:2609.03215v1cs.CLcs.LG

TL;DR

Whether language models can realistically simulate multidimensional student writing remains underexplored. SWIM formulates this problem as proficiency-conditioned essay generation and evaluates prompting, SFT, and GRPO with AES-based profile alignment. Explicit training substantially improves alignment, with GRPO and PAR providing further gains, but authentic low-proficiency writing remains difficult to reproduce.

  • Problem

    Student writing simulation remains largely unexplored despite writing’s multidimensional variation across content, organization, lexical choice, and language proficiency.

  • Method

    SWIM formulates student writing simulation as proficiency-conditioned essay generation and compares prompting, SFT, and GRPO with an AES-derived Proficiency Alignment Reward.

  • Results

    GRPO with PAR improves profile alignment across every writing trait and essay prompt, while SFT outperforms prompting and prompting provides limited control.

  • Takeaways & Limitations

    Explicit supervision enables stronger profile alignment than prompting alone, but profile alignment does not guarantee realistic student writing.

  • Takeaways & Limitations

    The evaluation relies on AES as a proxy for fidelity and uses the limited ASAP/ASAP++ benchmark, so generalization remains unclear.

Abstract

from arXiv · show

Writing proficiency manifests in how students develop content, organize ideas, choose words, and use language. Despite growing interest in LLM-based student simulation, whether LLMs can reproduce such multidimensional variation in extended writing remains largely unexplored. In this work, we explore if language models can realistically simulate student writing, and introduce SWIM, a task that formulates Student Writing sIMulation as proficiency-conditioned essay generation. We evaluate prompting, supervised fine-tuning (SFT), and reinforcement learning (RL) methods for writing simulation using automated essay scoring as a measure of profile alignment. Extensive experiments reveal that prompting provides limited proficiency control, even for strong proprietary LLMs with rubric-grounded strategies. In particular, while models can adjust content-oriented traits, they struggle to reproduce the lexical, grammatical, and organizational variation in different proficiency levels. SFT substantially improves alignment, while RL with the proposed proficiency-alignment reward yields further gains across all writing traits and essay prompts. Our findings suggest that explicit supervision enables substantially stronger profile alignment than prompting alone, while authentic low-proficiency writing remains challenging to reproduce.

1 Introduction

SWIM asks whether language models can generate extended student writing that reflects multidimensional proficiency profiles. Comparing prompting, SFT, and GRPO with PAR, the study finds that explicit training improves alignment, while authentic low-proficiency writing remains difficult to reproduce.

  • 1 Introduction: Writing simulation is challenging because proficiency varies across content, organization, lexical choice, sentence fluency, and language conventions.A simulator must control multiple proficiency-related characteristics simultaneously rather than only produce coherent essays.
  • 1 Introduction: Rubric-grounded prompting provides limited and uneven control, aligning better on content traits than on lexical, grammatical, and organizational dimensions.This pattern holds even for strong proprietary models with few-shot demonstrations.
  • 1 Introduction: SFT substantially strengthens proficiency alignment from authentic score–essay pairs, enabling open-source 7B and 4B models to move beyond near-random prompting alignment.Direct supervision is more effective than rubric descriptions alone.
  • 1 Introduction: SWIM formalizes student writing simulation as proficiency-conditioned essay generation evaluated through profile alignment.The task requires essays to reflect specified writing aspects while addressing a prompt.
  • 1 Introduction: GRPO with the Proficiency Alignment Reward consistently improves simulation fidelity across every writing trait and essay prompt.PAR is a dense AES-derived reward designed to optimize profile-level alignment.
  • 1 Introduction: Models reproduce high-proficiency writing more faithfully than low-proficiency writing, leaving low-proficiency linguistic form as a persistent bottleneck.Prompting tends toward superficial corruption, while trained models retain overly polished surface realization.

2 Related Work

LLM-based student simulation has supported educational-agent training and pedagogical evaluation, but prior work has emphasized short responses, dialogue, and learner actions. SWIM extends this landscape to student writing while drawing on automated essay scoring and controllable-generation methods.

  • LLM-based Student Simulation: LLM-based student simulators provide scalable alternatives to human studies for training educational agents and evaluating instructional interventions.Prior systems model learner responses in tutoring dialogues, interactive environments, and agent-training pipelines.
  • LLM-based Student Simulation: Existing student-simulation work has focused primarily on short responses, dialogue interactions, and learner actions rather than extended writing.The paper positions writing as an underexplored simulation target.
  • Writing Assessment and Student Writing Modeling: Educational writing research has largely centered on automated essay scoring, writing evaluation, and feedback generation.Recent AES systems report strong agreement with human raters at holistic and trait levels.
  • Writing Assessment and Student Writing Modeling: SWIM formulates student writing simulation as generation and leverages AES models as scalable evaluators of simulation fidelity.Its training framework combines supervised fine-tuning with GRPO and AES-derived rewards.
  • Controllable Text Generation: Controllable text generation uses prompting and training approaches to steer attributes such as topic, style, length, readability, and factuality.Prior work finds that controlling multiple attributes simultaneously remains challenging.

3 SWIM: Student Writing sIMulation

SWIM formulates student writing simulation as generating essays conditioned on prompts and multidimensional proficiency profiles. It compares rubric-grounded prompting, supervised fine-tuning, and reward-based optimization, evaluating alignment with a fixed multi-trait essay scorer.

  • 3.1 Task Formulation: SWIM generates essays that address a writing prompt while exhibiting behavior consistent with a target, trait-indexed proficiency profile.The target profile uses prompt-specific rubric dimensions and score ranges; inference can request combinations not necessarily observed in training data.
  • Evaluation Interface: A frozen multi-trait AES verifier scores generated essays, and QWK between target and predicted profiles measures simulation fidelity without generator–evaluator coadaptation.The shared input includes the prompt, trait–score configuration, and output constraints; the verifier remains fixed across methods.
  • 3.2–3.4 Methods: The framework evaluates rubric-grounded prompting, SFT on score–essay pairs, and GRPO refined with an AES-derived proficiency-alignment reward.These methods progressively increase the supervision signal used to obtain the generator.
  • 3.2 Prompting: Rubric-grounded prompting uses either coarse high-versus-low trait contrasts or score-specific rubric descriptors, with optional five-shot authentic writing examples.The five-shot variants retrieve development-set essays spanning the overall-score distribution, separating rubric grounding from exposure to student examples.
  • 3.3–3.4 Training: SFT learns from human-authored essays paired with observed proficiency signals, while GRPO further optimizes alignment using dense rewards from the frozen verifier.The reward averages trait-normalized accuracy, reaches 1 when predicted scores exactly match the target, and weights traits equally despite differing score ranges.

4 Experiments

Experiments use the trait-annotated ASAP/ASAP++ essay corpus and evaluate whether generated essays match their conditioning profiles. The protocol compares target scores with scores from a fold-specific, held-out AES verifier.

  • Dataset and Experimental Setup: ASAP/ASAP++ contains eight argumentative, narrative, and source-based writing prompts with prompt-specific traits and score ranges.Style and Voice are excluded because each appears on only one prompt, yielding insufficient samples for prompt-unified modeling.
  • Results: Table 2 reports trait-level QWK, using a single fold for prompting and five-fold means with standard deviations for SFT and RL.Five-shot prompting is marked FS, and bold entries identify the best value per column.
  • Dataset and Experimental Setup: The AES verifier is trained only on each fold’s training split and achieves an average prompt-level QWK of 0.722 on the corresponding test split.This fold-specific construction prevents the verifier from observing evaluation essays.
  • Evaluation Protocol: Evaluation conditions each generator on a test prompt and gold trait-score configuration, then scores its generated essay with the AES verifier.QWK is reported between the target and predicted profiles at multiple evaluation levels, as described in the evaluation protocol.

5 Results

Across prompting, SFT, and RL, explicit training progressively improves proficiency alignment, with GRPO using PAR producing broad gains across traits and prompts. Prompting controls content-oriented traits more successfully than form-oriented traits, while realistic low-proficiency writing remains difficult.

  • 5.1 Rubric Grounding Helps, but Prompting Alone Is Insufficient: Prompting aligns better with Content, Prompt Adherence, and Narrativity than with lexical, grammatical, and organizational dimensions, even with rubric grounding and few-shot examples.Score-specific rubric lookup generally outperforms contrastive trait prompting, and few-shot demonstrations improve both strategies without resolving the broader limitation.
  • 5.1 Rubric Grounding Helps, but Prompting Alone Is Insufficient: 0.695 average QWK on content-oriented traits versus 0.458 on form-oriented traits for Claude’s best prompting configuration, with Conventions weakest under every setting.Across eight prompting configurations and model families, content-oriented traits consistently align better than form-oriented traits.
  • 5.2 SFT Enables Proficiency-Conditioned Student Writing: 0.501 average trait-level QWK for SFT versus 0.036 for CTP+FS and 0.153 for SRLP+FS on the same Qwen2.5-7B-Instruct backbone and fold.Across five folds, Qwen2.5-7B-Instruct reaches 0.474 ± 0.023, compared with 0.428 ± 0.062 for Qwen3-4B.
  • 5.3 GRPO with PAR Consistently Improves Proficiency Alignment: +0.144 and +0.246 average trait-wise QWK gains over SFT for Qwen2.5-7B-Instruct and Qwen3-4B, respectively, with GRPOPAR improving across every trait and essay prompt.Qwen3-4B, which was weaker under SFT, benefits most and achieves the strongest overall alignment.
  • 5.3 GRPO with PAR Consistently Improves Proficiency Alignment: Zero-shot prompting produces a normalized mean Overall score of 0.74 with 25% of essays capped, whereas SFT and GRPO produce 0.57 and 0.53 against a human distribution of 0.58.The score distributions indicate that prompting overproduces high-scoring essays, while proficiency-aware training better matches the human range.
  • 5.4 Do the Gains Transfer to Independent Evaluators?: 0.598 versus 0.479 and 0.647 versus 0.501 for GRPO versus SFT under two independent evaluators, supporting gains beyond the original reward verifier.The corresponding improvements are +0.119 and +0.146, closely matching the original evaluation’s +0.144 gain.

6 Discussion

Discussion shows that explicit training improves proficiency alignment and better matches overall score and essay-length distributions, but authentic low-proficiency language remains difficult to reproduce. Data scarcity and long generations create a notable exception, while alignment gains do not appear to result from memorization or collapse.

  • Prompt 8 reverses the overall training advantage, with prompting outperforming SFT and GRPO despite Qwen3-4B + GRPO leading average prompt-level alignment overall.Prompt 8 contains only 723 training essays and requires responses averaging approximately 650 words, suggesting data scarcity and generation length remain bottlenecks.
  • Prompting produces a high-proficiency bias, whereas SFT and GRPO more closely match the human distribution of overall scores.Zero-shot prompting has a mean normalized score of 0.74, compared with 0.57 for SFT, 0.53 for GRPO, and 0.58 for humans.
  • Table 3 compares real and generated essays across linguistic metrics under low-proficiency targets, using error counts per 100 words and relative deviation from real essays.Darker blue marks smaller relative deviation, and bold identifies the closest generated value.
  • SFT and GRPO also recover realistic essay lengths more closely than zero-shot prompting.Median length is 304 words for zero-shot prompting, versus 158 for SFT, 141 for GRPO, and 167 for real students.
  • Low-proficiency simulation remains uneven: prompting uses superficial errors, while training recovers structure but produces too few surface errors.At high proficiency, the discrepancy narrows; SFT’s grammatical-error gap falls from 84% below the real rate at low targets to 5% below it at high targets.
  • Alignment gains are unlikely to reflect memorization or generation collapse.Non-source-based prompts show 0.00% near-duplicates and negligible 8-gram overlap, while no essay repeats within identical prompt-profile groups.

7 Conclusion

The conclusion presents SWIM as proficiency-conditioned essay generation and finds a hierarchy from limited prompting control to stronger SFT alignment and further GRPO gains. However, profile alignment does not ensure realistic low-proficiency writing.

  • SWIM formalizes student writing simulation as proficiency-conditioned essay generation.
  • Rubric-grounded prompting offers limited control, SFT learns stronger alignment from score–essay pairs, and GRPO with PAR improves across traits and prompts.
  • Models reproduce high-proficiency writing more readily than authentic low-proficiency writing, revealing that profile alignment does not guarantee behavioral realism.

Limitations

The study’s fidelity assessment, benchmark coverage, and static proficiency profiles limit how broadly its findings can be interpreted.

  • Evaluation relies on AES as a scalable proxy, but AES cannot fully replace human judgment of authentic student-writing resemblance.
  • ASAP/ASAP++ covers limited prompts and proficiency distributions, leaving generalization to other contexts, age groups, languages, and assessment frameworks unclear.
  • Static proficiency profiles omit evolving knowledge, misconceptions, motivation, and writing strategies over time.

Ethical Statement

The study uses publicly available ASAP/ASAP++ essays solely for research and reports no personal or sensitive student information.

  • The experiments use publicly available ASAP/ASAP++ datasets without personal or sensitive student information.
  • AI assistance was used for language editing and proofreading.

A Implementation Details

The experiments fine-tune Qwen models with QLoRA-based SFT and then refine the SFT policy with GRPO using the frozen AES verifier as a KL reference.

  • SFT: SFT fine-tunes Qwen2.5-7B-Instruct and Qwen3-4B with QLoRA, NF4 4-bit quantization, bf16 computation, and LoRA adapters.Adapters use rank 16, α = 32, and dropout 0.05 across attention and MLP projections.
  • SFT: SFT uses AdamW with a 10^-4 peak learning rate, cosine decay, 10% warmup, effective batch size 8, 2500-token sequences, and five epochs.The fold-specific ArTS checkpoint serves as the AES verifier.
  • RL: RL initializes from SFT and uses TRL’s GRPOTrainer with four generations per prompt, a 10^-5 learning rate, effective batch size 16, and one training epoch.The SFT checkpoint is the KL reference policy, with maximum prompt and completion lengths of 2500 and 1024 tokens.
  • Compute: All experiments run on one NVIDIA H100, requiring approximately 1 hour per SFT epoch and 8 hours per GRPO epoch.Evaluation uses the same fold-specific ArTS verifier.

B Additional Results

Additional evaluations show that GRPO-based training remains strongest across prompts, while generated essays exhibit qualitatively distinct proficiency-conditioned behaviors and rely on a reasonably aligned AES verifier.

  • Prompt-wise Results: GRPOPAR achieves the strongest performance across most essay prompts, while SFT consistently improves alignment over naive prompting on the same backbone.Prompt characteristics influence difficulty, with long-narrative Prompt 8 and its smallest training set underperforming across methods.
  • Qualitative Examples: Generated essays change qualitatively with target proficiency: length and structural sophistication increase monotonically from scores 1/4 to 4/4.Higher-target essays add source engagement, quotations, and multiple supporting examples.
  • Verifier Performance: The ArTS verifier reaches average QWK of 0.710 at the trait level and 0.722 at the prompt level, indicating substantial agreement with human raters.These scores are averaged across five cross-validation folds.

C Linguistic Analysis Details

The analysis measures error rates, lexical diversity, and syntactic complexity while detailing rubric-based prompting and the score profiles used to condition simulated essays.

  • Error-based metrics: Grammar, spelling, and casing errors are detected with LanguageTool and normalized as errors per 100 words.Grammar includes GRAMMAR and CONFUSED_WORDS matches; spelling uses TYPOS, and casing uses CASING.
  • Lexical diversity: Lexical diversity is measured with MATTR over lowercased alphabetic tokens using a 50-token sliding window.Essays shorter than 50 qualifying tokens use all available tokens, equivalent to conventional type–token ratio.
  • Syntactic complexity: Syntactic complexity is assessed through dependency-based clauses per sentence and dependency depth using spaCy parsing.Clause counts add one main clause plus clauses associated with specified dependency labels.
  • Rubric prompting: Rubric-grounded prompts instruct models to match proficiency across content, organization, word choice, sentence fluency, conventions, and voice.The prompts require essay-only output and prohibit mentioning scores or rubric language in the generated essay.
  • Score-level prompting: Score-level prompting supplies target overall and trait scores, with five-shot variants retrieving development examples spanning the overall-score distribution.The supplied score ranges and trait profiles operationalize the requested proficiency level during generation.
Loading 2609.03215v1…