PaperlayerView exact revision on arXiv

Source-linked AI summary

ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning

Jianghao Chen, Wei Sun, Qixiang Yin, Zhixing Tan, Jiajun Zhang

arXiv:2509.04903v3cs.CL

TL;DR

Long-form generation remains challenging, and existing methods can rely on scarce high-quality data and coarse rewards that miss task-specific demands. ACE-RL builds adaptive constraint checklists and uses their verification as RL reward signals. On WritingBench, it outperforms SFT and LLM-as-a-Judge RL baselines on average.

  • Problem

    Existing long-form generation approaches rely on costly or scarce response data and coarse rewards that miss fine-grained, instruction-specific requirements.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    ACE-RL converts each instruction into fine-grained adaptive constraints, scores responses through constraint verification, and uses the resulting signals for reinforcement learning.

    Source: §1 · p. 2

  • Results

    On WritingBench, ACE-RL-trained LLMs surpass SFT by 18.63% and LLM-as-a-Judge RL by 7.61% on average.

    Source: §5.1 · p. 6

  • Takeaways & Limitations

    ACE-RL offers a verifiable reward-based reinforcement-learning paradigm for long-form generation scenarios.

    Source: §6 · p. 9

Abstract

from arXiv · show

Long-form generation has become a critical and challenging application for Large Language Models (LLMs). Existing studies are limited by their reliance on scarce, high-quality long-form response data and their focus on coarse-grained, general-purpose metrics (e.g., coherence and helpfulness), overlooking the nuanced, scenario-specific requirements of real-world tasks. To address these limitations, we propose a framework utilizing Adaptive Constraint-Enhanced reward for long-form generation Reinforcement Learning (ACE-RL). ACE-RL first decomposes each instruction into a set of fine-grained, adaptive constraint criteria spanning key dimensions of long-form generation tasks. Subsequently, we design a reward mechanism to quantify the response quality based on their satisfaction over corresponding constraints, converting subjective quality evaluation into constraint verification. Finally, we leverage reinforcement learning to optimize LLMs using these fine-grained signals. Experimental results show that ACE-RL significantly outperforms existing SFT and RL baselines by 18.63% and 7.61% on WritingBench, and our top-performing model even surpasses proprietary systems like GPT-4o by 8.76%, providing a more effective training paradigm in long-form generation scenarios.

1 Introduction

Producing coherent, well-structured long-form text remains challenging, while existing approaches often miss instruction-specific quality demands. ACE-RL replaces coarse subjective rewards with adaptive constraint verification and reports gains over SFT, RL, and proprietary-model baselines.

1 Introduction

  • Long-form generation of coherent, well-structured, engaging content over thousands of words remains an unresolved challenge for current long-context LLMs.Source: §1 · p. 1
  • SFT can rely on costly synthetic data and lack quality-attribute optimization, while preference-based RL uses coarse evaluations and extra preference pairs.The paper identifies instruction-specific demands, such as a requested surprise ending, as a quality that general-purpose criteria may miss.Source: §1 · p. 1
  • ACE-RL turns explicit demands and implicit intents into fine-grained constraints on completeness, logic, and style, then uses constraint satisfaction as an RL reward.This converts subjective writing-quality evaluation into constraint verification.Source: §1 · p. 2
  • The work also introduces an automatic pipeline for constructing constraint-checklist datasets and states that the datasets will be released for future studies.Source: §1 · p. 2
  • ACE-RL significantly outperforms SFT and RL baselines and surpasses strong proprietary LLMs in the reported experiments.Source: §1 · p. 2

2 Related Work

Prior long-context research primarily improves handling of long inputs, while long-form output methods rely on coarse rewards and extra preference data. ACE-RL addresses the latter gap with instruction-adaptive constraint verification.

2.1 Long-context Understanding for LLMs

  • Long-context methods redesign architectures, attention, or positional encodings, or use continual pre-training and SFT; these studies mainly target long-context inputs, not high-quality long-form outputs.Source: §2.1 · p. 2

2.2 Long-form Generation for LLMs

  • Long-form output work uses SFT data synthesis or preference-based RL, but its coarse rewards and extra preference pairs miss fine-grained, instruction-specific writing demands.ACE-RL is proposed as a reward mechanism based on instruction-adaptive constraint verification.Source: §2.2 · pp. 2–3, Figure 2 · p. 3

3 Methodology

ACE-RL turns long-form instructions into adaptive, verifiable constraints and uses their satisfaction, alongside length, to guide reinforcement learning. Its construction pipeline prepares real-world instructions, generates constraint checklists, and applies rewards through RL training.

3.1 Data Preparation

  • ACE-RL collects and deduplicates WildChat-1M instructions, filters for long-form tasks, and adds target word counts using Qwen3-235B-A22B.The same model filters instructions and analyzes them to assign target counts.Source: §3.1 · p. 3

3.2 Adaptive Constraint Construction

  • For each open-ended instruction, ACE-RL generates verifiable constraints for content completeness, structural logic, and stylistic formatting.The checklist aims to capture explicit requests and implicit expectations such as logical flow, depth, and completeness.Source: §3.2 · pp. 3–4
  • Human evaluation of generated constraints yielded 82% coverage and 89% precision; instructions averaging above 0.85 verification across eight sampled responses were filtered out.The authors report that simple constraints gave nearly identical high scores across rollouts, making them ineffective for RL training.Source: §3.2 · p. 4

3.3 Reward Design and RL Training

  • GRPO trains the policy by sampling response rollouts and estimating each rollout’s advantage against the group’s average reward.The authors use GRPO for RL training; the supplied passages describe group-based reward comparison and its optimization objective.Source: §3.3 · pp. 4–5
  • ACE-RL averages length and constraint rewards, scoring length against a target and each checklist item with a verifier’s three-level judgment.The verifier assigns 0, 0.5, or 1 for not met, partially met, or fully met, and the constraint reward averages item scores.Source: §3.3 · pp. 5–6

4 Experimental Settings

The experiments train ACE-RL on an instruction dataset with constraint checklists and length requirements, then compare it with language-model, SFT, and judge-based RL baselines. Evaluation uses WritingBench and Arena-Write.

4.1 Training Dataset

  • The training dataset pairs each long-form generation instruction with a constraint checklist and a length requirement.Table 1 presents detailed dataset statistics.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 Training Configuration

  • Training uses VERL with GRPO for 200 steps, a batch size of 64, 32 rollouts, and a learning rate of 1×10−6.The maximum generation length is 8192; Qwen3-8B verifies constraints, and the KL penalty term is removed.Source: §4.2 · p. 6

4.3 Baselines

  • Baselines include leading proprietary, open-source, and writing-enhanced LLMs, LongWriter SFT, and LLM-as-a-Judge RL.The judge-based RL baseline removes pair comparison and uses the same judge model as ACE-RL constraint verification.Source: §4.3 · p. 6

4.4 Evaluation Benchmarks

  • WritingBench evaluates style, format, and length across 1,000 instructions; Arena-Write measures win rates over 595 comparisons against six baseline responses.WritingBench covers six major domains and 100 sub-domains, with a critic LLM reported to agree with human judgments over 80% of the time.Source: §4.4 · p. 6

5 Results and Analysis

Across benchmark and preference evaluations, ACE-RL outperforms the reported SFT and RL alternatives; analyses also test reasoning mode, reward discrimination, data filtering, self-rewarding, and human preference.

5.1 Main Results

  • 21.03%, 18.63%, and 7.61%: ACE-RL improves average WritingBench performance over original models, SFT, and LLM-as-a-Judge RL, respectively.Table 2 covers six domains and three writing requirements.Source: §5.1 · p. 6, Table 2 · p. 7
  • 84.22: Qwen3-8B exceeds Gemini-2.5-pro-preview (83.05), GPT-4o (74.46), and the listed writing-enhanced models on WritingBench.The compared writing-enhanced models are LongWriter-Zero-32B, LongWriter-llama3.1-8B, and LongWriter-glm-9B.Source: §5.1 · p. 7
  • 77.39%: ACE-RL-trained Qwen3-8B wins against six strong baselines on Arena-Write; human evaluation also finds higher win rates than the base model and LLM-as-a-Judge RL.Table 3 reports win, tie, and loss rates; Figure 5 presents the human preference evaluation.Source: §5.1 · p. 7, Table 3 · p. 8, §5.7 · p. 9

5.2 Superiority of RL over SFT

  • Across all model scales, RL-based methods outperform SFT-based methods: ACE-RL-trained Qwen2.5-3B averages 78.97, surpassing the 62.11 score of the larger Qwen2.5-7B model trained with LongWriter SFT.Source: §5.2 · p. 8

5.3 Effectiveness of Test-Time Scaling

  • On Qwen3-8B, the thinking mode—with intent analysis and planning before response generation—performs significantly better than non-thinking mode.Table 4 compares the two modes on WritingBench.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 Effectiveness of Reward Design

  • ACE-RL produces greater within-group reward-value variation than LLM-as-a-Judge, whose rewards cluster more narrowly and distinguish response quality less effectively.The comparison concerns the average group standard deviation of reward value; ACE-RL uses fine-grained constraint verification.Source: Figure 4 · p. 8, §5.4 · p. 9

5.5 Training Data Quality Analysis

  • 3.19%: filtering instructions with high average verification scores improves performance over the unfiltered variant across all six WritingBench domains.The average is based on constraint-verification scores for eight sampled responses per instruction; Table 5 compares results with and without filtering.Source: §5.5 · p. 9, Table 5 · p. 9

5.6 Self-Rewarding Enables Effective Bootstrapping

  • 84.22: in a fully self-bootstrapping setting, Qwen3-8B generates and learns from its own reward signals without larger or proprietary reward models.The authors attribute this capability to constraint verification being simpler than open-ended long-form generation.Source: §5.6 · p. 9

6 Conclusion

The conclusion presents ACE-RL as a constraint-verification reward framework for improving LLM long-form generation.

6 Conclusion

  • ACE-RL measures long-form response quality through instruction-adaptive constraint verification.It replaces coarse-grained subjective evaluation with fine-grained verification tasks.Source: §6 · p. 9
  • The authors report that ACE-RL-trained LLMs outperform other SFT and RL methods and surpass leading proprietary LLMs.Source: §6 · p. 9
  • The paper presents ACE-RL as a pathway toward verifiable reward-based reinforcement learning for long-form generation.Source: §6 · p. 9

A Prompt Templates

The appendix gives prompt templates for filtering long-form instructions, generating and verifying constraint checklists, and estimating response length.

A.1 Prompt for Data Filtering

  • The filtering prompt classifies whether a query is likely to require more than 2,000 words as Yes, Maybe, or No.It asks the model to assess instruction completeness, depth and complexity, scope and breadth, structural needs, and research and synthesis.Source: §A.1 · p. 13

A.2 Prompt for Constraint Generation

  • The checklist-generation prompt analyzes explicit requirements and implicit expectations across content, structure, style, and other quality criteria.It asks for concise, relevant checklist questions, typically two to eight, each focused on one aspect and answerable with a three-level rating.Source: §A.2 · p. 13
  • Checklist questions are to be precise, nonredundant, and phrased for Fully Met, Partially Met, or Not Met ratings.Source: §A.2 · p. 13

A.3 Prompt for Length Requirement

  • The length prompt estimates an acceptable response range from the instruction, checklist, and cumulative informational burden of satisfying its criteria.It first checks for an explicit length request.Source: §A.3 · p. 14
  • Length predictions use words for English instructions and characters for Chinese; bounds are multiples of 100, with a maximum range width of 2,000.The permitted overall bounds are 0 to 10,000.Source: §A.3 · p. 14

A.4 Prompt for Constraint Verification

  • The verifier assesses each response against its instruction and checklist, justifying a Fully Met, Partially Met, or Not Met rating with references to the response and instruction.Source: §A.4 · p. 14
  • The verifier prompt formats the analysis and rating separately for each checklist question, repeating the structure for all questions.Source: §A.4 · p. 14

B Constraint Checklist Examples

The appendix illustrates checklist generation with a literature-review request and a fictional-worldbuilding instruction.

B Constraint Checklist Examples

  • A fictional-setting example asks for worldbuilding about a theocratic dictatorship where birth collars symbolize ownership and removal triggers dehumanization.The generated checklist probes governance, the collar's social meaning, consequences of removal, and the tension between utopia and coercion.Source: B Constraint Checklist Examples · pp. 14–15
  • One example instruction requests an APA-style literature review on road-traffic-accident trauma among males in Riyadh.Source: B Constraint Checklist Examples · p. 15
  • Its checklist covers population and location, review structure, APA citation style, source synthesis, local context, and research gaps or implications.Source: B Constraint Checklist Examples · p. 15

C Checklist Quality Evaluation

The checklist evaluation measures whether generated constraints capture explicit instruction requirements and remain relevant to the instruction. Thirty randomly selected samples were assessed by computer science graduate annotators.

C Checklist Quality Evaluation

D Response Examples

The response examples illustrate plot designs for a modern, logically consistent mystery centered on business crime or family ethics, with no bloody violence. They range from missing-ledger investigations to intricate family and corporate dilemmas.

D Response Examples

  • The example task requests a logical, unexpected modern mystery led by a private investigator, focused on business crime or family ethics and avoiding bloody violence.Source: D Response Examples · p. 15
  • The Qwen2.5-7B-instruct example proposes a missing-ledger scheme tied to stock manipulation, with a twist involving a family member trying to protect the family name.Source: D Response Examples · pp. 15, 17
  • Another plot profile gives Elena Rossi forensic-accounting and family-law expertise, while its design emphasizes psychological tension and intellectual conflict over violence.Source: D Response Examples · pp. 15, 17
  • One plot follows investigator Elena Rossi through a Marconi family ledger disappearance involving suspected financial misconduct, family conflict, and a stock-price drop.Source: D Response Examples · pp. 15–16
  • The Marconi outline uses a red herring, multiple perspectives, foreshadowing, and logical links between clues, motives, and revelations.Source: D Response Examples · p. 16
  • The Qwen3-8B-thinking example builds a Kaneda Tech mystery around a missing ledger, fragmented records, corporate history, and an ethical choice about disclosure.Source: D Response Examples · pp. 15, 17
Loading 2509.04903v3…