Source-linked AI summary
Co-Evolving Actor-Conditioned Critics for Non-Verifiable Generation
Jinyoung Kim, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, Lu Wang
TL;DR
Non-verifiable generation makes critique difficult to credit because final improvement does not show whether feedback was followed or caused the intended change. The paper introduces TAISCORE, which evaluates the complete critique-guided revision for a target actor, and uses it in a co-evolving critic-actor loop. TAISCORE-trained critics and co-evolution outperform the stated comparison systems, while broader domains and model families remain to be tested.
Problem
Non-verifiable generation lacks deterministic success signals, and final revision quality does not establish whether a critique was followed or improved the intended aspect.
Method
TAISCORE evaluates the instruction, initial response, critique, and revision together, training an actor-tailored critic while critique-guided refinements form actor preference pairs for co-evolution.
Results
An 8B critic trained with TAISCORE outperforms a frozen gpt-oss-120B critic and outcome-only or critique-only critics, while co-evolution further improves performance.
Takeaways & Limitations
Effective critique supervision should be evaluated through the revision behavior it induces in the target actor and adapted as that actor changes.
Takeaways & Limitations
Evaluation covers creative writing and deep research, training focuses on Qwen-family models, and co-evolution uses a fixed number of rounds over disjoint data shards.
Abstract
from arXiv · showhide
Natural-language critiques provide supervision beyond scalar rewards for non-verifiable generation, which lacks deterministic verifiers. In critique-guided refinement, a critic gives feedback on an initial response and an actor revises it. However, final revision quality does not reveal whether the critique was actually useful: a capable actor may improve without following the feedback, while valid feedback may fail if the actor cannot execute it. We frame critique as actor-conditioned revision guidance, where usefulness depends on whether the feedback helps the target actor address the intended weakness. We introduce TAIScore (Targeted Actionable Improvement Score), a reward that evaluates the instruction, initial response, critique, and revision together, assessing whether the critique targets a real weakness, whether the actor follows it, and whether the intended aspect improves. We use this reward to train an actor-tailored critic with GRPO, and use critique-guided refinements to construct DPO preference pairs for the actor, forming a co-evolving critic-actor loop where the critic adapts to the actor's changing capability. Experiments show that an 8B critic trained with TAIScore outperforms both a zero-shot 120B critic and critics trained with outcome-only or critique-only reward signals. Co-evolving the critic and actor further improves performance, suggesting that effective critique supervision should adapt as the actor changes.
1 Introduction
Non-verifiable generation lacks deterministic success signals, making actionable critique important but difficult to evaluate. The paper frames critique usefulness as actor-conditioned, introduces TAISCORE to assess the full revision process, and reports gains from TAISCORE-trained and co-evolving systems.
- Open-ended generation lacks deterministic verifiers, while scalar scores rank outputs without identifying which dimension failed or what edit would help.
- Critique usefulness depends on whether the target actor can execute feedback and improve the intended aspect, not on critique quality alone.
- TAISCORE evaluates the instruction, initial response, critique, and revision together to reward valid issue identification, actor adherence, and targeted improvement.
- An 8B TAISCORE-trained critic outperforms a frozen gpt-oss-120B critic and critics trained with only refinement quality or critique quality.
- Co-evolution improves the Qwen3-8B actor from 72.33 to 76.72 on WritingBench, with the same trend on HelloBench and DeepResearch-Gym.
2 Related Work
Prior work uses natural-language feedback, learned evaluators, rubrics, and dynamic critics to improve non-verifiable generation. The paper distinguishes its approach by tying critic supervision to the revision behavior of a target actor.
- Self-refinement methods use natural-language feedback for iterative revision, including self-critique, verbal memory, and training from feedback at scale.
- RLHF and LLM-as-a-Judge provide learned or prompted evaluation for open-ended tasks, while rubric-based methods structure criteria into scalar optimization signals.
- Dynamic-critic approaches co-evolve supervision with policy changes or connect critic training to actor refinement behavior in interactive environments.
- This paper focuses on whether feedback helps a specific actor make the intended revision, rather than evaluating critique or final quality in isolation.
3 Motivation and Problem Formulation
The motivation analysis tests whether critique usefulness is intrinsic to feedback or conditioned on the actor executing it. Controlled comparisons show that actor capability predicts adherence and gains more reliably than critic scale alone.
- 3 Motivation and Problem Formulation: Critique-guided revision contains an instruction x, initial response y0, critique c, and revised response y1, with usefulness defined by the actor’s intended revision.
- 3.1 Motivating Analysis: Critique Usefulness Is Actor-Conditioned: The controlled study varies critic size while holding the actor fixed, or varies refiner size while holding the critique fixed.
- 3.1 Motivating Analysis: Critique Usefulness Is Actor-Conditioned: Figure 1 compares larger and smaller critics or refiners using gray and blue bars, respectively, across these three quantities.
- 3.1 Motivating Analysis: Critique Usefulness Is Actor-Conditioned: The analysis measures standalone critique quality, critique adherence, and downstream gain S(y1)−S(y0).
- 3.1 Motivating Analysis: Critique Usefulness Is Actor-Conditioned: Larger critics improve standalone critique-quality scores but do not reliably improve adherence or downstream refinement gains for a fixed actor.
- 3.1 Motivating Analysis: Critique Usefulness Is Actor-Conditioned: Larger actors show stronger adherence and larger refinement gains when given identical critiques, establishing actor-conditioned usefulness.
- 3.2 Actor-Conditioned Critic Training: These findings motivate rewarding critiques that identify real weaknesses, are followed by the target actor, and improve the targeted aspect.
4 Method
The method jointly adapts a critic to the current actor using TAISCORE-based GRPO and trains the actor on preferred critique-guided revisions with DPO. Alternating these updates keeps critique guidance aligned with the actor’s evolving weaknesses and capabilities.
- 4.1 Targeted Actionable Improvement Score: TAISCORE evaluates the instruction, initial response, critique, and revision together to assess validity, adherence, targeted gain, and faithfulness.Its final score serves as the reward for critic adaptation.
- 4.2 Critic Update: For each on-policy initial response, the critic samples multiple critiques, the actor produces corresponding revisions, and each rollout receives a TAISCORE reward.Using the same initial response and actor across critiques enables relative comparison of revision guidance.
- 4.2 Critic Update: GRPO updates the critic with group-relative advantages so it assigns higher probability to critiques that provide more useful guidance for the current actor.The diagnostic scores support analysis, while the final TAISCORE is the training reward.
- 4.3 Actor Update from Critique-Guided Refinements: Using the adapted critic, selected revisions that improve over their initial responses form preference pairs, with the revision preferred over the initial response.These pairs constitute the preference dataset used for actor optimization.
- 4.3 Actor Update from Critique-Guided Refinements: DPO initializes from the current actor and encourages it to produce responses closer to those generated after useful critique.The reference policy is a frozen copy of the actor before the update.
- 4.4 Co-evolution: Alternating critic adaptation and actor updates produces a sequence in which the critic tracks the actor’s changing weaknesses and revision capabilities.The critic and actor therefore co-evolve across training rounds rather than remaining static.
5 Experiments
Experiments test TAISCORE against alternative critic-training signals, co-evolution, and actor-critic matching across creative writing and deep research benchmarks. Results consistently favor actor-conditioned and adaptive critique supervision.
- Setup: Experiments evaluate TAISCORE against outcome-only and critique-only rewards, static versus co-evolving critics, and critics tailored to different actors.The study covers creative writing and deep research, using WritingBench, HelloBench, and DeepResearch-Gym with benchmark-specific judges and metrics.
- Setup: Table 1 compares an initial Qwen3-8B actor with off-the-shelf and trained critics, whose critiques construct DPO preference pairs.The trained critics use different reward definitions, while co-evolution updates the critic as the actor changes.
- Main Results: TAISCORE-trained critique supervision outperforms the frozen gpt-oss-120B critic and both reward ablations on all six reported metrics.Outcome-gain rewards final revision quality without establishing critique attribution, whereas critique-quality rewards plausibility without testing actor execution.
- Main Results: Co-evolving the critic and actor further improves performance across all benchmarks, supporting adaptation to changing revision capability.Blind human evaluation on 50 WritingBench prompts also prefers co-evolution over single-round TAISCORE and TAISCORE over the base actor and off-the-shelf critic.
- Direct Refinement: 72.33 to 75.11 (+2.78) is the WritingBench gain from the TAISCORE-trained critic during direct refinement before actor DPO.All conditions share prompts, initial responses, refiner, decoding, and evaluator; only feedback varies, and all revisions are retained.
- Actor Matching: 3.63 points is the largest gain from the critic tailored to the target Qwen3-8B actor, exceeding Qwen3-4B- and Llama-3.2-3B-tailored critics by 0.88 and 1.34 points.Critique behavior transfers across actors, but matching the critic to the target actor produces the greatest downstream improvement.
6 Conclusion
The paper presents critique-guided revision as actor-conditioned and introduces TAISCORE to reward feedback that produces targeted, adopted improvements. Across two domains, adaptive critic training further improves results.
- Conclusion: TAISCORE rewards critiques that identify real weaknesses, are followed by the target actor, and cause targeted improvements.It evaluates critique-guided revision through the interaction among instruction, initial response, critique, and revision.
- Conclusion: Across creative writing and deep research, TAISCORE outperforms outcome-gain and critique-quality rewards, while critic adaptation as the actor changes yields further gains.The conclusion supports evaluating critique supervision through the revision behavior it induces in the target actor.
Limitations
The evaluation covers two non-verifiable generation domains and focuses main training experiments on Qwen-family models. The co-evolution procedure also uses fixed rounds over disjoint data shards.
- Scope: The study evaluates actor-tailored critique training only on creative writing and deep research, leaving broader domains for future validation.Suggested extensions include dialogue, multimodal generation, and long-form instruction following.
- Model Scope: Main training experiments use Qwen-family models, while the controlled zero-shot analysis includes Qwen and Llama models.The authors report similar actor-conditioned trends in the controlled cross-family analysis but call for broader training validation.
- Training Procedure: Co-evolution uses a fixed number of rounds over disjoint data shards rather than adaptive update schedules or continuous feedback reuse.The authors identify adaptive scheduling and continuous co-training as future directions.
Ethical Considerations
The study uses licensed public resources, a small anonymized human evaluation, and filtering for unsafe or privacy-sensitive prompts. The authors also warn that improving generation can amplify undesirable outputs.
- Data and Evaluation: The evaluation uses publicly available artifacts under their licenses, with research-oriented use and citations to original creators.The authors state that produced artifacts are intended for research use only.
- Human Evaluation: Human evaluation collects only pairwise preferences from anonymized model outputs and does not solicit sensitive personal information.Query filtering excludes unsafe examples and prompts requesting private or personally identifying information.
- Risks: Underlying-model pretraining data may contribute biases or harmful patterns to generated outputs.The authors frame these risks as relevant to systems that improve language-model generation quality generally.
- Risks: Critique-guided revision may improve the fluency, persuasiveness, or apparent helpfulness of undesirable outputs without safeguards.The authors recommend safety filtering, monitoring, and domain-specific safeguards for downstream use.
A Details of the Zero-Shot Controlled Analysis
The controlled analysis isolates critic scale and actor capability by holding the other components fixed, using standardized critique, adherence, and downstream gain metrics.
- The analysis uses the same 1,000 official WritingBench examples for all controlled comparisons.
- Critic scaling fixes the instruction, initial response, and revising actor, isolating the effect of feedback from smaller versus larger critics.
- Actor scaling fixes the instruction, initial response, and critique, isolating the effect of smaller versus larger actor capability.
- Critique quality observes only the instruction, initial response, and critique, whereas critique adherence observes the full critique-guided revision context.
- Gain is computed as S(y1) − S(y0), where S is the official WritingBench evaluator score.
- Table 5 reports larger-model minus smaller-model deltas for both critic-scaling and actor-scaling comparisons.
B Qualitative Analysis of Critic–Actor Reachability
The qualitative analysis shows that standalone critique quality can diverge from actor-executable usefulness: broader feedback may be less reachable for a fixed refiner.
- 9.00 versus 8.33: the Qwen3-32B critique scores higher in standalone quality but produces 3.50 versus 8.75 adherence from the same Qwen3-8B refiner.
- Localized requests to introduce a rival and strengthen the mentor’s role are largely implemented by the Qwen3-8B refiner.
- The Qwen3-32B critique proposes coordinated changes across symbolism, motivations, climax, ending, and dialogue, but the refiner adopts only isolated elements.
- The study selects four English-language cases where the larger critic scores higher in standalone quality but lower in adherence for a fixed Qwen3-8B refiner.
- Across paired cases, larger-critic revisions implement isolated suggestions while failing constraints requiring coordinated changes or unavailable information.
- TAISCORE evaluates the instruction, initial response, critique, and revision together for weakness targeting, feedback incorporation, targeted improvement, and instruction faithfulness.
D.6 Critique-Content Controls
The critique-content controls separate generic revision effects from the value of correctly matched, instance-specific TAISCORE feedback.
- All four conditions use identical prompts, initial responses, reviser, revision prompt, decoding, and evaluator, differing only in supplied feedback.
- The no-critique condition requests improvement without providing feedback.
- The generic-critique condition gives every example the same broad request covering correctness, clarity, relevance, and overall quality.
- The shuffled condition preserves TAISCORE critic source and style while randomly breaking correspondence between critiques and instruction–response pairs.
- The TAISCORE critique condition supplies each example with its own model-generated, correctly matched critique.
- Together, the controls distinguish an extra revision pass, a general improvement request, and TAISCORE feedback with instance-specific relevance.
E Human Evaluation Protocol
The human evaluation protocol compares key system conditions through anonymized pairwise judgments, while the surrounding tables document the controlled-analysis metrics, prompts, and training settings.
- The evaluation compares TAISCORE with the base actor, the frozen gpt-oss-120B critic, and single-stage TAISCORE against co-evolution.
- 50 WritingBench prompts produce 150 response pairs and 450 individual judgments across the three comparisons.
- Three anonymized annotators independently choose the response that better fulfills the instruction overall or select a tie.
- Per-prompt outcomes use the choice selected by at least two annotators as the majority decision.
- Table 5 defines critique quality, critique adherence, and Gain, and specifies critic-scaling and refiner-scaling blocks with larger-minus-smaller deltas.
- Tables 6–9 provide prompts for critique generation, revision generation, critique-quality judging, and critique-adherence judging.
- Tables 10–14 document paired cases, query filtering, training hyperparameters, outcome-gain rewards, and TAISCORE reward prompts.