PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning with Comparative Evidence for Social Intelligence

Keane Ong, Yuriel Ryan, Sabri Boughorbel, Vladimir Necula, Jack Wei Lun Shi, Rui Mao, Roy Ka-Wei Lee, Adriel Kuek, Nancy F. Chen, Erik Cambria, Gianmarco Mengaldo, Paul Pu Liang

arXiv:2610.04072v1cs.AI

TL;DR

Social understanding relies heavily on human annotations, and ambiguous social cues lack a verification oracle. RLCE compares candidate interpretations using evidence tests against the input; across four benchmarks, it achieves the strongest performance among label-free reward methods, with gains up to +18.93 points over the strongest baseline.

  • Problem

    Social predictions can be ambiguous and lack a verification oracle, while social-understanding models remain heavily reliant on human-annotated training data.

    Source: §1 · pp. 1–2

  • Method

    RLCE compares distinct candidate answers using tests for observable evidence in the sample, aggregates test outcomes, and regenerates tests as policy answers change.

    Source: §1 · p. 2

  • Results

    Across four benchmarks, RLCE achieves the strongest performance among label-free reward methods, with gains over the strongest baseline of up to +18.93 points.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Analyses report that RLCE’s reward variation more often distinguishes correct from incorrect predictions and that its pairwise tests, aggregation, and test evolution benefit verification.

    Source: §1 · p. 2, §5 · p. 9

    Because rewards derive from the model’s own judgments about observable behavior, label-free self-training can entrench base-model priors rather than correct them.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Developing socially intelligent AI remains heavily dependent on human-annotated data, limiting the scale and breadth of social understanding models can acquire. Methods that derive training signals from unlabeled data offer a path beyond this dependence, but social predictions lack the verification oracles available in mathematics and coding. Moreover, core social targets such as affect, intent, preference, and pragmatic meaning are often ambiguous. The same behavior can support multiple plausible interpretations, making it difficult to verify which is best supported. To address this challenge, we introduce Reinforcement Learning with Comparative Evidence (RLCE), a reinforcement learning method that learns social understanding from unlabeled training data without constructing rewards from ground-truth annotations. Given distinct answers in a rollout group, RLCE constructs evidence tests that identify observable evidence favoring an answer over another, validates these tests against the input sample, and aggregates test outcomes to determine the best-supported interpretation. Tests are regenerated as the policy produces new answers, enabling them to evolve with the policy. Across four benchmarks spanning affect, pragmatics, communicative intent, and preference, RLCE attains the strongest performance among seven methods that use no ground-truth training labels for rewards, including consensus, policy LLM-judge verification, multimodal co-evolution, and rubric-based rewards. Gains over the strongest baseline reach up to +18.93 points. Analyses further show that RLCE exhibits a larger share of reward variation between correct and incorrect predictions than compared rubric methods, can overturn erroneous policy-derived preferences, and benefits from pairwise test construction, compositional test aggregation, and on-policy test evolution.

1 INTRODUCTION

The introduction frames social understanding as a label-dependent, ambiguous prediction problem without a verification oracle, then presents RLCE as a way to learn from unlabeled data by comparing observable evidence. It reports the strongest performance among label-free reward methods across four benchmarks.

1 INTRODUCTION

  • Social AI remains reliant on human-annotated data, while ambiguous behavior and the lack of a verification oracle complicate learning from unlabeled examples.The introduction notes that the same observations can support multiple plausible interpretations and that self-generated supervision risks reinforcing errors.Source: §1 · pp. 1–2
  • RLCE compares distinct answers with tests for observable evidence favoring one interpretation, validates the tests against the sample, and rewards the best-supported answer.It regenerates tests as the policy updates and produces new answers.Source: §1 · p. 2
  • RLCE achieves the best mean performance across four benchmarks, with gains over the strongest baseline of up to +18.93 points.The reported gains are +1.15 on IntentQA, +3.58 on MOSEI, +4.50 on UR-FUNNY, and +18.93 on PrefEval.Source: §1 · p. 2
  • Analyses report stronger separation of correct and incorrect predictions in reward variation, more frequent correction of erroneous policy-derived preferences, and benefits from RLCE’s three components.The reported components are pairwise test construction, compositional aggregation, and on-policy test evolution.Source: §1 · p. 2

2 RELATED WORK

Prior social-understanding methods commonly rely on human annotations, while unlabeled-data methods use agreement, self-judgment, or response criteria. RLCE instead compares interpretations against observable evidence in the input.

2 RELATED WORK

  • Unlabeled-data verification has used prediction agreement, policy self-judgment, or explicit response criteria, including fixed and adaptively constructed rubrics.Examples include Self-Consistency (Wang et al., 2022), Universal Self-Consistency (Chen et al., 2023), and Rubrics as Rewards (Gunjal et al., 2026).Source: §2 · p. 3
  • RLCE tests pairwise candidate answers for observable evidence in the input and aggregates test outcomes to resolve among plausible social interpretations.The overview describes evidence-test construction, compositional aggregation, and on-policy evolution.Source: §2 · p. 3, Figure 1 · p. 3
  • Social-understanding systems have largely relied on annotated labels or human-authored reasoning traces, limiting learning from naturally occurring data without training labels.Examples include sentiment and emotion labels in MOSEI and reasoning traces in Social Genome (Mathur et al., 2025).Source: §2 · p. 3

3 REINFORCEMENT LEARNING WITH COMPARATIVE EVIDENCE

RLCE learns from unlabeled social-understanding examples by sampling candidate interpretations, verifying them through comparative evidence tests, and using the selected answer to update the policy.

3.1 PRELIMINARIES

  • The training setting samples policy rollouts for a multimodal sample and question, assigning rewards without access to the training example’s ground-truth answer.A rollout answer represents an interpretation of a social target, and the same observations may support multiple plausible interpretations.Source: §3.1 · p. 4

3.2 VERIFICATION WITH COMPARATIVE EVIDENCE

  • RLCE constructs observable-evidence tests for pairs of distinct candidate answers, with tests checking evidence favoring one interpretation over the other.Semantically equivalent rollout outputs are merged before test construction; the frozen initial policy is used for canonicalizing open-ended outputs.Source: §3.2 · p. 4
  • A validator marks each test satisfied or unsatisfied against the sample, and may abstain when the evidence is insufficient.Abstained tests are excluded from aggregation.Source: §3.2 · p. 5
  • RLCE aggregates satisfied supporting and contradicting tests into a verification score and selects the distinct answer with the highest score.Aggregation can remain accurate when some tests are incorrect, ambiguous, or uninformative if the remaining tests are reliable; unresolved ties can prompt up to two additional tests, with no reward if the tie persists.Source: §3.2 · p. 5

3.3 POLICY UPDATE

  • The verified answer’s rollouts receive reward 1 and other rollouts receive 0 before policy optimization with GRPO.As the policy changes, RLCE regenerates tests for its new answers; the constructor and validator remain frozen.Source: §3.3 · p. 5, §3.2 · p. 5

4 EXPERIMENTS

The experiments evaluate RLCE against seven unlabeled-reward methods across four social-understanding benchmarks. RLCE leads on all four benchmarks, while analyses examine reward signals and evidence-test design.

4 EXPERIMENTS

  • The evaluation covers affect, pragmatics, intent, and preference, comparing RLCE with seven methods that construct rewards from unlabeled training data.Methods include rubric-based rewards, multimodal evolution, self-agreement, and a frozen Policy LLM-Judge; all methods share the same Qwen3.5-9B initialization and use no ground-truth training labels for reward construction.Source: §4 · p. 6

4.1 RESULTS

  • RLCE achieves the highest mean performance across all four benchmarks, with gains over the strongest baseline of +3.58 points on MOSEI, +4.50 on UR-FUNNY, and +18.93 on PrefEval.On IntentQA, the +1.15-point difference is within one standard deviation (58.48 ± 1.59 vs. 57.33 ± 0.50), indicating comparable performance.Source: §4.1 · p. 6
  • In test-time training, RLCE improves over the base model on all four benchmarks, whereas TTRL’s performance changes vary across tasks and rollout group sizes.The reported setup uses G=4 for RLCE and reports TTRL with G=4 and G=16.Source: Table 2 · p. 7, §4.1 · p. 7

4.2 ANALYSIS

  • On MOSEI, roughly 65–85% of RLCE’s reward variation distinguishes correct from incorrect predictions, versus about 30–36% for RLER and RaR.The analysis relates this separation to reward differences that determine relative advantages in GRPO.Source: §4.2 · p. 7, Figure 2 · p. 8
  • When USC selects an incorrect answer, RLCE assigns higher reward to the correct answer more often than RaR and RLER on MOSEI and UR-FUNNY.For Policy LLM-Judge errors, RLCE overturns 43% on UR-FUNNY and 50% on MOSEI; on MOSEI, RaR overturns 19% and RLER 24%.Source: §4.2 · pp. 7–8, Figure 2 · p. 8

4.3 ABLATIONS

  • Evolving evidence tests outperform static tests, with larger performance gains on free-form PrefEval than on closed-label MOSEI.Test performance is 77.77% vs. 46.57% on PrefEval and 71.73% vs. 69.40% on MOSEI for evolving versus static tests.Source: §4.3 · p. 8, Table 3 · p. 8
  • Aggregating more evidence tests raises verdict accuracy from 54% to 69% on MOSEI and from 56% to 66% on UR-FUNNY as tests increase from 1 to 12.With 0–20% incorrect constituent tests, verdicts remain correct for 96% of MOSEI and 100% of UR-FUNNY examples; with 20–40% incorrect tests, accuracy is 87% and 86%, respectively.Source: §4.3 · p. 8, Figure 3 · p. 9
  • Pairwise evidence-test construction outperforms pointwise construction on PrefEval (77.77% vs. 68.51% macro average) and IntentQA (59.61% vs. 56.61%).The ablation uses IntentQA and PrefEval, whose answers allow more distinct rollout-group answers and pairwise comparisons.Source: §4.3 · p. 9

5 CONCLUSION

The conclusion presents RLCE as a label-free learning signal for social understanding, with strong results across four benchmarks and analyses supporting its verification components. The accompanying statements delimit the study’s ethical scope, describe AI-assisted writing, and summarize reproducibility materials.

5 CONCLUSION

  • Pairwise evidence-test construction outperforms pointwise construction on IntentQA and PrefEval in a single training run.PrefEval reports the macro-average across its Choice, Persona, and Explicit settings.Source: Table 4 · p. 9
  • Across four benchmarks spanning affect, pragmatics, intent, and preference, RLCE achieves the strongest performance among the baselines, and analyses support its verification components.The reported analyses find that reward variation more often separates correct from incorrect predictions than compared rubric methods, and identify benefits from pairwise tests, compositional aggregation, and on-policy test evolution.Source: §5 · p. 9

AI USE STATEMENT

  • Generative AI tools were used only to polish writing; authors reviewed the edits and take responsibility for the paper.The statement excludes research ideation or execution, related-work retrieval, synthetic-data generation, and mathematical-claim formulation or proof.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The study warns that its benchmark results do not establish fitness for consequential decisions and that subgroup-disaggregated performance remains unevaluated.The authors state that the benchmarks do not support such subgroup analysis well and consider it necessary before deployment.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The appendix documents datasets, configurations, and baseline reimplementations; code and models are planned for release after review.Main results are reported as means and standard deviations over seeds.Source: REPRODUCIBILITY STATEMENT · p. 10

A DATASETS, BENCHMARKS, EVALUATION SETTINGS

The evaluation covers four social-understanding tasks using shared benchmark inputs and task-specific scoring procedures. The appendix also specifies prompt, answer-extraction, and checkpoint-selection protocols used to compare methods.

Datasets and inputs

  • The evaluation uses MOSEI-Sentiment, UR-FUNNY, IntentQA, and PrefEval, with the first three using HBA splits and PrefEval drawn from its official benchmark.Video-task inputs include sampled frames; text and video content varies by task, and IntentQA has no transcript.Source: A DATASETS, BENCHMARKS, EVALUATION… · p. 13

Evaluation protocol

  • At evaluation, methods receive the same task content and input evidence, while retaining the response-format instruction used during training.For video tasks, the prompt’s video line expands into frame tokens.Source: A DATASETS, BENCHMARKS, EVALUATION… · p. 13, Figure 4 · p. 16
  • Responses are parsed according to each method’s training format, then scored with the same task procedure; unextractable valid answers count as incorrect.The authors report negligible parse failures and argue that prompt-format differences are unlikely to explain method performance differences.Source: A DATASETS, BENCHMARKS, EVALUATION… · pp. 13–14

Checkpoint selection

  • Checkpoints are selected using the best validation score under each task’s metric, with validation labels excluded from training and reward construction.The authors preserve each method’s validation schedule and also report fixed-iteration checkpoints for VisPlay and Video-Zero.Source: A DATASETS, BENCHMARKS, EVALUATION… · p. 14

Task scoring

  • MOSEI-Sentiment is scored by weighted F1 after collapsing non-neutral labels to binary polarity and excluding neutral gold rows; UR-FUNNY uses weighted F1 for binary humor detection.Invalid or missing answers count against the gold class on both tasks.Source: A DATASETS, BENCHMARKS, EVALUATION… · pp. 14–15
  • IntentQA evaluates open-ended video-intent answers with HBA’s LLM judge and scores accuracy over 2,134 rows.The judge receives the question, candidate answer, and reference through two user messages; failed or unparseable verdicts after retries count as incorrect.Source: A DATASETS, BENCHMARKS, EVALUATION… · pp. 15, 18, Figure 6 · p. 18

PrefEval

  • PrefEval tests preference-following across explicit, choice, and persona forms, using a 70/10/20 split of preference–question pairs stratified by topic.The overall score is the macro-average of test accuracies across the three forms.Source: A DATASETS, BENCHMARKS, EVALUATION… · pp. 15–16
  • PrefEval applies four binary judge checks in sequence, distinguishing unhelpful, preference-unaware, hallucinated-preference, and inconsistent responses as error types.Hallucination is checked only after an acknowledged preference quote; a response is correct when it has none of the defined error types.Source: A DATASETS, BENCHMARKS, EVALUATION… · p. 15, Figure 7 · p. 19, Figure 8 · p. 20

B.1 FULL DATASET RESULTS

Table 6 reports results over four benchmarks and three training seeds, with RLCE attaining the highest mean performance across the compared methods.

Full dataset results

  • +1.15 points over Policy LLM-Judge on IntentQA; RLCE has the highest mean performance across four benchmarks, with margins of +3.58 on MOSEI, +4.50 on UR-FUNNY, and +18.93 on PrefEval.The reported margins are modest on IntentQA and larger on the other three benchmarks.Source: Table 6 · p. 23
  • Table 6 reports each task’s mean and standard deviation over three independent seed runs.Qwen3.5-9B is the common training initialization; Qwen3.5-27B and LLaVA-OneVision-2-8B are zero-shot reference models.Source: Table 6 · p. 23

B.2 FULL PREFEVAL RESULTS

Table 7 presents test accuracy for PrefEval’s three forms and their macro-average across three independent training runs.

Full PrefEval results

  • PrefEval’s reported accuracy is judge accuracy on the test split, with means and standard deviations over three independent training runs.The table’s average is the macro-average over the three forms and matches the overall PrefEval value reported elsewhere.Source: Table 7 · p. 23
  • The table separates results by PrefEval form and reports an aggregate macro-average across the forms.The caption identifies the average as the value reported in Tables 6 and 1.Source: Table 7 · p. 23

B.3 TEST-TIME TRAINING

In test-time training, RLCE improves over the base model on all four benchmarks, while TTRL’s results vary and collapse on free-form PrefEval. The comparison uses a shared base model and test-time adaptation protocol.

B.3 TEST-TIME TRAINING

  • RLCE improves over the base model on all four benchmarks in test-time training.Table 8 reports test-time training performance, with Table 9 giving the three PrefEval task settings.Source: §B.3 · p. 23, Table 8 · p. 24, Table 9 · p. 24
  • Methods share the Qwen3.5-9B initialization, test inputs, GRPO optimization, and adaptation-cycle budget, with cycles scaled to test-set size.RLCE uses G=4; TTRL is reported with G=4 and an expanded-budget G=16 setting.Source: §B.3 · p. 23, Table 8 · p. 24
  • TTRL falls below the base model in some settings and collapses on free-form PrefEval, more severely with the larger rollout group.The authors note this is consistent with consensus rewards reinforcing a wrong majority or a degenerate mode when free-form answers rarely coincide; RLCE may avoid this by ignoring rollout frequency.Source: §B.3 · p. 24

C RLCE IMPLEMENTATION DETAILS

This appendix section outlines the RLCE implementation details covered in the following subsections.

C RLCE IMPLEMENTATION DETAILS

C.1 IMPLEMENTATION DETAILS

RLCE uses a frozen verifier alongside a fully fine-tuned policy, constructs and validates comparative evidence tests, and optimizes the policy with GRPO.

C.1 IMPLEMENTATION DETAILS

  • The trainable policy is fully fine-tuned, while a fixed verifier handles free-text canonicalization, evidence-test construction, and validation without stronger external models or ground-truth reward annotations.Closed-label outputs map directly to answer options; free-text candidate pairs are checked for semantic equivalence before verification.Source: §C.1 · p. 25
  • Evidence-test construction uses temperature 1.0, top-p 0.95, and top-k 20; candidate answers are removed before validation against the sample.Initial test counts vary by task: one per direction for IntentQA and PrefEval, three for MOSEI-S, and six for UR-FUNNY.Source: §C.1 · p. 25

C.2 REWARD AND GRPO OPTIMIZATION

  • The reward includes a verified-answer match and a structural format reward, which encourages response-schema compliance but carries no answer-correctness information.When a rollout group has no reward variation, it contributes no relative policy-gradient signal.Source: §C.2 · p. 25
  • GRPO uses a per-token importance ratio between the current and rollout policies, with a KL penalty of β=0.01 against the frozen base model.The KL term is applied in the policy loss rather than in the rollout reward; training uses four rollouts per prompt and a learning rate of 1 × 10−6.Source: §C.2 · pp. 25–26

C.3 RLCE PROMPTS

  • A fixed construction prompt takes the sample, question, and candidate answers to generate checkable comparative tests, while a validator labels each test yes, no, or unknown against the sample.Source: §C.3 · p. 26, Figure 11 · p. 26, Figure 12 · p. 27
  • The policy uses a solver prompt, and a frozen verifier performs remaining free-text candidate-pair equivalence decisions in one greedy call.The referenced figures show the solver and answer-canonicalization prompts.Source: §C.3 · p. 26, Figure 13 · p. 27, Figure 14 · p. 27

D BASELINE IMPLEMENTATION

The baselines use unlabeled-data reward signals, with USC relying on rollout agreement and Policy LLM-Judge relying on pointwise frozen-policy judgments.

D BASELINE IMPLEMENTATION

  • The baseline suite adapts five current methods, including consensus, rubric-based, adversarial-critic, and policy-judge approaches, for comparison on unlabeled training data.The reported baseline performance is the mean and standard deviation over three runs.Source: D BASELINE IMPLEMENTATION · p. 28

D.1 UNIVERSAL SELF-CONSISTENCY

  • USC rewards answers that agree with the rollout-group consensus, selecting the unique most frequent closed-set answer and withholding reward when the top vote is tied.For free-form answers, a frozen LLM selects the response most consistent with the majority, using only the question and candidate responses.Source: §D.1 · p. 28, Figure 15 · p. 28

D.2 POLICY LLM-JUDGE

  • Policy LLM-Judge evaluates each candidate independently using the question, available input evidence, and one proposed answer, assigning reward 1 to a parsed “Yes” and 0 otherwise.For closed-set tasks the judge does not see the other answer options; duplicate free-form responses are deduplicated before evaluation.Source: §D.2 · p. 29, Figure 16 · p. 29
  • Policy LLM-Judge uses a frozen base-model copy, receives no ground-truth training annotations, and differs from RLCE in its pointwise judgment reward rather than comparative evidence testing.The USC and Policy LLM-Judge baselines otherwise use the same RL training setup as RLCE.Source: §D.1 · p. 28, §D.2 · p. 29

D.3 EVOLVING RUBRICS (RLER) & STATIC RUBRICS AS REWARDS (RAR)

This section describes rubric-based reward and multimodal co-evolution baselines for training from unlabeled inputs. It outlines their reward, criteria, pseudo-labeling, and filtering mechanisms, along with compute costs relative to RLCE.

D.3 EVOLVING RUBRICS (RLER) & STATIC RUBRICS AS REWARDS (RAR)

  • Rubrics carry +1 or −1 weights for positive or negative traits, and evolving rubrics are pruned when they do not distinguish rollouts.The active set keeps at most five criteria with the highest score standard deviation; persistent seed rubrics are exempt.Source: §D.3 · pp. 29, 33
  • Evolving Rubrics generates active criteria from each clip’s eight policy rollouts, while Static Rubrics scores only persistent seed rubrics throughout training.Both pipelines use GRPO, and neither uses gold labels.Source: §D.3 · pp. 30–31
  • The evolving generator is instructed to avoid mirror rubrics and define negative rubrics as explicit failure behaviors, preventing duplicated or misleading reward criteria.A negative rubric phrased as a prohibition could penalize good responses that comply with it.Source: §D.3 · pp. 31–32
  • Rewards combine rubric scores with a format check, using an 0.8/0.2 split, before group-relative policy optimization.The text-only judge scores each response against one rubric on a 0-to-2 scale, rescaled to 0, 0.5, or 1.Source: §D.3 · pp. 31, 33
  • Conflicting or near-redundant criteria can cancel reward differences and dilute relative advantages, a failure mode the paper calls criterion collapse.Figures 17–18 illustrate conflicting criteria and their reward effects.Source: §D.3 · pp. 33–34, Figure 18 · p. 35, Figure 17 · p. 35
  • The authors suggest short label-and-evidence responses offer fewer useful rubric dimensions, making evolving criteria more prone to overlap or conflict; static criteria can retain flaws from seeding.These limitations motivate RLCE’s use of evidence derived locally from the input and current candidate answers.Source: §D.3 · p. 34

D.4 VISPLAY

  • VisPlay’s Reasoner reward assigns 0.9 to matching the pseudo-label and 0.1 to correctly formatting the boxed answer.The pseudo-label is the frozen Reasoner’s majority answer over repeated candidate-question responses; retained candidates meet a confidence window of 0.3–0.8 and validity checks.Source: §D.4 · p. 38
  • VisPlay preserves released settings where possible, uses 30 prompts per step to cover 600 clips over 20 steps, and costs 26–44 GPU hours per run, three to six times RLCE.The implementation favors VisPlay by sampling Questioner clips from the full training split each iteration and selecting the best checkpoint after validation on the full validation split.Source: §D.4 · p. 39

D.5 VIDEO-ZERO

  • Video-Zero alternates Questioner and Solver training over up to five iterations, generating questions from unlabeled clips and using the frozen Solver to pseudo-label them without gold labels.For transcript-only tasks, video-dependent components are disabled.Source: §D.5 · p. 40
  • Video-Zero scores Questioner rollouts using format, learnability, question dependency on the clip, and evidence quality terms.Learnability peaks at Solver self-consistency c = 0.5; dependency compares with-clip and question-only answers, while evidence rewards are capped at 0.3.Source: §D.5 · pp. 40, 42
  • Video-Zero retains open-ended questions when pseudo-label confidence is 0.3–0.8; closed-space runs add a K-aware lower bound and reject weak plurality margins.For closed spaces, the lower bound is min(0.6, 1/K + 0.2), and the top answer must lead the runner-up by at least two votes.Source: §D.5 · pp. 42–43
  • Video-Zero Solver rollouts with incorrect answers score zero, while correct answers receive a base reward increased by alignment with the consensus target span.Predicted and target spans outside the clip or with end ≤ start receive no alignment reward.Source: §D.5 · p. 43
  • The Video-Zero implementation follows the paper’s five-iteration, 600-clip recipe, gives both roles transcripts, and selects checkpoints using validation rather than evaluating only iteration 4.The authors’ scaling ablation reported near-identical results for 600, 1000, and 2000 clips; a run takes 28–40 GPU hours, three to five times RLCE’s GPU-hours.Source: §D.5 · p. 44

D.6 RLAC

RLAC adapts adversarial co-evolution to social understanding, using a Generator, Critic, and frozen Validator because social tasks lack the original method’s verification sources. Its alternating training pipeline uses no gold labels.

D.6 RLAC

  • RLAC adapts a three-role adversarial co-evolution method to social tasks, where Wikipedia and executable test cases are unavailable as verification sources.The Generator and Critic are trained, while the Validator remains frozen.Source: §D.6 · p. 44
  • Each round alternates Generator and Critic GRPO phases, harvesting training pairs between them without including gold labels in the pipeline.The Critic and Validator are frozen during Generator training; the Generator and Validator are frozen during Critic training.Source: §D.6 · p. 44
  • A frozen meta-verifier checks that the Critic’s extracted claim originates in the response before the Validator judges support from the clip and transcript.The Validator does not see the source response or learn that the claim came from the Critic.Source: §D.6 · pp. 44–45
  • Capping Critic training at one harvested pair per clip gives both players 18 optimizer steps per round and keeps compute comparable across methods.The cap trades training on all Generator outputs for a sample of them.Source: §D.6 · p. 46
Loading 2610.04072v1…