PaperlayerView exact revision on arXiv

Source-linked AI summary

Reason in the Words You Speak: Idiolectal Paraphrasing Off-Policy Traces for Reasoning Distillation in VideoLLMs

Ji Soo Lee, Jinyoung Park, Seohyun Lee, Jongha Kim, Joonmyung Choi, Jinsung Yoon, Hyunwoo J. Kim

arXiv:2608.26684v1cs.CV

TL;DR

The paper addresses the difficulty of learning from stronger teachers when their reasoning traces fall outside the student policy’s distribution and trigger clipping on essential tokens. Echo-GRPO paraphrases those traces into the student’s idiolect using Dual-Reference Decoding, yielding consistent improvements for VideoEcho-R1 across backbones and benchmarks and extending to reinforcement learning and supervised fine-tuning.

  • Problem

    Privileged teacher traces are out of distribution for the student policy, so trust-region clipping can suppress semantically critical reasoning-token updates.

  • Method

    Echo-GRPO rewrites privileged traces into the student policy’s idiolect while preserving semantics and distribution alignment through Dual-Reference Decoding.

  • Results

    VideoEcho-R1 achieves consistent reasoning-distillation improvements across three multimodal LLM backbones and five benchmarks, while idiolectal rewriting improves reinforcement learning and supervised fine-tuning frameworks.

  • Takeaways & Limitations

    Policy-aligned supervision extends beyond vanilla GRPO as a plug-in module for reasoning distillation.

  • Takeaways & Limitations

    Echo-GRPO adds preprocessing overhead, depends on teacher reasoning quality, may yield smaller gains on fine-grained numerical tasks, and faces potential pretraining–benchmark overlap.

Abstract

from arXiv · show

Recent large language models achieve strong performance on complex reasoning tasks, where reinforcement learning with Group Relative Policy Optimization (GRPO) has emerged as a leading paradigm for optimizing models on self-generated trajectories. However, the on-policy nature of GRPO bounds the model to the reasoning skills it can already produce, restricting to learn more advanced capabilities. Prior works inject privileged reasoning traces from a stronger teacher policy to guide training, yet these traces are inherently out of distribution with respect to the student policy. We observe that this mismatch between on-policy and off-policy causes gradient clipping on semantically critical reasoning tokens, ultimately rewarding correct answers while leaving the reasoning that justifies them unlearned. Hence, we propose \textbf{Echo-GRPO}, a framework that lets the model reason in the words it speaks. Rather than imitating low-probability privileged traces from the teacher model, Echo-GRPO rewrites them into the student policy's own \textit{idiolect}, that is, its own characteristic vocabulary and expression patterns, while preserving their semantics via Dual-Reference Decoding. We instantiate this framework as \textbf{VideoEcho-R1} for video reasoning distillation, achieving consistent improvements across three multimodal LLM backbones and five benchmarks. Finally, we show that our idiolectal paraphrasing is a plug-in module that consistently improves both RL and supervised fine-tuning frameworks for reasoning distillation, demonstrating that policy-aligned supervision extends beyond GRPO.

1 Introduction

The paper identifies a mismatch between privileged teacher traces and the student policy as a source of clipped, semantically important reasoning updates. Echo-GRPO rewrites those traces into the student’s idiolect while preserving semantics, and VideoEcho-R1 improves reasoning distillation across backbones, benchmarks, and training frameworks.

  • On-policy GRPO is limited to the reasoning skills already produced by the current policy, motivating privileged teacher traces in Mixed-Policy GRPO.
  • Figure 1 illustrates the alignment using “school of fish” rewritten as “several fish,” keeping the meaning while placing tokens within the policy’s distribution.
  • Out-of-distribution privileged traces trigger trust-region clipping on low-likelihood but semantically essential tokens, rewarding correct answers without learning their justification.
  • VideoEcho-R1 achieves consistent improvements across three multimodal LLM backbones and five video reasoning benchmarks.
  • Echo-GRPO paraphrases privileged reasoning into the student’s characteristic vocabulary and expression patterns while preserving semantics through Dual-Reference Decoding.
  • Idiolectal paraphrasing consistently improves reasoning distillation in both reinforcement learning and supervised fine-tuning frameworks.

2 Related Works

Related work develops reinforcement-learning methods for language and multimodal reasoning, including teacher-supervised approaches that address the capability ceiling of on-policy GRPO. Video reasoning systems extend these ideas across visual tasks and temporal modeling.

  • Reinforcement Learning in LLM Reasoning: GRPO is a widely adopted on-policy paradigm for advancing LLM reasoning, with later work refining trust-region behavior, length bias, and KL regularization.
  • Reinforcement Learning in LLM Reasoning: Mixed-Policy GRPO and related methods inject privileged supervision from stronger teachers to overcome the capability ceiling of on-policy training.
  • Reasoning in Multimodal Large Language Models: Multimodal reasoning research extends reinforcement-learning approaches to image and video tasks, including temporal-aware methods and video-specific R1 systems.

3 Method

The method frames Mixed-Policy GRPO as distilling from both on-policy samples and a privileged teacher trace, then addresses the resulting clipping of essential reasoning tokens. Echo-GRPO replaces the privileged trace with an idiolectal rewrite generated by Dual-Reference Decoding, preserving semantics while aligning tokens with the policy’s native distribution.

  • Reasoning Distillation with GRPO: Mixed-Policy GRPO combines current-policy reasoning traces with one privileged trace from a separate teacher policy.The objective uses grouped candidate samples and an off-policy teacher trajectory as supervision.
  • Clipping-Induced Suppression of Essential Reasoning: Off-policy teacher traces create unstable importance ratios because the student is unlikely to generate their low-probability tokens.Ratios outside the trust region cause the corresponding gradient updates to be discarded.
  • Clipping-Induced Suppression of Essential Reasoning: Semantically important tokens can be clipped, allowing a trajectory to receive reward for a correct answer while suppressing gradients that teach why it is correct.The visualization highlights the key referent “robot” as an example of a clipped reasoning token.
  • Idiolectally Paraphrasing Privileged Traces: Echo-GRPO rewrites the privileged trace into an idiolectally paraphrased trajectory that preserves its semantics while matching the policy’s native distribution.The rewritten trajectory is intended to satisfy both semantic coherence and distribution alignment.
  • Idiolectally Paraphrasing Privileged Traces: Dual-Reference Decoding multiplies a semantic-reference distribution with a distributional-reference distribution to favor tokens supported by both.The semantic reference conditions on the privileged trace and supplies top-k candidates, while the distributional reference evaluates the policy without privileged conditioning.

4 Experiments

Experiments evaluate VideoEcho-R1 across three multimodal backbones and five video benchmarks, comparing Echo-GRPO with general training and reasoning-distillation baselines. Results show consistent gains, reduced clipping, stronger training dynamics and generalization, and benefits from idiolectal paraphrasing as a plug-in.

  • Main results: VideoEcho-R1 outperforms on average across backbones and benchmarks, reaching 57.2 on Qwen3-VL-4B, 52.7 on InternVL3.5-4B, and 58.2 on Qwen3-VL-8B.On Qwen3-VL-4B, it surpasses vanilla GRPO by 1.3 points and SFT→GRPO by 3.2 points; corresponding gains on InternVL3.5-4B are 1.8 and 3.5 points.
  • Main results: Mixed-Policy GRPO performs worse than vanilla GRPO across benchmarks, with drops of up to 16.4, 2.4, and 4.6 points on the three backbones.The results support the reported failure mode in which naive off-policy trace injection hinders reasoning distillation.
  • Reasoning-distillation comparisons: Echo-GRPO achieves the strongest reasoning-distillation performance, averaging 57.2 across five benchmarks and surpassing RL w/ SFT, LUFFY, OPSD-JSD, and OPSD-IKL by 6.8, 5.3, 4.5, and 5.4 points.The comparison covers reasoning-distillation frameworks under the Qwen3-VL-4B evaluation setting.
  • Plug-in evaluation: Idiolectal paraphrasing improves existing frameworks: RL w/ SFT rises from 50.4 to 53.0, while LUFFY rises from 51.9 to 52.2.The gains support its use as a plug-in beyond the GRPO objective.
  • Training analysis: Echo-GRPO reduces the share of semantically important tokens among clipped tokens from 71.8% to 67.5%, a reduction of 4.0 percentage points.This addresses clipping that suppresses updates on essential reasoning components.
  • Training dynamics and generalization: Echo-GRPO ultimately surpasses baselines while sustaining increases in accuracy and reward and preserving confidence; plug-in variants similarly improve confidence, accuracy, and reward throughout training.Echo-GRPO performs best in both in-distribution and out-of-distribution regimes, with scores of 78.3 and 57.2.
  • Rewriting ablations: DRD reaches 58.2 average performance, exceeding prompt-only student rewriting by 2.2 points and semantic-only guidance by 0.6, 2.1, and 1.4 points across three benchmarks.Semantic guidance outperforms distributional guidance, while combining both references performs best.
  • Beyond reinforcement learning: Idiolectal paraphrasing also improves SFT from 50.6 to 52.6 and SFT→GRPO from 55.9 to 58.8 on average.These results indicate effectiveness across different training objectives.

5 Conclusion

The paper identifies trust-region clipping of semantically critical tokens as a mixed-policy GRPO failure mode and proposes Echo-GRPO to rewrite privileged reasoning into the model’s native distribution. VideoEcho-R1 improves across backbones and benchmarks, while the rewriting module also generalizes to reinforcement learning and supervised fine-tuning.

  • Conclusion: Echo-GRPO rewrites privileged reasoning into the model’s native distribution via Dual-Reference Decoding.VideoEcho-R1 achieves consistent improvements across multiple backbones and benchmarks, and the module generalizes to reinforcement learning and supervised fine-tuning.

A Experimental Details

The experiments use three multimodal backbones, uniformly sampled video frames, fixed rollout and optimization settings, and a 2.4K-example training set. Baselines include SFT, GRPO, two-stage SFT→GRPO, Mixed-Policy GRPO, RL w/ SFT, LUFFY, and OPSD.

  • Implementation: The study evaluates InternVL3.5-4B, Qwen3-VL-4B, and Qwen3-VL-8B using uniformly sampled 8-frame video inputs.Qwen3-VL-4B is used for ablations unless otherwise noted.
  • Data and rollout protocol: Training uses 2.4K OneThinker-SFT-340K instances, six rollouts per sample, and one privileged-trace replacement when distillation is applied.The privileged traces are used only during training, not inference; DRD uses k=5.
  • Paraphrasing procedure: The idiolectal paraphrasing prompt prevents outputs from explicitly citing or copying the privileged trace’s structure.Without this instruction, the model tends to produce phrases such as references to the supplied trace rather than genuinely paraphrasing its semantics.
  • Baselines: GRPO is an on-policy reinforcement-learning algorithm using groupwise normalized advantages over multiple sampled rollouts and reward-weighted likelihood ratios.It does not rely on explicit supervision.
  • Baselines: SFT→GRPO first initializes the model with supervised fine-tuning and then optimizes it with GRPO, using 0.8K samples for initialization.The remaining data is used for GRPO training.
  • Baselines: Mixed-Policy GRPO replaces one sampled rollout with a ground-truth reasoning trace while retaining current-policy rollouts.This introduces a controlled off-policy signal into GRPO.
  • Baselines: RL w/ SFT jointly optimizes supervised and reinforcement-learning objectives, training privileged reasoning with SFT and other rollouts with RL.The hybrid objective combines explicit supervision with reward-driven exploration.
  • Baselines: LUFFY combines external reasoning traces with on-policy rollouts and uses regularized importance weighting to stabilize optimization under distribution mismatch.It is designed to improve generalization beyond purely on-policy methods.

C Evaluation Benchmarks

The evaluation suite covers general video understanding, domain-rich comprehension, multi-step deduction, visual-spatial intelligence, and numerical inference. Supplementary analyses examine SFT→GRPO with paraphrased traces and DRD sensitivity to top-k.

  • Benchmark overview: The benchmark suite evaluates reasoning-driven video question answering across general understanding, structured multi-step inference, and numerical reasoning.It includes both broad video comprehension and reasoning-intensive tasks.
  • Video-MMMU: Video-MMMU contains 900 questions paired with 300 specialized videos across six fields, requiring temporal visual cues and domain expertise.Its fields include Medicine, Engineering, Art, Business, Science, and Humanities.
  • MMVU: MMVU (MC) contains 1,529 professional-domain videos and 3,000 curated QA pairs across 27 subjects.The branches include Healthcare, Engineering, Science, and Humanities & Social Sciences.
  • Video-Holmes: Video-Holmes tests multi-step deductive reasoning on 1,837 questions from 270 suspense short films.Tasks include temporal causal inference, intention and motive chaining, and physical anomaly reasoning.
  • VSIBench: VSIBench assesses visual-spatial intelligence and numerical inference using 288 authentic videos and over 5,000 QA pairs.It covers tracking, configuration analysis, and measurement estimation.
  • Video-MME: Video-MME contains 2,700 QA pairs tied to 900 diverse videos categorized as short, medium, or long duration.The duration categories span under 2 minutes, 4–15 minutes, and 30–60 minutes.
  • Supplementary analyses: Supplementary analyses report SFT→GRPO results with idiolectal paraphrased traces and examine DRD sensitivity to top-k.These analyses address generality across objectives and robustness across decoding configurations.

D.1 Full results of SFT→GRPO with Idiolectal paraphrased rewriting.

Idiolectal paraphrasing improves SFT→GRPO across all five benchmarks, and DRD’s top-k setting balances semantic fidelity with distributional alignment. Echo-GRPO also generalizes beyond video question answering to text-based math reasoning.

  • SFT→GRPO results: 3.8 points: replacing offline privileged traces with idiolectal paraphrased traces improves SFT→GRPO from 54.0 to 57.8 across five benchmarks.Gains include +5.5 on Video Holmes, +0.7 on VideoMMMU, and +9.0 on VSI-Bench.
  • DRD sensitivity: At k = 5, DRD achieves the best average performance of 58.2 across VideoMMMU, MMVU, and Video Holmes.k = 1 achieves a competitive average of 57.0 through stricter semantic fidelity.
  • DRD sensitivity: DRD’s semantic reference candidate-set size k controls the balance between semantic fidelity and distributional alignment.The candidate set is denoted V_k.
  • Text reasoning generalization: Echo-GRPO’s policy-aligned rewriting extends beyond video question answering to text-based math reasoning.Experiments cover AIME24, AIME25, and HMMT25.

D.4 Scale-up Experiment

At larger scale, Echo-GRPO remains more stable than Mixed-Policy during video-reasoning optimization while achieving higher average benchmark performance.

  • Scale: The scaled-up experiment uses 9k training samples, addressing whether earlier findings were driven by a small training set.Mixed-policy optimization reaches an average of 51.7 compared with 35.9 for the original on the three major benchmarks.
  • Performance: Echo-GRPO outperforms Mixed-Policy by +6.5 points and vanilla GRPO by +4.7 points on three major video benchmarks.The average performance comparison covers VideoMMMU, MMVU, and Video-Holmes.
  • Optimization stability: 7.9% average clipping: Echo-GRPO remains stable during scaled-up training, versus 26.1% for Mixed-Policy in later stages.Mixed-Policy’s clipping ratio rises sharply around step 200.

E Computation Analysis of DRD

DRD adds a substantial one-time offline construction cost but improves performance beyond paraphrasing without DRD, while cached traces preserve standard training and inference speed.

  • Performance-cost trade-off: 17.5× overhead: full DRD improves performance by an additional 1.3 points over Echo-GRPO without DRD at 163.9 ms/token.Echo-GRPO without DRD already improves vanilla GRPO by 1.3 points at 9.4 ms/token.
  • Offline construction: DRD’s two forward passes per decoding step jointly evaluate semantic and distributional references during offline rewritten-data construction.The cost is incurred before training, not during deployment.
  • Deployment cost: Cached idiolectal paraphrased traces allow training and inference to proceed at standard speed after preprocessing.Deployment inference speed is unaffected.

F.2 Limitations

The method has several scope and evaluation limitations, including preprocessing cost, dependence on teacher reasoning quality, numerical-precision sensitivity, and possible benchmark overlap with pretraining data.

  • Supervision quality: Idiolectal paraphrased-trace quality is bounded by the teacher policy’s reasoning quality.This limits the quality of supervision available to Echo-GRPO.
  • Task scope: Performance gains may be reduced on tasks requiring fine-grained numerical precision.The paper identifies this as a limitation discussed in Section 4.2.
  • Evaluation: Potential overlap between pretraining content and evaluation benchmarks creates a risk of implicit data leakage.VideoEcho-R1 is fine-tuned on top of large pretrained models.
  • Computation: Dual-Reference Decoding introduces mild preprocessing overhead because it uses two forward passes per decoding step.Rewritten traces are cached before training and incur no inference-time cost.
Loading 2608.26684v1…