PaperlayerView exact revision on arXiv

Source-linked AI summary

PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents

Jiazhou Zhou, Hu Zhou, Yucheng Chen, Jinyuan Qu, Ying-Cong Chen, Lei Zhang

arXiv:2609.35303v1cs.CV

TL;DR

Multi-turn VLM agents can receive no GRPO gradient on uniform failures, and OPD/OPSD’s gains have poorly understood mechanisms. PIVOT internalizes pivot diagnosis and privileged token scoring into policy updates without environment rollbacks. It reports 0.90 overall accuracy on Qwen2.5-VL-3B and 0.92 on Qwen3-VL-2B across five tasks.

  • Problem

    GRPO can be silent on uniform failures and assigns coarse episode-level credit, while the mechanism behind OPD/OPSD gains in multi-turn VLM agents remains poorly understood.

    Source: §1 · p. 1

  • Method

    PIVOT uses a trajectory Analyzer to localize pivots and build diagnostic context, a detached Teacher to rescore failed tokens, and a Student to optimize joint GRPO and gated OPD without simulator rollbacks.

    Source: §1 · p. 2

  • Results

    0.90 overall accuracy on Qwen2.5-VL-3B (+8% over SFT+GRPO baseline) and 0.92 on Qwen3-VL-2B (+12% over SFT+GRPO baseline) across five multi-turn VLM agent benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Counterfactual probes indicate that precise physical state rollback at the pivot, rather than additional skill prompting, drives most recovery gains in the tested agents.

    Source: §1 · p. 2

    The stated task formulation is a multi-turn agent modeled as a partially observable Markov decision process, bounding the described setting.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) via Group-Relative Policy Optimization (GRPO) is widely used for multi-turn VLM agent training, yet it suffers from zero-gradient silence on uniform failures and coarse episode-level credit assignment. While On-Policy Distillation (OPD) and On-Policy Self-Distillation (OPSD) mitigate sparse rewards using hindsight information, their underlying mechanisms remain poorly understood. Through controlled counterfactual rollback probes across five multi-turn VLM agent benchmarks, we reveal that performance gains in OPSD/OPD are largely driven by physical state rollback at the pivot step, defined as the first unrecoverable action without remaining step budget. However, physical state rollbacks are computationally prohibitive and infeasible in real-world environments. To bridge this gap, we present Pivot-Aware Internalized Visual On-Policy Training (PIVOT), an RL framework that internalizes pivot localization and state restoration directly into token-level parameter updates, eliminating environment rollbacks during RL training and additional skill hints at test time. PIVOT unifies three functional roles within a single architecture: a failure Analyzer non-invasively localizes the pivot step and diagnoses failure modes from visual trajectory collages and action logs; a detached Teacher re-scores failed tokens under this privileged diagnostic context; and a Student optimizes joint GRPO and confidence-gated OPD objectives. At test time, both Teacher and Analyzer branches are stripped. Evaluated on five multi-turn VLM agent tasks across cognitive grid puzzles, 3D embodied control and navigation, and generative reasoning, PIVOT achieves 0.90 overall accuracy on Qwen2.5-VL-3B (+8% over SFT+GRPO baseline and +5% over previous SOTA) and scales to 0.92 on Qwen3-VL-2B (+12% over SFT+GRPO baseline).

1 INTRODUCTION

GRPO can provide no learning signal for uniform failures and assigns credit only at the episode level, while the mechanisms behind OPD/OPSD gains remain unclear. PIVOT internalizes pivot diagnosis into policy updates, and achieves strong accuracy across five multi-turn VLM agent benchmarks.

1 INTRODUCTION

  • GRPO can fall silent when group returns are identical and gives coarse episode-level credit without identifying the turn that makes a task unrecoverable.This motivates finer-grained credit assignment for multi-turn VLM agents.Source: §1 · pp. 1–2
  • Counterfactual probes show pivot-step state restoration provides most suffix-return lift, while later resets sharply reduce recovery and skill hints add only marginal gains.The pivot is the first unrecoverable action under the remaining step budget.Source: §1 · p. 2
  • PIVOT trains a trajectory Analyzer, detached Teacher, and Student within a shared architecture, internalizing pivot localization and diagnostic hints without simulator rollbacks.At test time, the Analyzer and Teacher are removed, leaving the unprivileged Student without added computational or skill-prompt overhead.Source: §1 · p. 2
  • 0.90 overall accuracy on Qwen2.5-VL-3B, +8% over SFT-GRPO and +5% over dense-reward SOTA; Qwen3-VL-2B reaches 0.92, +12% over SFT+GRPO.Evaluated across five tasks spanning cognitive grid puzzles, 3D embodied control and navigation, and generative reasoning.Source: §1 · p. 3

2 MECHANISTIC ANALYSIS: PHYSICAL RESTORATION DOMINATES HINDSIGHT GUIDANCE

Episode-level GRPO can miss the critical failure turn or produce no update when group returns are uniform. Counterfactual probes identify precise physical rollback at the pivot as the main recovery driver, motivating PIVOT’s parameter-internalized alternative.

2.1 LIMITATIONS OF EPISODE-LEVEL CREDIT ASSIGNMENT

  • GRPO normalizes returns within a rollout group and assigns the resulting scalar advantage uniformly across action tokens, obscuring which turn made the task unrecoverable.When return variance is zero, the advantage vanishes even when failed trajectories contain different errors.Source: §2.1 · p. 3

2.2 DISENTANGLING OPD/OPSD: PHYSICAL STATE ROLLBACK DOMINATES SKILL HINTS

  • The pivot is the first action that makes the goal unreachable within the remaining step budget, providing the rollback point used to test recovery.The probe compares resampled suffixes after resetting to that point or one step later, with and without skill hints.Source: §2.2 · p. 4
  • Rollback at t∗ lifts suffix returns by 3.3%–56.6% across tasks and reduces zero-variance groups, while a one-step-later reset sharply lowers gains and skill hints add little.On Sokoban, t∗+1 lowers lift from 3.3% to 0.9%; on PrimitiveSkill, it falls from 17.7% to 8.3%.Source: §2.2 · p. 4

2.3 FROM PHYSICAL ROLLBACKS TO PARAMETER-INTERNALIZED UPDATES

  • Because physical resets are costly and infeasible in non-rewindable environments, PIVOT internalizes pivot localization and diagnostic context into token-level updates on failed rollouts.The Analyzer constructs privileged context for the detached Teacher to rescore failed tokens; Analyzer and Teacher branches are stripped at inference.Source: §2.3 · pp. 4–5, Figure 2 · p. 5

3 PIVOT: PIVOT-AWARE INTERNALIZED VISUAL ON-POLICY TRAINING

PIVOT consolidates Student, Analyzer, and Teacher roles in one VLM policy to internalize pivot localization and visual state restoration without training-time environment rollbacks or test-time overhead. Stage I cold-starts visual diagnosis; Stage II uses the diagnosis to score failed action tokens and update the policy.

3 PIVOT: PIVOT-AWARE INTERNALIZED VISUAL ON-POLICY TRAINING

  • PIVOT combines Student, Analyzer, and Teacher roles in one VLM policy, internalizing visual state restoration without environment rollbacks during training or extra test-time cost.At deployment, the Analyzer and Teacher branches are removed.Source: §3 · p. 5, §3.2 · p. 8

3.1 STAGE I: VISUAL DIAGNOSTIC COLD-START FOR PIVOT STEP LOCALIZATION

  • Stage I trains the Analyzer on visual trajectory collages, action logs, and task prompts to predict each failed trajectory’s pivot step, failure mode, and optional guidance skill.The collage uses observable trajectory history without access to environment states or internal memories; SFT uses autoregressive cross-entropy.Source: §3.1 · pp. 5, 7

3.2 STAGE II: INTERNALIZING LOCALIZED PIVOT STEP HINDSIGHT INTO POLICY UPDATES

  • Stage II applies localized pivot hindsight to original failed rollouts through a detached Teacher that re-scores action tokens under privileged context, while the Student updates from standard histories.The joint objective combines GRPO with confidence-gated OPD; when every rollout in a group fails, GRPO gradients vanish while OPD maintains informative gradients.Source: §3.2 · p. 7
  • The Analyzer centers a three-frame visual crop on its predicted pivot and combines it with the predicted failure mode and skill to form the Teacher’s privileged diagnostic context.The crop uses zero-padding at trajectory boundaries.Source: §3.2 · p. 7

4 EXPERIMENTS

Experiments evaluate PIVOT across five VAGEN tasks and two compact VLM backbones, comparing it with GRPO and on-policy distillation baselines. Results cover overall performance, diagnostic-context ablations, pivot localization, and training efficiency.

4.1 EXPERIMENTAL SETUP

  • PIVOT is evaluated on five VAGEN tasks across cognitive grid puzzles, embodied 3D control, and generative reasoning, using two backbones and GRPO, SFT-GRPO, and SDAR baselines.Puzzle and embodied-control performance uses average Success Rate; SVG Reconstruction uses a composite score averaging DINO and DreamSim embeddings.Source: §4.1 · p. 8

4.2 MAIN RESULTS

  • 0.90 overall accuracy on Qwen2.5-VL-3B gives M+P+R a +8% gain over SFT-GRPO, while Qwen3-VL-2B reaches 0.92, +12% over SFT-GRPO.The reported comparisons span five VLM agentic benchmarks; Table 1 identifies the baselines and context variants.Source: Table 1 · p. 9, §4.2 · p. 9

4.3 ABLATION STUDY

  • The diagnostic-context ablation reports 0.90 on Qwen2.5-VL-3B and 0.92 on Qwen3-VL-2B for M+P+R, but added skill prompts slightly lower the 3B SVG composite from 0.84 to 0.82.On Qwen3-VL-2B, M+P+R reaches 0.85 on SVG, while M+P reaches the Sokoban peak of 0.97.Source: Table 2 · p. 9, §4.3 · pp. 9–10

4.4 FURTHER ANALYSIS

  • On FrozenLake, Analyzer pivot accuracy rises from about 0.56 to 0.90 by step 100 and later holds near 0.95; Sokoban RL updates take 95 s versus 102 s for Vanilla-GRPO.The FrozenLake accuracy is a 15-step moving measure; a frozen Qwen3-VL-8B teacher takes 373 s per step, while stripped diagnostic branches add no test-time overhead.Source: §4.4 · p. 10, Figure 5 · p. 10

5 RELATED WORK

Related work frames multi-turn VLM training as a credit-assignment problem: outcome rewards can be coarse, while hindsight methods may carry inference overhead. PIVOT is positioned as internalizing pivot-focused, token-level feedback into policy parameters.

RL and Hindsight Distillation for VLM Agents

  • Outcome RL and process supervision use trial-and-error or dense shaping, but provide coarse feedback that does not isolate unrecoverable decision points.Hindsight, reflection, and memory systems offer post-hoc signals but incur prompt-length and inference-latency overhead.Source: §5 · p. 10
  • PIVOT is distinguished from prior on-policy distillation approaches by localizing pivot turns from visual histories and distilling token-level hindsight without dense shaping, extra prompts, or environment rollbacks.The passage characterizes agentic self-teachers in prior work as structurally decoupled or prompt-dependent.Source: §5 · p. 10

Policy Entropy Collapse and Multi-turn Credit Assignment

  • In multi-turn settings, early blunders can distort later observations and execution, while trajectory- or group-level diversity methods do not identify the failure's pivot turn.PIVOT is described as using visual panels and confidence-gated token-level gradients on failed rollouts, including zero-variance failure groups.Source: §5 · pp. 10–11

6 DISCUSSION AND LIMITATIONS

The discussion identifies three boundaries: pivot supervision currently relies on offline heuristic labels, the pivot definition fits structured visual environments, and evaluation uses compact models in simulation.

Pivot Supervision Source & Discovery

  • Stage I currently depends on offline pivot-target labels generated by feasibility heuristics; pivot discovery in complex environments without explicit solvers remains future work.The authors suggest unsupervised or self-diagnostic discovery as a possible direction.Source: §6 · p. 11

Task Domain Applicability

  • The remaining-budget goal-reachability definition of a pivot naturally fits structured visual environments, while open-ended dialogue and continuous control may require adapted reachability criteria.Proposed adaptations include dynamic, probabilistic, or entropy-guided reachability.Source: §6 · p. 11

Model Scale and Physical Realization

  • Evaluation is limited to compact Qwen2.5-VL-3B and Qwen3-VL-2B backbones in simulated benchmarks; larger models and physical robotic systems remain future directions.Source: §6 · p. 11

7 CONCLUSION

PIVOT addresses multi-turn VLM agents’ outcome-based RL credit-assignment bottleneck by internalizing pivot localization and visual state restoration in a unified Student-Analyzer-Teacher policy. It avoids physical resets during training and prompt overhead at deployment while outperforming prior methods.

7 CONCLUSION

  • PIVOT internalizes visual state restoration in a unified Student-Analyzer-Teacher policy to address outcome-based RL’s credit-assignment bottleneck.It localizes the pivot step and distills gated hindsight onto failed rollouts.Source: §7 · p. 11
  • Post-failure recovery is driven by physical state restoration at pivot steps rather than textual skill guidance.PIVOT localizes the pivot step and distills gated hindsight onto failed rollouts.Source: §7 · p. 11
  • PIVOT removes physical resets during training and prompt overhead at deployment, outperforming SOTAs and prior distillation methods.Source: §7 · p. 11

–Supplementary Material (Text)–

The supplementary material defines notation and environment-specific pivot certificates, details GRPO grouping and zero-gradient groups, and specifies the counterfactual recovery protocol.

APPENDIX CONTENTS

  • The appendix covers notation, pivot certificates, GRPO grouping and zero-gradient groups, counterfactual recovery, environments and rewards, training objectives, and implementation details.Source: APPENDIX CONTENTS · p. 15

B PIVOT CERTIFICATES

  • Pivot certificates use reachability checks for grid puzzles and environment-specific terminal events, budget tests, or visual progression for other benchmarks.Navigation checks format, blocked moves, and stagnation; PrimitiveSkill and SVG use their own failure criteria.Source: B PIVOT CERTIFICATES · p. 15

C GRPO GROUPING AND ZERO-GRADIENT GROUPS

  • GRPO groups are contiguous blocks of eight rollouts sharing a prompt key, and All-fail and σ=0 measure distinct group-level conditions.σ=0 means all eight returns match at 10^-8 rounding; the rates can differ when failures receive partial credit.Source: C GRPO GROUPING AND ZERO-GRADIENT GROUPS · pp. 15–16

A NOTATION

D COUNTERFACTUAL RECOVERY

  • Counterfactual recovery restores failed untrained rollouts at the solver-pinned pivot, then resamples their suffixes with the base policy.A retry counts as a lift only when its suffix return exceeds the original; the t∗+1 control delays restoration by one step.Source: D COUNTERFACTUAL RECOVERY · p. 17

E ENVIRONMENTS AND REWARDS

The evaluation covers five VAGEN tasks spanning grid puzzles, 3D embodied tasks, and SVG reconstruction, with validation metrics and episode rewards specified by environment.

E ENVIRONMENTS AND REWARDS

  • The five VAGEN tasks span cognitive grid puzzles, embodied tasks, and SVG reconstruction.The task suite is evaluated using environment-specific validation metrics.Source: E ENVIRONMENTS AND REWARDS · p. 17, Table 9 · p. 18
  • Sokoban asks the agent to push boxes onto targets in a 6×6 grid, while FrozenLake requires reaching a goal on a 4×4 grid while avoiding holes.Both use discrete actions and a nine-step horizon; FrozenLake disables slippery dynamics.Source: E ENVIRONMENTS AND REWARDS · p. 17
  • Navigation tests first-person instruction following to find objects, while PrimitiveSkill tests Panda-arm tabletop manipulation grounded in 3D coordinates.Navigation averages success across Base and Common-sense splits; PrimitiveSkill reports mean success across four skills.Source: E ENVIRONMENTS AND REWARDS · pp. 17–18
  • SVG reconstruction tests whether generated code replicates a target image, with validation averaging DINO and DreamSim on the final submission.Source: E ENVIRONMENTS AND REWARDS · p. 18
  • GRPO uses discounted episode returns with γ=0.95, and invalid actions incur an additional 0.1 penalty.Where an environment lacks a separate success bit, rollouts below Rsucc=1.0 are marked failed for OPD masking.Source: E ENVIRONMENTS AND REWARDS · p. 18, Table 10 · p. 18

F TRAINING OBJECTIVES

Training combines offline Analyzer SFT with Stage II objectives that use a detached Teacher to re-score failed-rollout tokens under diagnostic context. The shared policy’s Student is trained with GRPO and confidence-gated OPD.

F TRAINING OBJECTIVES

  • Stage I trains the Analyzer on task prompts, visual trajectory collages, and action transcripts to predict solver-supervised diagnostic triplets.The triplets contain pivot step, failure mode, and optional guidance text; the resulting SFT weights initialize Stage II.Source: F TRAINING OBJECTIVES · p. 19
  • The Analyzer’s supervised objective is length-normalized autoregressive negative log-likelihood, optimized with AdamW at 2×10^-6 for three epochs per environment.Each environment’s SFT checkpoint initializes Stage II for that task.Source: F TRAINING OBJECTIVES · p. 19
  • In Stage II, the same policy acts as a detached Teacher that re-scores failed-trajectory action tokens under privileged diagnostic context.The rollout actions are not resampled, and the Student scores them using the unprivileged interaction history.Source: F TRAINING OBJECTIVES · p. 19
  • Confidence-gated OPD aggregates its failed-trajectory loss over valid action tokens, while the joint objective combines it with GRPO.The Teacher logits and hindsight gap are stop-gradient, so the update acts on Student parameters.Source: F TRAINING OBJECTIVES · pp. 18–19

G SFT DATA AND OPTIMIZATION

This appendix records how Analyzer supervision is filtered and assembled, shows environment-specific prompts and examples, and documents a matched SDAR reproduction for comparison.

G SFT DATA AND OPTIMIZATION

  • The factual filter retains records only when Analyzer JSON, failure-mode labels, solver replay, and logged trajectory are consistent, with task-specific checks for guidance when present.Any failed check removes the record; lift-based replay filtering is not applied when building the SFT dataset.Source: G SFT DATA AND OPTIMIZATION · p. 19
  • Analyzer SFT data come from failed rollouts, with a solver supplying pivot and failure-mode labels and a frozen Qwen3-VL-8B model writing optional guidance.The resulting Qwen2.5-VL-3B packs use 90/10 train/validation splits; Qwen3-VL-2B follows the same pipeline.Source: G SFT DATA AND OPTIMIZATION · p. 19

H ANALYZER COLLAGES, PROMPTS, AND SFT TARGETS

Sokoban

  • The Sokoban example asks the agent to choose one admissible move after reasoning about box positions, target paths, and traps.The prompt requires reasoning to appear within designated tags and the action within its own tags.Source: Sokoban · p. 21

Navigation

  • The Navigation Analyzer prompt defines the pivot as the earliest structural mistake and requires a 0-based step, one failure mode, and an actionable avoidance rule.It directs the Analyzer to use both the labeled screenshot collage and compact action log.Source: Navigation · p. 22

PrimitiveSkill

  • The PrimitiveSkill example presents a failed episode with empty actions before a final pick-and-place action, as context for Analyzer supervision.The prompt specifies returning a pivot step, failure mode, and failure reason as JSON.Source: PrimitiveSkill · p. 24

SVG Reconstruction

I SDAR REPRODUCTION

  • SDAR uses a frozen 7B teacher to generate an episode-level skill and re-score student tokens, without Analyzer SFT or pivot localization.Its protocol matches PIVOT on the five tasks and validation budget, but runs only on Qwen2.5-VL-3B.Source: I SDAR REPRODUCTION · p. 33

J TRAINING CURVES

The appendix plots Stage II validation learning curves for four training variants on two model sizes, marking the running-best validation points.

J TRAINING CURVES

  • The plotted running-best validation points are aligned with the scores reported in Table 1.This alignment applies to both model sizes and all four variants.Source: J TRAINING CURVES · p. 34
  • The Qwen2.5-VL-3B curves compare Vanilla-GRPO, M, M+P, and M+P+R, with dots marking the running-best validation point.The curves are presented in Figure 10.Source: Figure 10 · p. 34, J TRAINING CURVES · p. 34
  • The Qwen3-VL-2B curves compare Vanilla-GRPO, M, M+P, and M+P+R, with dots marking the running-best point.The curves are presented in Figure 11.Source: Figure 11 · p. 34, J TRAINING CURVES · p. 34
Loading 2609.35303v1…