Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation

Motivation

• When reliable rewards are unavailable, environmental feedback is a primary external information source, but it describes action consequences rather than directly supervising policy learning.The feedback includes tool outputs, code execution results, and environment state changes.Source: §1 · p. 1
• Prior feedback-conditioned self-distillation methods, including SDPO and OEL, yield only modest gains despite feedback containing information relevant to policy improvement.The authors identify how feedback is exploited during training as the key limitation.Source: §1 · p. 1

Approach

• SELF jointly optimizes environmental feedback modeling and hindsight self-distillation, using realized feedback both as a prediction target and as privileged context for teacher supervision.The paper reports that the two objectives mutually reinforce one another.Source: §1 · p. 2

Results

• With Qwen3-8B, SELF exceeds SDPO and GRPO by 6.4 and 4.1 percentage points in τ-bench success rate, and by 10.71 and 3.57 points in AppWorld task goal completion.The comparisons are respectively against SDPO and GRPO on each benchmark.Source: §1 · p. 2

Evaluation

• SELF is evaluated on τ-bench Retail and AppWorld across Qwen3-4B-Instruct and Qwen3-8B, outperforming feedback-learning baselines and reward-trained GRPO in task success and goal completion.τ-bench Retail covers multi-turn customer service with domain-specific tools; AppWorld evaluates code-based task completion across applications.Source: §1 · p. 2

Agentic Self-Distillation and Learning from Environmental Feedback

• Agentic self-distillation methods use privileged information or feedback to supervise a student, while feedback-learning approaches derive policy supervision from interaction experience.Examples include OPSD, SDPO, SDFT, RLSD, OCSD, HERO, OEL, and AHEAD.Source: §2 · p. 2

Environmental Feedback Modeling for Policy Learning

• Environmental feedback modeling work includes predicting environment outcomes during policy optimization, choosing decision-oriented targets, and using feedback prediction as an auxiliary objective.RLTF is identified as most closely related because it studies both feedback-conditioned self-distillation and feedback modeling, but predicts textual critiques as an auxiliary objective.Source: §2 · pp. 2–3

SELF Overview

• Figure 1 depicts SELF's two complementary uses of each interaction transition: feedback-conditioned teacher supervision and prediction of realized feedback from history and action.Both objectives are jointly optimized to produce the next-round policy.Source: Figure 1 · p. 3

Learning from Environmental Feedback

• SELF predicts environmental responses from interactive trajectories and uses the same realized feedback as both a prediction target and privileged context for hindsight self-distillation.The authors report that the two objectives mutually reinforce each other.Source: §2 · p. 3

3.1 PROBLEM FORMULATION

• Training uses on-policy trajectories of actions and observed feedback, without task rewards, outcome labels, or external teachers.Each transition associates an action with its observed consequence without requiring access to latent environment states.Source: §3.1 · p. 3

3.2 HINDSIGHT SELF-DISTILLATION

• Hindsight self-distillation compares student and feedback-conditioned EMA teacher predictions on the same on-policy action prefixes, with only the teacher seeing realized feedback.The collected action determines comparison contexts rather than serving as a ground-truth action target; the teacher is updated as an EMA of the student.Source: §3.2 · p. 4
• Feedback modeling is added because hindsight distillation alone does not explicitly learn feedback information, limiting how effectively the teacher can exploit it.The same interaction supplies a supervised action–feedback relation for learning how the environment responds.Source: §3.2 · p. 4

3.3 ENVIRONMENTAL FEEDBACK MODELING

• SELF predicts feedback from the interaction history and action using a dedicated prediction prompt and parameters shared with the student policy.Observed feedback is the target; the point-mass forward-KL objective is equivalent to token-normalized negative log-likelihood and does not treat the sampled action as a correct action target.Source: §3.3 · pp. 4–5

3.4 OVERALL SELF TRAINING OBJECTIVE

• SELF jointly optimizes environmental feedback modeling and hindsight self-distillation through a shared model, using realized feedback as both prediction target and privileged teacher context.The updated policy collects interactions for the next training round; deployment requires only the student policy.Source: §3.4 · p. 5
• The overall objective weights environmental feedback modeling and hindsight self-distillation separately, and averages token-normalized objectives across trajectories and interaction turns.Environmental feedback modeling updates the model that supplies subsequent hindsight supervision, while self-distillation also updates the representation used for feedback prediction.Source: §3.4 · p. 5

4.1 EXPERIMENTAL SETUP

• Evaluation uses τ-bench Retail for multi-turn customer service and AppWorld for code-based tasks, with Observation, Full, and Reward supervision settings.Qwen3-4B-Instruct and Qwen3-8B are the main backbones; ablations and capability evaluations use Qwen3-4B-Instruct.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS (RQ1)

• 38.7% SR with Qwen3-4B-Instruct exceeded SDPO and OEL by 8.3 and 8.1 points; SELF also beat GRPO by 5.4 and 4.1 SR points across the two backbones.The comparisons cover τ-bench Retail and AppWorld; under Full feedback, SDPO's SR fell on both backbones.Source: Table 2 · p. 6, §4.2 · p. 7

4.3 MUTUAL BENEFITS OF FEEDBACK MODELING AND SELF-DISTILLATION (RQ2)

• At λo=0.2, student SR reached 38.7% versus 29.4% without observation modeling, while teacher SR rose from 30.1% to 46.7%.The teacher instead peaks at 49.1% with λo=0.1, where student SR is 36.5%.Source: §4.3 · p. 8, Table 3 · p. 8
• At λa=1.0, prediction accuracy reached 73% on Retail and 56% on AppWorld versus 37% and 9% without distillation; Retail SR rose 20.1% to 38.7%, and AppWorld TGC 5.52% to 32.14%.Without self-distillation, task performance falls below Base on both benchmarks.Source: §4.3 · p. 9, Table 4 · p. 9

4.4 GENERAL CAPABILITY RETENTION (RQ3)

• The SELF student differs from the base model by at most 0.11 points across MMLU, MMLU-Pro, and IFEval; observation-based SDPO scores lower on all three.These results support capability retention rather than general capability improvement.Source: §4.4 · p. 9, Table 5 · p. 9

• SELF uses realized action consequences both as prediction targets for action–feedback relationships and as privileged context for self-teacher supervision.Source: §5 · p. 9
• Environmental feedback modeling strengthens hindsight self-distillation, while self-distillation improves feedback prediction.Source: §5 · p. 10
• Across τ-bench Retail and AppWorld, SELF outperforms existing feedback-learning methods and reward-trained GRPO while largely preserving base-model general capabilities.Source: §5 · p. 10

• τ-bench Retail evaluates multi-turn customer-service interactions using domain-specific tools, while AppWorld evaluates code-based task completion across applications.Source: A DATASETS, BENCHMARKS, AND MODELS · p. 13
• Interactive environments assess learning from action consequences.Source: A DATASETS, BENCHMARKS, AND MODELS · p. 13
• General capability benchmarks measure retention of knowledge, reasoning, and instruction following after agent training.Source: A DATASETS, BENCHMARKS, AND MODELS · p. 13

A.1 DATASETS AND BENCHMARKS

• τ-bench Retail evaluates multi-turn, policy-constrained tool interactions, while AppWorld tests code-based tasks across applications and their resulting application states.The benchmarks use official training splits and held-out test splits; training trajectories come from environment interactions rather than expert demonstrations.Source: §A.1 · p. 13
• MMLU, MMLU-Pro, and IFEval evaluate general capabilities separately from interactive training, covering knowledge, reasoning, and instruction following.The initial model, trained student, and EMA teacher are assessed using only benchmark inputs.Source: §A.1 · p. 13

A.2 MODELS

• Qwen3-4B-Instruct and Qwen3-8B are trainable backbones, while Qwen3-30B-A3B-Instruct provides an external model comparison.Qwen3-8B uses its non-thinking configuration.Source: §A.2 · pp. 13–14

B.1 HYPERPARAMETERS

• Training uses LoRA and AdamW, with a learning rate of 3 × 10^-4, batch size 16, and objective weights λo = 0.2 and λa = 1.0.Both objectives update the same student adapters.Source: §B.1 · p. 14
• Each round collects student trajectories and holds them fixed during optimization, which samples trajectories and turns uniformly and separately normalizes the two losses by target length.Recorded actions and feedback are fixed data; gradients do not propagate through sampling or environment execution.Source: §B.1 · p. 14
• Feedback prediction learns from the realized response to a history and action, while hindsight distillation aligns the student’s action distributions with a feedback-conditioned teacher.At deployment, action generation uses only the student policy, without a feedback-prediction or teacher-scoring step.Source: §B.1 · p. 14

B.2 DATASET SETUP

• Training uses official training tasks and evaluates on held-out Retail and AppWorld test partitions, with held-out interactions excluded from parameter updates.AppWorld evaluation uses test-normal.Source: §B.2 · p. 14
• SELF trains from environmental observations without separate task-success labels, while comparison conditions distinguish Observation, Full, and Reward supervision.Observation supplies environmental responses; Full adds ground-truth information and explicit success or failure feedback, while Reward supplies verifiable task rewards.Source: §B.2 · pp. 14–15
• Table 6 summarizes the default SELF training configuration, listing backbone-specific LoRA settings separately from shared settings.The caption does not specify the individual configuration values.Source: Table 6 · p. 15
• Feedback-prediction comparisons use a fixed transition collection generated by the initial Qwen3-4B-Instruct model over evaluation tasks in three passes.The same recorded transitions are used for all predictors and are distinct from the four attempts per task used for Retail pass@4.Source: §B.2 · p. 15

B.3 EVALUATION PROTOCOLS

• Retail reports success rate as pass@4 and average interaction turns, while AppWorld reports Task Goal Completion and Scenario Goal Completion as percentages.Task completion is determined by the benchmark evaluator.Source: §B.3 · p. 15
• Retail pass@4 counts a task as successful if at least one of its four attempts succeeds, giving each task equal weight.It measures success within four attempts, not success on all four.Source: §B.3 · p. 15
• Interaction length averages all attempts, including failures, and Turns should be interpreted alongside success rate.A short failed interaction does not indicate efficient task completion.Source: §B.3 · pp. 15–16
• Feedback-prediction evaluation scores generated responses conditioned on recorded histories and actions against withheld realized responses.The judge compares only the two feedback texts; agreement does not indicate whether the action was appropriate or completed the task.Source: §B.3 · p. 16
• Privileged-teacher task scores are reported separately because the teacher receives additional environmental feedback unavailable to the student at its original decision.Teacher scores do not represent ordinary inference without additional environmental information.Source: §B.3 · p. 16
• General-capability retention is assessed on MMLU, MMLU-Pro, and IFEval without privileged environmental feedback, outside the training objectives.The protocol evaluates the initial model, adapted student, and teacher checkpoint.Source: §B.3 · p. 16

C.1 EFFECT OF CROSS-ROLLOUT FEEDBACK MODELING

• The comparison keeps other training settings fixed and contrasts standard SELF’s shared transitions with independently sampled, intact transitions for feedback modeling.Each condition is evaluated in a single run.Source: §C.1 · pp. 16–17
• 38.6% vs 36.6% Retail SR, 10.3 vs 11.7 turns, and 32.16%/12.44% vs 31.21%/8.67% AppWorld TGC/SGC: standard SELF improves over cross-rollout training.The comparisons are standard SELF versus cross-rollout, respectively.Source: Table 7 · p. 17, §C.1 · p. 17
• Cross-rollout training reaches 36.6% Retail SR and 31.21% AppWorld TGC, outperforming the unadapted backbone and remaining competitive with full SELF.These results indicate feedback modeling can help even when its transitions differ from those used for hindsight distillation.Source: §C.1 · p. 17, Table 7 · p. 17

C.2 TRAINING COST

• 16.4 GPU-hours and 4.93 hours of wall time; SELF costs less than OEL and GRPO but more than SDPO (Observation).Table 8 aggregates costs across reported training runs.Source: Table 8 · p. 18, §C.2 · p. 18
• The cost accounting includes interaction collection and optimization, but excludes final evaluation and additional ablation experiments.GPU-hours sum device time across runs; wall time sums elapsed run durations.Source: §C.2 · p. 18, Table 8 · p. 18
• Against GRPO, SELF reduces GPU usage by 60.2% and wall time by 60.4%; against OEL, reductions are 19.2% and 19.4%.Source: §C.2 · p. 18
• Against SDPO (Observation), SELF adds 4.6 GPU-hours and 1.26 hours, increases of 39.0% and 34.3%, respectively.Source: §C.2 · p. 18
• SELF reuses collected transitions for feedback prediction and hindsight distillation without extra environment execution, and deployment uses only the student action policy.Source: §C.2 · p. 18

D SELF DETAILS

• The SELF details section presents the training procedure, hindsight-supervision construction, and two learning objectives, including loss alignment, normalization, and differentiation.Source: D SELF DETAILS · p. 18

D.1 TRAINING WORKFLOW

• SELF alternates between collecting transitions with the current student and optimizing on a fixed trajectory set, using feedback as a prediction target and teacher context.The student’s action and feedback objectives update shared trainable parameters; the teacher is initialized from the student and maintained by EMA updates. Minibatches sample trajectories and then turns uniformly, and losses are normalized by their target lengths.Source: §D.1 · pp. 18–19

D.2 HINDSIGHT SELF-DISTILLATION

• At each recorded action-token position, the hindsight teacher receives the observed feedback while student and teacher compare distributions over the same action prefix.The target is a learned distribution, not a corrected action label; feedback is privileged training context and is not available to the deployed student for that action.Source: §D.2 · p. 20

D.3 SELF OBJECTIVE

• Feedback prediction uses length-normalized maximum-likelihood learning from the realized response, which remains a valid target even when the action does not succeed.The objective models what follows an action rather than labeling the action as desirable, so it complements policy learning.Source: §D.3 · p. 21
• The two objectives update the same student parameters, and their effects can shape later teacher supervision, but gradient addition alone does not guarantee that they reinforce one another.Feedback-modeling updates can affect the EMA teacher used in future hindsight supervision, while policy distillation can also affect feedback prediction; the direction is empirical, and gradients may reinforce or oppose one another.Source: §D.3 · pp. 22–23

E ILLUSTRATIVE PROMPTS AND EXAMPLES

• The illustrations show feedback prediction correctly identifying a KeyError’s missing field and an agent fixing a script after test feedback until the tests pass.In the coding example, the feedback exposed a missing field and incorrect output formatting; after changes, the test passed and the program printed 47.25.Source: E ILLUSTRATIVE PROMPTS AND EXAMPLES · pp. 23–25
• Examples show feedback prediction misstating a delivered order as canceled and returning 0 instead of 17 pending orders, while correctly matching the contents of a configuration file.Source: E ILLUSTRATIVE PROMPTS AND EXAMPLES · p. 25

E.1 PROMPTS

• The student prompt generates the next action from task and interaction history, while the teacher additionally uses realized feedback as hindsight and the environmental-feedback prompt predicts feedback from history and action.Figure 4 summarizes these three prompting interfaces.Source: Figure 4 · p. 26, §E.1 · p. 26