PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL

Juzheng Zhang, Disha Makhija, Manoj Ghuhan Arivazhagan, Vinayshekhar Bannihatti Kumar, Rashmi Gangadharaiah

arXiv:2609.20715v1cs.LGcs.AIcs.CL

TL;DR

Standard agent SFT predicts actions but treats environment observations only as context, leaving open whether that is the best initialization for RL. ActObs also supervises those observations, and although the methods perform similarly after SFT, ActObs improves downstream GRPO results across terminal tasks and transfers to unseen code editing.

  • Problem

    Standard trajectory SFT excludes environment observations from prediction targets, and whether that convention provides the best initialization for subsequent RL has rarely been tested.

    Source: §1 · p. 1

  • Method

    ActObs changes the SFT loss mask to predict observations already present in trajectories alongside actions, without adding data, parameters, sequence tokens, or forward passes.

    Source: §1 · p. 2

  • Results

    After similar SFT performance, ActObs improves post-GRPO pass@k on Terminal-Bench 2.0 and transfers to aider-polyglot, including a 43% 4B advantage over ActionSFT at pass@1.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Joint action-and-observation supervision prevents one-sided gradient specialization and preserves consequence prediction while preparing policies for downstream exploration.

    Source: §1 · p. 2

    Experiments cover terminal agents, Qwen3 at 4B and 8B, and trajectories from one teacher, so benefits across other environments, model families, teachers, and data pipelines remain unestablished.

    Source: H LIMITATIONS · p. 28

Abstract

from arXiv · show

Agent trajectories record what an agent does and what happens next. Yet standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, using environment observations as context but not as prediction targets. We ask whether this convention provides the best initialization for subsequent reinforcement learning. We introduce ActObs, which also supervises the observation tokens already present in each trajectory. Although deployed agents never generate observations, learning to predict them encourages the policy to model action consequences without adding data, parameters, sequence tokens, or forward passes. The methods perform similarly after SFT but diverge after GRPO. On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every evaluated sampling budget than its action-only counterpart on Terminal-Bench 2.0. On Qwen3-8B, it trades some pass@1 reliability for higher pass@k (+3.4 pp at pass@16) and solves more distinct tasks. The advantage extends to cross-domain code editing on aider-polyglot (+4.2 pp at pass@1 at 4B), whose tasks are unseen during SFT and RL. ActObs retains more entropy during RL while requiring less policy movement, leaving the final policy closer to its SFT initialization. Our analysis traces this difference to SFT: action and observation gradients rapidly become orthogonal, while action-only training leaves a large residual observation gradient and degrades environment prediction below the base model. Joint supervision prevents this one-sided specialization, preserving consequence prediction and preparing the policy for downstream exploration.

1 INTRODUCTION

The paper tests whether environment observations should remain context only or also become SFT targets. ActObs supervises those observations and, despite similar SFT performance, improves downstream RL results and cross-domain transfer.

Motivation

  • Standard trajectory SFT predicts agent-authored actions but leaves environment observations in context rather than treating them as prediction targets.The paper tests whether that convention is suitable when SFT initializes subsequent RL.Source: §1 · p. 1

Results

  • At 4B, ActObs yields the strongest GRPO policy at every evaluated budget; at 8B, it gains 14% at pass@16 and solves 24 tasks versus ActionSFT’s 21.The advantage also extends to aider-polyglot, where the 4B ActObs GRPO policy leads ActionSFT by 43% at pass@1 and 24% at pass@4.Source: §1 · p. 2
  • During GRPO, ActObs retains more entropy and remains closer to its SFT initialization, while greater observation-loss weight trades some pass@1 reliability for repeated-sampling success.Matching ActionSFT’s inference temperature to ActObs entropy does not close the pass@k gap.Source: §1 · p. 2

Mechanism

  • Action and observation gradients rapidly become nearly orthogonal; ActionSFT leaves a residual observation gradient and predicts terminal feedback worse than the base model, while ActObs fits both objectives.The ActObs initialization retains more entropy at command arguments, flags, and paths, supporting broader sampling of command variants.Source: §1 · p. 2

Objective

  • ActObs adds observation-token prediction to the SFT loss without adding data, parameters, sequence tokens, or forward passes.The observations are already present in each trajectory; the change is to expose them to the loss alongside actions.Source: §1 · pp. 1–2

2 ACTOBS: LEARNING FROM THE FULL AGENT TRAJECTORY

ActObs changes the loss mask so the model learns from both expert actions and the environment’s responses already present in each trajectory. This makes each trajectory both a policy example and a transition example.

2 ACTOBS: LEARNING FROM THE FULL AGENT TRAJECTORY

  • ActObs trains on action and observation tokens already in the trajectory, changing only which tokens contribute to the loss.The data, model, context, and training procedure remain fixed.Source: §2 · p. 3
  • The observation-loss weight λ sets the objective: λ = 0 recovers ActionSFT, while the default λ = 1 gives action and observation tokens equal weight.Prompts remain masked, and the denominator preserves per-example loss scale as λ changes.Source: §2 · p. 3
  • At observation positions, predicting the environment’s response requires anticipating how the preceding action changes the environment.The shared representation used to predict consequences also informs later action choice.Source: §2 · p. 3

3 MAIN RESULTS

The three SFT initializations are closely matched, but separate after GRPO: ActObs improves results across Terminal-Bench sampling budgets and transfers to unseen multilingual code-editing tasks.

3.1 SETUP

  • The experiments compare action-only, joint ActObs, and sequential Obs→Act SFT, then evaluate on tasks disjoint from both the SFT corpus and RL set.SFT uses 50k terminal trajectories; GRPO uses 2,392 tasks, and headline Terminal-Bench evaluations use 16 attempts per task.Source: §3.1 · pp. 3–4

3.2 OBSERVATION SUPERVISION STRENGTHENS DOWNSTREAM RL

  • At 4B, ActObs leads ActionSFT→GRPO by 1.6, 1.4, 1.5, and 1.1 points at pass@1, pass@4, pass@8, and pass@16; at 8B, its pass@16 lead is 3.4 points.At 8B, ActionSFT leads at pass@1, but ActObs overtakes it at pass@4 and its advantage grows at larger budgets.Source: Table 1 · p. 4, §3.2 · pp. 4–5
  • At 8B, ActObs reaches task coverage of 24 versus ActionSFT’s 21; sequential Obs→Act does not achieve the same benefit despite equal observation supervision.ActObs also solves three tasks that no SFT policy or ActionSFT→GRPO can solve.Source: §3.2 · p. 5
  • With ECHO, ActObs leads or ties ActionSFT at seven of eight operating points; only the 8B pass@16 comparison favors ActionSFT.ECHO adds next-observation prediction during RL.Source: §3.2 · p. 5

3.3 CROSS-DOMAIN TRANSFER TO CODE-EDITING TASKS

  • On aider-polyglot’s unseen code-editing tasks, 4B ActObs leads ActionSFT→GRPO by 4.2 points at pass@1 and 4.9 points at pass@4.The 4B ActObs SFT checkpoint trails both alternatives before RL; the benchmark covers six programming languages.Source: §3.3 · p. 5

4 ACTOBS PRESERVES EXPLORATION DURING RL

ActObs initialization changes how the same GRPO procedure evolves: it retains more entropy with less policy movement, supporting stronger repeated-sampling performance.

4.1 TRAINING DYNAMICS DURING GRPO

  • At 4B, ActObs finishes GRPO with higher reward, rising late training entropy, and shorter rollouts than ActionSFT.At 8B, the entropy separation also appears, although training rewards converge.Source: §4.1 · p. 5, Figure 3 · p. 6

4.2 SELF-ENTROPY AT THE FINAL CHECKPOINTS

  • ActObs retains more post-RL self-entropy than ActionSFT while reaching higher pass@16.Higher entropy alone is insufficient: ECHO has higher endpoint entropy but does not match ActObs at pass@16, and temperature matching does not close the gap.Source: §4.2 · p. 6

4.3 LESS POLICY MOVEMENT

  • ActObs undergoes the smallest endpoint change from its SFT initialization while retaining more entropy and attaining the strongest pass@16.ECHO’s still-higher entropy does not match ActObs at pass@16, so entropy alone does not account for the result.Source: §4.3 · p. 6, Figure 4 · p. 7

4.4 VARYING OBSERVATION SUPERVISION STRENGTH

  • As observation-loss weight λ increases, the post-GRPO pass@8 advantage grows while pass@1 moves in the opposite direction.The sweep holds the data and downstream GRPO recipe fixed; stronger observation supervision trades some single-rollout reliability for greater repeated-sampling success.Source: §4.4 · p. 7

5 WHY JOINT SUPERVISION IS A STRONGER INITIALIZATION

Joint supervision changes the SFT checkpoint by preserving observation prediction after action and observation gradients separate, and its policy differences persist into RL.

5.1 GRADIENT DYNAMICS DURING SFT

  • Because the observation loss is absent from GRPO, its effect is carried by the SFT checkpoint handed to RL.Joint supervision keeps both action and observation signals active through that handoff.Source: §5.1 · p. 7
  • Within 10 to 20 SFT steps, action and observation gradients become nearly orthogonal; their norm ratio rises to about 41 under ActionSFT but stays near 0.5 under ActObs.ActionSFT leaves a large residual observation gradient and degrades environment prediction, while ActObs continues optimizing the orthogonal component.Source: §5.1 · p. 8, Figure 5 · p. 8

5.2 TOKEN-LEVEL EFFECTS

  • After GRPO, ActObs retains more entropy later in commands; its observation prediction improves on payload text, while ActionSFT performs below the base model across most of the sequence.The command-token difference opens later in commands rather than at their first tokens.Source: §5.2 · pp. 8–9, Figure 6 · p. 9

6 RELATED WORK

The related work situates ActObs among trajectory supervision, agent world modeling, and RL exploration methods, emphasizing its change to the SFT loss mask.

Trajectory supervision and agent RL

  • Representative trajectory-SFT pipelines use environment feedback as context rather than prediction targets, while ActObs extends loss to both action and observation tokens.ActObs holds the teacher, trajectories, and token sequence fixed.Source: §6 · p. 9

World modeling for language agents

  • Unlike world-model approaches requiring extra transitions, staged warm-ups, explicit state descriptions, or inference-time simulators, ActObs unmasks observations already in expert traces.The paper uses this design to isolate how the SFT loss mask shapes subsequent GRPO.Source: §6 · p. 9

Initialization and exploration in RLVR

  • Where many exploration methods intervene during RL, ActObs changes the checkpoint passed from SFT to otherwise unchanged RL.The cited work notes that similar pre-RL performance can conceal different initial policy distributions or action support.Source: §6 · p. 10

7 CONCLUSION

ActObs supervises both teacher actions and environment responses, and the paper reports stronger post-GRPO repeated-sampling outcomes alongside a gradient-based explanation.

7 CONCLUSION

  • ActObs learns from teacher actions and environment responses already present in trajectories, encouraging the shared model to represent action consequences.The reported benchmark performance changes little immediately after SFT, but the resulting initialization is stronger for GRPO.Source: §7 · p. 10
  • After GRPO, ActObs has higher pass@k at every evaluated budget at 4B and trades some single-shot reliability for greater repeated-sampling success at 8B.The paper also reports strong transfer to a cross-domain multilingual code-editing benchmark.Source: §7 · p. 10
  • The analysis links the gains to SFT geometry: orthogonal gradients leave ActionSFT with a residual observation gradient, while joint supervision avoids one-sided specialization.ActObs retains more entropy with less policy movement during GRPO.Source: §7 · p. 10

A EVALUATION PROTOCOL AND STATISTICAL PROCEDURES

The evaluation uses fixed task sets and attempt budgets to compare pass@k across sampling budgets, with controlled serving settings and bootstrap uncertainty estimates.

A EVALUATION PROTOCOL AND STATISTICAL PROCEDURES

  • Headline Terminal-Bench 2.0 comparisons use 16 attempts per task, and pass@k is estimated with the unbiased estimator of Chen et al. (2021).Other evaluations use four or eight attempts per task as indicated; model-caused failures score zero, while verified completions can count despite timeouts.Source: A EVALUATION PROTOCOL AND STATISTICAL… · p. 15
  • Each table reports a point estimate and one bootstrap standard error from 20,000 resamples that hold the task set fixed and resample attempts within each task.The resulting uncertainty captures finite-attempt sampling variation.Source: A EVALUATION PROTOCOL AND STATISTICAL… · p. 15
  • Headline checkpoints share the same task set, 16-attempt budget, and serving configuration, while a pinned vLLM setup controls latency-sensitive Terminal-Bench evaluation.The setup uses one engine across eight A100 GPUs, 12 concurrent trials, and decoding at T = 0.6, top-p 0.95, and top-k 20.Source: A EVALUATION PROTOCOL AND STATISTICAL… · p. 15

B COMPLETE RESULTS AND ABLATIONS

Across both model scales, ActObs's similar SFT performance gives way to stronger high-budget results after RL; controls indicate that correctly paired, jointly supervised observations matter.

Diagnostic controls

  • Controls indicate that the effect requires joint, sustained supervision on correctly paired observations: extra action-only SFT does not recover it, early-only observation supervision underperforms, and shuffled observations reduce pass@1 from 7.9 to 3.7.Observation-only training does not replace action learning, and supervision on complete terminal responses is stronger than selecting salient lines.Source: B COMPLETE RESULTS AND ABLATIONS · p. 15

8B results

  • At 8B, ActObs→GRPO reaches 23.6 pass@8 and 27.0 pass@16, solving 24 of 89 tasks versus ActionSFT→GRPO's 21, while the sequential Obs→Act method lacks the same high-budget advantage.The ActObs advantage appears at larger sampling budgets despite ActionSFT leading at pass@1 before RL.Source: B COMPLETE RESULTS AND ABLATIONS · p. 15, Table 2 · p. 16

ECHO comparisons

  • The 4B ActObs initialization also pairs with ECHO: ActObs→ECHO reaches 20.2 pass@16 and solves 18 tasks, the largest high-budget estimate and task count in that matrix.At 8B, the ECHO advantage is concentrated at smaller and intermediate budgets rather than the largest sampling budgets.Source: B COMPLETE RESULTS AND ABLATIONS · pp. 16–17

4B results

  • At 4B, ActObs→GRPO is strongest among the plain-GRPO policies at every reported sampling budget, improving on ActionSFT→GRPO by 1.5 points at pass@1 and retaining an advantage through pass@16.The SFT checkpoints are nearly indistinguishable at pass@1, with point estimates from 4.4 to 4.5.Source: B COMPLETE RESULTS AND ABLATIONS · p. 16, Table 3 · p. 17

Direct endpoint differences

  • At 8B, the ActObs-minus-ActionSFT difference crosses zero between pass@2 and pass@4 and reaches +3.4 points at pass@16 after GRPO, despite ActObs starting below ActionSFT at SFT.At 4B, a near-zero SFT difference becomes a positive post-GRPO difference of 1.1–1.5 points at every k.Source: B COMPLETE RESULTS AND ABLATIONS · p. 16, Table 4 · p. 17

Improvement during RL

  • ActObs's GRPO improvement at 8B remains positive from pass@1 through pass@16, reaching +4.5 points at pass@16, whereas ActionSFT's gain contracts and turns negative at pass@8 and pass@16.At 4B, ActObs also has larger plain-GRPO gains than ActionSFT from pass@1 through pass@8.Source: B COMPLETE RESULTS AND ABLATIONS · p. 17, Table 5 · p. 18

Observation-loss weight

  • At 8B, increasing observation-loss weight raises post-GRPO pass@8 from 21.0 at λ = 0 to 23.6 at λ = 1, while pass@1 falls from 12.3 to 11.0.Intermediate λ = 0.25 reaches 22.5 pass@8 after GRPO despite SFT pass@1 and pass@4 close to ActionSFT.Source: Table 6 · p. 18, B COMPLETE RESULTS AND ABLATIONS · p. 18

Overall pattern

  • The weight sweep and cross-scale comparison show that the downstream advantage is graded rather than an all-or-nothing endpoint effect, and appears as ActObs strengthens at larger k.Figure 7 shows the 4B point estimate higher at every k, while the 8B curves cross between k = 2 and k = 4.Source: B COMPLETE RESULTS AND ABLATIONS · pp. 18–19, Figure 7 · p. 19

C RL TRAINING DYNAMICS

ActObs follows a distinct entropy trajectory during GRPO, while its policy remains flatter on command probes; observation-supervised checkpoints also show higher pass@1 estimates under longer interaction budgets.

C RL TRAINING DYNAMICS

  • At 8B, GRPO reaches the same reward for ActObs and ActionSFT, but their on-policy entropy curves separate near step 20 and remain apart as rollout lengths end at similar values.ActObs is longer through the middle of training, so the persistent entropy gap is not a mechanical consequence of longer generations.Source: C RL TRAINING DYNAMICS · p. 19, Figure 8 · p. 20
  • Across both scales, ActObs maintains higher training entropy than ActionSFT, while sequential Obs→Act finishes below both, indicating observation exposure alone does not reproduce joint supervision.At 8B, the ActObs separation emerges earlier and grows through training; at 4B, ActionSFT loses entropy late in GRPO.Source: C RL TRAINING DYNAMICS · p. 19, Figure 9 · p. 20

D ENTROPY AND COMMAND-MARGIN PROBES

  • After GRPO at 8B, ActObs has 0.779 self-entropy versus ActionSFT's 0.716 and a flatter command margin of 7.24 versus 7.55, but higher entropy alone does not determine benchmark success.Before RL, ActObs is also flatter on fixed states, while shuffled observations reproduce its entropy and margin signature but perform much worse.Source: D ENTROPY AND COMMAND-MARGIN PROBES · pp. 20–21, Table 7 · p. 21

E EXTENDED-BUDGET EVALUATION

  • With doubled agent wall-clock limits, all four observation-supervised checkpoints have higher pass@1 point estimates; ActObs gains +2.2 points at SFT and +1.7 after GRPO.The standard ActionSFT checkpoint is the only SFT policy that does not benefit from the additional interaction time.Source: E EXTENDED-BUDGET EVALUATION · p. 21

F BEHAVIORAL STUDIES

On Terminal-Bench 2.0, ActObs's successes are distributed across more tasks rather than more strategies within already-solved tasks; trajectory studies describe less repetitive interaction and varied tactics.

F BEHAVIORAL STUDIES

  • ActObs→GRPO solves 24 tasks versus ActionSFT→GRPO's 21, distributing 6.5 successes per solved task rather than 8.3 across fewer tasks and achieving the highest win entropy.The two policies solve 19 tasks in common; ActObs adds five tasks while ActionSFT adds two different tasks.Source: F BEHAVIORAL STUDIES · pp. 21–22, Table 9 · p. 22
  • Blinded review finds that higher-entropy policies use more varied openings, fewer commands across more turns, and more often change tactics when progress stalls; eight of 12 reviewed successes involve less-correlated executions of the same high-level procedure.Judges counted 1.5–1.9 strategies per solved task, with 91%–94% exact inter-judge agreement across three analyses.Source: F BEHAVIORAL STUDIES · pp. 21, 23, Table 10 · p. 23
  • Before RL, ActObs has longer median trajectories than ActionSFT, 59 versus 51 steps, but shorter successful trajectories, 13 versus 21 steps.The sequential Obs→Act policy has the shortest successful trajectories at 11 median steps.Source: F BEHAVIORAL STUDIES · p. 23, Table 11 · p. 24
  • Across the three main SFT-to-GRPO comparisons, exact command repetitions fall from 34.9–48.1 per trajectory to 1.0–1.2, and identical retries after errors fall from 48.0–55.5% to 0.7–0.9%.Trajectory length does not uniformly shrink; shuffled observations instead have the highest error count and often repeat the preceding command after errors.Source: F BEHAVIORAL STUDIES · p. 23, Table 11 · p. 24

G REPRESENTATIVE TRANSCRIPTS

  • The reproduced qemu-startup and fix-git transcripts illustrate tactics behind selected ActObs successes, including changing QEMU memory settings and trying alternate Git integration methods.The appendix identifies blue boxes as agent turns and orange boxes as returned observations.Source: F BEHAVIORAL STUDIES · p. 22, G REPRESENTATIVE TRANSCRIPTS · pp. 23–24
  • Each transcript's first episode prompt combines scaffold instructions, task description, and initial terminal screen; later user messages contain only new terminal output.The appendix notes that long episodes may include scaffold-managed context summarization turns.Source: G REPRESENTATIVE TRANSCRIPTS · pp. 23–24

G.1 CASE STUDY: QEMU-STARTUP

ActObs→GRPO alone solves the QEMU-startup case after varying launch settings and increasing guest memory. In fix-git, methods find the lost commit, but ActObs→GRPO successfully integrates it after failed approaches.

G.1 CASE STUDY: QEMU-STARTUP

  • ActObs→GRPO alone solved QEMU-startup: after twelve launch variants, 1024M guest memory produced a reachable login prompt, and the verifier scored the trial 1.0.The trajectory was still waiting when the 900-second task limit expired, but the VM was correctly configured and reachable.Source: §G.1 · pp. 25–26
  • The successful QEMU search changed launch settings one at a time, including serial configuration, boot flags, drive interface, and guest memory.Earlier attempts varied these settings; the winning launch used 1024M rather than the default 128M, which was too little for the login prompt.Source: §G.1 · pp. 25–26

G.2 CASE STUDY: FIX-GIT

  • In fix-git, ActObs→GRPO recovered the dangling commit’s work and completed integration despite failed cherry-pick and patch attempts.ActionSFT attempts reached the integration step but lost progress to pager handling and an unsupported file rewrite.Source: §G.2 · pp. 27–28

H LIMITATIONS

The experiments are limited to terminal agents, two Qwen3 sizes, one teacher, and pipelines that differ across stages. The paper also describes its AI use and planned reproducibility release.

H LIMITATIONS

  • The experiments are confined to terminal agents; aider-polyglot adds cross-domain evaluation, but broader agent settings remain untested.The authors identify coding, web, GUI, and embodied environments with richer observations as directions for testing generality.Source: H LIMITATIONS · p. 28
  • The study compares Qwen3 at 4B and 8B, uses one teacher and one data pipeline, and includes only consequences realized after teacher actions.Additional model families, teachers, and data collection pipelines would separate these factors.Source: H LIMITATIONS · p. 28
  • SFT, RL, and evaluation differ in formatting, context construction, shell setup, and command handling, which may affect absolute scores and cross-stage comparisons.Methods within each stage share a scaffold, keeping the central ActObs-versus-ActionSFT comparisons controlled.Source: H LIMITATIONS · p. 28

I STATEMENTS

  • The authors report that generative AI assisted code development, language editing, and figure scripts, and that they reviewed and verified the outputs.All authors reviewed and approved these uses and take responsibility for the final content.Source: I STATEMENTS · p. 29
  • The paper specifies its training and evaluation procedures and states that code, data, outcomes, configurations, and analysis artifacts will be released upon acceptance.The loss-mask change is specified by Eq. 1.Source: I STATEMENTS · p. 29

Related papers

Loading 2609.20715v1…