PaperlayerView exact revision on arXiv

Source-linked AI summary

Attacca: Goal-Directed Control under State Continuity for Long-Horizon Embodied Agents

Gyusik Seo, Jaehong Yoon

arXiv:2610.07785v1cs.AI

TL;DR

Long-horizon embodied tasks require policies to continue from the state left by earlier tasks, while existing visual goal-conditioned policies are typically evaluated in settings where targets are already visible. Attacca trains a goal-conditioned policy on search-to-interact trajectories with goal images from another world. It achieves the highest short-horizon success across seen and held-out classes and completes 5–7 times as many long-horizon chains as the strongest baseline.

  • Problem

    Long-horizon embodied objectives require interdependent behaviors, but existing visuomotor policies are largely evaluated on short-horizon tasks and may face an unseen next target after a task transition.

    Source: §1 · pp. 1–2

  • Method

    Attacca trains a target-conditioned policy on search-to-interact trajectories using masked goal images from another environment, with auxiliary target-grounding and behavioral-phase objectives.

    Source: §1 · p. 2

  • Results

    Attacca completes 5–7 times as many long-horizon chains as the strongest baseline and achieves the highest short-horizon success on all tasks for seen and held-out classes.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The results suggest that jointly learning target acquisition and interaction from environment-decoupled goal images can improve robustness and generalization across targets and task horizons.

    Source: §1 · p. 2

    The evaluation isolates low-level Minecraft control; a scripted stage controller supplies task goals and detects completion, while helpers handle items at task transitions.

    Source: H LIMITATIONS · p. 24

Abstract

from arXiv · show

A central capability of embodied agents is to accomplish complex objectives through sequences of interdependent tasks. Yet existing visual goal-conditioned policies underlying these agents are typically evaluated on isolated interactions where the target is already visible, and thus do not capture the conditions that arise during continuous long-horizon task execution. In such settings, each task begins from the state left by the previous one: the agent may end at a different position and orientation, the world may have been modified, and the next interaction target may lie outside the current field of view. As a result, agents relying on such policies may struggle to proceed to the next task when they cannot ground their target in the current observation. To address this challenge, we propose Attacca, a new approach that trains visual goal-conditioned policies on complete search-to-interact trajectories using goal images decoupled from the execution environment. Attacca uses context-decoupled goal sampling to pair each demonstration with a class-compatible masked goal image from another world, removing direct scene and pose correspondence. It learns dense current-view grounding through a target-mask prediction head, providing auxiliary supervision beyond action imitation. We further introduce behavioral-phase conditioning that teaches the policy to distinguish Search, Approach, and Interact stages and adapt its control as execution progresses. We evaluate Attacca on multiple short- and long-horizon embodied tasks in Minecraft. Our method achieves 39.0-47.5% clean success, improving over the strongest baseline by 1.7-2.4x. On long-horizon tasks, it attains 54%, 30%, and 28% completion, yielding up to a 7x improvement.

1 INTRODUCTION

Long-horizon control requires each task to continue from the state left by the previous one, but existing visuomotor policies are largely evaluated on short-horizon tasks. Attacca trains a policy to search for, approach, and interact with targets across task transitions, achieving stronger results on short- and long-horizon tasks.

1 INTRODUCTION

  • A skill hand-off gap arises because even a successful plan leaves the policy to carry execution forward from the state produced by the previous task.Long-horizon objectives often require interdependent behaviors, each beginning from the state left by the preceding one.Source: §1 · p. 1
  • Existing visuomotor policies are largely developed and evaluated on short-horizon tasks, leaving successive tasks underexplored when the next target is not visible after a task changes the agent’s state.The figure depicts each task starting from the state left by the previous one, with search required when the target is initially out of view.Source: §1 · p. 2, Figure 1 · p. 2
  • Attacca trains a target-conditioned policy on search-to-interact trajectories, pairing demonstrations with masked target images from another environment and adding target-grounding and behavioral-phase supervision.The policy distinguishes Search, Approach, and Interact stages and adapts its control as execution progresses.Source: §1 · p. 2
  • Across short-horizon Mine, Hunt, and Place, Attacca achieves clean success of 39.0%, 47.5%, and 47.0%; on long-horizon tasks, it attains 54.0%, 30.0%, and 28.0%, with up to 7× gain over the strongest baseline.Source: §1 · p. 2

2 RELATED WORK

Prior work specifies goals through fixed objectives, text, video, or masked goal images, while hierarchical agents dispatch language subgoals to low-level policies. Attacca focuses on target acquisition and interaction by a low-level policy operating from inherited states.

Target specification in goal-conditioned policies

  • Masked goal images designate targets, but prior ROCKET-1 and ROCKET-2 pipelines require the target to be visible to construct the goal and use goals from the imitated trajectory’s scene.Attacca retains the masked-image interface while sampling the goal from another world, so it remains defined before the target enters view.Source: §2 · p. 3
  • Text goals can name targets but cannot show unfamiliar targets, while GROOT’s reference gameplay videos entangle targets with demonstrator behavior and surrounding layout.STEVE-1 conditions on MineCLIP embeddings of text or 16-frame behavior clips; JARVIS-VLA post-trains a vision-language model for instruction-conditioned control.Source: §2 · p. 3

Hierarchical execution and the skill hand-off gap

  • Hierarchical agents dispatch planner-generated language subgoals to low-level policies, but the controller must still acquire the next target from the preceding skill’s resulting state.This paper studies that hand-off within the low-level policy, using task chains that preserve the state left by each completed task.Source: §2 · p. 3

3 ATTACCA: GOAL-DIRECTED EMBODIED CONTROL UNDER STATE CONTINUITY

Attacca trains a visual goal-conditioned policy to find and interact with targets across successive tasks that inherit the preceding task’s state. It combines cross-world goal sampling, current-view target grounding, and behavioral-phase conditioning.

3.1 PROBLEM STATEMENT

  • Successive tasks preserve the agent’s pose, camera orientation, and world changes, while the next target may initially be outside the field of view.The policy acts from visual observations and goal images without target coordinates or external grounding and navigation modules.Source: §3.1 · p. 4
  • A frozen visual encoder and goal-fusion module feed a causal Transformer that predicts low-level camera motions and button presses from temporally fused context.The goal image and current observations provide visual features for the policy.Source: §3.1 · p. 4

3.2 CONTEXT-DECOUPLED GOAL SAMPLING

  • Each demonstration is paired with a visible, masked goal image of the same target class sampled from another world, removing direct scene and pose correspondence.Resampling varies the goal scene while keeping its target class fixed.Source: §3.2 · p. 5

3.3 GOAL-CONDITIONED TARGET GROUNDING

  • A target-mask head predicts the target’s visible support in current-view patches, using patch-averaged mask labels, binary cross-entropy, and soft Dice supervision.Frames where the target is not visible receive an all-zero target mask; predicted grounding is fed back to the policy through a residual vector.Source: §3.3 · pp. 5–6

3.4 BEHAVIORAL-PHASE CONDITIONING

  • A behavioral-phase head predicts Search, Approach, or Interact and uses the phase distribution to modulate the temporal readout before action prediction.Ground-truth masks and phase labels supervise training, while inference uses the policy’s own predictions.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments test Attacca on short-horizon tasks and state-carryover chains, with ablations and qualitative analyses examining its components and target-directed behavior.

4.1 EXPERIMENTAL SETUP

  • Attacca receives masked goal images from worlds other than the execution world and trains on demonstrations from 1,160 worlds, with target-mask and behavioral-phase labels.Baselines use their native goal formats; all except the released ROCKET-2 checkpoint train on the same dataset.Source: §4.1 · pp. 6–7
  • Short-horizon tasks require finding an initially out-of-view target and interacting with it; long-horizon chains carry the world, position, and camera state across stages.The study uses Mine, Hunt, and Place plus Diamond Pickaxe, Wolf Feeding, and Nether Portal chains.Source: §4.1 · pp. 6–7

4.2 SHORT-HORIZON TASK RESULTS

  • Attacca consistently achieves the highest success on seen and held-out short-horizon classes, using the same data, initialization, and goals as fine-tuned ROCKET-2.Current-view ROCKET pipelines stay at or below 5% on Mine and Place when targets are out of view; Hunt reaches 18–20% as animals wander into view.Source: §4.2 · p. 7

4.3 LONG-HORIZON TASK RESULTS

  • Attacca completes 14–27 of 50 episodes per long-horizon chain, while no baseline exceeds 4; it completes 5–7 times as many chains as the strongest baseline.Attacca maintains 85–93% next-stage completion at most transitions.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 ABLATION STUDY

  • Residual feedback and phase conditioning do not help consistently alone, but their combination improves Diamond Pickaxe, Wolf Feeding, and Nether Portal by 0.04, 0.18, and 0.10 over dense grounding alone.The authors suggest phase conditioning signals when to move from search to approach, while residual feedback supplies the target location.Source: Table 3 · p. 8, §4.4 · p. 8
  • Among four mask-injection pathways, the residual pathway performs best on all three chains; the authors hypothesize its single grounding vector handles variation in mask presence across phases.Cross-attention remains below residual feedback, and the authors suggest finer patch-level features add no benefit.Source: Table 4 · p. 8, §4.4 · p. 8

4.5 QUALITATIVE ANALYSIS

  • On diamond-stage episodes, Attacca’s paths converge on the ore and keep it in view, while ROCKET-2 repeatedly loses sight of it and spreads across the arena.Attacca completes the stage in 39 of 42 episodes, versus 4 of 27 for ROCKET-2 episodes where the ore entered view.Source: Figure 4 · p. 9, §4.5 · p. 9
  • ROCKET-2 succeeds by reproducing close same-world goal views, but far or different-world goals separate view matching from target pursuit; Attacca remains successful across all three goal-image conditions.ROCKET-2 success is 0.75 with close same-world goals, 0.24 with far goals, and 0.30 with different-world goals; Attacca is 0.81–0.89 across them.Source: Table 5 · p. 9, §4.5 · p. 9

5 CONCLUSION

Attacca addresses successive tasks from inherited states using target images from other worlds, and reports higher success across short-horizon tasks and more completed long-horizon chains than the strongest baseline.

5 CONCLUSION

  • Attacca achieves the highest success on all short-horizon tasks for seen and held-out classes and completes 5–7 times as many long-horizon chains as the strongest baseline.The conclusion also describes Attacca as one visual goal-conditioned policy that searches for, approaches, and interacts with targets from inherited states.Source: §5 · p. 10
  • Extending Attacca to other embodied domains and learned planners remains future work, and the paper notes limitations in its evaluation.The conclusion points to Section H for the evaluation limitations.Source: §5 · p. 10

A EXAMPLE OF CONTEXT-DECOUPLED GOAL SAMPLING

  • Context-decoupled sampling pairs a trajectory with a same-class goal image from another world, so target appearance—not shared scene content—is the usable cue.In contrast, context-correlated sampling draws the goal image from the trajectory’s world, where matching terrain could substitute for locating the target.Source: A EXAMPLE OF CONTEXT-DECOUPLED GOAL… · p. 15, Figure 6 · p. 15
  • Attacca succeeds in 0.68–0.96 of episodes across every tested goal background, versus 0.08–0.28 for fine-tuned ROCKET-2.The comparison covers mine diamond and hunt cow tasks, with 50 episodes per goal image.Source: A EXAMPLE OF CONTEXT-DECOUPLED GOAL… · p. 15

B SEARCH-TO-INTERACT DATASET

The HumanPlay dataset provides search-to-interact demonstrations with target-mask and behavioral-phase labels; the passages also describe errors in hindsight-generated supervision.

B SEARCH-TO-INTERACT DATASET

  • Hindsight labeling can misidentify held tools, spread masks onto the player’s body, or label visible targets absent, making visibility and grounding supervision noisy.The passages also state that search frames cannot be identified reliably from these labels.Source: B SEARCH-TO-INTERACT DATASET · p. 16, Figure 8 · p. 16
  • HumanPlay contains 1,160 demonstrations and 284,961 frames from experienced Minecraft players, spanning search, approach, and interaction in new worlds.The episodes cover Mine (822), Hunt (198), and Place (140), with counts including the 10% held out for validation.Source: B SEARCH-TO-INTERACT DATASET · p. 16
  • Renderer-derived masks label visible target instances, while behavioral phases mark frames as Search, Approach, or Interact and are used only as training targets.Search, Approach, and Interact account for 33%, 47%, and 20% of frames, respectively; frames without visible targets receive all-zero masks.Source: B SEARCH-TO-INTERACT DATASET · pp. 16–17

C IMPLEMENTATION DETAILS

The implementation initializes Attacca from ROCKET-2, samples goal frames during training, and specifies loss aggregation and weight sensitivity.

C IMPLEMENTATION DETAILS

  • Attacca is initialized from the released ROCKET-2 checkpoint, and training samples draw a new reference demonstration and target-visible goal frame each time.Goal frames are resized to 224 × 224; Table 6 lists the architecture and training settings.Source: C IMPLEMENTATION DETAILS · p. 17, Table 6 · p. 17
  • The behavior-cloning loss is summed over 128 window steps, while the phase loss is averaged over frames and the target loss averages visible- and absent-target frames.Table 7 describes the loss-weight sensitivity setup, varying one weight while holding the other at its default.Source: C IMPLEMENTATION DETAILS · p. 17, Table 7 · p. 17
  • Across a 16× weight sweep, Mine success changed at most 4.5 points for λtarget and 6.0 points for λϕ, within the ±6.7-point 95% confidence interval (n=200).Each weight was varied with the other held at its default.Source: C IMPLEMENTATION DETAILS · p. 17, Table 7 · p. 17

D SHORT-HORIZON TASK DETAILS

The short-horizon tasks test single targets that begin out of view across seen and held-out classes, using shared scenes and action budgets for all methods.

D SHORT-HORIZON TASK DETAILS

  • Each task evaluates 200 episodes, including 140 on seen classes, with a single initially out-of-view target and a goal image from another world.All methods use the same scenes and action budgets.Source: D SHORT-HORIZON TASK DETAILS · p. 18
  • Mine stages ores and held-out coral blocks on common terrain; Hunt tests mobs across layouts, while Place tests blocks placed onto target cells in worlds.Mine swaps staged target classes while keeping terrain fixed; Hunt and Place each include seven seen and three held-out classes.Source: D SHORT-HORIZON TASK DETAILS · p. 18
  • An episode counts as a clean success only if it achieves the requested outcome without any wrong-class interaction.Reaching the goal after a wrong-class interaction still counts as failure; termination and action budgets do not change.Source: D SHORT-HORIZON TASK DETAILS · p. 18
  • Tables 8–10 report per-class clean success and interaction precision, with 20 tests per class; Figure 10 shows four-frame successful episodes for seen and held-out classes.The figure’s upper row shows a training-seen class and its lower row a held-out class.Source: Figure 10 · p. 18, Table 8 · p. 19, Table 9 · p. 19

E LONG-HORIZON TASK DETAILS

The evaluation tests three multi-stage task chains with state carried across stages, then compares Attacca with baseline policies using their native goal interfaces.

E LONG-HORIZON TASK DETAILS

  • Diamond Pickaxe, Wolf Feeding, and Nether Portal require three, four, and five successive stages, respectively, with state carried over between stages.The chains use one fixed world and 50 paired policy seeds per method; no method is trained on these chains.Source: E LONG-HORIZON TASK DETAILS · p. 20, Figure 11 · p. 20, Figure 12 · p. 20
  • The fixed stage controller switches goals and supplies quotas, while macros and helpers handle selected item transitions.World state, agent position, and camera orientation carry over without teleportation or forced camera alignment.Source: E LONG-HORIZON TASK DETAILS · p. 21

F BASELINE IMPLEMENTATION

  • Baselines use their native goal interfaces, while ROCKET-1 and ROCKET-2 can receive online Molmo-and-SAM 2 goals or different-world goal images.All baselines except the released ROCKET-2 checkpoint are trained on the demonstrations.Source: F BASELINE IMPLEMENTATION · p. 21

F.1 GOAL SPECIFICATION PER METHOD

Methods are evaluated on the same task instances, differing in how they specify the target. Goal inputs include text, demonstration video, current-view targets, and masked images from another world.

F.1 GOAL SPECIFICATION PER METHOD

  • Text-goal policies name the target class, video-goal policies receive a demonstration clip, and ROCKET policies use current-view goals or no separate goal.STEVE-1 receives 16-frame clips and GROOT receives 128-frame clips; Molmo queries the current view every 30 steps for ROCKET-1 and ROCKET-2.Source: §F.1 · p. 22, Figure 14 · p. 22
  • ROCKET-2†, ROCKET-2, and Attacca receive the same masked goal image taken in a different world.The figure contrasts video goals, current-view goals, and different-world goals across methods.Source: §F.1 · p. 22, Figure 14 · p. 22
  • Short-horizon Mine and Place goal images share one camera pose and target position, while the figure set depicts cropped goals for all target classes.The goals show the target mask outlined in magenta.Source: Figure 15 · p. 22

G ADDITIONAL FAILURE ANALYSIS

The failure analysis contrasts online current-view goal construction, ROCKET-2 goal use, and Attacca on diamond and oak scenarios. The described rollouts show target-search failures for the online pipeline and successful task progress for Attacca.

G ADDITIONAL FAILURE ANALYSIS

H LIMITATIONS

The evaluation isolates Attacca’s low-level goal-directed control within a hierarchical agent, leaving other long-horizon task functions to the evaluation system. The authors have not evaluated an integrated planner system, and experiments are limited to Minecraft.

H LIMITATIONS

  • Evaluation isolates low-level control: a scripted stage controller issues goals and detects completion, while helpers handle task-transition items.For example, a macro moves cooked beef to the hotbar and a helper equips it.Source: H LIMITATIONS · p. 24
  • An integrated planner system has not been evaluated, and extending Attacca beyond Minecraft remains future work.A planner would handle subgoal achievement and dispatch; language subgoals would still need conversion into masked goal images.Source: H LIMITATIONS · p. 24
Loading 2610.07785v1…