PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards a Systematic Study of Embodied System 2 over Full Task Execution Cycles

Yuxi Wei, Ye Fang, Shuai Yang, Tong Wu, Tai Wang, Xiaojuan Qi

arXiv:2610.17657v1cs.RO

TL;DR

Open-loop evaluations use fixed observations, while some closed-loop settings omit execution-time evidence or assume idealized outcomes, limiting evaluation of evidence accumulation, monitoring, and recovery. SCOPER combines a benchmark with abstract goals, ongoing visual observations, shared rule-based actions, and controlled failures, and a training-free harness that links evidence and verified outcomes to an adaptive agenda. SCOPER-harness raises overall SR by 24.8/28.6 percentage points on BEHAVIOR and 16.9/11.1 on ProcTHOR for Qwen3.5-9B/Qwen3.8-27B, respectively; harnessed Qwen3.5-9B surpasses direct Qwen3.8-27B on both backends.

  • Problem

    Open-loop evaluations use fixed observations, while some closed-loop settings omit execution-time evidence or assume idealized outcomes, limiting evaluation of evidence accumulation, monitoring, and recovery.

    Source: §1 · pp. 1–2

  • Method

    SCOPER combines a benchmark with abstract goals, ongoing visual observations, shared rule-based actions, and controlled failures, and a training-free harness that links evidence and verified outcomes to an adaptive agenda.

    Source: §1 · p. 2

  • Results

    SCOPER-harness raises overall SR by 24.8/28.6 percentage points on BEHAVIOR and 16.9/11.1 on ProcTHOR for Qwen3.5-9B/Qwen3.8-27B, respectively; harnessed Qwen3.5-9B surpasses direct Qwen3.8-27B on both backends.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    SCOPER provides a setting for diagnosing and improving embodied reasoning through a complete task execution cycle.

    Source: §6 · p. 10

Abstract

from arXiv · show

An embodied brain, or System 2, must turn an abstract goal into actions while discovering relevant scene conditions and checking whether executed skills achieved their intended effects. Open-loop evaluations cannot test observations shaped by an agent's decisions, while some closed-loop evaluations provide limited evidence during execution or assume idealized skill outcomes. We introduce SCOPER, a suite for studying observation-grounded planning, execution, and recovery. SCOPER-bench combines high-level goals, ongoing visual observations, a shared rule-based System 1 for low-level actions, and controlled execution failures. Its construction and validation pipeline yields 870 tasks across BEHAVIOR and ProcTHOR, complemented by about 3,500 situated questions tied to the evaluated agent's execution. Benchmark results reveal difficulties in retaining scene evidence, monitoring progress, and recovering from unexpected outcomes. To address these difficulties, SCOPER-harness combines Evidence Memory and Progress Verification with a dynamic task agenda and bounded exploration, without additional model training. Across two evaluated open-source backbones and two simulators, the harness improves task success by 11.1-28.6%; a harnessed 9B model outperforms a directly executed 27B model on both simulators. In an offline next-action evaluation using observations from 200 real-robot demonstrations, the harness also improves full-call accuracy by 14.7% and 15.3%. SCOPER provides a setting for diagnosing and improving embodied reasoning throughout a complete task execution cycle.

1 INTRODUCTION

Existing evaluations leave important parts of embodied System 2 untested: agents must acquire decision-relevant evidence and recover from execution failures in a closed loop. SCOPER combines a benchmark for those demands with a harness that improves performance across evaluated models.

Evaluation gap

  • Open-loop evaluations use fixed observations, while some closed-loop benchmarks omit execution-time evidence, assume idealized outcomes, or provide detailed procedures.These settings limit evaluation of evidence accumulation, outcome monitoring, and recovery.Source: §1 · pp. 1–2

SCOPER-bench

  • SCOPER-bench evaluates abstract-goal decomposition, ongoing visual observation, execution, and recovery using a shared rule-based System 1 across 870 tasks.Its construction and validation pipeline spans two simulators and includes controlled failures.Source: §1 · p. 2

SCOPER-harness

  • SCOPER-harness combines Evidence Memory and Progress Verification with a dynamic agenda and anti-deadlock control to balance information gathering and task advancement.It maintains descriptions of observed scene information, checks expected outcomes after execution, and updates pending subtasks from evidence and verification.Source: §1 · p. 2

Experiments

  • The strongest directly evaluated model achieves 50.5% task success on BEHAVIOR and 35.1% on ProcTHOR, while SCOPER-harness improves performance across evaluated open-source models.Reported harness gains reach 28.6 percentage points, and a smaller harnessed model performs comparably to the strongest evaluated model.Source: §1 · p. 2

2 RELATED WORK

Prior embodied evaluations cover reasoning over recorded observations and interactive simulated execution, but leave important questions about decision-dependent evidence and recovery unresolved. SCOPER-bench isolates these demands with shared, controllable skills, multi-frame observations, and explicit failure injection.

Open-loop Evaluation of Embodied Reasoning

  • Open-loop benchmarks assess reasoning over recorded observations, which remain independent of the evaluated agent’s decisions.They cannot establish whether agents acquire missing evidence through action, revise decomposition during execution, or recover from failures.Source: §2 · p. 3

Closed-loop Embodied Benchmarks

  • Closed-loop benchmarks often emphasize observations at action boundaries and do not systematically test recovery from deviations between intended and actual skill effects.End-to-end performance in BEHAVIOR also makes high-level reasoning difficult to disentangle from low-level execution.Source: §2 · p. 3

Agentic designs for embodied intelligence

  • SCOPER-harness couples scene discovery, evidence-driven agenda revision, and verified progress to determine an abstract goal’s required work.This emphasis distinguishes its approach from related agentic designs combining planning, memory, and execution feedback.Source: §2 · p. 3

3 SCOPER-BENCH

SCOPER-bench tests information acquisition, task decomposition, subtask selection, and plan revision during execution, with completion and situated questions assessing outcomes and understanding. Its shared execution infrastructure, controlled perturbations, and validated task construction provide a common evaluation setting across two simulators.

3.1 TASK FORMULATION AND EVALUATION PROTOCOL

  • Tasks test System 2 information acquisition, goal decomposition, subtask selection, and plan revision from observations and outcomes.Four task groups cover routine execution, observation-dependent decisions, failure recovery, and composite cross-step dependencies.Source: §3 · p. 3, §3.1 · p. 3
  • A task instance specifies an instruction, object-role constraints, initial conditions, goal predicates, optional execution events, and attached questions.Evaluated System 2 receives instructions, visual observations, and prescribed public tool information, not privileged construction-time guidance.Source: §3.1 · p. 4

3.2 ORACLE SYSTEM 1 AND CONTROLLED EXECUTION

  • A shared rule-based Oracle System 1 provides active observation, navigation, and manipulation skills, while controlled events alter behavior or outcomes to require recovery or replanning.The common skill infrastructure controls low-level variation across evaluated models.Source: §3.2 · p. 4

3.3 TASK CONSTRUCTION AND VALIDATION

  • The construction pipeline produces 870 executable tasks from 35 base templates and their compositions, using static checks and privileged teacher validation.Failure-injection instances must recover to completion; about 3,500 situated questions are screened against teacher traces and answered from the evaluated episode’s state.Source: §3.3 · p. 5

4 SCOPER-HARNESS

SCOPER-harness addresses evidence retention and unsuccessful execution detection with a training-free system that connects scene information and verified outcomes to an evolving task agenda. It updates memory and progress from observations, supports local recovery, and uses anti-deadlock control to balance exploration with task advancement.

4 SCOPER-HARNESS

  • SCOPER-harness uses Evidence Memory, Progress Verification, a dynamic agenda, and anti-deadlock control without additional privileged simulator information or model training.Its model-based components share a fixed-parameter backbone through role-specific prompts.Source: §4 · p. 5

4.1 EVIDENCE MEMORY AND ADAPTIVE TASK DECOMPOSITION

  • Evidence-grounded agenda revision activates conditional requirements and updates or reopens task items based on committed memory.Observed running water can activate faucet shutoff work, but does not establish that the faucet has been turned off.Source: §4.1 · p. 6

4.2 PROGRESS VERIFICATION AND RECOVERY

  • Progress Verification compares expected effects with before-and-after images, recording success, failure, or ambiguity to guide planning and recovery.Verified effects update memory and the agenda; unresolved or failed outcomes guide further observation, renewed approach, or retry.Source: §4.2 · p. 6

4.3 ANTI-DEADLOCK CONTROL

  • Anti-deadlock control tracks exploration and retry budgets, then restricts tools or requests reselection at exploration limits to promote approach and interaction.Budget exhaustion can terminate execution or allow continuation with unresolved state, depending on the recovery branch.Source: §4.3 · pp. 6–7

5 EXPERIMENTS

The experiments compare eight models under a shared execution interface and evaluate a training-free harness against direct execution. The harness improves task success across models and simulators, and also improves next-call accuracy on real-robot observations.

5.1 EXPERIMENTAL SETUP

  • Eight models use shared Oracle System 1 and 512×512 RGB observations without privileged simulator state; episodes allow 24/36/48 high-level calls for one/two/three subtasks, including QA.The evaluation compares direct execution and the same-backbone, training-free harness for Qwen3.5-9B and Qwen3.8-27B.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • GPT-6 Astra leads direct execution with 50.5%/35.1% overall SR on BEHAVIOR/ProcTHOR, while 58.6% of its failures exhaust the action budget and embodied-specialized models show no consistent advantage.Astra's necessary-action coverage is higher on both backends, and only one of its 500 failed episodes ended in a parsing error.Source: §5.2 · pp. 7–8
  • SCOPER-harness raises overall SR by 24.8/28.6 percentage points on BEHAVIOR and 16.9/11.1 on ProcTHOR for Qwen3.5-9B/Qwen3.8-27B, respectively; the 9B model surpasses direct 27B execution on both backends.SR, progress, and necessary-action coverage improve in every task group for both models on both backends.Source: §5.2 · p. 8, Table 1 · p. 8
  • Harness trigger rates rise in all four backbone–backend combinations and triggered-answer accuracy rises in three, but execution paths activate different question sets.Triggered accuracy therefore is not an identical-question comparison.Source: §5.2 · p. 8
  • Across 4,813 failed normal-image episodes, 47.7% stopped prematurely, 30.7% exhausted the action budget, and 40.2% had at least three consecutive identical tool calls; indicators overlap.Parsing failures accounted for 12.8%, and reference-binding issues appeared in 17.3%.Source: §5.2 · p. 8

5.3 HARNESS COMPONENT ABLATIONS

  • Removing anti-deadlock control lowers overall SR by 11.8/17.6 percentage points on ProcTHOR/BEHAVIOR; removing recovery lowers failure-recovery SR from 39.2% to 27.7% and 65.4% to 21.5%, respectively.Without recovery, routine execution and observation-dependent performance remain largely stable or improve slightly; anti-deadlock removal also reduces progress and necessary-action coverage in every task group.Source: §5.3 · pp. 8–9
  • Removing Evidence Memory lowers observation-dependent SR from 37.8% to 33.3% on ProcTHOR and from 34.4% to 17.8% on BEHAVIOR.Composite SR on BEHAVIOR also falls from 30.0% to 23.0%, with a smaller change on ProcTHOR.Source: §5.3 · p. 9

5.4 REAL-OBSERVATION EVALUATION

  • On 200 real demonstration episodes, SCOPER-harness raises full-call accuracy from 33.9% to 48.6% for Qwen3.5-9B and from 44.3% to 59.6% for Qwen3.8-27B.Observation-critical call accuracy improves by 26.7/27.8 percentage points, respectively, and repair-call accuracy by 28.4 points for both.Source: §5.4 · pp. 9–10

6 CONCLUSION

SCOPER studies embodied System 2 through closed-loop, observation-grounded planning and recovery. Its evaluation protocol specifies what agents observe and how execution, stopping, and outcomes are scored.

6 CONCLUSION

  • SCOPER-bench combines high-level goals, execution-time observations, and controlled failures in 870 tasks across two simulators, while SCOPER-harness connects scene evidence and verified outcomes to an evolving agenda.The conclusion identifies richer, more continuous action spaces and more complex, diverse, dynamic scenes as future extensions.Source: §6 · p. 10

A.1 INTERACTION, OBSERVATIONS, AND INFORMATION ACCESS

  • Episodes proceed from task and scene initialization to a model skill request, reference binding, System 1 execution, returned images and records, and evaluator checks.The model receives public inputs; ground-truth states, progress, and teacher trajectories remain internal.Source: §A.1 · p. 17
  • Navigation, scanning, and search return up to eight process frames, while manipulation returns before-and-after images; direct models can receive up to 32 frames and the harness routes memory with up to four recent frames.The harness maintains observations by uniformly sampling up to eight frames from navigation, scanning, and search.Source: §A.1 · p. 17

A.2 EVALUATION PROMPTS AND PER-DECISION CONTEXT

  • Each decision turn supplies the current step, remaining budget, prior public requests, and current visual input, and the model assesses action outcomes from images rather than success labels.Prompts allow one tool call at a time; prior reasoning is not retained as permanent environmental evidence.Source: §A.2 · p. 18

A.3 BUDGETS AND TERMINATION

  • Atomic, two-subtask, and three-subtask episodes share budgets of 24, 36, and 48 high-level calls, counting observation, manipulation, answering, and stopping.Internal verification, memory, and reselection calls are tracked separately; after stopping, pending questions use an answer-only phase without resumed physical actions.Source: §A.3 · p. 18
  • Task completion is scored from the environment state and applicable process conditions at scoring time, with a stop call freezing the goal score and budget exhaustion without stopping leaving the stopping requirement unmet.Runner failure and physical task success are reported separately; composite-task handoffs do not replace the parent task’s goal check.Source: §A.3 · pp. 18–19

A.4 METRICS AND AGGREGATION

This section defines task, progress, action-coverage, QA, and step metrics and their aggregation. It also cautions that average call counts must be read alongside task completion.

A.4 METRICS AND AGGREGATION

  • Success rate measures the fraction of episodes satisfying final success conditions, including goal, applicable process, and stopping requirements.The scored episode set includes valid evaluation successes and agent failures; infrastructure errors and unfinished runs are recorded separately.Source: §A.4 · p. 19
  • Task progress averages each episode’s fraction of satisfied goal predicates, so tasks with more predicates do not automatically receive greater weight.Goal items may include environmental predicates or evaluator-recorded process conditions.Source: §A.4 · p. 19
  • Necessary action coverage measures execution-confirmed matches to a validated teacher-reference action set, normalized within each task.It is distinct from shortest-path efficiency and final goal correctness.Source: §A.4 · p. 19
  • Situated QA reports trigger rate and triggered-question accuracy separately; triggered but unanswered questions count as incorrect, while untriggered questions reduce trigger rate.Different execution paths can expose different question sets, so accuracy is interpreted alongside trigger rate.Source: §A.4 · p. 20
  • Average steps counts high-level calls across successful and failed episodes, excluding low-level simulator actions and internal harness calls.Fewer steps can reflect early failure, and episode metrics are weighted by episode counts while QA metrics use their respective question denominators.Source: §A.4 · p. 20

B HIGH-LEVEL ACTION COUNTS

  • On ProcTHOR, RoboBrain2.5-8B uses 10.08 steps with 5.8% success, while Qwen3.8-27B uses 20.68 steps and achieves 21.1% success.The section cautions that shorter trajectories do not necessarily indicate more effective execution.Source: B HIGH-LEVEL ACTION COUNTS · p. 20, Table 4 · p. 20

C BLIND EVALUATION

The blind evaluation replaces visual inputs with black images to test whether SCOPER-bench requires visual observations. Models show near-zero task completion under this condition.

C BLIND EVALUATION

  • With visual inputs replaced by black images, all models exhibit near-zero task completion, indicating that benchmark performance requires visual perception rather than simple instruction decomposition.Tables 5 and 6 report results for BEHAVIOR and ProcTHOR, respectively.Source: C BLIND EVALUATION · p. 20, Table 5 · p. 21, Table 6 · p. 21

D REAL-ROBOT NEXT-TOOL-CALL EVALUATION

The real-robot next-tool-call evaluation uses demonstrations segmented into decision nodes, then tests call prediction from restricted observations and reference history. The accompanying skill interface standardizes low-level execution while varying in backend-specific tools.

D.1 DATA AND DECISION-NODE CONSTRUCTION

  • The evaluation uses 200 distinct AgiBotWorld2026 source episodes, grouped into normal execution, observation-dependent, and anomalous or incomplete-action repair cases.The groups contain 80, 60, and 60 episodes, respectively; each sample covers a contiguous task scope in one source video.Source: §D.1 · p. 21
  • Decision nodes follow semantic action segments and predict the next reference call from the reference prefix, beginning after the first segment.Inputs use fixed-grid head-camera frames within the current segment and withhold future frames, labels, and reference answers.Source: §D.1 · p. 21, §D.2 · p. 22

D.2 OBSERVATIONS AND CHALLENGE CATEGORIES

  • Observation-dependent instructions require inspection without revealing the state to discover, while repair cases distinguish naturally occurring anomalies from constructed incomplete-action continuations.The constructed continuations do not imply an actual grasp failure or controller retry.Source: §D.2 · p. 22

D.3 HARNESS STATE ALONG REFERENCE EXECUTION

  • Harness memory and agenda updates follow the reference prefix using current-segment images and earlier observations retained through memory, without executing predicted calls or committing their expected effects.Subsequent updates use reference video and actions without future frames or answer labels.Source: §D.3 · p. 22

D.4 OUTPUT INTERFACE AND METRICS

  • Call Accuracy requires the predicted tool and all necessary arguments to match a complete reference call, while Tool Accuracy checks only the tool name.Because predictions do not control the robot or subsequent observations, this evaluation reports no task success, goal progress, action coverage, or execution steps.Source: §D.4 · p. 22

E ORACLE SYSTEM 1 AND TEACHER POLICY

  • Oracle System 1 supplies shared rule-based skill execution for evaluated models and the teacher, converting high-level requests into scene changes and visual observations.The teacher selects skill sequences for reference execution and validation.Source: E ORACLE SYSTEM 1 AND TEACHER POLICY · p. 22, §E.1 · p. 23

E.1 UNIFIED SKILL ABSTRACTION

  • Skill rows describe shared semantic capabilities rather than identical backend API names, and available tools differ between ProcTHOR and BEHAVIOR.Models receive only the tool definitions available in their current backend.Source: Table 7 · p. 23, §E.2 · p. 23

E.2 TOOL ARGUMENTS AND MODEL-FACING OUTPUTS

  • System 1 state transitions are not exposed as state values or effect labels; models receive prescribed execution images and a request-completion summary.BEHAVIOR exposes some dedicated state-processing tools, while corresponding ProcTHOR effects arise through container, device, and placement operations.Source: §E.2 · p. 24

E.3 NAVIGATION AND OBSERVATION IMPLEMENTATION

Navigation and observation support scene discovery during execution, while synchronized rule-based manipulation and controlled interventions create tasks that require outcome checking and recovery. The section specifies how failures and disturbances are activated, applied, and validated.

E.3 NAVIGATION AND OBSERVATION IMPLEMENTATION

  • Navigation uses backend-specific reachability searches to reach interaction poses, while execution callbacks deliver images from scanning and target search for action selection.ProcTHOR searches reachable positions; BEHAVIOR uses an occupancy grid that accounts for agent footprint and prevents diagonal corner cutting.Source: §E.3 · p. 24

E.4 MANIPULATION AND STATE SYNCHRONIZATION

  • Manipulation updates object poses, relations, and properties through rules rather than low-level contact dynamics, with synchronized visual state and observations supporting outcome checks.The teacher organizes skill calls into reference trajectories, and evaluated models use the same System 1 skill interface.Source: §E.4 · pp. 24–25

F CONTROLLED FAILURE INJECTION AND SCENE DISTURBANCES

  • Controlled interventions convert routine skill execution into closed-loop tasks by changing skill outcomes or the scene after milestones, pairing the same goal with different recovery challenges.Source: F CONTROLLED FAILURE INJECTION AND… · p. 25

F.1 ACTIVATION AND SCHEDULING

  • Events specify an eligible skill or milestone, target, activation conditions, state intervention, and firing bound; the scheduler records event history and whether changes were applied.Source: §F.1 · p. 25

F.2 FAILURE MECHANISMS

  • Six execution-failure mechanisms alter holding, placement, openness, or navigation outcomes, requiring later actions to respond to observed results.Incomplete navigation leaves the agent at an intermediate or pre-terminal pose, requiring further navigation without adding a physical obstacle.Source: §F.2 · pp. 25–26, Table 9 · p. 25

F.3 STATE-CHANGING DISTURBANCES

  • Scene disturbances can reverse previously achieved states or relocate objects, requiring the model to revisit facts and restore disrupted prerequisites or goals.Validation checks final goals and required interventions, while models assess consequences from protocol-provided observations and public interaction information.Source: §F.3 · p. 26

G TASK TEMPLATES AND CONSTRUCTION

This section describes SCOPER’s task templates and construction pipeline, covering routine, observation-dependent, recovery, and composite goals through validated scene instances and instructions grounded in observations. It also explains how public action requests are bound to scenes and resolved at runtime while preserving the need to inspect observations and verify required states.

G.1 DATASET STATISTICS AND SCENE SAMPLING

  • 870 tasks were constructed: 420 in BEHAVIOR and 450 in ProcTHOR, with 35 base templates and four task groups represented across the benchmarks.The target was 450 tasks per simulator; 30 BEHAVIOR instances were excluded because of limited scene matches. ProcTHOR scenes came from its 12,000-scene pool, while BEHAVIOR matching used 35 filtered scenes.Source: §G.1 · p. 26

G.2 TEMPLATE SPECIFICATION AND INSTANTIATION

  • Templates specify object roles, affordances, relations, initial states, goal predicates, instruction patterns, and event configurations; scene matching and validation check constraints and task feasibility.Matching filters by object type and affordance before checking spatial constraints; initialization configures concrete objects, and validation tests predicates, reachability, and whether goals already hold.Source: §G.2 · pp. 26–27
  • Instructions expose objectives, scope, and handling rules without revealing instance bindings or action sequences, so agents must identify targets, prerequisites, and operations from observations.For universal inspection tasks, construction includes all actionable issues within the scoring scope and initializes other relevant objects compliantly; clean-dishware storage distinguishes clean targets from dirty distractors.Source: §G.2 · p. 27

G.3 REPRESENTATIVE TEMPLATES

  • Representative templates span placement and preparation workflows, observation-driven inspection and cleaning, injected execution failures, and multi-child composites with subtask-scoped events and goals.Examples include safety inspection of an open container and running faucet, dropped-placement recovery, and composites whose handoffs are managed by the evaluator.Source: §G.3 · p. 27

G.4 TEMPLATE STRUCTURE AND PREDICATE CATALOG

  • The benchmark has 35 templates grouped by capability, with placement, distractor-exclusion, optional closure, and event predicates distinguishing task goals and execution challenges.Conditional container inspection requires a valid, visually discriminable interior view; merely requesting an observation does not satisfy inspection, and initial openness alone cannot complete the task.Source: §G.4 · pp. 27, 28, 30

G.5 PROGRAMMATIC INSTANTIATION AND COMPOSITION

  • Programmatic recipes instantiate five observation-conditioned inspection and workflow templates by configuring concrete issues, roles, initial states, and handling goals; long-horizon tasks combine two or three compatible subtasks.Composite construction retains child roles, events, and goal ownership while scheduling events and QA within each child’s scope.Source: §G.5 · p. 30

G.6 TASK INSTRUCTIONS AND EXECUTION MECHANISMS

  • The 35 atomic templates comprise 13 routine, nine observation-dependent, and 13 recovery templates; recovery covers four initially present anomalies, six skill-outcome failures, and three mid-execution disturbances.Category labels follow template families rather than exclusive capabilities; for example, routine clean-dishware selection still requires visual judgment, and construction mechanisms are not additional model input.Source: §G.6 · pp. 30–31
  • Representative instructions ask agents to find and place unseen objects, inspect rooms for safety or misplaced items, and discover dirty dishes or coexisting issues before acting.These examples pair public instructions with task-specific execution mechanisms; situated QA can be attached separately without changing the underlying task category.Source: Table 12 · p. 31, §G.6 · pp. 31–32
  • Task instructions require inspection and conditional handling, such as washing dirty dishware, switching off faucets, and closing containers when contents or initial states warrant it.Examples also pair placement or food storage with additional room checks, making required work depend on scene conditions.Source: §G.6 · pp. 32–33
  • Injected failures require agents to reacquire dropped or slipped objects, continue incomplete navigation, redirect from distractors, and finish partial opening or closing.Milestone-triggered disturbances can also reclose a destination, relocate an object, or reverse cleanliness, requiring the agent to restore the intended state.Source: §G.6 · p. 32
  • Composite tasks combine routine, observation-dependent, and recovery requirements while preserving child goals and event scopes.Templates include added runtime instructions and condition-dependent container tasks, alongside disturbances that require recovery after placement, grasping, navigation, or state changes.Source: §G.6 · pp. 32–33

H.1 CONSTRUCTION-TIME BINDING AND PUBLIC REFERENCES

  • Construction-time binding maps template roles to concrete scene objects using category, affordance, and spatial constraints, supporting initialization, event scheduling, scoring, and runtime resolution.The model selects public object categories and operations; infrastructure maps those requests to scene instances.Source: §H.1 · p. 33, H BINDING PROCEDURE · p. 33
  • Object requests must use complete public category names or listed aliases, and navigation must use public room labels; private roles and raw simulator identifiers are invalid.Broad concepts such as dishware or storage must be translated through observation into interface-supported categories.Source: §H.1 · p. 33

H.2 RUNTIME RESOLUTION AND INTERACTION READINESS

  • At runtime, resolution uses admission rules and may prioritize recent interaction focus or task-designated instances before aliases, activated observations or memory, and permitted scene references.If the applicable rules cannot resolve ambiguity, the request remains unresolved rather than selecting an arbitrary object.Source: §H.2 · p. 33
  • A valid public name does not guarantee an object exists, is localized, or is ready for manipulation; interaction also requires approach and satisfied access and affordance preconditions.After an object moves or drops, its retained identity does not guarantee the previous interaction pose is still valid.Source: §H.2 · pp. 33–34
  • Resolution can use task mappings and recent focus without revealing simulator identifiers, but the model must still choose the operation and use returned images to verify outcomes.Likewise, resolving dishware does not reveal whether it is dirty or decide whether it should be washed.Source: §H.2 · p. 34

H.3 EXECUTION RESULTS AND THE MODEL INFORMATION BOUNDARY

This appendix specifies how execution information is bounded and exposed, how situated questions assess episode understanding, and how the harness maintains task state and handles uncertain outcomes. It details the component inputs, evidence rules, and control limits behind the execution loop.

H.3 EXECUTION RESULTS AND THE MODEL INFORMATION BOUNDARY

  • System 1 executes admitted skills while System 2 selects actions, interprets observations, and assesses outcomes; the model receives public skill requests and images, not private binding diagnostics.Unresolved or rejected requests remain in internal accounting.Source: §H.3 · p. 34

I.1 QUESTION TYPES AND CONSTRUCTION

  • Situated QA uses episode-resolved questions to assess object states, spatial relations, action outcomes, issues, obligations, and task progress.Eight templates are generated from task semantics, goal predicates, and reference traces; candidates are screened for valid activation and uniquely resolvable answers.Source: I SITUATED QA · p. 34, §I.1 · p. 34

I.3 SCORING

  • QA is answered in a separate model call without further exploration or manipulation, and is scored separately from physical task completion.Accuracy is interpreted with trigger rate because execution paths can produce different question sets; valid posted questions left unanswered count as incorrect.Source: §I.1 · p. 35, §I.3 · p. 35

I.2 ACTIVATION, QUEUING, AND ANSWER TIME

  • Questions activate at configured tool results, action thresholds, handoffs, new instructions, or stop milestones, and each fires at most once.Action-outcome answers snapshot the triggering action; state and progress answers freeze when posted.Source: §I.2 · p. 35

J HARNESS DETAILS

  • The harness uses role-specific calls for observation, agenda revision, action selection, and verification, while deterministic code handles structural checks, memory routing, and bounded control.Observation maintenance samples up to eight frames; the policy uses the latest four, and verification compares existing pre- and post-action frames.Source: J HARNESS DETAILS · p. 35, §J.1 · pp. 35–36, §J.5 · p. 38

J.2 AGENDA REPRESENTATION AND UPDATES

  • The agenda tracks each requirement’s mode, status, evidence, and optional next intent, with at most eight items and one active item.Only evidence supporting the requirement can establish completion; later contradictory observations reopen affected items, and instruction or relevant-memory changes trigger revision.Source: §J.2 · p. 36

J.3 MEMORY MAINTENANCE AND ROUTING

  • Memory separates scene facts, execution incidents, and obligations, then reconciles observations and routes relevant records to the policy.Unknown or unobservable assessments do not authorize negation; stale facts and resolved records are preferentially removed, while unresolved incidents and obligations cannot be silently deleted.Source: §J.3 · pp. 36–37

J.4 VERIFICATION, RECOVERY, AND PLANNING ASSUMPTIONS

  • Visual verification distinguishes supported effects, observable discrepancies, and insufficient evidence; recovery can request observation, approach, prerequisite satisfaction, or retry.Repeated exploration, recovery observations, retries, and model reselection have finite budgets; temporary assumptions cannot count as verified facts or support agenda completion.Source: §J.4 · p. 37, §J.5 · p. 37

K EXECUTION TRACE VISUALIZATION

The execution traces illustrate how the harness responds to failures, discovers observation-dependent requirements, and handles dependencies between actions. Real-observation examples also distinguish evidence revealed during execution from synthetic early-stop points.

K EXECUTION TRACE VISUALIZATION

  • Examples show recovery from misplaced objects, slipped grasps, wrong-target navigation, partial drawer openings, and a cabinet that closes before a second object is stored.The traces depict locating or retrieving objects, correcting navigation, inspecting and retrying, and reopening the cabinet.Source: K EXECUTION TRACE VISUALIZATION · p. 38, Figure 5 · p. 40, Figure 6 · p. 41
  • Other traces show observations activating cleaning or safety requirements and action dependencies, such as putting down a knife before placing apple slices.The mug and bowl examples proceed from discovering dirt to washing and storage; the safety example identifies a running faucet and open fridge.Source: K EXECUTION TRACE VISUALIZATION · p. 38, Figure 11 · p. 46, Figure 12 · p. 47
  • Real-data examples show movement and reinspection revealing relevant evidence, while synthetic early-stop nodes truncate reference actions rather than represent recorded malfunctions.Later reference frames showing eventual outcomes are excluded from model inputs.Source: K EXECUTION TRACE VISUALIZATION · p. 38, Figure 17 · p. 52, Figure 18 · p. 53
  • Algorithm 1 sequences evidence observation and memory commitment, agenda revision, memory routing, guarded action selection, and effect commitment when verification is required; Algorithm 2 bounds policy requests and applies recovery-exhaustion rules.Source: K EXECUTION TRACE VISUALIZATION · pp. 39, 43

L HARNESS MODEL-CALL OVERHEAD

SCOPER uses additional model calls around action selection while retaining the same high-level environment-step budgets as direct execution.

L HARNESS MODEL-CALL OVERHEAD

  • Approximately 3.7× as many model calls as direct execution are used on the complete 870-task benchmark, while internal calls add no environment-interaction steps.Calls support discovery, reconciliation, agenda revision, and verification in addition to policy action selection.Source: L HARNESS MODEL-CALL OVERHEAD · p. 54
Loading 2610.17657v1…