PaperlayerView exact revision on arXiv

Source-linked AI summary

BT-Agent: Deploying Onboard Robot Agents with Behavior Trees and Multi-Fidelity Evaluation

Fabio Hübel, Adam Lilja, Yue Yin, Siming He, Claire Tomlin, Shankar Sastry, Jonas Frey, Marco Pavone

arXiv:2610.13862v1cs.RO

TL;DR

Onboard robot agents must combine task reasoning with responsive execution, but the reviewed approaches lack a shared interface for skill state, execution conditions, and interruption. BT-Agent pairs model-driven decisions with reactive behavior-tree skills and evaluates designs across text and simulation. Onboard trials demonstrate task completion, with local checks able to interrupt motion without another model decision.

  • Problem

    Onboard robot agents need to reason about tasks and execute physical actions responsively, but existing approaches lack a shared interface for skill state, execution conditions, and interruption.

    Source: §I · p. 1

  • Method

    BT-Agent separates model-driven task decisions from predefined behavior-tree skills that recheck execution conditions and return structured feedback.

    Source: §I · p. 1

  • Results

    Across five tasks, BT-Agent completed 11 of 15 onboard trials, and local checks could interrupt motion without another model decision.

    Source: 1) Onboard task execution · p. 7, §VI · p. 8

  • Takeaways & Limitations

    Onboard trials combine task reasoning with reactive execution, while text evaluation supports inexpensive development and simulator trials expose skill and decision failures.

    Source: §VI · p. 8

    The Actor can sequence and parameterize available skills but cannot change their implementations or guards at runtime, so new capabilities require skill authoring and platform-specific integration.

    Source: A. Limitations and Future Work · p. 8

Abstract

from arXiv · show

Deploying language-model agents onboard mobile robots requires task reasoning and responsive execution within limited compute and memory. We propose BT-Agent, an onboard architecture combining LLM-driven task decisions with predefined behavior-tree (BT) skills. These skills enforce execution conditions and return structured failure feedback. We demonstrate fully onboard operation with a 9B model on a quadruped across five task categories. Intervention tests show that local checks stop ongoing navigation without waiting for another model decision, while the agent remains active to retry or report constraints. To compare agent designs efficiently, we introduce a 161-task multi-fidelity benchmark, evaluated in text on all tasks and in Habitat on 39. Text supports 10.9$\times$ higher evaluation throughput, while simulation reveals how perception and motion failures change task outcomes. BT-Agent's largest success gains over the strongest Act/ReAct baseline occur at 4B and 9B: approximately 10 and 5 percentage points. Ablations identify structured tool calls and informative failure feedback as the most useful components.

I. INTRODUCTION

Onboard robot agents must combine task reasoning with responsive physical execution, yet existing approaches lack a shared interface for skill state, execution conditions, and interruption. BT-Agent pairs model-driven decisions with predefined behavior-tree skills, and the paper introduces a multi-fidelity benchmark and reports onboard trials.

I. INTRODUCTION

  • Onboard operation must accommodate task reasoning and responsive physical execution while sharing finite compute and memory with perception and control.Robots must track partial information and adapt when actions fail during search, inspection, and manipulation.Source: §I · p. 1
  • BT-Agent separates task-level model decisions from predefined behavior-tree skills that enforce execution conditions and return structured feedback.Local guards recheck conditions on each tick and can interrupt a running action without another model decision.Source: §I · p. 1
  • Fully onboard trials demonstrate task completion with a 9B model, while intervention tests validate reactive skill execution.The evaluation also includes five task categories across the paper’s robot demonstrations.Source: §I · p. 2
  • The 161-task benchmark connects text evaluation with simulation through shared task goals and skill interfaces, with Habitat trials on 39 primary tasks.Text supports controlled fault injection; paired Habitat trials examine how perception and motion change outcomes.Source: §I · pp. 1–2
  • Text evaluation achieves 10.9× higher throughput than Habitat at eight concurrent workers.The contributions also characterize task success and inference cost across model sizes and execution backends.Source: §I · p. 2

II. RELATED WORK

Prior robot-agent work selects skills or composes programs, while the reviewed approaches lack a shared skill interface for execution state, enforced conditions, and interruption. BT-Agent uses fixed behavior-tree skills within a feedback-driven model loop.

II. RELATED WORK

  • Prior work spans planning-and-acting behavior trees, visually conditioned or generated trees, failure analysis, and contract-grounded synthesis; BT-Agent instead selects fixed skills with local checks and interruption support.These approaches include PA-BT, VLM-BT [10], BTGenBot-2 [11], Ahmad et al. [12], and contract-grounded synthesis [13].Source: §II · p. 2

III. BT-AGENT

  • BT-Agent assigns task-level choice to the model and execution enforcement to predefined behavior trees.The Actor selects a skill and arguments; the supervisor runs it and returns status and results before the next model call.Source: §III · p. 2, A. Agent · p. 2

A. Agent

  • The Actor uses task history and stored results to select skill arguments or claim completion, then uses returned feedback to retry, change actions, or report constraints.The supervisor converts a completion claim into terminal submission, and cancelling the task interrupts its running skill.Source: A. Agent · p. 2

B. BT Skills

BT skills provide fixed, reviewable execution logic with preconditions and reactive guards. Their feedback supports recovery, while physical response remains dependent on the checks, state feedback, and controller behavior.

B. BT Skills

  • Physical skills require eligible preconditions, including a known navigation target and any required prior observations, before execution.The framework validates arguments and resolves object references before accepting calls.Source: B. BT Skills · p. 2
  • Reactive guards recheck cancellation, elapsed time, and skill-specific conditions on every tick, interrupting a branch when a check fails without another Actor call.The Actor cannot disable these guards through skill arguments.Source: B. BT Skills · p. 2
  • Failed guards or leaves provide diagnostic feedback when available, but physical response depends on specified checks, state feedback, and controller stopping behavior.Each skill is a predefined behavior tree exposed as a named tool whose execution logic developers can review independently of model choices.Source: B. BT Skills · p. 2
  • BT-Agent does not prescribe a fixed skill library; the inspection benchmark uses the skills described in Section IV.Source: B. BT Skills · p. 2

IV. SIMULATION FRAMEWORK

The simulation framework pairs inexpensive text counterparts with externally supplied Habitat and Isaac environments, aligning tasks and skill interfaces while retaining backend-specific perception, dynamics, and control. Its scene coverage supports both primary-suite development and transfer-task evaluation.

IV. SIMULATION FRAMEWORK

  • Text counterparts make repeated agent and skill tests less costly while shared tasks, skill interfaces, and goal checks align the text and physics-simulation backends.Text also supports controlled fault injection.Source: §IV · p. 3

A. Scene Coverage

  • The primary suite contains 50 tasks across four scene configurations, with 39 Habitat-derived and 11 Isaac-derived tasks; 111 additional-home tasks test transfer with model weights unchanged.The primary suite includes 32 externally supplied tasks and 18 added indoor tasks varying ambiguity, target count, dependencies, and infeasibility.Source: A. Scene Coverage · p. 3

B. Aligned Execution

  • Text execution updates position, visibility, and interaction state from records, whereas Habitat uses rendered images, VLM perception, and depth-based navigation.Because VLM outputs can miss or misidentify objects, perception and simulated motion can change feedback and task outcomes despite shared instructions and completion requirements.Source: B. Aligned Execution · p. 3

C. Task Design

The task suite combines search, documentation, inspection, interaction, and infeasibility reporting, with text tasks aligned to simulator tasks. Fault injection tests how agents respond to skill-call failures under matched conditions.

C. Task Design

  • Tasks span search and visual documentation, inspection and reporting, physical interaction, and reporting absent or unreachable targets; most object-finding tasks require a new in-episode capture.The primary suite includes pressing or pushing and infeasibility-reporting tasks; transfer tasks vary instruction phrasing and may withhold targets until local survey.Source: C. Task Design · p. 3
  • Simulator geometry and camera records produce scene memory, navigation graphs, and task definitions; Habitat and Isaac tasks receive text counterparts, while simulator grading covers 39 primary Habitat tasks.Instance masks are construction evidence, not VLM input.Source: Figure 2 · p. 4
  • Table III gives shortened examples of task instructions and required evidence, while task checks include completion rules, reference sequences, and whether required information can be observed.Agents do not receive the scorer’s target lists or goal predicates.Source: Table III · p. 4, C. Task Design · p. 4

D. Fault Injection

  • Text fault injection changes execution state, returned observations, or both on eligible skill calls, including cases that report success despite incomplete execution.Matched nominal and fault conditions retain the same tasks, completion requirements, and agent settings.Source: D. Fault Injection · p. 4, Table IV · p. 4

V. EXPERIMENTS

  • The experiments ask which model and harness choices support onboard task execution, how aligned text and simulation aid development, and whether BT-Agent handles interruptions with local skill conditions.Source: §V · p. 4

A. Evaluation Setup

The evaluation compares task-level orchestration over shared skills and execution checks, across models and harnesses. BT-Agent matches or exceeds the strongest Act/ReAct baseline at every model size, with its largest combined-suite gains at 4B and 9B.

A. Evaluation Setup

  • Act selects textual skill commands, ReAct adds reasoning before commands, and adapted CaP-X generates and revises Python programs with persistent state.Methods share instructions, skills, and execution checks within each backend, so comparisons assess task-level orchestration over a common skill layer.Source: A. Evaluation Setup · p. 4

B. Model Size and Harness Design (Q1)

  • The main comparisons use 50 primary and 111 transfer tasks, three paired seeds per task, and Qwen3.5 NVFP4 models at 4B, 9B, 27B, and 122B-A10B.Act, ReAct, and BT-Agent receive 1,024 output tokens per action-selection call; adapted CaP-X receives 20,480 per program.Source: B. Model Size and Harness Design (Q1) · pp. 4–5
  • Success and median/P99 output tokens per episode are reported, including failures and context exhaustion.Median and P99 characterize typical and unusually expensive episodes.Source: B. Model Size and Harness Design (Q1) · p. 5
  • 10.1 and 5.0 percentage points are BT-Agent’s largest combined-suite gains over the strongest Act/ReAct baseline, at 4B and 9B, respectively.BT-Agent matches or exceeds that baseline at every size; transfer gains are positive at every size and largest at 9B, at 4.8 points over ReAct.Source: 1) Nominal task performance · p. 5

1) Nominal task performance:

Nominal traces show BT-Agent completing required task procedures that ReAct may omit, while component ablations indicate that structured calls and informative failure feedback contribute to success. Fault tests and output-cost results further distinguish agent configurations.

1) Nominal task performance:

  • In a 4B safe-task trace, ReAct visits the panels but submits without pressing them, while BT-Agent completes the sequence and photographs the medication.In an inscription task, both identify the printed word, but only BT-Agent records it using the report skill.Source: 1) Nominal task performance · p. 5

2) Component ablations:

  • Replacing structured tool calls lowers success at every size, most at 4B and 9B, where median output rises 1.9× and 1.7× on nominal tasks.Reducing failure feedback to a skill name and status lowers nominal success by 2–6 points across sizes; removing capture-specific guidance has little effect.Source: 2) Component ablations · p. 5
  • At 4B, BT-Agent reduces ReAct’s P99 output by 42% across 161 tasks; at 27B, ReAct matches BT-Agent’s nominal success with lower median and P99 output.Act has the lowest median output, consistent with direct action selection without reasoning.Source: 2) Component ablations · p. 5
  • The optional response watchdog lowers P99 at every size with minor impact on success, stopping episodes after repeated unusable outputs, equivalent queries, failed-skill outcomes, or token and call limits.Token checks follow complete responses, allowing small overshoots.Source: 2) Component ablations · p. 5
  • Under injected faults, BT-Agent has the highest observed success at every size; at 27B, its nominal tie with ReAct becomes a lead.In a paired 4B breakfast trace, BT-Agent changes viewpoint after a route blockage and completes, whereas ReAct retries the blocked route and repeats unsuccessful captures until exhausting its budget.Source: 2) Component ablations · p. 5

4) Manipulation transfer:

In Robosuite manipulation, BT-Agent leads combined success through its extension-task performance, while CaP-X leads the reference suite at some model sizes. Individual tasks illustrate a trade-off between programs for regular sequences and stepwise decisions when observations determine the next action.

4) Manipulation transfer:

  • Precise insertion is excluded because eight of nine code-policy successes exceed the 1,000-step simulator limit, which is checked only between program generations.Source: 3) Inference cost and fault robustness · p. 5
  • Stepwise agents use the reference-based action budget, whereas code policies have no action-count cap.Source: 3) Inference cost and fault robustness · p. 6
  • BT-Agent leads combined success at all three model sizes through its advantage on the extensions, while shared-skill CaP-X leads the reference suite at 9B and 27B.The 14-task evaluation combines six CaP-X reference tasks and eight extensions; it uses 20 repetitions per task.Source: 3) Inference cost and fault robustness · p. 6, Table VI · p. 6
  • At 27B, shared-skill multi-turn CaP-X completes 14/20 wiping trials while BT-Agent completes none within its time budget; on three-cube restacking, BT-Agent completes 15/20 versus CaP-X’s 4/20.Wiping follows a regular 101-call raster; restacking requires responding to the observed stack order.Source: 3) Inference cost and fault robustness · p. 6

O6 A8 All O6 A8 All O6 A8 All

The paired evaluation compares text and Habitat on shared tasks, showing broadly aligned method ordering but substantial differences in task-level outcomes. Text evaluation is much faster, while Habitat exposes effects of perception and motion.

1) Paired text–Habitat outcomes:

C. From Text to Simulator Evaluation (Q2)

  • The comparison covers 39 primary Habitat tasks in two configurations of one home, with four 9B-model agent configurations and three paired seeds per task.Isaac-derived tasks and the transfer suite are evaluated quantitatively in text only; both backends share instructions, scene priors, and a 900-second budget.Source: 1) Paired text–Habitat outcomes · p. 6

2) Evaluation cost:

  • 950 episodes/hour in text versus 87.3 in Habitat yields a 10.9× throughput difference with eight workers, excluding timeout attempts.The comparison uses a shared 9B model; oracle episodes average 0.245 seconds in text and 57.7 seconds in Habitat with ground-truth perception.Source: Figure 6 · p. 7, 2) Evaluation cost · p. 7

1) Paired text–Habitat outcomes:

  • Paired traces show that outcomes can diverge through decisions and stopping behavior as well as perception and motion.On wallet/backpack, survey and capture disagree about visibility in the same image, causing repeated failed captures; an absent-object Habitat run reports an unsuccessful lookup and submits where its text counterpart times out.Source: 1) Paired text–Habitat outcomes · p. 7

2) Evaluation cost:

Text trials support rapid evaluation and development, while simulator trials reveal skill and decision failures before deployment. Onboard trials demonstrate task execution with the full system on a quadruped.

2) Evaluation cost:

  • Text enables inexpensive development, while paired simulator trials expose skill failures and changed decisions before onboard deployment.The benchmark connects task goals and skill contracts across evaluation backends.Source: 2) Evaluation cost · p. 7

D. Onboard Operation and Reactive Skill Execution (Q3)

  • The onboard system runs the Actor, visual skills, retrieval, behavior-tree execution, planning, and control on a Spot quadruped with Jetson Thor and a 9B model.Motion requires operator approval; the scene graph is prebuilt from an earlier RGB-D mapping run.Source: 1) Onboard task execution · p. 7

1) Onboard task execution:

  • BT-Agent completes 11 of 15 trials across five onboard tasks, including multi-stop navigation and capture and operation of a chest through its floor panel.It finds an unmapped screwdriver using a mapped landmark; in the reporting task it captures the display but misreads the small clock faces and reports the wrong time.Source: 1) Onboard task execution · pp. 7–8
  • A guard rejects pressing before a survey, prompting the Actor to survey and retry; at a simulated 9% battery reserve, navigation is rejected and the Actor reports the constraint and abandons the task.Source: 2) Local guards and interruption · p. 8

2) Local guards and interruption:

Local behavior-tree checks stop or reject physical actions without waiting for another model decision, while the Actor remains available to retry or report constraints. The fixed skill interface limits runtime expansion and leaves reliable interpretation of ambiguous observations as an open challenge.

2) Local guards and interruption:

  • Across nine motion interventions, commanded velocity reaches zero a median 19 ms after local acceptance, before the next model-selected action 3.2–3.9 seconds later in interruption and timeout trials.The Actor receives timeouts as ordinary skill failures and retries navigation; a 5% battery reserve instead triggers a persistent motion hold and sit-down command, after which the Actor reports the constraint without retrying motion.Source: 2) Local guards and interruption · p. 8

VI. CONCLUSION

  • Onboard trials show the agent coordinating multi-step tasks while local checks enforce conditions and interrupt motion without waiting for another model decision.Useful components depend on the task and inference cost; text supports inexpensive development, while simulator trials expose skill and decision failures.Source: §VI · p. 8

A. Limitations and Future Work

  • The Actor can sequence and parameterize available skills, but their implementations and guards remain fixed at runtime; new capabilities require skill authoring and platform-specific integration.Reliably conveying task-relevant evidence remains an open challenge when observations are incomplete or ambiguous.Source: A. Limitations and Future Work · p. 8
  • Future work could let task-level reasoning run alongside skill supervision and use tree structure and failed leaves to support failure diagnosis and recovery.The paper also proposes evaluating frontier models and using the executable text environment for policy learning followed by simulation and robot evaluation.Source: A. Limitations and Future Work · p. 8
Loading 2610.13862v1…