PaperlayerView exact revision on arXiv

Source-linked AI summary

Fly-by-Code: Embodied Coding Agents for Aerial Manipulation with Active Visual and Physical Feedback

Jaewoo Lee, Jeongyeon Seo, Sihyun Cho, Gyeongrak Choe, Yutong Wang, Bavin Saravanan, Jia-Bin Huang, Furong Huang, Sebastian Scherer, Guanya Shi, H. Jin Kim, Seungjae Lee, Dongjae Lee

arXiv:2610.12549v1cs.ROcs.CVcs.LG

TL;DR

Aerial manipulation can leave task outcomes unseen and failure causes unresolved, limiting what passive execution feedback tells coding agents. Active Feedback (AF) gathers targeted visual and physical evidence through separate diagnostic programs; across six simulated tasks, it achieves 75% final task success versus 31.7% for terminal feedback and 48.3% for execution-trace feedback.

  • Problem

    Aerial manipulation can leave task-critical outcomes outside the robot’s observations and make failure causes difficult to distinguish, limiting the evidence available for program refinement.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    Active Feedback (AF) has the agent synthesize a separate diagnostic program to acquire missing outcome observations from informative viewpoints (View) and test failure hypotheses through targeted physical interactions (Probe).

    Source: §1 · pp. 2–3, §3.3 · p. 5

  • Results

    Across six aerial manipulation tasks in two simulated environments, AF achieves 75% final task success, versus 31.7% for terminal feedback and 48.3% for execution-trace feedback.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Task programs refined with AF in simulation achieve zero-shot transfer to a physical aerial manipulator through shared robot APIs, without task-program source-code modification.

    Source: §1 · p. 3, §5 · p. 10

    The protocol provides no operational safety guarantee for unrestricted physical probing directly in the real world.

    Source: §6 · p. 11

Abstract

from arXiv · show

Embodied coding agents construct robot policies from natural-language instructions by generating executable programs and refining them from execution feedback. This refinement assumes that execution feedback reveals whether the task succeeded and, after a failure, provides sufficient evidence to guide program revision. Aerial manipulation violates this assumption in two ways. Payload constraints restrict onboard camera coverage and leave task-critical outcomes outside the robot's view (partial observability), while strong dynamic coupling and limited control margins make physical interactions fragile (limited interaction robustness) and leave failure causes unresolved. Re-examining the same feedback cannot recover observations that were never collected or separate failure causes that the available evidence does not distinguish. To acquire this missing evidence, we introduce Active Feedback (AF), an active "View and Probe" framework. When execution feedback leaves unresolved questions, the agent synthesizes a separate diagnostic program that runs in simulation to inspect unseen outcomes from informative viewpoints (View) or test hypotheses about failure causes through targeted physical interactions (Probe). Across six simulated aerial manipulation tasks with 60 trials per condition, AF reduces false completion declarations from 25 to 5 and raises final task success from 31.7% to 75% compared with terminal feedback under the same coding-agent configuration and task-execution limit. Using shared robot APIs, task programs refined with AF in simulation run on a physical aerial manipulator without task-program source-code modification and complete multi-stage manipulation tasks in the real world. These results show that executable diagnosis fills gaps in execution feedback for embodied coding agents, improving both completion assessment and program refinement in aerial manipulation.

1 INTRODUCTION

Execution feedback can miss task outcomes or leave failure causes unresolved, especially in aerial manipulation. Active Feedback (AF) acquires additional diagnostic evidence before the agent decides whether to finish or revise its program; it improves simulated task success and is demonstrated on physical hardware.

1 INTRODUCTION

  • Partial observability can hide task-critical outcomes, while limited interaction robustness can leave physical failure causes difficult to distinguish.Re-examining the same feedback cannot recover uncollected observations or distinguish causes that the evidence does not resolve.Source: §1 · p. 2
  • Unlike feedback gathered during task execution or embedded verification procedures, AF synthesizes an attempt-specific diagnostic procedure after execution and separately from task evaluation.Diagnostic actions can be selected for their information value rather than their immediate contribution to task completion.Source: §1 · p. 3
  • 75% final task success, versus 31.7% with terminal feedback and 48.3% with execution-trace feedback, across six aerial-manipulation tasks in two simulated environments.The comparison uses the same agent configurations and execution limits.Source: §1 · p. 3
  • Simulation-refined task programs repeatedly complete two multi-stage tasks on a physical aerial manipulator through shared APIs, without task-program source-code changes.The physical sensing and control implementations differ from simulation.Source: §1 · p. 3
  • AF inserts a post-execution diagnostic stage in which the agent generates a separate program to gather evidence through informative viewpoints or targeted physical interactions.The acquired observations and measurements inform completion assessment and subsequent program revision.Source: §1 · pp. 2–3

2 RELATED WORK

Related research spans executable-program synthesis and feedback-driven refinement, active and interactive perception, and aerial manipulation. Fly-by-Code focuses on post-execution diagnosis that actively gathers evidence for completion assessment and program revision.

Robot program synthesis and feedback-driven refinement

  • Robot program-synthesis systems compose perception and control APIs, with iterative refinements using textual or visual feedback, execution traces, human corrections, reusable repairs, or verification interfaces.AF instead targets cases where existing execution evidence is insufficient, generating an attempt-specific diagnostic program to acquire visual or physical evidence.Source: §2 · pp. 3–4

Active and interactive perception

  • Active and interactive perception methods acquire task-relevant information, often to support ongoing perception or task execution; AF generates a separate post-attempt diagnostic program for completion assessment and refinement.The cited approaches include viewpoint selection, coordinated perception–manipulation under occlusion, and epistemic actions.Source: §2 · p. 4

Learning-based and language-guided aerial manipulation

  • Aerial-manipulation research includes platform design, contact-aware control, planning, perception, and learned policies, while Fly-by-Code actively gathers post-execution evidence to assess and revise executable programs.Language-guided examples include predefined flight-safe skills and viewpoint changes for target observability under occlusion.Source: §2 · p. 4

3 ACTIVE FEEDBACK FOR PROGRAM SYNTHESIS IN AERIAL MANIPULATION

Fly-by-Code adds active diagnosis to aerial-manipulation program refinement because passive execution feedback may not reveal task outcomes or failure causes. The agent generates a diagnostic program for the current attempt, uses View or Probe to gather evidence, then applies that evidence to its completion judgment or next task program.

3 ACTIVE FEEDBACK FOR PROGRAM SYNTHESIS IN AERIAL MANIPULATION

  • AF inserts an active diagnostic stage before the agent decides to finish or revise the task program, using a separate program to acquire evidence for unresolved questions.The task score is recorded before diagnosis, so diagnostic interactions do not affect the evaluation.Source: §3 · p. 4, §3.2 · p. 4

3.1 MOTIVATION: LIMITATIONS OF PASSIVE FEEDBACK

  • Partial observability can leave outcomes outside the terminal camera view, while limited interaction robustness can make a static image insufficient to distinguish contact errors from kinematic constraints.Fly-by-Code supplements passive logs and observations with actively acquired evidence.Source: §3.1 · p. 4

3.2 DIAGNOSTIC PROGRAM SYNTHESIS IN AF

  • After reviewing execution logs, terminal observations, and a semantic-map summary, the agent writes a diagnostic program using the task program’s shared robot APIs to target uncertainties from that attempt.Because the score is recorded before diagnosis, diagnostic actions gather information rather than advance the evaluated task.Source: §3.2 · p. 4, Figure 2 · p. 5

3.3 DIAGNOSTIC ACTIONS: VIEW AND PROBE

  • View selects viewpoints to inspect outcomes, while Probe performs targeted physical tests; in the cited examples, View exposed fallen tools and Probe measurements informed carton-contact selection.The agent writes both using shared robot APIs, may combine them, and chooses what to inspect or test for the current attempt.Source: §3.3 · p. 5, Figure 3 · p. 6

3.4 CLOSING THE FEEDBACK LOOP

  • The agent combines diagnostic observations, inspection notes, and console measurements with the original execution logs to choose FINISH or REGENERATE.The environment resets before the next attempt, while diagnostic information persists to inform the revised task program.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments evaluate Active Feedback (AF) across six simulated aerial manipulation tasks, examining task success, completion assessment, diagnostic-guided revision, model/interface variation, and physical transfer. AF outperforms passive feedback baselines in simulation, and simulation-refined programs also complete multi-stage tasks on physical hardware without source-code changes.

4.1 EXPERIMENTAL SETUP

  • The evaluation spans six aerial manipulation tasks, comparing AF with terminal and trace feedback under a shared agent setup and task-execution limit.Tasks cover object transport, sorting, articulated cabinet interaction, unknown payload constraints, and contact-dependent pushing; each trial allows an initial execution and up to ten revisions.Source: §4 · p. 6, §4.1 · pp. 6–7, Figure 4 · p. 7

4.2 TASK SUCCESS AND COMPLETION ASSESSMENT

  • 75% overall success for AF across six tasks, versus 31.7% for CaP-X and 48.3% for CaP-X+Trace; AF also leads on every task.On 2-Objects Pick-and-place, AF reaches 90% versus 40% and 60%; on Payload it reaches 100% versus 30% and 60%, and on Cabinet 40% versus 10% and 0%.Source: §4.2 · pp. 7–8, Figure 5 · p. 8
  • Across 60 trials, AF produces 5 false completion declarations and 90.0% precision, versus 25 and 43.2% for CaP-X and 19 and 60.4% for CaP-X+Trace.Table 1 reports correct declarations, incorrect declarations, and trials reaching the step limit without a declaration.Source: §4.2 · p. 8, Table 1 · p. 8

4.3 HOW DOES ACTIVE FEEDBACK SUPPORT PROGRAM REFINEMENT?

  • In 2-Objects Pick-and-place, View inspection exposed unsuccessful placements; in Align and Payload, Probe results guided control and load-planning revisions.The Align agent compared pushes at different contact points to initialize a push-response model; after a refused 1.2 kg lift, the Payload agent regrouped bottles and successfully lifted 1.1 kg.Source: §4.3 · pp. 8–9, Figure 6 · p. 9

4.4 GENERALITY ACROSS MODELS AND INTERFACES

  • AF matches or exceeds passive CaP-X across evaluated model–interface pairs, although the size of its gains varies.On Cabinet, reported AF success rises from 0% to 90% with Astra API and from 60% to 100% with Claude Code Opus 5.5 and Codex Astra; a No harness condition performs below AF on the reported comparisons.Source: §4.4 · p. 9

4.5 SIM-TO-REAL TRANSFER AND PHYSICAL DEPLOYMENT

  • Simulation-refined task programs run on physical hardware without source-code modification, achieving 80% success on 2-Objects Pick-and-place and 60% on Cabinet.These trials require about 8 to 9 minutes of autonomous operation; the programs use real-world observations through shared APIs rather than replaying simulated trajectories.Source: §4.5 · p. 10, Table 2 · p. 10

5 CONCLUSION

The conclusion presents Active Feedback as a post-execution diagnostic stage that acquires targeted visual and physical evidence for completion judgments and program revision. Across six simulated tasks it improves performance over passive feedback, and simulation-refined programs transfer to physical aerial manipulation without source-code changes.

5 CONCLUSION

  • Active Feedback synthesizes a separate post-execution diagnostic program to gather informative viewpoints and targeted physical-test evidence for completion assessment and program revision.The conclusion reports improved success and completion assessment across six simulated tasks, plus repeated zero-shot physical task completion through shared robot APIs.Source: §5 · p. 10

6 LIMITATIONS

The diagnostic stage adds inference and interaction overhead, and its evidence remains bounded by sensing and viewpoint access. Physical probing safety and generalization beyond known settings are also not established.

6 LIMITATIONS

  • Active Feedback adds model-inference and robot-interaction overhead during diagnosis.The limitation concerns the explicit diagnostic stage itself.Source: §6 · p. 10
  • Post-execution inspection cannot guarantee correct completion judgments when onboard perception is flawed or informative viewpoints are unreachable.Diagnostic interactions occur in simulation; the paper does not provide an operational safety guarantee for unrestricted physical probing.Source: §6 · pp. 10–11
  • The authors do not claim zero-shot generalization to entirely unseen physical environments or radically different embodiment morphologies.Successful deployment also relies heavily on the underlying motion-planning and flight controllers.Source: §6 · p. 11

A IMPLEMENTATION DETAILS

Implementation details describe how AF reviews existing evidence, generates diagnostics, and returns findings for task-program decisions, alongside feedback baselines, simulation environments, and shared robot APIs. The same task-level interfaces support simulation and hardware execution while allowing environment-specific sensing and control implementations.

A.1 PROGRAM SYNTHESIS AND ACTIVE FEEDBACK

  • AF reviews task requirements against existing feedback, then generates diagnostic code for unsettled outcomes or failure questions unless the evidence already suffices.Diagnostics execute from the task program’s resulting state without an intervening reset; the environment resets before each revised task program. Selected images, notes, positions, reasoning, and console output are returned to the agent, which declares FINISH or supplies a revised program.Source: §A.1 · p. 17

A.2 FEEDBACK CONDITIONS AND GENERATOR CONFIGURATIONS

  • The primary comparison holds the Claude Code–Opus 5 configuration fixed, while trace feedback adds primitive-call records and keyframes and AF adds separately acquired diagnostic evidence.Cross-configuration differences are not treated as isolated language-model effects because interfaces can differ; revised task programs start from the initial environment state and must complete the full task.Source: §A.2 · pp. 17–18, Figure 8 · p. 18

A.3 SIMULATION ENVIRONMENTS AND ROBOT CONFIGURATION

  • The simulation uses six AM-Bench tasks, a dynamics-based aerial-manipulation model, a warehouse asset, and a laboratory digital twin reconstructed from images and videos.The simulated platform follows the physical coaxial quad-tiltrotor design, with four actuated tilt joints, a four-degree-of-freedom arm, gripper, and wrist-mounted RGB-D camera.Source: §A.3 · pp. 18–19

A.4 SHARED ROBOT APIS AND EXECUTION BACKEND

  • Task and diagnostic programs compose the same robot APIs for observation, perception, motion, and manipulation; AF adds no separate View or Probe primitives.The semantic map supplies observation-derived spatial references, while collision-aware planning verifies whole-body trajectories; simulation and hardware share task-level calls but may use different sensing and control implementations.Source: §A.4 · pp. 19, 21

B.1 TASK INSTRUCTIONS AND SETUP

The primary evaluation covers six aerial-manipulation tasks across a laboratory digital twin and a warehouse simulation. The setups vary in object destinations, visibility, payload constraints, and interaction demands.

Evaluation tasks

  • The six-task evaluation uses the laboratory digital twin for four tasks and the warehouse simulation for Sort and Align.The laboratory tasks are 1-Object Pick-and-place, 2-Objects Pick-and-place, Cabinet, and Payload.Source: §B.1 · p. 21

Pick-and-place tasks

  • Pick-and-place tasks require moving a hammer, or a hammer and wrench, from a shared pegboard to destinations across the hall.The hammer destination is outside the source view; the two-object task has spatially separated destinations.Source: §B.1 · pp. 21–22

Cabinet and Sort

  • Cabinet requires retrieving a bottle behind closed, outward-opening doors, while Sort requires matching a bottle to a tote by sticker color.Sort places the bottle on a high rack shelf and the nine totes on another rack.Source: §B.1 · p. 22

Payload

  • Payload requires moving five filled bottles and two baskets across the hall while leaving three empty bottles behind.Bottle masses and the carrying limit are undisclosed; motion commands are rejected when the carried load exceeds the payload limit.Source: §B.1 · p. 22

Align

  • Align requires pushing two protruding, yellow-stickered cartons flush and parallel to neighboring fronts.Their centers of mass are offset from their geometric centers, which is not visible or disclosed and can make a geometric-center push induce rotation.Source: §B.1 · p. 22

B.2 TASK COMPLETION AND PROGRESS

This section defines task completion and progress scoring across the six tasks. Progress is normalized by each task's rubric maximum, and the reported curves show AF's clear average-progress advantage.

Completion criteria

  • A trial counts as successful only when the final task-program execution meets task-specific completion criteria and the agent declares FINISH.A false FINISH remains a failure.Source: §B.2 · p. 22

Aggregation and progress

  • The evaluation runs ten trials per task and condition with equal task weighting, making the task-macro-average equal to the pooled success rate.Progress scores are normalized by task rubric maxima, and early-terminated trials retain their final progress through turn 11.Source: §B.2 · pp. 22–23

Task rubrics

  • Cabinet progress scores door opening, bottle grasp, extraction, placement, and FINISH, with each stage contributing one point.The door-opening threshold is 55° in this setup.Source: §B.2 · p. 23
  • The pick-and-place rubrics distinguish grasp, transport, placement, retention, and FINISH, while 2-Objects Pick-and-place scores the hammer and wrench separately.Payload awards points for loading, delivery, preserving the empty bottles, and satisfying the final crossing and placement conditions.Source: §B.2 · pp. 23–24
  • Sort scores grasp, extraction, transport, release, retention, and FINISH, while Align combines precision scoring with a FINISH criterion of emax ≤5 cm.Stopped trials keep their final score; standing scores can decrease after revision, unlike the running maximum.Source: §B.2 · p. 24

Progress curves

  • AF shows a clear advantage in average progress across all six tasks, with generally comparable or higher progress on individual tasks.Figure 10 plots progress across task-program executions; early-terminated trials retain their final score through turn 11, and scores are normalized by task rubric maximum.Source: Figure 10 · p. 25

B.3 COMPLETION JUDGMENTS AND STATISTICAL ANALYSIS

The evaluation distinguishes terminal completion judgments from intermediate decisions, specifies trial-level uncertainty for progress curves, and limits agent feedback to execution evidence rather than evaluator labels.

B.3 COMPLETION JUDGMENTS AND STATISTICAL ANALYSIS

  • Completion-judgment precision is successful terminal FINISH declarations divided by all terminal FINISH declarations.True FINISH, false FINISH, and execution-limit exhaustion without FINISH are reported separately from errors over intermediate decisions.Source: §B.3 · p. 25
  • Progress-curve bands are pointwise 95% percentile-bootstrap intervals from 1,000 trial-level resamples, and comparisons characterize the evaluated task suite.Repeated executions within a trial are not independent samples, and matching trial indices across conditions are not assumed to be paired.Source: §B.3 · p. 25

B.4 INFORMATION ACCESS

  • Agents receive observations and execution outputs but not evaluator-only ground-truth states, task scores, or success labels.Grader logs are recorded separately from agent-visible execution output.Source: §B.4 · p. 25

C.1 TASK PROGRESS IN THE MAIN EVALUATION

  • Figure 10 shows task progress across successive task-program executions for three feedback methods on all six tasks.The figure's accompanying caption reports AF's clear advantage in average progress across the tasks.Source: §C.1 · p. 25

C.2 TASK PROGRESS ACROSS MODELS AND INTERFACES

Figure 11 presents task progress across models and agent interfaces, organizing tasks by row and feedback methods by column.

C.2 TASK PROGRESS ACROSS MODELS AND INTERFACES

  • Figure 11 organizes task-progress results across models and interfaces with tasks in rows and feedback methods in columns.The figure includes Gemini 3.8 Flash API (Google DeepMind, 2026).Source: §C.2 · p. 25

C.3 SEMANTIC MAP ABLATION

Removing semantic-map support reduced task success and increased false FINISH declarations, while the case studies show AF gathering evidence to assess outcomes and revise programs. The physical-system description situates these findings in an aerial manipulation platform with onboard egocentric sensing.

C.3 SEMANTIC MAP ABLATION

  • Task success fell from 9/10 to 4/10 on 2-Objects Pick-and-place and from 5/10 to 2/10 on Sort; false FINISH declarations rose from 1 to 5 and 0 to 3, respectively.Table 5 defines the ablation scope and metrics: ten trials per task and condition; success counts successful trials and FP counts false FINISH declarations.Source: Table 5 · p. 26, §C.3 · p. 27
  • Without the map, distant basket localization became less accurate and mean SAM3 calls per task-program execution increased from 14 to 54.Execution records suggest the map supports spatial search; its background detections provide cues about objects outside the current view.Source: §C.3 · p. 27
  • All three false FINISH declarations in Sort without map support followed an AF step that skipped observation and treated target objects as absent.In one map-supported trial, map-provided target candidates prompted continued REGENERATE after repeated SAM3 queries found no targets; estimated model costs also rose without the map on both tasks.Source: §C.3 · p. 27
  • Because map support was removed throughout the loop, this ablation does not isolate its contributions to task execution from diagnosis.The map supplies inspection targets rather than definitive completion evidence; AF images and close-range perception provided evidence when wrench loss was recognized.Source: §C.3 · p. 27

D.1 EVIDENCE FOR COMPLETION ASSESSMENT

  • A release trace did not verify basket placement, while AF views found both tools near floor level outside their destinations and led the agent to REGENERATE.The next program revised release and placement actions, but the immediately following execution failed; the trial succeeded after further refinement on the fourth task execution.Source: §1 · p. 29, §2 · p. 30, §5 · p. 31

D.2 EVIDENCE FOR PROGRAM REVISION

  • In Align, the first high-y edge push produced less rotation per unit advance than centre and low-y edge pushes, and neither sideways slide reversed rotation.The next program used a linear push-response model to select lateral contact; the following execution pushed on the high-y side and precisely aligned the boxes.Source: §2 · p. 32, §4 · p. 33
  • A roughly 1.2 kg basket load was refused, indicating infeasibility without identifying the exact carrying limit; later, both planned 1.1 kg basket loads lifted successfully.The agent revised its planning limit to 1.1 kg; a separate intended 1.5 kg test supplied no additional capacity bound because the handle was not grasped.Source: §3 · pp. 33–34, §4 · p. 34, §5 · p. 34

E.1 PLATFORM AND EXECUTION SYSTEM

  • The aerial platform uses a wrist-mounted RGB-D camera for task-program observations, while external cameras record experiments but do not supply images to the program.Physical starting poses vary across trials, and the physical scene does not exactly reproduce simulated object placement or surrounding obstacles.Source: §E.1 · p. 35

E.2 PHYSICAL EXPERIMENTS AND OUTCOME ASSESSMENT

The physical experiments include successful Cabinet executions and trial-level records of both completed stages and failures. The scene differs from simulation, and robot starting position and heading vary across trials.

E.2 PHYSICAL EXPERIMENTS AND OUTCOME ASSESSMENT

  • The physical scene contains corresponding task objects but does not exactly reproduce simulated object placement or surrounding obstacles, and starting position and heading vary across trials.The starting poses are recorded for every evaluated trial; object-position comparisons use a shared frame and are approximate.Source: §E.2 · p. 36
  • Figure 12 shows a successful real-world Cabinet execution, while Figure 13 compares that task in simulation and on the physical robot from the wrist camera.The comparison shows the left and right doors opened and the bottle grasped in both settings, followed by the physical execution approaching the basket before release.Source: §E.2 · p. 36, Figure 12 · p. 37, Figure 13 · p. 37
  • Trial-level Cabinet records include failures after the gripper stuck on a handle and after a grasp occluded the wrist camera, preventing close-range basket measurement and bottle placement.Other listed trials completed the recorded stages; the table defines check marks as completed and crosses as failed.Source: §E.2 · p. 37, Table 10 · p. 37
Loading 2610.12549v1…