PaperlayerView exact revision on arXiv

Source-linked AI summary

TEAR: Task-Evidence Authority Routing for Robotic Manipulation

Shicheng Zhang, Hui Liu, Haoliang Li

arXiv:2610.30819v1cs.RO

TL;DR

Changing communication and task conditions make it difficult to rely on teleoperation or autonomy alone for reliable manipulation. TEAR routes authority between them using network and task evidence, achieving success close to continuous teleoperation with substantially less teleoperation use in the evaluated settings.

  • Problem

    Teleoperation can adapt to contact-rich and ambiguous manipulation but depends on a stable communication loop, while local autonomy may stall or fail under semantic ambiguity and unforeseen contingencies.

    Source: §I · p. 1

  • Method

    TEAR is an edge-deployable robot-side framework that combines network and task evidence to route authority between human teleoperation and local VLA control.

    Source: §I · pp. 1–2

  • Results

    TEAR achieves task success close to continuous teleoperation while using teleoperation for 19.8–27.4% of control steps and improving success by 30–40% points over autonomy-only execution.

    Source: §I · p. 2

  • Takeaways & Limitations

    Across the evaluated simulation and real-world settings, locally computed authority routing achieved near-teleoperation task success with substantially less teleoperation use, at the cost of longer successful completion times.

    Source: §VI · p. 7

    The evaluation focuses on two related pick-and-place tasks, and the visual task evidence excludes fine contact and force information.

    Source: §V · p. 7

Abstract

from arXiv · show

Reliable robotic manipulation in field-deployed settings requires more than executing a capable controller. As the execution conditions change, the system must decide, online and on the robot side, which control source should hold authority. Human teleoperation adapts readily to contact-rich and semantically ambiguous situations but depends on a stable communication link. Local autonomy is executed without continuous human intervention, but may fail under semantic ambiguity, long-horizon drift, and unforeseen contingencies. These two control sources exhibit complementary reliability profiles. We formulate this complementarity as a control authority routing problem and present Task-Evidence Authority Routing for Robotic Manipulation (TEAR), an edge-deployable framework that allocates authority between human teleoperation and local autonomy. TEAR arbitrates on the robot side using network-state evidence, which governs the usability of teleoperation, and task-semantic evidence, which reflects whether autonomous execution remains consistent with the current task. A lightweight temporal arbitrator observes both forms of evidence together with robot state and current authority to time authority transfers. Across simulated and real-world trials under dynamic network conditions, TEAR achieves task success rates close to pure teleoperation while limiting teleoperation to 19.8-27.4% of control steps and improves success by 30-40% over autonomy-only execution in our settings. These results demonstrate the potential of complementary authority routing to sustain successful manipulation with substantially reduced dependence on continuous human control.

I. INTRODUCTION

Teleoperation adapts to contact-rich and ambiguous manipulation but depends on a reliable link, while local autonomy can stall or fail under semantic ambiguity and unforeseen conditions. TEAR routes authority between them using network and task evidence, achieving success close to continuous teleoperation with substantially less human control.

I. INTRODUCTION

  • Teleoperation adapts to unexpected contacts and semantic ambiguity, but latency and packet loss can prevent timely delivery of human commands.Local policies avoid dependence on continuous commands but can stall, drift, or fail under ambiguity and unforeseen contingencies.Source: §I · p. 1
  • The routing problem requires evidence about both remote-control usability and local autonomous execution, with decisions available on the robot as conditions change.TEAR’s overview shows network and task evidence entering a temporal handoff predictor alongside robot state and current authority.Source: §I · p. 1, Figure 1 · p. 2
  • TEAR achieves task success close to continuous teleoperation while using teleoperation for only 19.8–27.4% of control steps and improving success by 30–40% points over autonomy-only execution.These results span simulated and real-world manipulation under time-varying network conditions.Source: §I · p. 2
  • TEAR combines network-state and task-semantic evidence to predict handoffs over time, then uses a stateful gate with hysteresis and cooldown to make authority transfers.Evidence estimation and routing run locally without online external-model queries.Source: §I · p. 2

II. RELATED WORK

Prior work addresses communication constraints, autonomous policies, and human–autonomy coordination through distinct approaches. TEAR routes authority between teleoperation and a local VLA policy using network and visual task evidence and recent execution history.

II. RELATED WORK

  • Teleoperation is constrained by latency, jitter, bandwidth variation, and packet loss, motivating the use of link conditions to guide control allocation.TEAR treats network state as a decision signal while retaining a locally executable manipulation policy.Source: §II · pp. 2–3
  • Autonomous policies increasingly map observations and task specifications to local actions, while TEAR adds visual task evidence to decide when control should pass to a human operator.The evidence includes task progression failures monitored with VLM-based detection.Source: §II · p. 3
  • Shared-autonomy and mode-switching work includes intent inference, command blending, and automatic mode selection, while TEAR predicts handoffs from network observations and visual task cues.Expert handoffs supervise upcoming-event prediction, and a stateful gate regulates when predictions change authority.Source: §II · p. 3

III. METHOD

  • TEAR targets remote manipulation with network degradation and task disturbances such as stalling, using temporal evidence to transfer authority between a human and local VLA policy.The method proceeds through routing formulation, evidence construction, temporal prediction, and handoff execution.Source: §III · p. 3

A. Problem Setup and Routing State

The router selects either remote teleoperation or local VLA control for each execution interval and represents network, robot, visual-task, and authority information in its state. Visual task cues are constructed from task-specific text pairs and image observations, then smoothed and filtered before local routing.

A. Problem Setup and Routing State

B. Complementary Evidence Construction

  • Visual task cues cover goal progress, stagnation, target–gripper engagement, and undesired states; stagnation is interpreted jointly with progress to distinguish possible completion from a stall.Task-specific positive and negative text pairs are selected offline using task specifications and labeled demonstrations, then fixed before deployment.Source: B. Complementary Evidence Construction · p. 3
  • A frozen VLM scores image–text agreement, while causal smoothing and persistence filtering suppress transient visual changes before routing.These scores do not directly estimate physical contact or failure probability; visual scoring, routing, and low-level control operate asynchronously and locally.Source: B. Complementary Evidence Construction · p. 4

C. Learning Sparse Handoff Events

TEAR trains a causal temporal predictor on expert handoff annotations to anticipate transfers from recent routing evidence, with current authority determining transfer direction.

C. Learning Sparse Handoff Events

  • Expert handoff annotations are expanded into an H-update pre-handoff supervision window for either transfer direction, while deployment inference remains causal.This lets the predictor learn evidence that develops before an annotated transfer.Source: C. Learning Sparse Handoff Events · p. 4
  • A causal temporal convolutional network maps the preceding K routing states to an event score.Source: C. Learning Sparse Handoff Events · p. 4
  • The TCN uses three residual blocks with causal dilated convolutions, temporal pooling, a scalar output, and focal loss for event-sample imbalance.Source: C. Learning Sparse Handoff Events · p. 4
  • The same event predictor serves both transfer directions; current authority determines whether an accepted event switches to teleoperation or local VLA control.Source: C. Learning Sparse Handoff Events · p. 4

D. From Event Scores to Authority Transfers

A stateful gate converts predicted handoff events into authority transfers while suppressing repeated triggers and limiting transfer frequency.

D. From Event Scores to Authority Transfers

IV. EXPERIMENTS

The experiments evaluate TEAR in simulation and real-world manipulation, covering network effects, task-level outcomes, and handoff prediction.

IV. EXPERIMENTS

  • Experiments evaluate TEAR in both simulation and real-world manipulation.Source: §IV · p. 4
  • The evaluation first characterizes teleoperation under network degradation.Source: §IV · p. 4
  • The remaining analyses assess task-level performance and teleoperation use, then examine routing evidence and temporal handoff prediction.Source: §IV · p. 4

A. Experimental Setup

The setup evaluates a shared pick-and-place routing design across simulation and real-world tasks, with impaired remote commands, time-varying network conditions, and separately developed routers.

A. Experimental Setup

  • Simulation uses an SO-101 arm and GR00T-N1.5 fine-tuned on 50 demonstrations for orange pick-and-place; real-world trials use an SO-101 follower, remote leader, cameras, and GR00T-N1.7-3B for duck placement.The tasks share a pick-and-place structure while varying objects, policies, and deployment conditions.Source: A. Experimental Setup · p. 4
  • Remote commands face software-imposed latency, jitter, and packet loss; stale real-world commands stop issuing targets while the position controller holds its last goal, whereas local VLA actions bypass that path.Source: A. Experimental Setup · p. 4
  • Task-level network impairment cycles through RTT settings of 50, 250, and 500 ms, jitter of 4, 50, and 100 ms, and packet loss of 0%, 25%, and 100%, updating every 8 s.Source: A. Experimental Setup · p. 5
  • The router runs at 60 Hz in simulation and 30 Hz in real-world trials, while visual evidence updates at 1.5 Hz and settings are fixed before evaluation.The authority gate uses genter = 0.40, gexit = 0.30, and a 20-update cooldown.Source: A. Experimental Setup · p. 5
  • Task-level comparisons use ten valid rollouts per method and setting, including valid failures and timeouts; real-world TEAR trials were supervised and included learned-router events and manual switches.Source: A. Experimental Setup · p. 5
  • The evaluation reports task success, teleoperation use as the fraction of low-level steps under teleoperation, and successful-episode completion time with sample variability and Wilson confidence intervals for success.Source: A. Experimental Setup · p. 5

B. Teleoperation under Network Degradation

Across the two evaluated settings, TEAR matched Always-Teleop’s 90% success in simulation and approached its 100% real-world success while exceeding Always-VLA; teleoperation use remained below 28%. Fixed network impairments also revealed distinct effects of latency and packet loss on teleoperation.

B. Teleoperation under Network Degradation

  • Increasing latency mainly lengthened successful-trial completion time, while severe packet loss reduced observed success and affected completion time.Moderate packet loss had limited effect on successful execution; the impairment sweeps motivate routing evidence for both command timeliness and continuity.Source: B. Teleoperation under Network… · p. 5, Figure 2 · p. 5
  • Teleoperation usability depends on command timeliness and continuity, motivating latency, jitter, and packet-loss evidence alongside a locally executable VLA policy.Source: B. Teleoperation under Network… · p. 5

C. Task Outcomes in Simulation and Real-World Deployment

  • TEAR achieved 90% success in simulation and 90% in the real world, versus Always-VLA’s 60% and 50% and Always-Teleop’s 90% and 100%, respectively.The two settings shared a pick-and-place structure and routing interface, but used different objects, policies, and separately developed routers.Source: C. Task Outcomes in Simulation and… · p. 5, Figure 3 · p. 6
  • Teleoperation accounted for 19.8% of control steps in simulation and 27.4% in real-world TEAR trials, with longer successful-episode completion times than either single-controller baseline.Mean successful-episode times were 45.2 s for TEAR versus 15.3 s for Always-Teleop and 33.4 s for Always-VLA in simulation; real-world means were 63.7 s, 28.3 s, and 59.5 s, respectively.Source: C. Task Outcomes in Simulation and… · pp. 5–6, Figure 3 · p. 6
  • Real-world TEAR outcomes came from supervised trials that recorded 18 learned-router events and four manual switches across two episodes, including one terminal switch.Source: C. Task Outcomes in Simulation and… · p. 6

D. Analysis of Routing Evidence

The analysis finds that visual task cues vary with outcome and that the evidence preceding handoffs differs by transfer direction. Network degradation precedes Teleop-to-VLA handoffs, while VLA-to-Teleop evidence is more heterogeneous.

Visual task cues

  • Successful calibration episodes showed higher mean progress and engagement and lower stagnation and anomaly for both CLIP and SigLIP, though cue values varied and sometimes approached saturation.Stagnation alone did not distinguish task completion from an unsuccessful stationary state; its interpretation depended on concurrent progress and engagement evidence.Source: D. Analysis of Routing Evidence · p. 6, Table I · p. 6

Handoff evidence

  • Before Teleop-to-VLA handoffs, latency, jitter, and packet loss increased significantly relative to ordinary teleoperation periods after adjustment across comparisons.The comparison uses the two seconds before handoff against ordinary same-mode execution, with repeated same-direction events averaged within episode.Source: D. Analysis of Routing Evidence · pp. 6–7
  • Before VLA-to-Teleop handoffs, visual cues changed less and none met the same significance criterion; progress increased before both transfer directions.The observations support interpreting task evidence jointly over time rather than relying on a single visual cue.Source: D. Analysis of Routing Evidence · p. 7

E. Offline Handoff-Event Prediction

The offline evaluation compares temporal handoff prediction with a rule-based comparator, then interprets the TCN’s results as selective rather than complete recovery of expert switching. The analysis also separates predictor performance from deployed authority-transfer logic.

E. Offline Handoff-Event Prediction

  • Evaluation uses ten held-out simulation episodes with seven expert handoffs, matching predicted and expert events one-to-one within ±5 router samples.Event F1 summarizes precision and recall; events per episode measures prediction frequency.Source: E. Offline Handoff-Event Prediction · p. 7
  • The TCN achieves event F1 0.308 with 0.6 predicted events per episode, versus 0.171 and 2.8 for the rule-based comparator.The table covers ten test episodes; the rule is the highest-F1 configuration in the evaluated sweep.Source: E. Offline Handoff-Event Prediction · p. 7, Table II · p. 7
  • TCN recall is 0.286 versus 0.429 for the rule-based comparator, so some expert handoffs are missed.The result is better read as selective event prediction, not complete recovery of expert switching behavior.Source: E. Offline Handoff-Event Prediction · p. 7
  • Because authority transfers are intended to be sparse, fewer predicted events reduce unnecessary opportunities for switching, while lower recall leaves infrequent handoffs difficult to recover.Recall is reported separately because handoffs are sparse and missed events remain relevant to routing.Source: E. Offline Handoff-Event Prediction · p. 7
  • Offline analysis characterizes the temporal predictor independently; deployed routing instead applies the two-threshold gate.Offline events use threshold crossings with a cooldown.Source: E. Offline Handoff-Event Prediction · p. 7

V. LIMITATIONS

TEAR’s evaluation covers two related pick-and-place tasks and controlled time-varying network impairment, while its task evidence is visual and excludes fine contact and force information. The conclusion reports success close to pure teleoperation with less teleoperation use, but longer successful completion times.

V. LIMITATIONS

  • The evaluation covers two related pick-and-place tasks with different objects, local VLA policies, and separately developed routers.Broader task families, additional operators, and more varied handoffs remain to be evaluated.Source: §V · p. 7
  • Network testing uses controlled time-varying impairment and omits operational effects such as congestion, retransmission, correlated delay and loss, and coupled command-video degradation.Recorded and live network traces are proposed as extensions while retaining the same network-evidence interface.Source: §V · p. 7
  • Current task evidence uses visual observations for progress, stagnation, engagement, and anomaly, leaving fine contact and force information outside its representation.Additional sensory signals can be incorporated into the same routing state.Source: §V · p. 7

VI. CONCLUSION

  • Across evaluated simulation and real-world settings, TEAR achieves success rates close to pure teleoperation with substantially less teleoperation use, at the cost of longer successful completion times.The framework combines network and task evidence, with a frozen VLM supplying visual cues and a causal temporal router predicting handoffs.Source: §VI · p. 7

Related papers

Loading 2610.30819v1…