Source-linked AI summary
Watch, Recall, Act: Always-On Robots in Concurrent Embodied Streams
Ding Yi, Peiwen Sun, Chenchu Rong, Jianan Wang, Xili Dai, Xiangyu Yue, Xi Lin
TL;DR
Always-on robots must act on live-stream cues and their own distant past actions while operating two arms, a combination not sustained by existing models. ARMS adds three asynchronously updated context modules and agent-causal self-history to a pretrained π0.5 backbone. It reaches ∼45% on the combined task versus ∼28% for the strongest main baseline, with each of three key components necessary in ablations.
Problem
Always-on dual-arm robots must combine streaming memory, mid-task concurrency, and continuous operation, while existing models provide these capabilities only in isolation.
Source: §1 · pp. 1–2Method
ARMS augments a pretrained π0.5 backbone with context modules for live perception, embodied state, and past actions, plus agent-causal self-history.
Source: §1 · p. 2, §3.2 · p. 4Results
∼45% on the combined task versus ∼28% for the strongest of four main baselines; ablations find the memory module, embodied-state head, and asynchronous concurrency each necessary.
Source: §1 · p. 2, §4.3 · pp. 7–8Takeaways & Limitations
ARMS sustains proactive concurrency, self-history recall, and priority arbitration together on composed streams, a real robot, and long horizons.
Source: §4.4 · p. 8, §5 · p. 8Real-robot evidence is small-scale on a single Cobot Magic platform, with only ∼200 staged always-on streams.
Source: §5 · p. 8
Abstract
from arXiv · showhide
An always-on robot faces an endless stream that never resets: instructions arrive and lapse, the scene changes, and its own past actions reshape what it must reason about. Today's action models are built for the opposite: a fixed instruction, no mid-task intervention, single-step reasoning. In an open-ended world a robot must watch a live stream for far-future cues, recall its own far-past actions, and act on them under dual-arm concurrency. We present ARMS (Always-on Robot in Multi-modal Streams), a deliberately simple streaming policy: a single pretrained $\pi$0.5 backbone augmented by three lightweight modules that turn live perception, embodied states, and the robot's own past actions into context the backbone reads before it acts. The modules update this context asynchronously, so watching and recalling never block acting and the two arms act at once. Rather than inventing new mechanisms, ARMS integrates these learned context providers with an agent-causal self-history that logs which arm did what, and when. To supervise them without extra annotation, we build ARMS Dataset, whose staged construction script itself labels every module from real dual-arm teleoperation. Trained on it, ARMS reaches 45% on the combined task against 28% for the strongest of our four main baselines, and ablations confirm the memory module, the embodied-state head, and asynchronous concurrency are each necessary.
1 Introduction
Always-on robots must coordinate live monitoring, recall of their own past actions, and physical dual-arm behavior across an unreset stream. ARMS integrates three context modules with a pretrained backbone and reports a higher combined-task score than its strongest baseline.
1 Introduction
- The robot must watch for far-future triggers, recall its own far-past actions, and act on a real dual-arm robot rather than answer about video.Instructions and decision-relevant information can arrive far apart in the never-reset stream.Source: §1 · pp. 1–2
- Existing models provide streaming memory, mid-task switching, or always-on operation in isolation, but not their combination on a dual-arm robot.The combined setting requires sustained watching, self-history for self-referential commands, and awareness of potential arm collisions.Source: §1 · p. 2
- ARMS reaches ∼45% on the combined task versus ∼28% for the strongest of four main baselines.Ablations also find the memory module, embodied-state head, and asynchronous concurrency each necessary.Source: §1 · p. 2
- ARMS uses one pretrained π0.5 backbone with three lightweight context modules for live perception, embodied state, and the robot’s past actions.The contribution is their integration with agent-causal self-history, not a new network or planner.Source: §1 · p. 2
2 Related Work
Prior work covers components of always-on, dual-arm streaming, including memory, live-stream response, and robot foundation models. ARMS combines these demands around concurrent physical action and queryable self-history.
2 Related Work
- Prior systems cover only parts of the five-axis regime; ARMS covers all five axes it targets.The axes are always-on, proactive, retrospective, interruptible, and concurrent operation.Source: §2 · p. 2, Table 1 · p. 3
- Prior robot-memory approaches are passive or world-state-centric, while ARMS logs which arm performed each action and when for language queries.Recording the acting arm lets “the one the left arm stowed” resolve across concurrent records.Source: §2 · pp. 2–3
- Proactive video systems answer with recognition, whereas ARMS replaces a blocking reaction track with embodied dual-arm action.Prior control systems do not combine foreground-arm execution, inserted actions, queryable self-history, and overload arbitration.Source: §2 · p. 3
- ARMS builds on robot foundation models, including π0.5 [31], and contributes an orthogonal always-on, dual-arm, self-history layer.The paper identifies spatially grounded encoders [32] [33] and discrete skill-tokenization vocabularies [34] [35] as complementary foundations it does not require.Source: §2 · p. 3
3 ARMS
ARMS addresses always-on dual-arm control, where a robot must monitor a never-reset stream, recall its own past actions, and act as its arms alter a shared scene. It uses one pretrained backbone with context from live perception, embodied state, and action memory.
3 ARMS
- ARMS adds context for live perception, embodied state, and past actions to a single pretrained π0.5 backbone, rather than adding a separate planner for each demand.The modules update context asynchronously while the action forward reads it on demand.Source: §3 · p. 3, §3.2 · p. 4
3.1 Problem setup
- A guard is a persistent rule that may fire later, while a recall instruction requires a record of the robot’s own past actions rather than the current observation.The never-ending episode and shared scene require the two arms to coordinate around possible collisions.Source: §3.1 · pp. 3–4
3.2 Overview: one backbone, three context providers
- Three context providers share one backbone: a streaming visual trigger, an embodied-state head, and compressed action memory, all of which supply tokens or text for action generation.The modules update asynchronously, so perception and memory can continue while the arms execute; a high-frequency gate wakes the lower-frequency action path when a trigger fires.Source: §3.2 · p. 4
3.3 Visual perception: a streaming trigger gate
- The visual gate continuously checks active guards using π0.5’s vision head and invokes the action expert only when a guard’s condition is met.A negative trigger keeps scanning without waking the action expert; positive-trigger visual tokens enter the backbone context.Source: §3.3 · p. 4
3.4 Embodied-state head: action primitive and collision in one sentence
- The embodied-state head adds each arm’s current action primitive and a collision check, letting the backbone choose concurrent action when paths are clear or serialize when they are not.Execution also uses joint velocity limits and an operator-controlled emergency stop.Source: §3.4 · p. 5
3.5 Self-action memory by compression
- Compressed memory records what each arm did, to which object, and when, then retrieves relevant actions so language queries can distinguish even same-class objects by provenance and order.Retrieved actions are rewritten as a short sentence in the backbone context.Source: §3.5 · p. 5
3.6 Training
- The pretrained π0.5 backbone is fine-tuned for dual-arm control, while the added modules are trained from dataset-derived labels and recorded histories rather than policy rollouts.ARMS runs inference on a single 80 GB accelerator.Source: §3.6 · p. 6
3.7 ARMS Dataset
- ARMS Dataset uses about 200 staged, never-reset streams and about 1,200 teleoperated skill demonstrations; its construction script supplies module supervision from labels written during staging.The Online protocol evaluates end-to-end action across four task families and six metrics; the complementary Probe protocol measures recognition alone. Evaluation uses about 150 real-robot trials per method, with LIBERO as an optional scale-up.Source: §3.7 · pp. 6–7
4 Experiments
ARMS is evaluated on the combined always-on dual-arm task, through module ablations, and in real-robot deployment. The experiments compare it with policies built for individual ingredients and assess execution over long streams.
4.1 Setup
- Four baselines target individual ingredients and share ARMS's pretrained initialization, fine-tuning data, and backbone training budget.They include Fixed-rate VLA, MemoryVLA, SwitchVLA, and Recognition→Controller.Source: §4.1 · p. 7
- EAS requires uninterrupted foreground completion, guard service without collision, correct past-object recall, and priority-consistent overload ordering in one episode.Overall EAS is weighted by evaluation counts across task families; diagnostic metrics use their applicable subsets.Source: §4.1 · p. 7
4.2 The combination gap
- 45% EAS versus 28% for SwitchVLA, the strongest baseline; ARMS remains ∼40 points below the ≥85% human/oracle ceiling.Source: §4.2 · p. 7
4.3 Ablations
- Removing memory collapses recall from 72 to 31; removing temporal order leaves category recognition but lowers EAS to 36, while removing embodied state or asynchronous concurrency harms continuation and non-interference.Replacing asynchronous concurrency with a hard interrupt lowers continuation from 76 to 41 and non-interference from 78 to 38.Source: §4.3 · p. 7
- Removing memory, the embodied-state head, or asynchronous concurrency drops EAS from 45 to 32, 34, and 30, respectively; removing agent-causal labels reduces recall/history from 72 to 50 and 82 to 55.Source: §4.3 · p. 8
4.4 Real-robot deployment and qualitative results
- In a real-robot episode, ARMS stows a banana then a carrot and recalls the carrot as the second item; because they are distinct, appearance could aid recall.Same-class recall and full overload are quantified on ARMS Dataset; large-scale real-robot evaluation remains future work.Source: §4.4 · p. 8
- Over ≥10-minute never-reset streams, ARMS retains ∼67% versus ∼30% (Table 3, last row).Table 3 reports pooled percentages from three rounds of 20 trials per method/segment.Source: §4.4 · p. 8, Table 3 · p. 8
- On one 80 GB accelerator, asynchronous modules keep median trigger-to-decision latency under 200 ms while both arms execute.Source: §4.4 · p. 8
5 Conclusion and Limitations
ARMS integrates three lightweight context modules with a pretrained π0.5 backbone and agent-causal self-history, with staged dataset construction providing module supervision. The paper reports that every module is necessary, while noting constraints on real-robot evidence and task specifications.
5 Conclusion and Limitations
- ARMS integrates visual perception, embodied state, and compressed agent-causal memory with a single π0.5 backbone, and builds a dataset whose construction supplies module supervision.Source: §5 · p. 8
- Real-robot evidence is small-scale on one Cobot Magic platform with ∼200 staged streams; standing guards are predefined and overload priorities use fixed dataset-defined tiers.Large-scale evidence, especially for full overload, remains limited.Source: §5 · p. 8