Source-linked AI summary
HIRE: History-Conditioned Interaction Reasoning and High-Rate Execution for Visually Aliased Precision Manipulation
Rongji Li, Wenhao He, Cewu Lu, Xingyu Chen, Xu-Yao Zhang
TL;DR
Visually similar observations can reflect different physical interaction histories and require different actions, while existing methods leave long-horizon reasoning and precise execution largely decoupled. HIRE couples wrench-history-conditioned reasoning with high-rate manifold-structured execution; experiments report improvements across interaction regimes in task completion, state disambiguation, execution precision, and generalization.
Problem
Visually similar configurations may reflect different prior physical interactions and latent states, while existing approaches leave long-horizon interaction-state reasoning and precision-critical execution largely decoupled.
Source: §I · p. 2Method
HIRE couples an ISR that uses ordered wrench history as persistent physical evidence with a high-rate IME that structures contact execution around interaction progress and correction.
Source: §I · p. 2, §VI · pp. 7–8Results
Across transient surface, constrained insertion, and sustained rotational interactions, HIRE demonstrates consistent improvements in task completion, interaction-state disambiguation, contact-critical execution precision, and generalization.
Source: §VI · p. 8Takeaways & Limitations
HIRE treats physical interaction both as task execution and as persistent evidence for future state reasoning under history-dependent partial observability.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Precision manipulation with contact-critical interactions is often history-dependent: visually similar observations can correspond to different latent interaction states and therefore require different actions, while small execution errors can alter task outcomes. Policies relying on the current visual observation alone cannot resolve such ambiguity; force-aware and memory-augmented methods enrich physical or temporal context, while reactive high-rate policies improve local contact response, yet long-horizon temporal reasoning and precision execution remain largely decoupled in existing methods, limiting reliable progression in visually aliased precision manipulation. To bridge this gap, we introduce History-Conditioned Interaction Reasoning and Execution (HIRE), a cross-rate framework comprising a history-conditioned Interaction-State Reasoner (ISR) and a high-rate Interaction-Manifold Executor (IME). ISR encodes ordered wrench history with a temporal wrench encoder and Force Perceiver as persistent physical evidence for state-consistent action generation, while IME structures contact-critical motion into intrinsic progress and transverse correction for precise execution; their cross-rate loop allows the resulting physical traces to inform subsequent reasoning. In real-robot experiments across surface, insertion, and rotational interactions, HIRE achieves at least 90% completion across all evaluated task stages while improving interaction-state disambiguation, execution precision, and generalization. More broadly, HIRE provides a unified reasoning--execution perspective on precision manipulation under history-dependent partial observability, where physical interaction both realizes task intent and reveals latent-state evidence for future decisions. Code will be released upon publication.
I. INTRODUCTION
Visually similar configurations can reflect different histories of physical interaction and require different actions, a form of history-dependent partial observability. HIRE couples wrench-history-conditioned reasoning with high-rate manifold-structured execution, using interaction outcomes as evidence for subsequent reasoning.
I. INTRODUCTION
- Visually similar observations can follow different physical interaction histories and imply different latent states and actions, making current visual observations alone insufficient for reliable progression.For example, near-surface views after a clearance transition may call for continued probing or retreat toward grasping, depending on whether contact has occurred.Source: §I · pp. 1–2
- HIRE couples an Interaction-State Reasoner that uses ordered wrench history as persistent physical evidence with a high-rate Interaction-Manifold Executor in a closed reasoning–execution loop.Physical traces produced during interaction inform subsequent state reasoning.Source: §I · p. 2
- HIRE is validated on real-robot surface, insertion, and rotational interactions, with consistent improvements in stage-wise completion, state disambiguation, execution precision, and generalization.The three regimes cover transient surface interaction, constrained insertion, and sustained rotational interaction.Source: §I · p. 2
II. RELATED WORK
Prior policies use visual, force, memory, or reactive feedback, but challenges remain in retaining decision-relevant interaction history and structuring precise execution. HIRE pairs force-history-conditioned reasoning with manifold-structured high-rate execution.
A. Multimodal and Memory-Augmented Robot Policies
- Force-history-conditioned reasoning preserves mechanically grounded evidence from prior interactions, while HIRE couples it with high-rate physical execution in a closed loop.Perceptual or action memories may remain ambiguous when relevant events are visually indistinguishable; force-based memory retains history but leaves precise execution challenging.Source: A. Multimodal and Memory-Augmented… · p. 2
B. High-Rate Interaction Execution
- HIRE structures high-rate execution around a local interaction manifold, separating intrinsic progress from transverse correction rather than leaving interaction geometry implicit in generic action representations.The structure coordinates progress with correction of interaction-induced deviations while regulating reference wrench and force–position control subspaces.Source: B. High-Rate Interaction Execution · p. 2
A. Visual Interaction-State Aliasing
- Visual interaction-state aliasing occurs when visually similar observations reflect different latent interaction states shaped by prior physical interactions and therefore require different actions.This differs from ambiguity in appearance alone: the relevant distinction is what physical interaction has occurred.Source: A. Visual Interaction-State Aliasing · p. 3
B. Contact-Critical Execution Requirements
Contact-critical phases make task completion sensitive to execution accuracy and latency. Reliable progression therefore requires both history-dependent state disambiguation and responsive, precise control during contact.
B. Contact-Critical Execution Requirements
- Small execution errors can change whether contact is established, maintained, or mechanically completed, making closed-loop interaction sensitive to latency and accuracy.Contact-critical phases require a faster control timescale and tighter execution tolerances than task-level visual reasoning.Source: B. Contact-Critical Execution… · p. 3
- Reliable task completion in visually aliased precision manipulation requires resolving history-dependent interaction-state ambiguity while maintaining responsive, precise control during contact.Source: B. Contact-Critical Execution… · p. 3
IV. METHOD
HIRE pairs a 5 Hz Interaction-State Reasoner (ISR) with a 50 Hz Interaction-Manifold Executor (IME). ISR supplies context for contact-critical execution, and physical interaction provides evidence for subsequent ISR decisions.
IV. METHOD
- ISR operates at 5 Hz and integrates visual observations, language, and temporally ordered physical interaction history to generate continuous actions and a Cross-Rate Context Embedding (CCE).IME operates at 50 Hz, combining local sensory feedback with the latched CCE for manifold-structured interaction execution.Source: §IV · p. 3
- The two pathways form a cross-rate closed loop in which IME produces physical evidence for subsequent ISR decisions.ISR determines task progression before and after contact, while IME executes during contact-critical intervals.Source: §IV · p. 3
- The common HIRE interface distinguishes long-horizon wrench history for ISR from short-horizon local motion–wrench history for IME.It also includes visual views and the CCE latched for the corresponding contact-critical interval.Source: §IV · p. 3
A. Force-History-Conditioned Interaction-State Reasoner
ISR uses ordered wrench history as persistent physical evidence for action generation, allowing visually similar observations to prompt different actions when their interaction histories differ. It compresses causal wrench features into force-event tokens and uses auxiliary future-wrench forecasting during training.
A. Force-History-Conditioned Interaction-State Reasoner
- Perceptually similar observations can induce different actions when preceding wrench histories provide different evidence about the latent interaction state.ISR keeps interaction-state inference implicit in continuous action generation rather than using a separate phase, success, or termination classifier.Source: A. Force-History-Conditioned… · p. 3
- A causal temporal convolution processes a task-relevant wrench history sampled from the 100 Hz six-axis stream and reduced to 25 Hz with peak-preserving pooling.Pooling retains the complete six-axis sample with the largest translational-force magnitude in each temporal bin; the history is normalized and time-encoded before feature extraction.Source: A. Force-History-Conditioned… · p. 3
- A Force Perceiver compresses variable-length causal wrench features into a fixed budget of eight force-event tokens.Learned queries selectively aggregate evidence from the complete causal trace, with temporal information retained through time-encoded features.Source: A. Force-History-Conditioned… · p. 3
- ISR inserts force-event tokens alongside visual and language tokens, then extracts a Cross-Rate Context Embedding (CCE) by masked mean pooling.The CCE gives IME a compact summary of visual, linguistic, and force-history context rather than a phase label or prescribed local trajectory.Source: A. Force-History-Conditioned… · pp. 3–4
- An auxiliary future-wrench forecasting objective regularizes the force-event representation during training and requires no future wrench at inference.Forecasting supplies a physical learning signal for interaction dynamics that may otherwise be under-used by the primary flow-matching objective.Source: A. Force-History-Conditioned… · p. 4
B. High-Rate Interaction-Manifold Executor
IME uses local sensing, short-horizon motion–wrench history, and a latched CCE for high-rate execution. It separates progress along an interaction manifold from transverse correction and complements this motion structure with hybrid force–position regulation.
B. High-Rate Interaction-Manifold Executor
- IME operates at 50 Hz during contact-critical intervals, using wrist-centric visual feedback, short-horizon relative motion–wrench history, and the latched CCE.Its central design organizes execution according to the local geometry of physical interaction.Source: B. High-Rate Interaction-Manifold… · p. 4
- The interaction-manifold representation separates intrinsic progress along the manifold from corrective motion transverse to it.This factorization avoids entangling both components in a single Cartesian motion prediction and can be instantiated across interaction geometries.Source: B. High-Rate Interaction-Manifold… · p. 4
- IME pairs manifold-structured motion with a reference wrench and a selection subspace that chooses wrench regulation or manifold-derived motion for each degree of freedom.The learning objective supervises intrinsic progress and transverse correction, plus the interaction-regulation variables, using synchronized high-rate physical demonstrations.Source: B. High-Rate Interaction-Manifold… · p. 4
- The same factorization represents surface unloading and tangential correction, insertion progress and lateral compensation, and rotational progress and bounded radial correction.Across these geometries, IME retains intrinsic progress, transverse correction, and hybrid force–position regulation.Source: B. High-Rate Interaction-Manifold… · p. 5
C. Cross-Rate Reasoning–Execution Loop
The cross-rate loop transfers execution authority at contact onset and confirmed release, while measured interaction traces return to ISR as evidence for later reasoning. Contact-critical execution therefore both realizes the current interaction intent and informs subsequent decisions.
C. Cross-Rate Reasoning–Execution Loop
- At contact onset, the final pre-contact CCE is latched for the IME interval, while the ISR action chunk physically establishes the contact-entry condition observed by IME.The realized condition is not encoded in the CCE or sent as an additional symbolic message.Source: C. Cross-Rate Reasoning–Execution Loop · p. 5
- Contact evidence with hysteresis transfers execution authority from ISR to IME at contact onset and back to ISR after confirmed contact release.IME observes the realized physical configuration through wrist-centric visual and motion–wrench feedback.Source: C. Cross-Rate Reasoning–Execution Loop · p. 5
- IME’s measured wrench trace is accumulated into the subsequent ISR history, so each contact-critical interval both realizes interaction intent and acquires physical evidence for future decisions.This evidence can help resolve later visually aliased states according to what physically occurred during the preceding interaction.Source: C. Cross-Rate Reasoning–Execution Loop · p. 5
V. EXPERIMENTS
The evaluation asks whether HIRE handles diverse visually aliased interactions, whether its reasoning and execution components help, and whether it generalizes beyond training.
Evaluation questions
- Q1 tests reliable manipulation across diverse interaction regimes.The evaluation includes transient surface, constrained insertion, and sustained rotational interactions.Source: §V · p. 5
- Q2 and Q5 examine whether ISR reasons from force history and retains interaction-state evidence.The questions distinguish reasoning reliability under visual aliasing from the evidence retained in wrench history.Source: §V · p. 5
- Q3 and Q4 test whether manifold-structured IME improves contact precision and whether HIRE generalizes beyond training.These questions cover execution precision and out-of-distribution generalization.Source: §V · p. 5
A. Experimental Setup
The experiments use a force-sensing, two-camera robot platform to evaluate three interaction regimes against seven baselines, with shared task demonstrations and protocols.
Platform
- Experiments use a Flexiv Rizon 4S, a two-finger gripper, a six-axis force/torque sensor, and global and wrist RGB-D cameras.The cameras are Orbbec Gemini 2 units.Source: A. Experimental Setup · p. 5
Tasks
- The tasks cover transparent-object pick-and-place, EV connector mating and demating, and quarter-turn latch actuation.Surface and rotational tasks use global and wrist views; insertion uses the wrist view only.Source: A. Experimental Setup · p. 5
Baselines
Tasks
- Table I covers stage-wise completion rates across the three visually aliased precision-manipulation tasks.The caption identifies the table's scope but supplies no individual rates.Source: Table I · p. 6
Implementation Details
- The training set contains 300 real-robot demonstrations per task, and all learning-based methods share the demonstrations and evaluation protocol.The collection pipeline synchronizes multimodal trajectories and screens rollouts for quality.Source: A. Experimental Setup · p. 6
B. Results and Analysis
Results address task completion, interaction-state reasoning, contact-critical execution, and generalization across visually aliased precision-manipulation settings.
Overall task completion
- HIRE demonstrates consistent improvement in stage-wise completion across transient surface, constrained insertion, and sustained rotation; Table I summarizes cumulative rates over 30 rollouts per task.The supplied passages do not state individual table values.Source: B. Results and Analysis · p. 6
Interaction-state reasoning
- Without persistent interaction evidence, visually similar surface, connector, and latch states can prompt repeated approaches, premature termination, or continued action after completion.These reasoning failures can consume the execution horizon and reduce later-stage completion.Source: B. Results and Analysis · p. 6
Contact-critical execution
- Corrupted depth can cause premature grasping or excessive probing, while precise contact execution is sensitive to both control latency and execution accuracy.These risks are described for transparent manipulation and contact-critical phases.Source: B. Results and Analysis · p. 6
- Low-frequency force-aware policies lack a high-rate loop for brief contact transitions, while local execution can accumulate lateral or radial drift during prolonged interaction.HIRE uses high-rate manifold-structured progress and transverse correction alongside history-conditioned reasoning.Source: B. Results and Analysis · p. 6
Interaction-state reasoning
- Ordered force history improves balanced reasoning accuracy over vision-only and current-wrench inputs, while shuffling that history markedly degrades performance.The evaluation decodes task-progression decisions from predicted continuous actions and reports the comparison in Fig. 4.Source: B. Results and Analysis · pp. 6–7, Figure 4 · p. 6
Contact-critical execution
- During latch rotation, HIRE performs best across five execution metrics, supporting manifold-structured progress and transverse correction; Table II reports the comparison.The metrics cover angular and radial error, normal-force tracking, radial-force peaks, and contact retention.Source: B. Results and Analysis · p. 7, Table II · p. 7
Generalization
- Generalization is evaluated on unseen transparent objects varying in shape, height, placement, and orientation; Table III reports this evaluation scope.The passage describes visual-semantic transfer and adapting contact unloading after light-touch establishes a physical height reference.Source: B. Results and Analysis · p. 7, Table III · p. 7
Force-history evidence
- Force histories distinguish visually aliased states through temporal and cross-axis wrench patterns, including different latch signatures for end-stop, obstruction, and contact loss.End-stop loading is tangential-dominant, obstruction loading is radial-dominant, and contact loss produces rapid tangential-force decay.Source: B. Results and Analysis · p. 7, Figure 5 · p. 7
VI. CONCLUSION
HIRE couples history-conditioned interaction reasoning with high-rate, manifold-structured execution, treating physical interaction as both action and evidence for later decisions.
Conclusion
- HIRE unifies history-conditioned interaction reasoning and high-rate manifold-structured execution for visually aliased precision manipulation.ISR preserves ordered wrench history as physical evidence for state-consistent action generation.Source: §VI · p. 7
- IME structures contact-critical motion around intrinsic interaction progress and transverse correction, while cross-rate coupling makes the resulting physical trace evidence for subsequent reasoning.This closes the reasoning–execution loop.Source: §VI · p. 8
- Experiments across surface, insertion, and rotational interactions show improvements in task completion, state disambiguation, and execution precision, with generalization to unseen configurations.The paper frames HIRE as an approach to manipulation under history-dependent partial observability.Source: §VI · p. 8