Robo-Harness K1: Harnessing Robot-Use Agents via Perception Augmentation

• VLMs understand objects and instructions, but VLAs may compromise pretrained understanding and generalize poorly with changed objects or layouts, while RGB-only control is costly and capability-dependent.Source: §1 · p. 2
• K1 supplies persistent identity anchors, calibrated spatial measurements, and grasp hypotheses through tools, leaving the VLM to choose targets, motions, and recovery.Source: §1 · p. 2
• 77.8% on matched LIBERO-PRO cases for Gemini with K1 surpasses RGB-harness GPT-6 Astra at 61.1%; K1 raises Astra to 88.9%.Source: §1 · p. 2
• K1 transfers without target-task fine-tuning to three RoboSuite arms and RoboTwin dual-arm tasks, reaching 28.0% on RoboTwin Hard.Source: §1 · pp. 2–3
• A Qwen3.5-9B student fine-tuned on 107 teacher episodes outperforms action-policy baselines across evaluation splits and alone generalizes where every VLA baseline scores 0.0%.Source: §1 · p. 3

• VLA approaches represent actions as language tokens or continuous actions, but action adaptation can damage pretrained visual-language representations and generalization depends on demonstration coverage.Source: §2 · p. 4
• Agentic systems expose skills, programs, spatial maps, feedback, or motion units; K1 instead augments the evidence used by one RUA model at each decision.Source: §2 · p. 4
• K1’s perceptual focus contrasts with RoboHarness and Harness-VLA, which orchestrate heterogeneous task-specific policies through memory-driven planning.Source: §2 · p. 4

3.1 Perception as Tools

• Four tool families ground visible entities, measure their spatial relations, track references across actions, and propose grasps from observed geometry.The agent selects generic motions using the returned evidence; the harness does not supply task-specific pick/place primitives or hidden object poses.Source: §3.1 · pp. 4–6
• The interaction model separates reasoning calls that gather evidence without changing state from action calls that advance the environment.Receipts are appended to the running context, and only physical action steps produce a state transition.Source: §3.1 · p. 5
• Bounded fine motions and direct moves support different control granularities, while action feedback reports achieved motion and residual error.Anchor co-motion provides evidence of grasp success.Source: §3.1 · pp. 5–6
• Memory retains the instruction, current evidence, recent exchanges, and prior images; archived exchanges can be searched and read.Decision notes record evidence, intent, and expected outcomes for student supervision.Source: §3.1 · pp. 6–7

3.2 From Tool Interaction to Policy Learning

• Tool-call traces align with next-token prediction, motivating student training on serialized calls rather than a separate action output.Qwen3.5-9B is adapted with language-side LoRA and a frozen vision encoder using 107 successful Gemini episodes; the frozen harness remains available at test time.Source: §3.2 · p. 7

4.1 Experimental Setup

• The evaluation covers LIBERO-PRO frontier and student tests, with splits for exact training configurations, new initial states, and held-out task conditions.Frozen Gemini is also tested without target-task fine-tuning on RoboSuite and RoboTwin; student methods share 107 successful Gemini episodes.Source: §4.1 · pp. 7–8, §4 · p. 7

4.2 Perception tools unlock manipulation capability

• K1 reports 77.2% overall LIBERO-PRO accuracy, with Object at 86.7%, Spatial at 81.7%, and Goal at 63.3%.Goal tasks involve articulated or multi-stage contact sequences, extending beyond spatial perception.Source: §4.2 · p. 8
• 77.8%: Gemini with K1 surpasses RGB-only GPT-6 Astra at 61.1% on matched cases, while K1 raises Astra to 88.9%.K1’s augmentation of Astra is a gain of 27.8 percentage points.Source: §4.2 · p. 8

4.3 Training compact RUAs from tool-call traces

• 44.2%: Qwen K1 RUA reaches this accuracy on new initial states, versus 30.2% for OpenVLA; on held-out task conditions it reaches 13.9%, while action baselines score 0.0%.At epoch 5, Qwen K1 RUA reaches 51.2% in-domain; Qwen VLA with continuous action regression remains at 0.0%.Source: §4.3 · pp. 8–9

4.4 Zero-shot transfer across environments and embodiments

• 90.0% on Panda, 88.8% on UR5e, and 86.2% on IIWA: K1 with Gemini transfers across the three tested arms without fine-tuning.Cube tasks remain fully successful across arms, while nut assembly accounts for the accuracy differences; handover reaches only 5.0% across the full Panda task set.Source: §4.4 · pp. 9–10
• 32.0% on RoboTwin Easy and 28.0% on Hard: Gemini with K1 transfers without RoboTwin fine-tuning, a 4.0-point gap.Hard randomizes lighting, background textures, clutter, and table height; the cited RDT and π0 references decline from 47.6% to 19.2% and 64.0% to 25.2%, respectively.Source: Figure 5 · p. 10, §4.4 · p. 10

4.5 Perceptual evidence in practice

• Across 180 LIBERO-PRO trajectories, perception accounts for 25.5% of Gemini decisions, alongside 67.5% motion requests.Longer text history tends to support behavioral consistency, while image history beyond one frame can add redundant or stale context.Source: §4.5 · pp. 10–11

5 Conclusion

• K1 exposes calibrated 3D evidence through visual reference lines and persistent anchors, letting agents reason about object relations and execute generic motions from measurements.The VLM remains responsible for choosing targets, motions, and recovery.Source: §5 · p. 11
• With K1, Gemini Flash surpasses GPT-6 Astra on matched tasks, and the harness further augments Astra itself.The conclusion also reports that a compact student trained on 107 episodes outperforms action-policy baselines.Source: §5 · p. 11
• Modest held-out accuracy and the simulation-only setting bound the current conclusion about K1-trained RUAs.The authors identify perceptual evidence acquisition as a promising target for robot post-training.Source: §5 · p. 11

A.1 Tool families and contracts

• Tool contracts cover perception, measurement, grasp proposals, and motion, but measurements and execution do not certify task success.Image measurements specify camera and frame, and pixel arguments declare their coordinate convention.Source: §A.1 · p. 17, Table 3 · p. 17

A.2 Segmentation, tracking, and geometry

• Segmentation returns visible geometry, while stored anchor identity does not guarantee accurate current coordinates.Tracking can fail from correspondence drift or a depth-surface change; lost correspondence requires reinspection.Source: §A.2 · p. 18

A.3 Grasp proposal and ranking

• Grasp ranking scores visible-space compatibility along five approach-path positions and presents at most three candidates as hypotheses, not executed grasps.Near-duplicate hypotheses within 1.5 cm and 20° are suppressed; backend scores and rank labels remain hidden from the agent.Source: §A.3 · p. 18

A.4 Control and feedback

• Bounded servo steps, anchored rotation targets, and persistent pose targets reduce local undertracking and accumulated drift without global path planning.Full-target motion stops on tolerance, stall detection, or its step budget, while feedback omits internal diagnostics and private grasp scores.Source: §A.4 · pp. 18–19

A.5 Perception-tool illustrations

• The tool atlas distinguishes measured references, visible geometry, automatic anchor updates, and grasp hypotheses from executed poses.The same wrist-camera anchor moves in pixel space with an estimated 0.89 mm world displacement before becoming invalid; grasp panels show hypotheses, not executed grasps.Source: §A.5 · p. 19

B.1 Data construction and filtering

• The training pool retains 107 replay-verified native successes from 43 conditions after withholding 32 of 139 successful Gemini episodes.Tool-call supervision retains 2,482 of 2,564 targets (96.8%) after excluding execution-error targets, malformed responses, and rejected completions.Source: §B.1 · p. 19
• Filtering a target leaves later inputs unchanged, preserving the teacher-visible causal history; depth and action supervision are regenerated through verified native replay.The tool dataset uses original pre-action requests, tool schemas, and model-visible images.Source: §B.1 · p. 19

B.2 Architectures and optimization

• The agentic Qwen arm uses a frozen vision encoder and language-side adaptation, while the action baselines include action regression, original OpenVLA weights, and the official π0.5 checkpoint.The listed baselines differ in action representation and trained components.Source: §B.2 · p. 19, §1 · p. 21

1 REGION GROUNDING

• Figure 6 expands the perception-tool examples into an atlas of tool-family feedback, including cross-view projection, tracked anchors, and grasp hypotheses.Its panels use archived images shown to the agent, and crops do not define new measurement coordinates.Source: Figure 6 · p. 20

B.3 RGB-to-action baseline

• The RGB-to-action baseline learns JSON step, target, or gripper commands from teacher decisions and executes one native action before observing again.Labels represent commanded actions at each boundary, not post-contact displacement.Source: §B.3 · p. 22
• Epoch 5 of the RGB-to-action baseline yields five native successes, alongside 74 budget terminations and 43 invalid actions.The authors warn that format robustness materially contributes to this baseline’s failure and should not be conflated with geometric failure.Source: §B.3 · p. 22

B.4 RGB baseline and paired interface comparison

• The paired systems share task, initial state, cameras, resolution, horizon, call limit, and evaluator, but the comparison tests K1 as a complete interface.A matched Gemini RGB-only evaluation is not included.Source: §B.4 · p. 22

C.1 Benchmark protocol and data splits

• Student split C holds out twelve task conditions, while five conditions without successful teacher demonstrations are excluded from splits A, B, and C.Split C includes new task conditions, such as new object relations within shared scenes and manipulation skills.Source: §C.1 · pp. 22, 24
• The full evaluation covers 180 trials across ten tasks in each of the Spatial, Object, and Goal suites, two variants, and three initial states.Published LIBERO-PRO rows use different trial counts and are external references, not reruns.Source: §C.1 · pp. 22–23
• 139/180 (77.2%) is Gemini’s full-evaluation native success rate; the paired subset records 11/18 for GPT RGB, 14/18 for Gemini K1, and 16/18 for GPT K1.The paired GPT evaluation uses task indices 0–2 and state 0 in all six suites; Gemini’s paired result comes from the full evaluation.Source: §C.1 · p. 23

C.2 Execution and scoring

• Native simulator success is replay-verified from the same initial state, with maximum TCP deviation capped at 10^-6 m.Gemini’s post-release accuracy is 131/180 (72.8%), versus 139/180 (77.2%) native accuracy, so the release diagnostic is reported separately.Source: §C.2 · p. 24

C.3 Frontier summary and paired comparison

• On the same 18 configurations, GPT-6 Astra with RGB-only Inspect Robots scores 61.1%, Gemini with K1 77.8%, and GPT-6 Astra with K1 88.9%.Figure 1 combines these local matched results with published baseline averages evaluated under their respective protocols.Source: §C.3 · p. 24, Table 7 · p. 25
• The budget-limited Astra subset covers task indices 0–2 at initial state 0, balancing Spatial, Object, and Goal and the swap and task variants.Each category contributes six cases and each variant contributes nine, matching the full Gemini evaluation's category balance.Source: §C.3 · p. 24
• Gemini with K1 scores 77.2% on the full 180-trial evaluation and 77.8% on the matched 18-case subset, a 0.56-point difference before rounding.The estimates are dependent because the subset is contained in the full evaluation; descriptive Wilson intervals are 70.6–82.7% and 54.8–91.0%, respectively.Source: §C.3 · p. 24
• The three local systems share task identities and initial states, but published baselines use their own protocols and trial counts.Table 7 compares the local systems on six cases per category, eighteen total; the GPT comparison changes perception, memory, control interface, and completion feedback together.Source: §C.3 · p. 24, Table 7 · p. 25

C.4 Full checkpoint results

• Qwen VLA (RGB-D) at epoch 5 completes all 122 cases with zero success; the RGB-only evaluation stopped after consistently unsuccessful results.Unrun cases are excluded from denominators.Source: §C.4 · p. 25
• OpenVLA-7B group C records 0/12 (0.0%) in Spatial, Object, and Goal, while π0.5 group C also records 0/12 (0.0%) in each category.The ledger distinguishes incomplete groups and unrun groups.Source: Table 8 · p. 25, §C.4 · p. 26
• At epoch 5, the agent records five successes across four task conditions: Goal task 6/state 40, Object swap 0/state 42, Spatial swap 0/states 40 and 41, and Spatial task 5/state 42.The passage identifies these as the epoch-5 agent's successes on held-out task conditions.Source: §C.4 · p. 26
• In the paired GPT comparison, K1-only outcomes include six successes versus one RGB-only success, while both systems solve ten cases and fail one.The exact two-sided McNemar test gives p = 0.125, which does not reach the 0.05 significance threshold in this small paired evaluation.Source: §C.4 · p. 26

D.1 RoboSuite: cross-embodiment transfer

• The RoboSuite adapter retains the K1 runtime and frozen Gemini 3.7 Flash without target-task demonstrations or fine-tuning.Panda covers seven tasks, while UR5e and IIWA cover the first four; all three retain the PandaGripper morphology on non-wiping tasks.Source: §D.1 · p. 26
• UR5e and IIWA wiping and dual-arm configurations are untested rather than failed, pending validation of setup and reachability requirements.Extensions require checking dual-arm base placement, reset poses, shared reachability, controller/camera configuration, and, for wiping, sponge attachment and contact alignment.Source: §D.1 · p. 26
• The adapter provides calibrated scene and wrist RGB-D, robot kinematics, and robot-owned collision geometry, but no task-object poses, instance masks, or shaped rewards.Generic world-frame commands can move multiple arms simultaneously; unspecified arms hold position, and the wiping configuration hides tools requiring an actuated gripper.Source: §D.1 · p. 26
• RATs reports 141/350 successes for CaP-Agent0 and 172/350 after adding RATs skills, but its published reference differs in trial count and evaluation conditions.The reference uses ten trials per randomization versus four here and differs in reset states, cameras, models, controllers, instructions, and budgets.Source: §D.1 · p. 27
• K1 totals 72/80 on Panda, 71/80 on UR5e, and 69/80 on IIWA across the four common tasks; IIWA reaches 69/80 (86.2%).Panda's seven-task total is 101/140, including wiping, handover, and two-arm lifting; totals across different task coverage should not be used as robot-to-robot comparisons.Source: §D.1 · pp. 27–28, Table 10 · p. 27
• OpenVLA base scores 0/100 on tested Panda tasks, 0/80 on UR5e, and 0/80 on IIWA; π0.5 base scores 0/100 on Panda and 0/80 on UR5e.These official-base VLA rows are distinct from the LIBERO-fine-tuned models in Figure 4.Source: Table 10 · p. 27, §D.1 · p. 28

D.2 RoboTwin: bimanual transfer

• The zero-shot RoboTwin setup uses Aloha AgileX dual arms in SAPIEN, native head and two-wrist RGB-D views, robot state, perceptual tools, and generic commands.Hard conditions vary clutter, illumination, textures, and table height; zero-shot means deployment without target-task model updates.Source: §D.2 · p. 28
• Of 200 planned RoboTwin episodes, 18 fail native initialization, leaving 96 Easy and 86 Hard episodes; all retained policy failures and execution errors count toward accuracy.The ten selected tasks exclude Open Laptop, Pick Dual Bottles, and Place Object Basket from the published table.Source: §D.2 · pp. 28–29
• K1 reaches 32.0% task-macro accuracy on Easy and 28.0% on Hard RoboTwin conditions.Episode-micro rates are 31/96 (32.3%) and 25/86 (29.1%), respectively, or 56/182 (30.8%) pooled; the task-macro and episode-micro values differ because valid episode counts are unequal.Source: §D.2 · p. 29, Table 11 · p. 29
• K1's Easy-to-Hard drop is 4.0 percentage points, versus 28.4 for RDT and 38.8 for π0 on the same task subset.The experiment combines lighting with other perturbations, so their individual effects remain unresolved; results also vary by task.Source: §D.2 · p. 30

E.1 Tool-use distribution

• Across 6,282 LIBERO-PRO responses, motion accounts for 4,238 requests (67.5%), perception 1,604 (25.5%), memory/progress 245 (3.9%), and completion checks 142 (2.3%).The remaining 0.8% are output-limit responses; these are model decisions, not native simulation steps.Source: Figure 7 · p. 30, §E.1 · p. 30
• Perception and memory/progress make up 1,849 non-motion decisions (29.4%), and 1,036 perception requests occur after the initial frame across 159 trajectories.The analysis distinguishes observable information-gathering steps from internal reasoning; private provider reasoning is excluded.Source: §E.1 · p. 30

E.2 Memory-context ablation

• The ablation varies prior image rounds K and recent text exchanges N while keeping retrieval, progress memory, perceptual tools, generic motions, and completion feedback enabled.Each setting uses the same 18 existing paired-comparison cases; this is a prompt-context ablation, not an ablation of the entire memory system.Source: §E.2 · pp. 31–32
• 83.3% at K = 0 falls to 61.1% at K = 3 (N = 8), favoring selective image history over longer automatic image windows.The 18-case evaluation retains retrieval and progress notes even at K = 0.Source: Table 14 · p. 32, §E.2 · p. 32
• 55.6% at N = 4 versus 77.8% at N = 8 (K = 1) shows a non-monotonic text-history effect.The shortest windows average 70.0 and 72.6 calls, compared with 33.0 at N = 8.Source: Table 14 · p. 32, §E.2 · p. 32
• Text records preserve decisions and milestones, while images provide time-sensitive visual evidence, so their automatic context windows need not be equally long.The authors retain K = 1, N = 8 as the main configuration and caution that each setting has only 18 episodes without repeated evaluation seeds.Source: §E.2 · p. 32

E.3 Qualitative interaction traces

• The LIBERO-PRO bowl trace reaches native success after 33 model calls, while its paired RGB episode ends without native success.The K1 trace uses an observed lift to update its carrying milestone before measuring placement; native success is checked separately from voluntary release.Source: §E.3 · p. 33
• After unsuccessful nut transports, the UR5e trace returns to the observed nut rather than treating gripper closure as proof of a successful grasp.The nut-assembly episode succeeds after regrasping, on a UR5e with a Panda gripper and without target-task fine-tuning.Source: §E.3 · p. 34
• In a randomized RoboTwin Hard scene, the dual-arm trace succeeds after 12 model calls and 31 native action requests, including a rejected completion claim before the joint lift.Sparse completion feedback distinguishes a verbal claim from physical success.Source: §E.3 · p. 35