Gamma: Grounded Multi-Agent Memory with Mechanical Audit for Memory-Intensive Manipulation

1 INTRODUCTION

• Directly mapping episodic history to action chunks performs poorly, while oracle grounded subgoals nearly solve the suite and isolate memory-to-subgoal generation as the unsolved stage.The policy succeeds at 0.179 end-to-end and 0.841 with oracle grounded subgoals; the target stream must supply content, coordinates, and timing.Source: §1 · pp. 1–2
• Single-VLM keyframe memory can drop decisive frames, misground coordinates, struggle with noisy unbounded frame context, and let hallucinations poison later decisions.The paper identifies pixels as the wrong medium for episodic memory and calls for verifiable replacement memory.Source: §1 · p. 2
• GAMMA uses SAM-3 grounding, a VLM writer that adds at most one event line per replan window, an append-only text bank, and a VLM reasoner that emits grounded subgoals.A propose–verify harness admits, defers, corrects, or rejects claims against visual evidence, and the same contract shapes training supervision and deployment.Source: §1 · pp. 2–3, Figure 2 · p. 3
• GAMMA reaches 0.660 average success against the oracle’s 0.841, recovering 78.5% of the privileged ceiling with the policy fixed.It reaches the oracle ceiling on several tasks.Source: Figure 1 · p. 2, §1 · p. 3
• The paper contributes a fixed-policy decomposition, verified multi-agent memory, state-of-the-art RoboMME performance, transfer to a second benchmark, and a diagnostic protocol for attributable memory defects.Readable memory lets failures be traced to a bank line or its absence.Source: §1 · p. 3

Vision–language–action policies

• The paper treats the fixed subgoal-conditioned VLA as a competent executor and studies the memory stage above it, since common VLA training data emphasizes tasks observable in the current frame.The cited VLA approaches map observations and instructions to short action chunks.Source: §2 · p. 3

Memory for embodied agents

• Architectural memory is latent and therefore cannot be inspected, audited, or corrected at deployment; newer VLA work compresses frames, attaches memory modules, or adds explicit memory.Other directions include audits of frozen VLA history use and continuous latent reasoning.Source: §2 · p. 3
• MemER stores VLM-selected keyframes and reasons over them in a second VLM pass, whereas GAMMA uses per-tick verified grounded text that preserves checkable facts.The comparison is presented as sharing a decompose-to-subgoal philosophy while replacing pixel memory and keyframe selection.Source: §2 · p. 4

Benchmarks

• RoboMME isolates episodic memory in sixteen tasks involving counts, orders, swaps, and routes, with a privileged grounded-subgoal oracle; the paper evaluates on RoboMME and RoboMemArena.Standard manipulation suites often chain subtasks that are individually Markov and therefore need no episodic store.Source: §2 · p. 4

VLM agents with tools and verification

• GAMMA delegates pixel-exact grounding to SAM-3 and checks claims against mechanical perceptual evidence rather than relying on model judgments of model text.The paper relates textual observations and append-only natural-language memory to earlier agent work.Source: §2 · p. 4

3 PROBLEM FORMULATION

• Each episode has an instruction, a demonstration video prefix, and an execution phase, and the sixteen tasks require remembering counts, object locations, orders, routes, or delayed reveals.The relevant information was observable earlier but is absent from the current frame at execution time.Source: §3 · p. 4
• The privileged planner emits one grounded subgoal sentence per action-chunk tick, naming the current subtask and an object’s image coordinate when relevant.The fixed π0.5 policy reaches 0.841 success with oracle grounded subgoals, versus 0.179 when finetuned end-to-end without subgoals.Source: §3 · p. 4
• The goal is to emit subgoals from the observable episode so the fixed subgoal-conditioned policy recovers oracle success, with correct content, pixel-level grounding, and completion-timed switches.Early switches ask the policy to skip undone work; late switches stall it.Source: §3 · p. 4

4.1 SYSTEM STRUCTURE

• On each policy tick, SAM-3 detects task objects, and a writer distils sampled frames and detections into at most one grounded identity–dynamics line or NONE.The agents share no hidden state and collaborate through the bank.Source: §4.1 · p. 5
• The append-only bank is the only history carrier; it preserves checkable facts, avoids selector-driven information loss, and compresses long episodes into a handful of lines.The bank is never rewritten or summarized.Source: §4.1 · p. 5
• A reasoner reads the instruction and full bank without pixels, then emits the next subgoal using coordinates supported by admitted bank lines.The action policy uses the oracle row’s weights and is never trained on GAMMA’s outputs.Source: §4.1 · p. 5

4.2 THE PROPOSE–VERIFY HARNESS

• The harness checks typed event, progress, and grounding claims against window evidence, admitting, deferring, correcting, or rejecting each claim.Deferrals preserve the prior subgoal for retry; corrections rebind grounding to evidence, while rejections discard claims.Source: §4.2 · pp. 5–6

4.3 TARGETS AND TRAINING

• Training targets are evidence-gated and coordinate-knowable: event lines attach to windows showing their evidence, while unsupported coordinates are demoted.Targets include oracle-boundary events and tracker-derived observations such as container moves and per-stroke demonstration records.Source: §4.3 · pp. 6–7
• The writer is trained first on verified lines, then the reasoner on writer-predicted banks; evaluation episodes are held out and deployment reproduces the training geometry.Reasoner targets are oracle subgoals with coordinates snapped to the bank.Source: §4.3 · p. 7

5.1 SETUP

• The evaluation uses held-out RoboMME episodes and a fixed subgoal-conditioned π0.5; the 9B configuration is tested across three serving seeds, with 30 episodes per task per seed.Baseline and oracle rows are reported by RoboMME; GAMMA’s rows are means over the three serving seeds.Source: §5.1 · p. 7

5.2 MAIN RESULTS

• On shuffle and reveal tasks, GAMMA roughly doubles the best baseline: VUS 75.6 vs. 38.0, VRP 65.6 vs. 30.4, and VPO 66.7 vs. 36.0.It also leads press-counting (BU 84.4, BUS 35.6) and recovers the oracle on route tracing (PL 98.9, RS 56.7).Source: §5.2 · p. 7
• 66.0 average across sixteen tasks, +21.5 over FrameSamp+Modul (44.5) and 78.5% of GroundSG (84.1); GAMMA reaches the oracle ceiling on three tasks.GAMMA is the best non-privileged method on nine of sixteen tasks.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 HARNESS ABLATION: WHICH VERDICT, AND AT WHICH SCALE

• Without DEFER, BU falls 84 →57 and BUS 36 →13; without REJECT, PL falls 99 →3 and RS 57 →7; removing CORRECT costs four to seven points on binding tasks.The ablation disables one verdict at a time at 9B; Table 3 reports losses of at least four points in affected cells.Source: Table 3 · p. 8, §5.3 · p. 8
• At 0.8B, the harness adds +25.1 points (32.3 →57.4), versus +14.8 at 9B (66.0 against 51.2); the verified 0.8B system exceeds the 44.5 best prior non-privileged result.On thirteen tasks with identical evidence extractors across scales, the small-scale gain is +17.8 (34.9 →52.7).Source: Table 3 · p. 8, §5.3 · p. 8

5.4 TICK-LEVEL DIAGNOSIS: AUDITING A VLM AS A MEMORY WRITER

• The tick-level audit logs each serving step and assigns failures to memory, grounding, timing, or policy defects, connecting episodes to data, model, or serving.Figure 4 shows a deferred ButtonUnmaskSwap press claim while the button remains detected, followed by defect attribution and closed-loop predicates.Source: §5.4 · p. 8, Figure 4 · p. 9
• Audits find that writers report progress from prior rather than evidence, fabricate unobservable grounding, lose identity across swap chains, and read motion direction at chance; single-VLM pipelines fail by binding, timing, or content.DEFER raises BU 57 →84 and BUS 13 →36; REJECT raises PL 3 →99 and RS 7 →57. MemER retains no demonstration frame.Source: §5.4 · p. 9

5.5 A SECOND BENCHMARK: ROBOMEMARENA

• On held-out RoboMemArena layouts, GAMMA reaches 38.8 task and 59.0 subtask success, above reported memory-VLA and keyframe methods and level with the benchmark’s end-to-end memory system.The transfer uses the contracts unchanged, with two geometry constants, detector prompts, and five evidence predicates adapted for the benchmark.Source: §5.5 · p. 9, Table 4 · p. 9

6 DISCUSSION AND LIMITATIONS

• The contribution and controlled comparisons are limited to simulation; real-robot use would change the perception front end and executor, not the contracts.The comparison uses a privileged ceiling and 480 held-out episodes per configuration.Source: §6 · p. 9
• Each tick costs one detector pass and two VLM generations, and the closed-loop evaluation waits for calls rather than measuring their latency.The proposed deployment runs agents asynchronously at the subgoal timescale while the policy acts at control rate.Source: §6 · p. 10
• The release includes the pipeline, trained adapters, evaluation tools, trace logger, and RoboMemArena port needed to reproduce the reported numbers.It also provides the fixed executor weights and scripts for corpus generation and finetuning.Source: REPRODUCIBILITY STATEMENT · p. 11

A HARNESS INSTANTIATIONS AND THE LIMITING CASE

• The harness admits, defers, corrects, or rejects claims using evidence such as visible completion, compatible detections, and detected container movement.It also rejects duplicate admitted lines and claims that conflict with authoritative extractors.Source: A HARNESS INSTANTIATIONS AND THE… · p. 15, Figure 5 · p. 15
• RouteStick and PatternLock expose motion judgments that the finetuned writer guessed at chance despite six attempts to reveal the signal.Their direction cues are distributed across many frames, including fading ink and small marker movements.Source: A HARNESS INSTANTIATIONS AND THE… · p. 15
• Deterministic motion readers score 33/33 RouteStick strokes and 10/10 PatternLock episodes against development annotations.They emit demo-record lines for the writer and advance stroke progress by detecting arrival at the known endpoint.Source: A HARNESS INSTANTIATIONS AND THE… · p. 15, Table 5 · p. 16
• The readers encode benchmark regularities, so transfer requires re-deriving them for a new environment or a VLM that can genuinely read motion.The paper identifies motion reading as an open research problem.Source: A HARNESS INSTANTIATIONS AND THE… · p. 15
• RouteStick success (56.7) matches the privileged oracle’s (55.6), although reader errors cluster in seven demonstrations.Those errors do not lower success below the policy’s reported ceiling.Source: A HARNESS INSTANTIATIONS AND THE… · p. 16

B TARGET CONSTRUCTION FOR THE WRITER AND THE REASONER

• Writer targets use role-based event lines, while candidate events come from oracle steps, detection trackers, or detector scene lines.Targets are placed only in windows containing evidence; unsupported candidates are dropped, and windows without an evidenced event receive NONE.Source: B TARGET CONSTRUCTION FOR THE WRITER… · pp. 17–18
• A knowability audit ensures target coordinates are visible in the evidencing window or already present in the bank, demoting unsupported coordinates.Writer and reasoner records pair targets with their deployment inputs, including sampled frames and detections for the writer and the full bank for the reasoner.Source: B TARGET CONSTRUCTION FOR THE WRITER… · p. 18

C DEVELOPMENT EVIDENCE: CLOSED-LOOP ROUNDS

• Moving completion registration to the reasoner collapsed counting tasks; returning it to the writer restored them, with a one-tick delay for invisible press effects.The reasoner declared completion on schedule rather than on evidence.Source: C DEVELOPMENT EVIDENCE: CLOSED-LOOP… · p. 18
• The hardest shuffle task rose from 4/10 to 10/10 when each move was recorded with detected endpoints and an evidence gate, followed by a summary.Folding the admitted move chain also provides coordinate authority when the reasoner mis-folds long chains.Source: C DEVELOPMENT EVIDENCE: CLOSED-LOOP… · p. 18
• Serving at 16 steps after training at 8 broke both video-placement tasks, while restoring verbatim training-side pairing recovered them.The mismatch was invisible in open-loop metrics.Source: C DEVELOPMENT EVIDENCE: CLOSED-LOOP… · p. 18
• Predicting the Nth object arrival and pressing one tick early raised counting-press success from 2/10 to 5/10, but is an exception to evidence-based verification.The press effect has zero pixel change, so it cannot be completed from visual evidence alone.Source: C DEVELOPMENT EVIDENCE: CLOSED-LOOP… · p. 18
• A deterministic demonstration record alone moved RouteStick from 0/10 to 4/10 before arrival-based stroke advancement was added.Source: C DEVELOPMENT EVIDENCE: CLOSED-LOOP… · p. 19
• Adding the current frame to the reasoner improved no task in screening because progress judgment belongs to the writer and harness, while admitted lines carry fresh coordinates.The frame added context without adding evidence.Source: C DEVELOPMENT EVIDENCE: CLOSED-LOOP… · p. 19

Protocol and memory coverage

• MemER’s keyframe memory holds 1–9 execution frames per episode across six binding and press tasks, covers 41 of 59 oracle transitions within 16 steps, and never stores demonstration frames.Its released implementation attaches the demonstration as raw video to each query instead of distilling it into keyframe memory.Source: D TICK-LEVEL DIAGNOSIS: PROTOCOL,… · p. 19

Three-way statistics

• The three systems fail along distinct axes: binding by grounding, press-counting by timing, and direction by content.The diagnosis uses 10 episodes per task and seed 7; content is strict per-tick string match, grounding is pixel distance on content-matched ticks, and premature switches are tracked separately.Source: D TICK-LEVEL DIAGNOSIS: PROTOCOL,… · p. 19, Figure 7 · p. 20, Table 10 · p. 21

Press-counting case

• On ButtonUnmaskSwap, MemER switches to the second press at step 64 while its oracle holds the first until step 144, then advances without returning.GAMMA defers its first completion claim while the button remains detected, admits it one tick later, and subsequently follows the oracle; its pick-up coordinate is 1.4 px from the oracle.Source: Table 9 · p. 20, Figure 7 · p. 20

Binding case

• On VideoRepick, both single-VLM pipelines target the cube’s demonstration-time position, 22–24 px from the oracle, and both episodes fail within eight steps.The episode’s correct cube moves twice before execution; GAMMA’s bank records the moves and its reasoner cites ⟨84, 102⟩, 1.4 px from the oracle.Source: CASE STUDIES · pp. 20–21

Three-way statistics

• The tick-level table reports content match, wrong-instance grounding, and premature switches; GAMMA adopts 91–100% of oracle transitions with zero median lag.Deferred switches lower its content-match rate on binding tasks.Source: Table 10 · p. 21

Direction case

• For direction, GAMMA’s stream matches the oracle at every tick, while the single-VLM pipelines misread or repeat stroke direction.GAMMA’s demonstration record preserves all three stroke senses, and admitted completion claims advance stroke progress.Source: CASE STUDIES · p. 21

E SEED VARIANCE

• GAMMA’s suite averages are 66.7/66.3/65.2 across three serving seeds, a 1.5-point spread, with a stable per-family pattern.The same agent checkpoints and configuration were used; only the environment seed differed.Source: E SEED VARIANCE · p. 21
• The 0.8B agents’ suite averages are 59.2/57.5/55.6 across three serving seeds, a 3.6-point spread.The scale comparison therefore uses three seeds at both agent sizes.Source: E SEED VARIANCE · p. 21

MemER

• In the VideoRepick case study, both single-VLM pipelines target the cube’s demonstration-time position, while GAMMA’s bank records its two moves.GAMMA’s bank includes the demonstration pick and drop and both observed cube movements before execution.Source: Table 12 · p. 22, MemER · p. 22

Models and finetuning

• Both agents share a Qwen3.5-9B base with separate LoRA adapters, trained for one epoch at effective batch 64.The adapters use rank 16, α = 32, and learning rate 10^-4.Source: F CONFIGURATION DETAILS · p. 22

Corpus

• The training corpus contains about 55k writer records and matching reasoner records from 1,600 recorded episodes, split by episode index for validation.Source: F CONFIGURATION DETAILS · p. 23

Context geometry and serving

• Each tick is one 16-step action chunk; the writer sees five sampled frames per window and the last eight bank lines, while the reasoner sees the full bank.Frames use stride 4 with window boundaries included; /reset replays the demonstration prefix in 16-step windows, and /tick runs writer then reasoner once per 16 steps.Source: F CONFIGURATION DETAILS · p. 23

Transfer setup

• The RoboMemArena port changes the tick to 10 steps, the frame window to three frames, detector prompts, and evidence predicates based on executor state.The contracts require a detector, tick grid, recorded episodes, and an evidence stream against which claims can be verified.Source: G ROBOMEMARENA DETAILS · p. 23
• The agent code, corpus generator, and harness logic remain those of Section 4, while the reasoner follows the recorded fixed subtask sequence.The memory stage supplies the timing for handing the executor its next subtask.Source: G ROBOMEMARENA DETAILS · p. 24

Transfer results

• With the writer, RoboMemArena task/subtask success is 38.8 / 59.0; with harness predicates alone it is 38.7 / 58.6.For fixed plans, the writer’s role is to raise completion claims that the predicates can also raise, so the verified hand-off carries the gain.Source: G ROBOMEMARENA DETAILS · p. 24, Table 16 · p. 24

Hand-off timing

• Admitting a verified writer claim on the tick it is raised instead gives 19.2 / 46.8 task/subtask success in the reported 10–20 episodes.The claims are correct but early; the benchmark predicates can fire before a segment is complete, handing off mid-motion.Source: G ROBOMEMARENA DETAILS · p. 24, Table 16 · p. 24
• Waiting two ticks after a predicate holds yields 38.7 task success, while advancing on segment-completion conditions yields 19.2.Segment completion waits for conditions such as object release or the wrist returning, rather than the benchmark predicate’s earlier event.Source: G ROBOMEMARENA DETAILS · p. 24

• The released Qwen3.5-9B corpus contains 54,829 training and 5,904 validation writer records across 1,600 episodes; 97.5% of training-target coordinates lie within 2 px of window detections or visible bank coordinates.The corresponding validation knowability rate is 97.3%; at least 8 px endpoint displacement occurs in 94.5% of training move lines and 96.2% of validation move lines.Source: H BANK QUALITY AND ADDITIONAL MATERIAL · pp. 24–25
• The writer is evaluated autoregressively on its own predicted bank, and the reasoner on writer-predicted banks; direction tasks are where writer lines are weakest, so demonstration records are reader-owned at deployment.Validation covers 5,904 ticks, including 1,211 event ticks; the passage defines exact match, recall, precision, and coordinate-known metrics but supplies no values for them.Source: H BANK QUALITY AND ADDITIONAL MATERIAL · p. 25, Table 17 · p. 25
• SAM-3 uses fixed text prompts, a 0.35 score threshold, a 0.5 mask threshold, and a 14 px hit radius consistently during corpus construction and deployment.Table 18 lists the text prompts per object class.Source: H BANK QUALITY AND ADDITIONAL MATERIAL · p. 25, Table 18 · p. 25
• The writer observes the last 16 control steps and emits grounded event line(s) or NONE; the reasoner reads bank events and emits the current control-chunk subgoal using bank coordinates.The writer’s window uses five sampled frames at stride four, and the reasoner answers with a lowercase subgoal sentence.Source: H BANK QUALITY AND ADDITIONAL MATERIAL · pp. 25–26
• Temporal counting uses admitted completed-pick lines, while spatial retrieval reads a state summary after per-move swap lines; both are lookups or counts over the bank.The spatial example also keeps the press count in completion lines.Source: H BANK QUALITY AND ADDITIONAL MATERIAL · p. 26
• For VideoRepick, tracker lines carry the demonstrated cube’s identity through a shuffle, so the reasoner selects its final coordinate rather than the demonstration-time position cited by single-VLM baselines.The bank walkthrough identifies the same physical cube across the demonstration, shuffle, and execution frames.Source: H BANK QUALITY AND ADDITIONAL MATERIAL · p. 26, Figure 11 · p. 27
• A bank holds a median of 8 lines (5th–95th percentile 3–20; maximum 73) versus median episode length of 240 control steps (95th percentile 1,296), enabling whole-history reads each tick.These figures are from three serving seeds of the released configuration.Source: H BANK QUALITY AND ADDITIONAL MATERIAL · p. 27
• For RouteStick, a deterministic reader extracts stroke direction from demonstration ink; the writer’s own direction words are at chance, while admitted completion advances stroke progress.The writer records arc-tip observations around the reader-owned demonstration strokes.Source: H BANK QUALITY AND ADDITIONAL MATERIAL · p. 27, Figure 12 · p. 28