PaperlayerView exact revision on arXiv

Source-linked AI summary

inr2action: Encoding Robot Episodes as Neural Implicit Representations for Recall and Action

Leann Alhashishi, Christos Margadji, Sebastian William Pattinson

arXiv:2610.28882v1cs.ROcs.AI

TL;DR

Long-horizon robots need memory for information no longer visible in the current scene, without storing a stream whose cost grows with operating time. The paper encodes each episode as a weight delta from a shared meta-learned initialization and uses projected tokens for recall and action prediction. On RoboMME, question answering exceeds chance across question types and action decoding reaches mean R2 of 0.51.

  • Problem

    For long-horizon robotic operation, relevant information may leave the field of view, while feeding the full observation stream back into a policy makes cost scale with operating time.

    Source: §I · p. 1

  • Method

    The system fits an INR to each episode from a shared meta-learned initialization and projects its weight delta into tokens for language-model question answering and joint-command regression.

    Source: §I · p. 1, A. Task definition · p. 3

  • Results

    Question answering exceeds chance across all question types, with 47.2% overall accuracy against a pooled 34.6% chance rate, and action decoding reaches mean R2 of 0.51.

    Source: A. Weight-Space Question Answering · p. 5, B. Memory-Language-Action Decoder · p. 6

  • Takeaways & Limitations

    The weight delta functions as a persistent, individually addressable episode memory that supports recall and action prediction without frame re-access.

    Source: §V · p. 7

    Evaluation uses two training seeds and a limited source-episode corpus; closed-candidate likelihood scoring tests discrimination rather than free-form generation, and memory construction is offline.

    Source: §V · p. 7

Abstract

from arXiv · show

Long-horizon robotic operation requires memory that scales with essential information rather than execution time. We present a parameter-space memory framework where each episode is stored as a separable object. A neural implicit representation is fitted to episode video from a shared, meta-learned initialization; the resulting weight delta constitutes the memory. Crucially, this delta remains a fixed size regardless of whether the episode spans a hundred or a hundred thousand frames. A learned projector maps the weight delta into token embeddings, conditioning a language model for question answering and a decoder for joint command regression, without frame retrieval at query time. On RoboMME, the language model exceeds chance across all question types, and the decoder achieves a mean $R^2$ of 0.51. Replacing the delta with matched noise drops performance by 11.4 points, while random-origin weights are non-decodable, proving the meta-learned initialization is essential for embedding episode content. By representing individual experiences as fixed-size weight deltas in a common parameter space, this approach decouples memory cost from time horizon and provides a basis for scalable streaming memory.

I. INTRODUCTION

Long-horizon robots need persistent memory when relevant evidence has left the field of view, but storing the full observation stream makes cost grow with operating time. This work stores each episode as a parameter-space memory that supports language-based recall and next-command prediction without frame access at query time.

I. INTRODUCTION

  • Long-horizon robots need memory when evidence relevant to an action is no longer visible, sometimes seconds, minutes, or hours after it was observed.The introduction calls settings non-Markovian when information needed at time t is distributed across the episode rather than contained in a single frame.Source: §I · p. 1
  • Transient activations are bound to the current computation, while external storage must be retrieved before reasoning can proceed.The introduction also distinguishes parameter-based memories that continually update one running memory from this work’s separable episode memories,.Source: §I · p. 1
  • Each episode is encoded as an INR fitted from a shared meta-learned initialization, and its weight delta is projected into tokens for question answering and next-joint-command prediction without frame access.The work evaluates on RoboMME, a benchmark of 16 non-Markovian manipulation tasks spanning temporal, spatial, object, and procedural memory.Source: §I · p. 1, Figure 1 · p. 2
  • Matched-noise and within-task controls, plus an initialization ablation, test whether recall and action decoding depend on episode content and the shared meta-learned origin.The paper states that the meta-learned origin is the condition that renders the weights decodable.Source: §I · p. 1

II. RELATED WORK

Neural implicit representations encode signals as continuous functions in network parameters, and prior work extended them to scenes and video. Their fitted weights have largely served reconstruction and compression rather than direct querying.

A. Neural Implicit Networks

  • A neural implicit representation encodes a signal as a continuous function mapping input coordinates directly to signal values, and reconstructs it through forward passes over learned parameters.SIREN uses periodic sine activations to counteract ReLU spectral bias and produce high-fidelity reconstructions of images, objects, and audio.Source: A. Neural Implicit Networks · p. 2
  • NeRF extends implicit representations to 3D scene geometry and appearance, while LeRF enables open-vocabulary semantic queries over encoded scenes using distilled CLIP and DINO features.Source: A. Neural Implicit Networks · p. 2
  • NeRV represents video as a function of time, predicting a full RGB frame from one temporal coordinate and fitting and reconstructing faster than pixel-wise INRs.Source: A. Neural Implicit Networks · p. 2
  • Fitting-cost reductions—from alternative parameterizations,, amortized weight prediction, or meta-learned initializations —make per-instance fitting practical online, but prior work mainly uses fitted weights for reconstruction and compression.Source: A. Neural Implicit Networks · p. 2

B. Weight Space Learning

Weight-space learning treats fitted network parameters as data, with a shared origin supporting comparison across instances. Robotic manipulation memory systems more commonly retain records, features, or activation states than individual parameter-space episodes.

B. Weight Space Learning

  • Functa treats each fitted INR function as a data point in a learned representation space, where classifiers over modulation vectors generalize to unseen instances; this relies on a common origin [19].Source: B. Weight Space Learning · p. 2
  • A shared initialization lets fitted weights remain comparable across instances; MAML learns one from which a few gradient updates produce task-specific parameters.COIN++ [21] applies shared initialization and fast per-instance fitting to INRs, and Gielisse et al. combine meta-learned SIREN initializations with a transformer over resulting weights.Source: B. Weight Space Learning · p. 2
  • Encoders over fitted INR weights can support retrieval, clustering, and classification for 3D shapes [23], while LLaNA projects fitted NeRF weights into a language model for natural-language scene queries.Source: B. Weight Space Learning · p. 2

C. Memory in Robotic Manipulation

  • Robotic manipulation systems commonly store history as external records, activations, or parameter memories; symbolic and perceptual approaches retain abstractions or features, while recurrent approaches compress history into fixed-size latent states.The pipeline projects each episode’s weight-delta tensors into 56 tokens that condition recall or action heads without frame access.Source: C. Memory in Robotic Manipulation · pp. 2–3, Figure 2 · p. 3

III. METHODS

Each video episode is fitted as an INR from a shared, task-agnostic initialization, and the difference in weights serves as its memory. A learned projector and language-model-based heads use that memory for question answering or next-command regression.

A. Task definition

  • Each episode’s video is fitted by an INR to produce parameters φ, and the episode memory is the delta from shared, task-agnostic initialization θ.Source: A. Task definition · p. 3
  • A learned projector maps the delta into token embeddings, which a language model conditions on alongside the natural-language question for question answering.Source: A. Task definition · p. 3
  • For action decoding, a regression head replaces linguistic decoding and uses the language model’s final hidden state to predict the absolute joint-space command at the observation endpoint.The task instruction supplies q for this output head.Source: A. Task definition · p. 3

B. Dataset generation

RoboMME samples manipulation episodes across four memory suites and creates multiple temporally distinct memories per episode. Each memory is an INR fitted from a shared initialization, with its delta representing the episode-specific change.

B. Dataset generation

  • RoboMME’s four suites target temporal, spatial, object-reference, and procedural memory across 16 ManiSkill tabletop-manipulation task types.The benchmark uses a 7-DOF Franka Panda arm.Source: B. Dataset generation · p. 3
  • Each of 16 task types contributes 100 source episodes; 10 random cut points per episode yield 16,000 fitted INRs per configuration.The source corpus is split 70/15/15, and labels are read at each cut frame.Source: B. Dataset generation · p. 3
  • The label fields range from cut-frame gripper state to the current action, sub-step, episode instruction, and commanded joint vector.Task goals within a task can be near-duplicates differing in one attribute, such as color or repetition count.Source: B. Dataset generation · p. 3, Figure 3 · p. 4
  • Each episode’s INR is fitted from a shared MAML-learned initialization, and its delta isolates the change induced by that episode.Taking the difference from the shared initialization also makes deltas comparable across instances.Source: B. Dataset generation · pp. 3–4
  • Meta-initialized deltas cluster by suite in t-SNE, including for task types absent from meta-training, while random-initialized weights show no visible suite separation.Counting and Reference interleave; Permanence and Imitation form tight, separated clusters.Source: B. Dataset generation · p. 4

C. System architecture

The system encodes each episode as an INR weight delta, then projects that representation into tokens for downstream decoders. The INR reconstructs video from time, using a shared initialization adapted to each episode.

C. System architecture

  • The architecture comprises an INR encoder that produces an episode delta, a projector that maps it to weight-tokens, and two output heads over those tokens.The heads support question answering and action decoding.Source: C. System architecture · p. 4

1) INR Encoder:

  • The INR uses a positional encoding of normalized time, a three-layer MLP stem, upsampling NeRV blocks, and a final convolution to produce RGB frames.The blocks use 3×3 convolutions, PixelShuffle, and GELU activations.Source: 1) INR Encoder · p. 4
  • The INR weight-space figure compares meta-learned and random initialization embeddings across memory suites, using the same eight tasks and 100 samples per task.The t-SNE embeddings use PCA-100 and perplexity 30; axes are arbitrary.Source: Figure 4 · p. 4
  • INR fitting combines L1 reconstruction with structural dissimilarity at λ = 0.7; meta-training learns a per-parameter inner-loop learning rate over K = 3 steps.L1 encourages pixel accuracy, while SSIM preserves local structure that L1 alone can blur.Source: 1) INR Encoder · p. 4

2) Weight-Token Projector:

A tensor-wise projector preserves the layer structure of each INR delta while converting it to a fixed sequence of tokens. Those tokens condition a language model for questions or an action model for joint-command regression.

2) Weight-Token Projector:

  • Each of seven active INR tensors is projected independently, pooled to eight tokens, and concatenated into a 56-token sequence.Per-tensor maps project rows to latent width 256 before a shared LayerNorm and linear map lift tokens to the backbone dimension.Source: 2) Weight-Token Projector · p. 4

3) Output Heads:

  • Question answering uses TinyLlama-1.1B-Chat; action decoding uses Pythia-400M with its vision tower, multimodal projector, and diffusion head discarded.A three-layer MLP regresses an eight-dimensional joint command from the final non-padding hidden state using Huber loss.Source: 3) Output Heads · pp. 4–5

4) Training and Implementation Details:

  • Both heads train in two stages: first updating the projector and placeholder embeddings with the backbone frozen, then jointly fine-tuning projector and backbone at a lower learning rate.A hinge penalty compares each example’s loss with its own weight-tokens against loss with another example’s tokens substituted; action readout receives 10 Stage 2 epochs.Source: 4) Training and Implementation Details · p. 5

IV. EXPERIMENTS

Experiments evaluate weight-token question answering on RoboMME with prior-debiased candidate scoring and controls intended to test reliance on episode information. After Stage 2, accuracy exceeds chance for every question type.

IV. EXPERIMENTS

  • For task goal and subgoal, distractors come from the ground-truth task’s own label vocabulary, requiring distinction among labels within the same task.Candidate scores subtract the no-weight-token score from the score with weight-tokens, so predictions reflect the tokens’ shift in answer preference rather than model priors.Source: §IV · p. 5

A. Weight-Space Question Answering

  • Stage 2 exceeds chance on every question type, with overall accuracy of 47.2% against a pooled 34.6% chance rate.Gains are +16.1 points for choice action, +19.1 for gripper, +9.5 for subgoal, and +3.5 for task goal.Source: A. Weight-Space Question Answering · p. 5, Table I · p. 5
  • Figure 5 illustrates prior-debiased candidate scoring on a correctly answered subgoal example, where the true candidate is separated from distractors by a clear margin.The prediction is the candidate with the minimum score.Source: Figure 5 · p. 5
  • With the language model frozen, Stage 1 clears chance substantially only on gripper, by +12.4 points; the other question types remain at their floors.After unfreezing, fine-tuning multiplies the choice-action gain roughly 8× and doubles the subgoal gain.Source: A. Weight-Space Question Answering · p. 5
  • Task goal is the most demanding label, with a +3.5-point margin and κ = 0.309; its same-task candidates differ by a single attribute such as color or repetition count.Across question types, Cohen’s κ ranges from 0.31 to 0.44; pooled Stage 2 κ is 0.446.Source: A. Weight-Space Question Answering · p. 5

B. Memory-Language-Action Decoder

The same weight-space memory can condition a decoder to predict the robot’s next joint command, not just support retrospective question answering. Evaluation measures offline action decoding at the episode’s final observed frame.

B. Memory-Language-Action Decoder

  • The decoder predicts RoboMME’s seven target joint angles and binary gripper directive at the cut frame, measuring behavior cloning at a single decision point.Because the input episode must end before the memory exists, this is offline decoding rather than closed-loop policy performance.Source: B. Memory-Language-Action Decoder · p. 6
  • Stage 2 reaches a mean R2 of 0.51 across eight action dimensions, with improvement over Stage 1 in every dimension and mean absolute error of roughly 0.15 radians.Figure 6 reports pooled R2 of 0.53 across the seven continuous joints.Source: B. Memory-Language-Action Decoder · p. 6, Figure 6 · p. 6
  • Replacing the weight delta with matched noise drops mean test R2 from +0.48 to −0.39 for Stage 1 and from +0.51 to −0.55 for Stage 2.Every individual dimension falls below zero under noise.Source: B. Memory-Language-Action Decoder · p. 6
  • Within-task mean R2 remains positive across all 16 tasks, ranging from roughly 0.19 on InsertPeg to 0.80 on StopCube.Unmasking tasks decode best, while fine-manipulation and ordering tasks decode worst; the passage suggests coarse scene and pose structure may be represented more faithfully than fine detail.Source: B. Memory-Language-Action Decoder · p. 6

C. Ablation Studies

The ablations compare meta-learned versus random INR initialization and alternative supervisory signals. Meta-initialization is the clearer factor, while auxiliary supervision has more modest and mixed effects.

1) Varying INR Initialization and Supervision:

  • Meta-initialized Flavor 2 exceeds its random-origin counterpart Flavor 1 by +9.2 percentage points overall, with the widest separation on gripper.Random-origin configurations remain at the 50% gripper floor, while meta-initialized configurations reach 66 to 69%.Source: 1) Varying INR Initialization and… · p. 6, Table II · p. 7
  • Flavor 3, which uses DINO supervision without pixel reconstruction, scores 37.0%, below random-origin Flavor 1; Flavor 4 trails Flavor 2 by 2.2 points.Flavor 5 tracks Flavor 1 within a point on every question type, while Flavor 6 trails Flavor 2 overall but has the table’s best subgoal and task-goal scores.Source: 1) Varying INR Initialization and… · p. 7, Table II · p. 7
  • Flavor 2’s per-suite accuracy stays within roughly 44 to 52%, so its pooled result spans temporal, spatial, object, and procedural memory suites.The passage reports this range as evidence that one tractable suite does not carry the pooled result.Source: 1) Varying INR Initialization and… · p. 7

2) Matched Noise Control:

Matched-noise controls distinguish episode-informed weight tokens from performance that persists without episode information. The conclusion presents the deltas as individually addressable episode memories, while noting limits in data, evaluation, and offline construction.

2) Matched Noise Control:

  • For meta-initialized Flavor 2, matched noise lowers overall accuracy from 47.2% to 35.9%, while random-origin Flavor 1 remains essentially unchanged at 38.0% with real deltas and 37.7% with noise.Flavor 2’s choice-action score loses 18.5 points and falls below chance; the passage attributes Flavor 1’s persistent margin to learned question and candidate statistics rather than episode content.Source: 2) Matched Noise Control · p. 7

V. CONCLUSION

  • The conclusion states that meta-learned parameter origins are necessary for projected INR weight-delta tokens to make episode information decodable for both question answering and action decoding.The delta is described as a persistent, individually addressable memory that complements continually overwritten parametric memories such as Titans.Source: §V · p. 7
  • The evaluation uses two training seeds and a limited source-episode corpus, so differences of one to three points between adjacent configurations should not be over-read.The paper also notes that closed-candidate likelihood scoring tests discrimination rather than free-form generation, and that memory construction is offline.Source: §V · p. 7

Related papers

Loading 2610.28882v1…