Test-time Offline Reinforcement Learning on Goal-related Experience

1 Introduction

• GC-TTT specializes a pre-trained goal-conditioned policy to each test goal using only the pre-training dataset and pre-trained agent.The policy is fine-tuned during evaluation rather than remaining frozen.Source: §1 · p. 1
• The framework reuses pre-training data for test-time adaptation, while privileged or auxiliary data is also possible.The proposed setup does not require information beyond the pre-training dataset and pre-trained agent.Source: §1 · p. 1
• GC-TTT produces significant performance gains on standard benchmarks when combined with GC-BC, GC-IQL, and SAW.Figure 1 presents the benchmark comparison across these offline RL backbones.Source: Figure 1 · p. 2, §1 · p. 2
• GC-TTT selects experience relevant to the current state and optimal for the evaluation goal, then updates the actor for a few gradient steps.The procedure repeats periodically in a receding-horizon fashion to adapt to the current trajectory.Source: §1 · p. 2
• The method works with expert and play-like data, includes a critic-free variant that retains good expert-data performance, and is agnostic to the backbone RL algorithm.The study also varies test-time training frequency and compares test-time training with increased model size.Source: §1 · p. 2
• GC-TTT significantly outperforms existing algorithms when inference FLOPs are matched by scaling baseline network sizes.This evaluates compute allocation at inference rather than only comparing unscaled models.Source: §1 · p. 2

Goal-conditioned reinforcement learning

• Goal-conditioned RL learns policies for specified goals using sparse rewards, relabeling, self-supervision, and specialized value-based, contrastive, or quasimetric methods.These methods support learning across many goals from arbitrary experience and can be adapted to offline settings.Source: §2 · p. 2
• Existing goal-conditioned policies generally keep their parameters fixed throughout evaluation, whereas this work trains policy weights at test-time.GC-TTT can be combined with the value-based algorithms discussed in prior work.Source: §2 · p. 2

Test-time training

• Test-time training specializes models to individual prediction tasks through a few gradient steps, with applications spanning control, language, reasoning, and video generation.Prior methods often select data from the pre-training dataset without adding privileged information.Source: §2 · p. 3

Test-time reinforcement learning

• Test-time offline RL differs from prior test-time training because trajectories come from different policies pursuing different goals and therefore should not be imitated directly.GC-TTT reuses the offline dataset for policy improvement while avoiding the need for known environment dynamics.Source: §2 · p. 3
• Unlike subgoal-search methods that remain limited by the best pre-trained goal-conditioned policy, GC-TTT dynamically trains policy parameters during evaluation of a single goal.The approach combines goal-conditioned pre-training with continuously training on experience for one task.Source: §2 · p. 3

• Goal-conditioned RL models families of behaviors by conditioning a policy on a specified goal, commonly represented with sparse rewards.Source: §3 · p. 3
• The standard pipeline pretrains a policy on offline trajectories using a state-goal distribution and then evaluates it on a single goal per episode.Source: §3 · pp. 3–4
• Average optimization over broad goal distributions can produce locally suboptimal policies for specific goals, especially with noisy data or limited model capacity.Source: §3 · p. 4
• Test-time offline reinforcement learning fine-tunes the pretrained policy during evaluation to specialize it to the agent’s current state and goal.Source: §3 · p. 4

4.1 What to train on? Selecting Relevant and Optimal Experience

• GC-TTT selects sub-trajectories near the current state and optimal for the evaluation goal, then fine-tunes the policy on them for a few gradient steps.Relevance uses a temporal-distance criterion, while optimality uses rewards along the sub-trajectory and an estimated final-state value.Source: §4 · p. 4, §4.1 · pp. 4–5
• The distance threshold can be fixed or adapted to control dataset size, and only local accuracy of the distance function is required.Source: §4.1 · p. 4
• When a critic is unavailable, GC-TTT can rank relevant experience using behavioral-policy trajectory returns instead.Source: §4.1 · p. 5

4.2 When to train? Receding Horizon Training

• Every K steps, GC-TTT resets to pretrained weights, fine-tunes on data selected for the current state-goal pair, rolls out for K steps, and repeats.This receding-horizon procedure supports local task focus and dynamic trajectory correction without requiring a model.Source: §4.2 · p. 5

4.3 Computational Efficiency

• Test-time cost scales linearly with training frequency and gradient steps, with additional data-selection overhead and periodically concentrated computation.Source: §4.3 · p. 5
• GC-TTT completes a 1000-step evaluation episode in approximately 70 seconds with an average control frequency above 10 Hz under a generous compute budget.A critic-free, less frequent variant exceeds 75 Hz, while the pretrained policy reaches approximately 190 Hz.Source: §4.3 · pp. 5–6

Overall evaluation

• GC-TTT improves backbone policies across four environments and three algorithms, including increasingly complex navigation tasks and cube relocation.The evaluation averages success across fixed goals and reports gains for most backbone-environment combinations without harming the remaining ones.Source: §5 · p. 6

Critic-free variant

• The critic-free variant retains much of GC-TTT’s effectiveness on expert data but is limited because non-goal-reaching play trajectories share the same return.It replaces critic-based H-step returns with discounted trajectory returns and therefore cannot infer optimality from unsuccessful trajectories.Source: §5 · pp. 6–7

Data-selection ablation

• Selecting both state-relevant and goal-optimal data produces the substantial gain, whereas random or singly filtered data provides little improvement.The ablation uses GC-IQL in pointmaze and antmaze play environments.Source: §5 · p. 7, Figure 5 · p. 8

Test-time compute

• More frequent TTT improves performance in complex antmaze play, while pointmaze performance saturates at frequency 1/200.Long-horizon value estimates become inaccurate beyond 200 steps in antmaze play, making infrequent updates harmful.Source: §5 · pp. 7–8, Figure 5 · p. 8
• At matched inference FLOPs, increasing GC-TTT frequency consistently outperforms scaling policy model size.The comparison assumes compute scales linearly with TTT frequency and quadratically with policy width.Source: §5 · p. 8, Figure 5 · p. 8

Conclusion

• GC-TTT selects relevant and optimal trajectory data and periodically fine-tunes the pretrained policy during evaluation.The method adapts to the agent’s current state and goal through a self-supervised selection scheme.Source: §6 · p. 8

Limitations

• GC-TTT’s main practical limitation is compute, while gradient updates may also require constraints for safety-critical systems.High-frequency control may require lower update frequency or a lazy variant, and the benchmark did not observe instability.Source: §6 · p. 8
• GC-TTT may provide limited gains when value estimates are poor, related data is scarce, or the state-goal space is immense.The paper gives complex multi-object manipulation as an example of a setting where these conditions may arise.Source: §6 · p. 8

Future work

• The framework does not use freshly collected test-time data beyond the current state, leaving this as a direction for future research.The authors also suggest investigating why offline GCRL algorithms do not accurately fit every trained task.Source: §6 · pp. 8–9

Impact Statement

• The work’s potential impacts are described as similar to general deep reinforcement learning concerns, including automation and hardware misuse.Source: Impact Statement · p. 9

A Taxonomy of Test-time Training

• Test-time training updates model parameters at test time for each task, and the paper organizes existing approaches into a taxonomy.Source: A Taxonomy of Test-time Training · p. 12, Table 2 · p. 12

B.1 Behavior Cloning

• Behavior cloning learns a policy by supervised reconstruction of dataset actions and converges to the behavioral policy when that policy lies in the policy class.Source: §B.1 · p. 12

B.2 Implicit Q-Learning

• Implicit Q-learning avoids critic queries on out-of-distribution actions by estimating values through expectile regression, then extracts policies with AWR or a BC-regularized objective.Source: §B.2 · p. 12

B.3 SAW

• SAW combines implicit Q-learning and AWR with an additional alignment term between low-level policies for close and distant goals.Source: §B.3 · pp. 12–13

C.1 Ablation on Test-time Training Parameters

• Increasing test-time gradient steps initially helps local fitting, but excessive updates can reduce performance through small-data training and value overestimation.Higher learning rates adapt faster and show a slight peak-performance advantage, while both learning rates become comparable as steps increase.Source: §C.1 · p. 13

C.2 Ablation on Data Selection Parameters

• Extreme relevance thresholds degrade performance by either making selected data scarce or admitting irrelevant sub-trajectories.The environment-defined distance criterion sets ϵ directly; in antmaze, ϵ is 1.Source: §C.2 · p. 13
• Dynamic horizon selection matches hand-selected fixed horizons, supporting the default of adapting H to each sub-trajectory.Both Monte Carlo and H-step returns provide reasonable optimality estimates under their respective policy assumptions.Source: §C.2 · p. 14
• GC-TTT is robust to the optimality quantile q, although extremely low or high values can impose overly strict or lax selection criteria.Unlike ϵ and H, q has no fundamental connection to another quantity and is treated as a fixed hyperparameter.Source: §C.2 · p. 14

C.3 Value-based Relevance Criterion

• A value-based relevance criterion does not significantly change GC-TTT performance in antmaze compared with the original relevance criterion.The replacement uses a value threshold C to control the maximum temporal distance between the current state and selected trajectories.Source: §C.3 · p. 14

C.5 Results for Fixed Hyperparameters

• Fixed test-time hyperparameters cause only a moderate performance drop across environments, with no significant further average drop when also fixed across backbones.These evaluations correspond to the Table 1 setting and use the fixed-hyperparameter comparisons in Tables 4 and 5.Source: §C.5 · p. 14, Table 4 · p. 15, Table 5 · p. 15

C.6 Results for GC-IQL-AWR

• The critic-free variant is evaluated alongside GC-TTT across loco-navigation and manipulation backbones under fixed environment hyperparameters.The supplied table caption identifies GC-BC, GC-IQL-DDPG, and SAW as the evaluated backbones.Source: Table 4 · p. 15

C.7 Results for QRL

• GC-TTT consistently improves QRL performance on play-like datasets, supporting its algorithmic-agnostic behavior.The evaluation uses fixed settings of α = 3e−4, N = 200, K = 500, and a 50% uniform-sample mixture; QRL already performs strongly on expert data.Source: §C.7 · p. 16, Table 6 · p. 16

C.8 Robustness to Noisy Value Estimates

• GC-TTT’s value-based data selection remains effective under modest value-estimation noise, with degradation appearing only under strong perturbations such as α = 1.The authors argue that ranking sub-trajectories requires less value accuracy than policy extraction.Source: §C.8 · p. 16, Table 7 · p. 16

C.9 Statistical Tests

• Welch’s t-tests confirm significant improvements in 14/21 environment/backbone combinations, while many non-significant cases use the already strong SAW backbone.Table 1 defines significance through non-overlapping standard-error intervals, and Table 8 reports the tests with p < 0.1.Source: §C.9 · p. 16

D Visualizing GC-TTT

• Two GC-TTT iterations solve pointmaze: selected obstacle-avoiding sub-trajectories change an initially wrong action direction, while the actor loss decreases during adaptation.After 200 gradient steps, the second test-time update completes the remaining task segment.Source: D Visualizing GC-TTT · p. 16, Figure 9 · p. 18

E Discussion with Respect to Graph-based Test-time Planning

• GC-TTT performs local parameter-space improvements with gradient descent, whereas TTGS solves a global problem through explicit planning.The two approaches are presented as orthogonal and potentially combinable.Source: E Discussion with Respect to… · p. 18
• TTGS can match GC-TTT on maze domains but scales less favorably because graph construction requires quadratic pairwise-distance computation, unlike GC-TTT’s linear buffer dependence.GC-TTT instead incurs gradient-update overhead and is described as nearly negligible with respect to buffer size.Source: E Discussion with Respect to… · p. 18

F.1 Hyperparameters

• GC-TTT adds q = 0.05, horizon K, gradient-step count N, and fine-tuning learning rate α to the base implementation’s default hyperparameters.The percentile q is fixed, while K, N, and α are tuned.Source: F Implementation Details · p. 18, §F.1 · p. 18, Table 9 · p. 19

F.2 Estimating FLOPs

• Under the stated MLP assumptions, a 1000-step episode costs C_no-TTT ≈ 4000w^2 without test-time training.The estimate assumes two hidden layers, negligible input/output size, batch-independent forward cost, and backward cost twice that of a forward pass.Source: §F.2 · pp. 18–19
• With width w = 512, GC-TTT costs approximately 1.6 · 10^9, 2.2 · 10^9, or 4 · 10^9 FLOPs at frequencies 1/1000, 1/500, or 1/200.The corresponding no-TTT widths for matched budgets are approximately 624, 732, and 992.Source: §F.2 · pp. 18–19

F.3 Wall-clock Overheads

• On an RTX4090, initial relevance computation takes under 15 seconds, while each later selection takes under 1 second and each optimization step under 30 milliseconds.Sampling each fine-tuning batch takes under 10 milliseconds; the critic-free variant skips the initial critic forward pass.Source: §F.3 · p. 19
• For L = 1000, N = 100, and K = 100, total GC-TTT overhead is approximately 65 seconds, averaging 65 milliseconds per environment step.The overhead is nonuniform, with much of it occurring before the first environment step and during test-time training.Source: §F.3 · p. 19