Source-linked AI summary
OpenTinker: Separating Concerns in Agentic Reinforcement Learning
Siqi Zhu, Jiaxuan You
TL;DR
Agentic SFT and RL need consistent management of mutable LoRA policy states across training, rollout, and environment interaction. OpenTinker provides a managed multi-LoRA runtime with versioned snapshots and a unified masked trajectory interface, and validates the resulting training loops across representative agentic workloads.
Problem
Agentic workloads require separately trained policies while keeping mutable LoRA state, optimizer state, rollout versions, and training-data attribution consistent.
Method
OpenTinker treats LoRA adapters as managed policy objects and unifies SFT and RL through masked token trajectories with explicit snapshot synchronization.
Results
Representative validation tasks show non-degenerate learning behavior and validate SFT/RL-style training loops over agent trajectories with coordinated policy state and rollout sampling.
Takeaways & Limitations
The framework supports shared-base multi-LoRA training while keeping adapter updates, checkpoints, rollout snapshots, and token-level learning signals isolated and reusable across modes.
Takeaways & Limitations
The fused mixed-adapter rollout kernel is a backend implementation strategy, not a requirement of the current service-backed execution path.
Abstract
from arXiv · showhide
We introduce \textsc{OpenTinker}, an open infrastructure for training large language model (LLM) agents with many LoRA-backed policies over shared execution resources. Modern agent workloads mix supervised fine-tuning (SFT), online reinforcement learning (RL), rollout generation, validation, and multi-turn environment interaction. In such workloads, LoRA adapters are not static inference artifacts: they are frequently updated policy states whose optimizer state, rollout snapshot, and training data attribution must remain consistent. \textsc{OpenTinker} centers the runtime around this policy lifecycle. Users define environments, agents, and learning objectives, while the system manages training clients, rollout samplers, checkpoint handles, and policy-version refresh. The same data path supports SFT and RL by converting trajectories into token sequences with explicit masks: context and environment observations condition the model, while generated action tokens carry supervised weights or RL advantages. This design enables multi-LoRA SFT/RL training in which many users, tasks, or agents can share a base model while keeping adapter updates, checkpoints, and rollout snapshots isolated. We describe the system architecture, the adapter lifecycle, the service-backed snapshot handoff used by the current implementation, the backend contract for mixed-adapter rollout kernels, and the training scheduler that isolates adapter-local gradients and optimizer state. Representative validation tasks exercise single-turn, multi-turn, LoRA, and multi-agent agentic training.
1 Introduction
OpenTinker addresses the difficulty of training many mutable LoRA-backed policies for agentic SFT and RL. It manages policy-version consistency while unifying trajectories, token masks, and execution across training modes.
- LoRA adapters become mutable policy states whose optimizer state, checkpoint history, rollout versions, and training-data attribution must remain consistent.
- OpenTinker treats adapters as first-class policies and synchronizes mutable training state with explicit rollout snapshots.Rollout data is associated with the policy version that generated it, while updates and sampler refresh occur at explicit synchronization points.
- The same token-sequence interface supports SFT weights and RL advantages while excluding context, observations, and tool outputs from the learning loss.Generated response or action spans receive the learning signals; surrounding trajectory content remains conditioning context.
- OpenTinker provides a service-oriented architecture for shared resources, multi-LoRA lifecycle management, snapshot handoff, mixed-adapter rollout kernels, and adapter-local scheduling.The framework preserves adapter-local gradients, optimizer state, and snapshot publication across SFT and RL workloads.
- Validation covers single-turn math, LoRA-backed math training, multi-turn games, vision-language interaction, and two-agent gameplay.
2 Related Work
OpenTinker builds on distributed RLHF, agent-training, and multi-adapter serving systems but centers the lifecycle of actively trained LoRA policies. Its architecture coordinates shared resources, user-facing agent runtimes, and policy-versioned execution.
- Distributed RLHF and SFT Frameworks: OpenTinker complements distributed RLHF systems by focusing on multi-LoRA policy lifecycle management rather than primarily training and generation scale.The lifecycle includes many adapters trained through SFT or RL over shared resources.
- The control plane dispatches training, rollout, and validation workers that share base-model resources and a LoRA adapter pool.Workers exchange versioned policy snapshots and use the same trajectory path for SFT weights or RL advantages.
- Agent Training Systems: The architecture separates user-side agent and environment logic from managed execution while coordinating reusable environments and explicit policy-version semantics.
- LoRA and PEFT Serving: Unlike mostly static adapter-serving systems focused on throughput and memory, OpenTinker coordinates optimizer updates and SFT/RL state for actively trained adapters.
3 Approach
OpenTinker manages agent training around first-class policy objects, separating user task logic from shared execution while coordinating adapter state and rollout versions. It unifies SFT and RL through masked token trajectories and supports multi-turn, multi-policy serving with pinned adapter versions.
- System architecture: The client defines environments, agents, prompts, rewards, and objectives, while scheduler and task-server components allocate resources and dispatch training, rollout, and validation work.Workers share a frozen base model and LoRA adapters, while task-owned execution state is coordinated centrally.
- Policy lifecycle: Policies are managed runtime objects with mutable adapter state, optimizer state, checkpoint history, and immutable rollout snapshots.Logical policy identity is separated from snapshot versions so rollout workers can sample explicit versions while training continues.
- Policy lifecycle: Training updates mutable adapters through SFT or RL, then publishes checkpoint handles that rollout workers use to refresh samplers at explicit synchronization points.The service-backed implementation exports sampler weights and creates or refreshes sampling clients from opaque snapshot paths.
- Unified data path: SFT and RL share tokenized trajectories with shifted next-token targets, while masks restrict learning to demonstrated or generated action spans.Context, dialogue history, observations, and tool outputs condition the model but receive zero training weight; RL additionally aligns old log probabilities and advantages.
- Unified data path: Multi-turn workflows append observations and tool outputs as context while training only model-generated action tokens across later turns.The same masking rule applies during inference without loss computation or optimizer updates.
- Multi-LoRA rollout: Multi-LoRA serving batches active token rows by pinned adapter version, reusing the shared base projection before computing and scattering adapter-specific residuals.This supports mixed prefill and decode workloads while preserving the version contract for every adapted linear layer.
4 Experiment
OpenTinker’s experiments validate a unified runtime for SFT/RL agent training across varied policy, trajectory, and interaction settings. The results indicate that reward attribution, policy updates, rollout refresh, and token-level learning signals remain coordinated across these scenarios.
- Supported Training Scenarios: The representative scenarios span full and LoRA-backed policies, supervised and RL trajectory use, and single-turn, multi-turn, language-only, vision-language, and multi-agent settings.These settings are summarized as the evaluation scope for the current implementation.
- Functional Validation: OpenTinker validates reward attribution to generated action tokens, non-degenerate policy improvement, and rollout refresh from published LoRA snapshots without changing environment code.These are the three runtime properties stated as evaluation targets.
- Functional Validation: Validation metrics show non-degenerate learning behavior across single-turn math, vision-language geometry, multi-turn gomoku, and tool-augmented geometry tasks.The reported behavior is attributed to consistent connections among the environment loop, token masking, reward assignment, and optimizer updates.
- LoRA SFT/RL Interface: SFT and RL share shifted token sequences, action masks, the LoRA policy object, and checkpoint/refresh mechanisms, with supervised weights replacing RL advantages for SFT.The shared interface supports supervised warm-up followed by RL fine-tuning.
- Multi-Turn and Multi-Agent Execution: Multi-turn and two-agent tasks restrict learning to generated actions, assign rewards to the acting agent, and produce expected competitive dynamics in zero-sum gomoku.Environment observations remain conditioning context rather than objective-bearing tokens.
- Functional Validation: Together, the experiments validate SFT/RL-style training over agent trajectories while coordinating policy state, rollout sampling, and token-level learning signals.This is the paper’s stated main runtime contract.
5 Conclusion
OpenTinker treats LoRA adapters as managed policy states and separates user environments from execution infrastructure. Its unified trajectory interface supports SFT and online RL through versioned snapshots, while the current service-backed implementation provides explicit snapshot handles and sampler refresh.
- Conclusion: OpenTinker’s central abstraction is a LoRA adapter managed as mutable policy state rather than a static serving artifact.The conclusion connects this abstraction to a unified path across supervised warm-up, online RL, validation, and multi-turn interaction.
- Conclusion: The system separates user-side environment logic from managed execution and reuses token-level masks to represent SFT weights and RL advantages.This allows environments and policy objects to be reused across training modes.
- Conclusion: The current service-backed path coordinates training clients, sampler-compatible snapshot handles, and explicit sampler refresh for policy-version consistency.The interface also specifies batching for mixed-adapter serving with shared base-model computation and version-pinned low-rank residuals.