Source-linked AI summary
MinT: Managed Infrastructure for Training and Serving Millions of LLMs
Mind Lab, :, Song Cao, Vic Cao, Andrew Chen, Kaijie Chen, Cleon Cheng, Steven Chiang, Kaixuan Fan, Hera Feng, Huan Feng, Arthur Fu, Jun Gao, Hongquan Gu, Aaron Guan, Nolan Ho, Mutian Hong, Hailee Hou, Peixuan Hua, Charles Huang, Miles Jiang, Nora Jiang, Yuyi Jiang, Qiuyu Jin, Fancy Kong, Andrew Lei, Kyrie Lei, Alexy Li, Lucian Li, Ray Li, Theo Li, Zhihui Li, Jiayi Lin, Kairus Liu, Kieran Liu, Logan Liu, Xiang Liu, Irvine Lu, Maeve Luo, Runze Lv, Pony Ma, Verity Niu, Anson Qiu, Vincent Wang, Rio Yang, Maxwell Yao, Carrie Ye, Regis Ye, Wenlin Ye, Josh Ying, Danney Zeng, Yuhan Zhan, Anya Zhang, Di Zhang, Ruijia Zhang, Sueky Zhang, Ya Zhang, Wei Zhao, Ada Zhou, Changhai Zhou, Yuhua Zhou, Xinyue Zhu, Murphy Zhuang
TL;DR
MinT addresses the infrastructure challenge of managing many evolving policies over a small number of expensive base-model deployments. It keeps bases resident and moves exported LoRA revisions through the post-training and serving lifecycle, achieving frontier-scale operation, faster adapter-only handoffs, and million-scale addressability. The system’s sparse-attention support remains limited because it does not reconstruct every DSA indexer selection.
Problem
Full-checkpoint workflows do not scale when many continuously changing policy variants share a few expensive frontier base-model deployments.
Method
MinT manages LoRA adapter revisions and durable policy records through rollout, update, export, evaluation, serving, and rollback over resident base-model deployments.
Results
MinT validates the adapter-revision path beyond 1T total parameters, reduces measured handoff steps by 18.3× and 2.85×, and builds an artifact-backed 1M adapter catalog.
Takeaways & Limitations
Multi-tenant LoRA training and serving can grow policy counts through revisions, policy records, cache tiers, and controlled readiness while selected revisions use bounded resident working sets.
Takeaways & Limitations
MinT does not reconstruct every DSA indexer selection and instead applies rollout correction when training and rollout probabilities leave a trusted band.
Abstract
from arXiv · showhide
We present MindLab Toolkit (MinT), a managed infrastructure system for Low-Rank Adaptation (LoRA) post-training and online serving. MinT targets a setting where many trained policies are produced over a small number of expensive base-model deployments. Instead of materializing each policy as a merged full checkpoint, MinT keeps the base model resident and moves exported LoRA adapter revisions through rollout, update, export, evaluation, serving, and rollback, hiding distributed training, serving, scheduling, and data movement behind a service interface. MinT scales this path along three axes. Scale Up extends LoRA RL to frontier-scale dense and MoE architectures, including MLA and DSA attention paths, with training and serving validated beyond 1T total parameters. Scale Down moves only the exported LoRA adapter, which can be under 1% of base-model size in rank-1 settings; adapter-only handoff reduces the measured step by 18.3x on a 4B dense model and 2.85x on a 30B MoE, while concurrent multi-policy GRPO shortens wall time by 1.77x and 1.45x without raising peak memory. Scale Out separates durable policy addressability from CPU/GPU working sets: a tensor-parallel deployment supports 10^6-scale addressable catalogs (measured single-engine sweeps through 100K) and thousand-adapter active waves at cluster scale, with cold loading treated as scheduled service work and packed MoE LoRA tensors improving live engine loading by 8.5-8.7x. MinT thus manages million-scale LoRA policy catalogs while training and serving selected adapter revisions over shared 1T-class base models.
1 Introduction
MinT treats LoRA adapters as managed policy units over resident base-model deployments, replacing repeated full-checkpoint movement with an adapter-only lifecycle. It scales this design across frontier model size, handoff efficiency, and large policy populations.
- Motivation: Full-checkpoint workflows no longer scale for continuously changing populations of policy variants over expensive base-model deployments.Post-training combines scheduling, versioning, deployment, and data movement across many variants.
- Core design: MinT keeps the base model resident and transfers compact, serving-compatible LoRA adapter revisions instead of full model checkpoints.Adapters represent task behaviors, product branches, experimental versions, tenant variants, and rollback points.
- Scale Up: Scale Up validates LoRA RL with distributed training and serving paths on frontier-scale dense and MoE architectures beyond 1T total parameters.The path includes model-parallel training, sparse-route consistency, and distributed adapter export.
- Scale Down: 18.3× and 2.85× reductions in the measured handoff step occur on 4B dense and 30B MoE models, respectively, with concurrent GRPO wall time also shortened without higher peak memory.Concurrent multi-policy GRPO shortens wall time by 1.77× and 1.45× on the same respective models.
- Scale Out: Scale Out separates durable policy addressability from CPU/GPU working sets while supporting a 1M packed adapter catalog and scheduled cold loading.The system materializes 1,000,000 revisions with zero build errors and audits selected revisions across shards.
- Managed lifecycle: MinT manages adapter revisions through export, evaluation, serving, rollback, and reproducible lifecycle state behind a service interface.Its reproducibility paths include a Tinker-compatible API and cookbook recipes for several post-training workflows.
2 From LoRA Adapters to Managed Policy Revisions
MinT separates executable adapter revisions from durable policy records and coordinates their movement across training, rollout, evaluation, serving, and recovery. This separation supports shared resident bases while bounding active working sets, with explicit limits for sparse-attention replay.
- Managed policy revisions: An adapter revision is a fixed serving-layout LoRA snapshot, while the policy record stores durable lifecycle state for reproducibility, reload, and rollback.The revision is frozen at a specific training step and tied to a compatible resident base.
- Adapter lifecycle: After an RL update, MinT exports a serving-compatible adapter revision because samplers consume fixed PEFT files rather than trainer checkpoints.Optimizer moments, scheduler position, gradients, and rollout records remain on the training side.
- Service state: Policy records separately identify compatible bases, resumable checkpoints, selectable revisions, and current adapter-byte locations across GPU, CPU, and shared storage.These facts change at different times and therefore remain distinct.
- Serving: Serving resolves a policy name to a revision and loads adapter bytes from GPU, CPU cache, or shared storage before decoding.This allows a large addressable catalog while bounding CPU-cache and same-batch working sets.
- Training restoration: When switching training policies, MinT restores adapter and optimizer state over a resident base, while policy records preserve adapter-specific shapes.Policies may use different LoRA ranks or target-module sets.
- Sparse-model caveat: For sparse models, unmappable MoE expert paths are omitted from replayed policy-gradient terms, while DSA indexer selections are not fully reconstructed.IcePop-style correction zeroes importance weights outside a trusted band rather than recovering the exact sparse-attention token set.
3 System Design
MinT presents LoRA RL and serving as a managed service over resident base-model deployments. Its service plane schedules durable, pollable operations, while workers time-slice policy state and export compatible adapter revisions for inference.
- Service plane: MinT exposes rollout, gradient, optimizer, export, evaluation, serving, and polling operations through a managed service over resident dense and MoE bases.Policy records and operation ids preserve retryable links among rollouts, checkpoints, exported revisions, and workers.
- Service plane: The service queues validated operations, admits them to compatible workers, and makes outputs visible only after recording completed files and pollable results.Uncommitted files remain unselectable after worker failure, allowing retries and cleanup of unreferenced attempts.
- Policy state: MinT resolves each training or sampling request through a policy record that specifies base compatibility, adapter shape, state locations, and exported revisions.Training restores mutable tensors, optimizer state, gradients, scheduler position, and rollout metadata; sampling selects a fixed exported revision.
- Time-sliced training: Each trainer time-slices one policy at a time: it writes the previous policy’s adapter and optimizer state out, then restores the next policy before updating.The resident base remains in GPU memory while inactive policy state can reside in CPU memory or storage.
- Distributed training and export: Single-worker PEFT and distributed Megatron paths restore sharded LoRA state, update it over model-parallel bases, and checkpoint the result.Distributed export gathers tensor-parallel slices, collects expert tensors, deduplicates replicated data, and emits vLLM-compatible PEFT layout.
- Adapter serving: Exported adapters are admitted to serving only when their base family, rank, target modules, tensor layout, and configured buffers are compatible.Evaluation and serving use the fixed revision from a particular export even if the policy continues training.
4 Three Scaling Axes
MinT scales shared-base LoRA post-training and serving by extending the same adapter-revision path across large model placement, compact handoff, and large policy populations. The system separates durable catalog addressability from active working sets and makes adapter loading an explicit serving stage.
- Scale Up: Scale Up supports LoRA RL and serving on model-parallel dense and MoE bases, including distributed placement and sparse-route consistency requirements.Megatron shards dense-module and per-expert LoRA tensors according to tensor- and expert-parallel ownership.
- Scale Up: MinT records MoE expert routes with rollout data and masks tokens when routes are unavailable or cannot map to the training layout.This preserves route alignment where supported but can reduce usable training tokens when provenance is missing.
- Scale Up: DSA support fixes several implementation mismatches, but probability mismatch can remain and the system does not prove that training used the inference engine’s exact sparse-attention token set.IcePop-style correction assigns zero importance weight outside a configured trusted probability-ratio band.
- Scale Down: Adapter-only handoff preserves a serving-compatible exported revision instead of materializing or transferring a merged full checkpoint.The export path removes optimizer state and rank-local training files while retaining serving tensor layout and compatibility metadata.
- Scale Out: MinT separates user-facing policy names, durable adapter files, actor-local caches, and same-batch GPU slots so catalog scale does not equal active working-set scale.A revision may be addressable, CPU-cached, GPU-active, evicted, or reloaded at different times.
- Scale Out: 1,000,000 packed adapter revisions are materialized and audited, while serving experiments select bounded cached working sets from that catalog.Cold misses fetch, materialize, register, and activate adapters; routing, backpressure, readiness states, and prewarming make this scheduled service work.
- Scale Out: 8.5–8.7× live-load speedups result from packing MoE LoRA tensors from 37,248 objects to 672 before engine registration.Admission-aware rollout prevents adapter-loading time from appearing in first-request latency after readiness, while cold first-touch can otherwise raise warm-tenant TTFT p95 above 20 seconds.
5 Evaluation
The evaluation exercises MinT’s shared adapter lifecycle across training paradigms, model scales, and policy-population serving. Results support scale-up beyond 1T parameters, faster adapter-only handoff and concurrent training, large catalogs with bounded working sets, and controlled cold activation.
- Scale Up Across Training Paradigms and Model Scales: MinT applies the same rollout-update-export-evaluate lifecycle across SFT, DPO, GRPO, dense models, sparse MoE models, and a 1.04T-parameter base.The evaluation includes Qwen3-235B-A22B, GLM-5.1, and Kimi K2 paths.
- Scale Down: 1.77× and 1.45× concurrent-training speedups shorten three-policy GRPO wall time on Qwen3-4B and Qwen3-30B without increasing peak memory.Both concurrent configurations use the same peak memory as their sequential baselines.
- Scale Up Across Training Paradigms and Model Scales: The same adapter lifecycle carries supervised, preference-based, and rollout-based updates without per-paradigm checkpoint surgery.The endpoint evidence includes held-out SFT gains and exported GRPO sampler evaluations.
- AutoResearch: AutoResearch uses proxy screening followed by full LawBench confirmation, rejecting a high-proxy v11 candidate and retaining v23 after both stages.v11 scores 0.4858 versus v10’s 0.4889 on full LawBench; v23 reaches 0.5079.
- Policy-Population Serving: A 1,000,000-revision catalog is separated from bounded engine-local caches, while warm serving reaches 100% TTFT≤5s through 2 rps for 64 adapters.The serving measurements distinguish durable addressability from CPU-cache residency, batch diversity, and cold-load capacity.
- Policy-Population Serving: 8.5–8.7× live engine-load improvements follow MoE tensor packing, while admission and two-phase readiness protect warm tenants from cold activation interference.Ready-path requests avoid load cost only after prewarming; cold activation remains scheduled work.
6 Related Work
Related work spans service interfaces, RL execution systems, training-serving consistency, parameter-efficient tuning, multi-LoRA serving, and large-model infrastructure. MinT connects these areas by making exported adapter revisions part of the managed training-to-serving lifecycle.
- Service interfaces: Remote post-training interfaces make training loops programmable, while Tinker-style systems expose reusable service primitives and benchmark coverage.The cited systems include Tinker, SkyRL tx, and OpenTinker.
- RL execution systems: RL execution systems study rollout scheduling, colocated or disaggregated execution, asynchronous optimization, utilization, failure isolation, and end-task quality.These systems focus on execution organization rather than MinT’s adapter-revision lifecycle.
- Training-serving consistency: Training-serving consistency work addresses probability gaps, numerical mismatch, and MoE router disagreement between rollout and training paths.Examples include truncated importance sampling and unified precision flow.
- Parameter-efficient tuning and multi-LoRA training: LoRA, AdaLoRA, and QLoRA provide parameter-efficient tuning through frozen bases, adaptive rank allocation, or quantized bases with LoRA updates.These methods motivate compact policy updates but do not by themselves define MinT’s managed service lifecycle.
- Multi-LoRA serving: Multi-LoRA serving systems optimize batching, memory, scheduling, kernels, compression, rank heterogeneity, placement, and routing after adapters exist.MinT extends this serving focus to connect catalogs and cache misses to trained policy revisions.
- MinT’s positioning: MinT’s distinction is that an addressable adapter is an exported revision of a trained policy, and cache misses reload that revision from its exported file.This connects serving-catalog mechanisms to the training lifecycle.
- Large-model infrastructure: Large-model infrastructure supplies distributed execution, model parallelism, memory partitioning, and MoE-parallel techniques for practical large-model training.The cited infrastructure includes Ray, Megatron-LM, ZeRO, and MoE Parallel Folding.
7 Conclusion
MinT addresses the scalability limits of full-checkpoint workflows by managing exported LoRA revisions over resident base models. Measurements validate this design across scale-up, scale-down, and scale-out, including million-scale catalogs and controlled activation.
- Problem: Full-checkpoint workflows make each policy variant resemble a complete deployment, which does not scale for continuously changing populations over shared base models.The workload couples rollout, update, evaluation, serving, scheduling, and data movement.
- Managed unit: MinT makes exported LoRA adapter revisions the managed unit while policy records preserve reproducibility, scheduling, and durability across lifecycle stages.Base-model deployments remain resident as revisions move through rollout, update, export, evaluation, serving, and rollback.
- Three scaling axes: 18.3× and 2.85× handoff-step reductions, plus 1.77× and 1.45× concurrent-GRPO speedups, validate scale-down efficiency without higher peak memory.These results compare 4B dense and 30B MoE settings under resident-base allocation.
- Three scaling axes: Scale-out validation builds 1,000,000 packed revisions with zero errors and serves selected revisions through bounded local working sets.Warm serving maintains 100% TTFT≤5s through 2 rps for the measured 64-adapter workload.
- Serving boundary: 8.5–8.7× packed live-load improvements and admission-aware readiness show that cold activation interference, rather than catalog registration, is the dominant online risk.Prewarming shifts loading before readiness so ready-path requests avoid load cost while incurring explicit prewarm delay.
- Conclusion: The results support multi-tenant LoRA services that grow policy populations through revisions, records, cache tiers, and controlled readiness over bounded resident working sets.The conclusion frames LoRA as a service-level unit for large-scale post-training.
A Author List
The author list identifies the paper’s core contributors and broader team. Names are presented alphabetically.
- Organization: Names are listed alphabetically.
- Core Contributors: The paper lists Andrew Chen, Cleon Cheng, Steven Chiang, Nolan Ho, Andrew Lei, Lucian Li, Kieran Liu, Irvine Lu, Pony Ma, Rio Yang, Di Zhang, and Adrian Zhou as core contributors.
- Team: The broader team list includes contributors from Song Cao through Anya Zhang, Ruijia Zhang, Sueky Zhang, Ya Zhang, Wei Zhao, Ada Zhou, Sizer Zhou, Xinyue Zhu, and Murphy Zhuang.
B Additional Serving Measurements
The serving measurements separate catalog addressability from cache state, cold-loading behavior, and active-wave capacity. They identify cold activation and adapter representation as key control points while documenting compatibility and scheduling boundaries.
- Catalog addressability: 1,000,000 adapters were built with zero errors across 100 shards and audited by sampling 256 adapters across all shards.The artifact-backed claim concerns a built and readable packed catalog, not merely a namespace-sized model.
- Catalog addressability: 64 warmed adapters from the 1M packed catalog were used for an open-loop Poisson capacity sweep, measuring selected-revision capacity rather than universal residency.The sweep ran for 180 s with 1024-token prompts, maximum output length 64, and TTFT≤5 s.
- Rollout control: Two-phase readiness moves cold-activation waiting into rollout and prewarming, so user-visible load latency measures only adapters marked ready.Admission serializes cold work to protect existing warm traffic; readiness therefore reports a prepared path rather than zero activation time.
- Compatibility boundary: The native vLLM 1M packed-layout row is a format-compatibility failure, while standard-PEFT baselines cover only small static adapter sets.The standard PEFT 1k rows are context rather than apples-to-apples performance comparisons for MinT’s million-scale packed lifecycle.
- Rollout control: Aggressive SLO-aware activation scheduling shortened rollout wall time but hurt warm tails, shifting attention to executor/add-LoRA activation interference.The paper retains the conservative fixed policy as the safe rollout control rather than claiming higher cold-load concurrency solves the bottleneck.
- Representation: Packed representation reduces tensor fanout from 37,248 to 672 tensors while changing adapter bytes from 110.75 MB to 105.58 MB.The measurements attribute cold-load overhead to object and registration costs from many small tensors, not only total byte volume.
- Cold-load behavior: Unique cold adapter loading into one engine is the measured bottleneck, while API queue wait is small and duplicate missing-adapter requests can share load work.Distinct cold adapters remain separate jobs, so backpressure and bounded cold loading remain necessary controls.
- Fleet capacity: The fleet-level model uses a 2300-distinct-adapter active wave, a 60 s warm-response target, and 2.57 req/s/engine as its warm-throughput input.This is a deployment-sizing model based on measured single-engine limits, distinct from the artifact-backed 1M catalog evidence.