PaperlayerView exact revision on arXiv

Source-linked AI summary

JustFit: Just-in-Time State Management for Local LLM Serving

Yuhua Chen

arXiv:2609.17475v3cs.AIcs.PF

TL;DR

Local agents need enough memory to retain working history while model execution and temporary allocations compete for a laptop’s unified memory. JustFit coordinates compressed KV execution, phase-dependent component loading, and request-state transitions. On a 24 GiB M4 Pro MacBook, it reaches 327,680 occupied page-capacity slots across two requests, while a single request completes the model’s full 262,144-position native window.

  • Problem

    Long-running agents accumulate context, while unified-memory devices must fit weights, request state, workspaces, and other allocations within one shared budget.

    Source: §1 · pp. 1–2

  • Method

    JustFit coordinates compressed-KV execution and checkpointing, phase-dependent component loading, and request transitions to manage overlapping state and computation lifetimes.

    Source: §1 · p. 2, §3.1 · p. 3, Let a new prompt progress without… · p. 5

  • Results

    327,680 occupied page-capacity slots across two active requests are 10.67× the evaluated historical mlx-vlm configuration’s 30,720-position single-request record numerically, not a like-for-like single-request gain.

    Source: §5.1 · p. 7, §8 · p. 10

  • Takeaways & Limitations

    Coordinating execution and state lifetimes makes compact representations more useful for retaining longer histories on personal hardware.

    Source: §8 · p. 10

    The single-platform runs and 250-ms watchdog do not establish continuous-service reliability, and the measured image scope excludes video, multiple images, and B4-plus-vision.

    Source: §7 · p. 10

Abstract

from arXiv · show

Local agents need memory for model execution and working history. We present JustFit, an MLX runtime that coordinates their overlapping allocations: KVExec executes and checkpoints four-bit KV with bounded workspace, PhaseSwap loads phase-dependent components, and StateTrans preserves history across execution modes. With Qwen3.8-27B MXFP4 on a 24 GiB M4 Pro MacBook, JustFit reaches 327,680 retained positions across two requests, 10.67 times the evaluated baseline's 30,720-position single-request record. One request completes the full 262,144-position native window at median 5.986 tokens/s. Each shape completes 16,384 outputs per request in three fresh processes: B1 uses one cold build and two prefix extensions; B2 uses three ordered prefix extensions (Section 4). Image encoding can proceed while preserving a live 196,608-input text request. A controlled, repetitive 32K+6K workload reaches median 18.284 tokens/s at 15,626 MiB; a separate AIME 2026 evaluation scores 29/30. Coordinating execution and state lifetimes makes longer histories feasible on personal hardware.

1. Introduction

Local agents need to retain working history and sustain execution, but long histories and overlapping allocations strain a laptop’s shared memory. JustFit coordinates compressed state and component lifetimes to make more context usable on the same device.

1. Introduction

  • 320K occupied page-capacity slots across two requests exceed the evaluated historical mlx-vlm configuration’s 30K single-request record numerically, not as a like-for-like single-request gain.A single request reaches the model’s 256K native window.Source: Figure 1 · p. 1, §1 · p. 2
  • Repository work accumulates files, tool results, and earlier decisions; compaction can omit details, while larger retained context gives agents more room before summarizing or rereading.Prefix caching complements retained context by reusing unchanged history.Source: §1 · p. 1
  • CPU staging, GPU workspaces, weights, and request state compete for unified memory, so sharing memory between processors does not remove the need to control simultaneous allocations.The fixed MXFP4 setting illustrates the pressure: its non-vision weights occupy 13.31 GiB, while an FP16 KV cache adds 16 GiB at the native window.Source: §1 · p. 2
  • JustFit combines KVExec for compressed-history execution and checkpointing, PhaseSwap for phase-dependent components, and StateTrans for preserving request history across execution modes.The contribution is coordinating these lifetimes rather than introducing another weight quantizer.Source: §1 · p. 2

2. Background: One Memory Budget, Different Lifetimes

JustFit’s memory problem arises because histories, model components, and temporary computation overlap under one memory budget. Its design accounts for those lifetimes while keeping compressed KV reusable and bounded during execution.

2. Background: One Memory Budget, Different Lifetimes

  • Qwen3.8-27B combines full-attention KV that grows with context length and fixed-size recurrent state per active request; model weights are shared, but request histories generally differ.The model has 16 full-attention layers interleaved with 48 Gated DeltaNet layers.Source: §2 · p. 2
  • Unified CPU/GPU arrays do not create a second memory budget, so simultaneous model, state, workspace, and host allocations determine the memory pressure.The accounting counts aliased storage once and aims to reduce peak simultaneous allocation without discarding useful history; it is conceptual, not an attribution of measured process footprint.Source: §2 · p. 2
  • TQ4 stores each 256-element vector as 128 packed bytes plus a two-byte norm, and the paged pool assigns 256-token blocks to requests and reuses pages after completion.Including norms, TQ4 uses 25.4% of the corresponding FP16 KV payload.Source: §2 · p. 2
  • Prefill uses floating-point MLX SDPA, so compressed KV must be reconstructed for that consumer; reconstructing two heads at a time lowers the 240K-input K/V operand from 960 to 480 MiB.MLX’s mx.eval boundary lets JustFit finish one group or layer before building the next, limiting accumulated temporary dependencies.Source: §2 · pp. 2–3

3. JustFit Design

JustFit separates retained request history from the temporary resources needed for each computation. KVExec, PhaseSwap, and StateTrans coordinate compressed-history execution, component residency, and request transitions under one memory budget.

3. JustFit Design

  • JustFit separates history from resources needed for the next computation: KVExec controls compressed-history use, PhaseSwap controls component residency, and StateTrans selects requests and execution mode.Request ownership persists while temporary components change.Source: §3 · p. 3, Figure 2 · p. 4

3.1. KVExec: Executing Compressed KV

  • KVExec uses page-native decode and fused page-to-operand prefill reconstruction, while dispatching multi-query attention to bounded grouped reconstruction or SDPA as eligible.At the 192K mechanism probe, two-head grouping reduced incremental MLX peak from 3,123.8 to 1,589.3 MiB with a 3.6% latency increase; single-query attention avoids full-cache expansion.Source: §3.1 · p. 3

3.2. PhaseSwap: Load Components for Their Next Use

  • PhaseSwap makes components available for their required phases and uses leases to keep a component attached while another request still needs it.It can encode image features, release the vision tower, and restore text components; changing ownership references also makes detaching the embedding effective in the mixed-image trace.Source: §3.2 · pp. 3–4

3.3. StateTrans: Add Requests Without Rebuilding History

  • StateTrans finishes a speculative round before switching an eligible singleton from MTP to batched AR, preserving target KV and recurrent state rather than copying or rebuilding history.The existing state can return to MTP when peers leave.Source: §3.3 · p. 4

Let a new prompt progress without monopolizing

  • With active decoders, N4/PF64 advances every active row for four batched AR forwards, then permits at most 64 prefill tokens; without a decoder, chunks can contain up to 256 tokens.This bounds work per opportunity, not total waiting time for an arbitrary workload.Source: Let a new prompt progress without… · p. 5
  • Admission reserves output pages as well as input space, guarantees the first 8,192 output tokens under the four-lane policy, and allows only limited overtaking of aged blocked requests.An overdue image gets an admission opportunity when a lane and KV reservation fit and the active retained-token sum passes the vision gate.Source: Let a new prompt progress without… · p. 5

3.4. Keep Checkpointing from Undoing the Savings

  • Checkpointing reads existing page runs while the request retains ownership, streams bounded pieces, and restores packed runs into the existing pool instead of first copying the full cache.At 224K, borrowing rather than copying reduced incremental snapshot allocation from 3,640 to 227.5 MiB; asynchronous callers still take an owning snapshot.Source: §3.4 · p. 5

4. Experimental Setup

The evaluation uses fixed Qwen3.8-27B MXFP4 weights on a 24 GiB M4 Pro MacBook and compares against a historical mlx-vlm configuration. Workloads test complete generation, development-stage throughput, and image admission alongside active text generation, with distinct measurement protocols.

Platform and comparison

  • Experiments use an M4 Pro MacBook with 24 GiB unified memory and fixed Qwen3.8-27B MXFP4 weights; the single-request target is its full 256K native window.The released setup sets iogpu.wired_limit_mb to 22,016 MiB, distinct from the 21,000-MiB process-footprint guard.Source: §4 · pp. 5–6
  • The baseline is a historical mlx-vlm snapshot, not the latest release or an ideal FP16 reference, and its effective cache storage type is not separately verified.The preserved launch manifest disables paged KV, segmented batch KV, and fused TQ reconstruction.Source: §4 · p. 6

Capacity workloads

  • Each final B1/B2 workload runs in three fresh processes and generates all 16K output tokens per request, using cold builds or exact-prefix extensions as specified.A separate cold pair uses 128K+16K requests; measured input work and end-to-end times retain their actual work amounts.Source: §4 · p. 6

Development workloads

  • The fixed 32K+6K development study repeats each successful configuration in three fresh processes, with MTP verification width three and APC disabled for stages 1–7, then enabled with zero prefix hits for stages 8–10.Capacity inputs repeat a word and use greedy selection with EOS suppression; token counts, output agreement, a subsequent request, and page reuse are checked.Source: §4 · p. 7

Image workload

  • In image trials, an image arrives about 0.75 seconds after a long text request’s first output, and each trial uses a fresh process and frozen-prefix clone.The image has 453 prompt positions and up to 64 outputs; text generates 512 outputs, and experimental active-context gates vary by input-size arm.Source: §4 · p. 7

Counters and timing

  • A watchdog samples physical process footprint every 0.25 seconds, while pool high-water reports 256-position page-capacity equivalents rather than a census of initialized KV entries.TG is measured over the common active interval for concurrent requests; end-to-end wall rate includes preparation and non-overlap.Source: §4 · p. 7

5. Evaluation

Across the evaluation, JustFit completes a single request at the model’s native window and reaches greater occupied page capacity across two requests. Additional tests measure throughput and memory tradeoffs, image admission during text generation, reusable history, and compressed-KV reasoning.

5.1. The Full Model Window and a Larger Shared Pool

  • 327,680 occupied page-capacity slots across two requests reached 10.67× the evaluated baseline’s 30,720-position single-request record; one request completed 262,144 native positions.The three 256K trials each generated 16,384 tokens, with median TG of 5.986 tokens/s; three 320K trials generated the full output for both requests, with median aggregate TG of 9.735 tokens/s.Source: Table 1 · p. 7, §5.1 · p. 7

5.2. How the Design Changes Memory and Throughput

  • 18.284 tokens/s at 15,626 MiB is the final system’s median on the fixed 32K+6K workload, 3.06× faster and 2,599 MiB lower than the first completing TQ4+MTP configuration.The figure’s development series uses three processes per stage; its APC change is not an isolated verifier ablation.Source: §5.2 · p. 7, Figure 4 · p. 8
  • At a shared 8K+64 workload, TG rises from 15.63 to 24.31 tokens/s with identical output tokens; measured whole-process peak-growth slopes are 169.9 versus 12.5 MiB per 1K input positions.The slopes are fitted over 8–24K for the baseline and 8–64K for JustFit; intermediate designs include a 13.291 tokens/s segmented-KV result and an 11.539 tokens/s paged path that uses less memory and supports shared state.Source: §5.2 · pp. 7–8

5.3. Reductions in Temporary Allocations

  • At 192K history, two-head reconstruction reduces incremental single-layer MLX peak from 3,123.8 to 1,589.3 MiB, including SDPA scratch.Latency rises 3.6%, from 271.220 to 280.897 ms, and the selected path matches the all-head reference bitwise at tested lengths.Source: §5.3 · p. 8
  • At a 224K prefix, borrowing page-run views instead of extracting a contiguous copy reduces snapshot peak allocation from 3,640 to 227.5 MiB.A separate writer-only comparison reduces application-accounted writes from 7.63 to 3.82 GB and median save time from 2.976 to 1.507 seconds.Source: §5.3 · p. 8

5.4. An Image Can Finish Before the Long Text Request

  • In the image-admission trial, image encoding completes in 21.636 seconds while a 196,608-input text request is generating; both streams finish at a 20,852-MiB peak below the 21,000-MiB guard.The text retains its target pages through image encoding, shares AR execution with the image request, and then resumes MTP.Source: §5.4 · pp. 8–9, Table 2 · p. 9

5.5. Reusable Prefixes, Safe Departures, and Decode Choice

  • Tests cover exact-prefix restoration during concurrent decoding, MTP-to-AR-to-MTP transitions, client departure with page reuse, and a staggered four-stream admission trial.The staggered trial reports 22.570 tokens/s during common-active decoding and 12.147 output tokens/s end to end; it is not a matched speedup comparison with earlier runs.Source: §5.5 · p. 9

5.6. Extended Reasoning with Compressed KV

  • Paged TQ4 answers 29/30 AIME 2026 problems, generates 696,834 tokens at 15.04 token-weighted tokens/s, and records a maximum footprint of 19,043 MiB.Uniform INT8 answers 28/30 at 14.64 tokens/s; both methods reach an output limit on one problem.Source: §5.6 · p. 9

6. Related Work

Related work addresses weight and KV compression, memory-constrained execution, and serving with reusable history. JustFit fixes its target weights and cache representation while coordinating execution and component lifetimes.

Model and cache compression

  • Weight-compression methods such as AWQ and MXFP4, and ternary Bonsai 2, reduce static model storage, while KIVI, KVQuant, and TurboQuant compress growing KV state.JustFit fixes both the target weights and a TurboQuant-based cache representation, then optimizes their execution and lifetimes.Source: §6 · p. 9

Memory-constrained execution

  • Prior memory-constrained systems coordinate weights, activations, and KV or optimize weight availability and I/O; KVExec adds page-addressed fused reconstruction, sequential head-group execution, and compact persistence.The comparison describes a dense-backbone setting where persistent history and selected phase-dependent components share one physical budget.Source: §6 · p. 9

Serving and reusable history

  • Serving systems schedule iterations, allocate KV blocks, interleave prefill with decode, or share prefixes; JustFit couples serving concerns to component lifetimes so mode changes and image admission preserve accumulated request state.Source: §6 · pp. 9–10

7. Discussion and Limitations

JustFit makes long histories usable, but latency and the scope of the evaluations constrain what the results establish. Capacity trials do not establish general comprehension, service reliability, or performance on newer hardware.

7. Discussion and Limitations

  • B1 fills the native window, while beyond-native single-request operation requires a changed model configuration and separate evaluation.Later turns use cached prefixes rather than rebuilding the full prefix plus suffix; the 256K trials do not repeat identical early input work.Source: §7 · p. 10
  • The 320K trials use different initializations and do not repeat the same early input work, so their peaks are not identical-run comparisons.Reported peak footprints range from 20,432 to 20,953 MiB.Source: §7 · p. 10
  • 67.7 minutes for a 240K cold build and 46.0 minutes for 16K generation make latency a practical constraint.Reuse avoids repeated ingestion, but new tokens still attend to a long history; newer-hardware performance needs direct measurement.Source: §7 · p. 10
  • The AIME evaluation tests extended reasoning, not full-window comprehension, repository-level success, or compaction frequency.Capacity inputs are repetitive and use workload-specific MTP acceptance; isolated probes measure local allocations rather than full-system attribution.Source: §7 · p. 10
  • Single-platform runs and a 250-ms watchdog do not establish continuous-service reliability; image tests cover only one bounded image beside one text request.Video, multiple images, and B4-plus-vision remain outside the measured scope.Source: §7 · p. 10

8. Conclusion

JustFit coordinates compressed state and component lifetimes to support long histories on a 24 GiB laptop. It fills a 27B model’s native window and reaches 320K occupied page-capacity slots across two requests.

8. Conclusion

  • 320K occupied page-capacity slots across two requests are 10.67× the evaluated historical configuration’s 30,720-position single-request record, not a like-for-like single-request gain.A single request fills the model’s native 256K window.Source: §8 · p. 10
  • KVExec bounds reconstruction and checkpointing, PhaseSwap supplies components when needed, and StateTrans preserves history as requests and execution modes change.The setting is a 27B model on a 24 GiB laptop.Source: §8 · p. 10
  • The results support managing execution and storage lifetimes alongside compact representations to make longer histories feasible on personal hardware.Evidence includes complete generation, memory probes, mixed image/text progress, and reasoning evaluation.Source: §8 · p. 10

Impact Statement

The supplementary material describes the study’s evaluation scope, implementation paths, storage, protocols, and development cohorts. It reports complete workloads and operating points under specified protocols, rather than universal service guarantees.

Impact Statement

  • The study evaluates execution and memory behavior; privacy, energy use, and availability depend on deployment.Local inference can support user control of code and documents and reuse existing hardware.Source: Impact Statement · p. 10

A.1. Codec and Attention Dispatch

  • The prefill facade presents one request-local row of the shared pool without making another KV copy, while dispatch selects among direct inverse, packed verification, and fallback paths.Figure 5 also shows grouped prefill reconstructing the current two-head group and persistence borrowing page views until synchronous serialization completes.Source: §A.1 · p. 12, Figure 5 · p. 12

A.2. Component and Cache Storage

  • The 327,680-position pool uses 5,200 MiB of backing arrays, while the 262,144-position pool uses 4,160 MiB.Packed arrays require 16,640 bytes per retained position; recurrent state is separate from the KV pool.Source: §A.2 · p. 13

A.3. Controlled Capacity Protocol

  • Capacity requests repeat a word with EOS suppression to reach the requested output length, and the harness checks token counts, output agreement, subsequent requests, and page reuse.The controlled 32K+6K study uses three fresh processes per successful configuration; acceptance rates and throughput depend on the workload.Source: §A.3 · p. 13

B.1. Development Configurations

  • Configuration 10 completes 256K B1 and reaches 320K page occupancy in B2, while fixed 32K+6K measurements form a separate series.Table 4 records available completed workloads, not a maximum remeasured at every development stage.Source: §B.1 · p. 13, Table 4 · p. 13

B.2. Earlier Fully Cold Single-Request Repeats

  • The earlier fully cold single-request cohort completes 196,608 uncached input positions and 16,384 generated tokens in each of three fresh processes.All output sequences agree and postflight/page-reuse checks pass; its 212,992 occupied page-capacity slots are below the configured 229,376-position pool.Source: §B.2 · p. 14, Table 6 · p. 14

B.3. Earlier Concurrent Workloads / B.4. Current Four-Stream Qualification

  • The current four-stream qualification completes 12,288 outputs per stream after a 131,072-token incumbent begins generating, with no head-phase deferrals or guard trips.The separate earlier concurrent operating points are summarized in Table 7; the current qualification is one staggered trial.Source: Table 7 · p. 14, §B.4 · p. 15, Table 8 · p. 15

B.5. Earlier Short-Output Probes / B.6. Residency-Manager Refactor

  • Short-output probes use a common 8K input point for direct baseline/JustFit comparison, while the residency-manager refactor reports paired workload results and passing output-agreement and page-reuse checks.Table 10 identifies each component-specific manager-to-common-manager pair; the B4 peak is contextualized against an earlier observation to show run-to-run variation.Source: §B.5 · p. 15, Table 9 · p. 15, §B.6 · p. 15

C.1. AIME Protocol and Error Cases

The AIME evaluation compares integrated TQ4 and INT8 runs under a seeded protocol, while the reproduction materials distinguish current release evidence from historical results. The reported score is not an isolated KV-bit-width comparison or a general accuracy claim.

C.1. AIME Protocol and Error Cases

  • Each arm evaluates the same 30 problems in one seeded run using temperature 1, top-p 0.95, top-k 20, and xhigh reasoning.Both arms enable MTP with draft block size 3 and disable prefix caching.Source: §C.1 · p. 16
  • Paged TQ4 answers 29/30 AIME 2026 problems, versus 28/30 for uniform INT8, in an integrated capability check rather than an isolated KV-bit-width experiment.The arms use different output ceilings and recovery procedures; the authors caution that the Q19 trajectory difference is not evidence that TQ4 is generally more accurate.Source: §C.1 · p. 16
  • TQ4 produces 696,834 output tokens at 15.04 token-weighted tokens/s, while INT8 produces 680,071 at 14.64 token-weighted tokens/s.The arithmetic mean of per-question decode rates is 17.59 for TQ4 and 17.84 for INT8; these denominators should not be interchanged.Source: §C.1 · p. 16

C.2. Reproducibility Materials

  • The immutable justfit-repro-v4 tag provides the current serving code and public results, while historical measurements retain their original runtime provenance.The supplementary M3 runs are descriptive rather than a controlled chip-only comparison; the long M3 workloads are single observations, and the M3 and M4 hosts have 36 GiB and 24 GiB, respectively.Source: §C.2 · pp. 16–17, Table 11 · p. 16
  • The tensor-level audit matches all 1,349 non-vision target tensors and all 333 production BF16 vision tensors to their pinned public components.MTP weights are also matched by digest; the container differs in inactive checkpoint-side vision storage.Source: §C.2 · p. 16

D. Decode-Mode Selection: AR versus Batched MTP

The measured batched-MTP path is near AR at two requests but slower and higher-peak at three or four, motivating the runtime’s conservative mode choice. The experiments also report completed capacity outcomes and diagnostic evidence about the observed performance differences.

D. Decode-Mode Selection: AR versus Batched MTP

  • The AR-versus-MTP pairs use fresh processes, greedy selection, matched pool capacity, and aligned EOS suppression; Table 12 defines TG as common-active-interval aggregate throughput.Only within-pair comparisons are controlled because the short and long suites use different backing volumes.Source: D. Decode-Mode Selection: AR versus… · p. 17, Table 12 · p. 17
  • All 12 measured arms completed without guard trips and matched output hashes, but one fresh-process observation per arm leaves small differences, especially at B2, statistically unestablished.The synthetic workloads are not application-level latency or quality evaluations.Source: D. Decode-Mode Selection: AR versus… · p. 17
  • B2 is near parity with AR, while B3/B4 are slower and require more peak memory in this implementation.Figure 6 reports matched-pair comparisons, not averages over repetitions.Source: Figure 6 · p. 17
  • At B2/S=3, the 5120 →17408 projection takes 0.704 ms for MXFP4 and 0.643 ms for affine/group-64; native AR takes 0.353 ms for MXFP4.The format comparison uses a synthetic dense matrix, not full-model checkpoints or cross-format output parity.Source: D. Decode-Mode Selection: AR versus… · pp. 17–18

E. Current Full-Window and Shared-Capacity Experiments

  • The final configuration completes 262,144 positions for one request and 327,680 occupied page-capacity slots across two requests, without runtime truncation, sliding-window eviction, or summarization.These executions use fresh processes, repetitive text, greedy selection, aligned EOS suppression, and a 21,000-MiB process guard.Source: E. Current Full-Window and… · p. 18

E.1. Three completed B1 native-window executions

The section reports completed single-request native-window and two-request shared-capacity runs, alongside controlled probes of memory use and serving policy. Results distinguish complete workload outcomes from preparation-route measurements and synthetic mechanism tests.

E.1. Three completed B1 native-window executions

  • Median TG across three complete 245,760-input/16,384-output B1 executions is 5.986 tokens/s, with a median sampled peak of 20,449 MiB.All three executions produce identical output tokens and pass postflight and page reuse; Table 13 groups them by their shared endpoint. Checkpoint-initialized runs record packed page-run reads of 0.5332 and 0.5380 seconds, which are host-call observations rather than a full restore decomposition.Source: §E.1 · p. 18, Table 13 · p. 18

E.2. Three completed B2 320K aggregate executions

  • Three B2 executions reach 327,680 page-capacity slots and a 9.735 tokens/s median aggregate common-active-interval TG across 32,763 generated tokens.The first computes 10,241 input positions while reruns compute two; the 335,872-position next shape trips the guard during restoration.Source: §E.2 · pp. 18–19, Table 14 · p. 19

E.3. Fully cold B2 construction

  • A separate cold B2 process completes two 131,072-input/16,384-output requests, totaling 294,912 positions, with 10.341 common-interval aggregate TG and a 20,858-MiB peak.It processes all 262,144 input tokens at 81.399 tokens/s; the headline aggregate record remains the 320K B2 shape.Source: §E.3 · p. 19

E.4. Controlled mechanism probes

  • At 192K, two-head grouping reduces incremental MLX peak by approximately 49.1% while increasing latency 3.6%.The measured head-grouping probe uses synthetic single-layer attention; the selected two-head implementation matches the all-head reference bitwise at reported target lengths. Table 15 includes reconstruction, synchronization, SDPA, concatenation, and workspace in its measurements.Source: §E.4 · p. 19, Table 15 · p. 19
  • Synchronous borrowed views reduce snapshot-probe peaks from 520 to 32.5 MiB at 32K and 3,640 to 227.5 MiB at 224K; writer peaks also fall across all three tested sizes.The writer old/new peaks are 16.25/8, 32.5/16, and 227.5/112 MiB; these probes isolate different operations and are not full-model physical-memory savings.Source: §E.4 · pp. 19–20, Table 16 · p. 19

F.1. Serving Contract and Admission Policy

  • Image admission requires a candidate gate of 198,656 positions, a free lane within the four-lane limit, a fitting reservation, and at least 500 ms of image age.At the serialized boundary, the image path releases eligible head/embedding leases for tower loading, then restores them while preserving incumbent target KV pages and recurrent rows.Source: §F.1 · p. 20

F.2. Workload and Measurements

The mixed text/image trials measure admission boundaries, image completion, and memory residency changes under a specified single-image workload. The accompanying validation section reports runtime and launcher tests and describes failure handling.

F.2. Workload and Measurements

  • All five mixed text/image arms load the tower and encode the image, and all text outputs match; the 200,704-input boundary trial records 452/453 image-prefix positions restored.The workload uses one image arriving about 0.75 seconds after the first text token; Table 17 identifies the experimental gates and whole-process guard.Source: §F.2 · p. 20, Table 17 · p. 20
  • At 196,608 input positions, embedding MLX-active allocation falls by 644.14 MiB, while the independent whole-process peak is 20,852 MiB.Tower load, embedding restore, and head restore take 0.151, 0.113, and 0.110 seconds in host intervals; Table 18 covers text-side timing in these trials.Source: §F.2 · p. 20, Table 18 · p. 20

F.3. Failure Handling and Validation

  • Focused validation passes 475 runtime tests and six launcher-policy tests; failed encoding restores text leases, failed tower release blocks further large allocations, and a missing embedding artifact stops startup.The final routing change keeps video on the existing path and was unit-tested after the image runs.Source: §F.3 · pp. 20–21

Related papers

Loading 2609.17475v3…