Source-linked AI summary
Nanbeige4.2-3B on Apple Silicon: Fixing Deployment Bugs and Decreasing Looped Transformer Memory Overhead
John T. Halloran
TL;DR
Nanbeige4.2-3B cannot reliably run on Apple Silicon because deployment bugs and Looped Transformer memory overhead block evaluation. The paper fixes these issues with chunked prefilling and additional patches, enabling 30% completion on a real MCPMark subset while multi-tool reliability remains limited.
Problem
The released checkpoint has deployment and memory-scaling problems that prevent reliable agentic evaluation on Apple Silicon.
Method
The paper fixes five deployment bugs, introduces chunked prefilling, and resolves system-prompt and MPS-memory issues for reproducible evaluation.
Results
30% of MCPMark Filesystem tasks were completed, while single-tool BFCL performance was reliable but multi-tool tests mostly failed.
Takeaways & Limitations
The patched model becomes evaluable for real agentic tasks on Apple Silicon, but multi-tool calling remains a distinct capability limitation.
Takeaways & Limitations
The model still fails most multi-tool tests and can exceed memory capacity or time out during context accumulation.
Abstract
from arXiv · showhide
Nanbeige4.2-3B is a 3B-parameter agentic model built around a Looped Transformer (LT) that reuses one stack of layers for a second forward pass, adding effective depth without additional parameters. Evaluated on Apple Silicon (MPS), we identify five independent bugs which prevent the released checkpoint from running via Hugging Face transformers out of the box (including a silently-zeroed RoPE buffer and calls to removed transformers cache APIs). Furthermore, we show that fixing these bugs is still not sufficient for agentic tasks, due to the LT's layer-reuse strategy (which effectively doubles peak attention memory) used to achieve parameter efficiency. We thus introduce a chunked-prefill strategy which alleviates the incurred memory-capacity penalty, extending allowable context width by $2.7 \times$ on 32~GiB shared memory. However, even with the reduced memory overhead, we show that patches are required to render Nanbeige4.2-3B usable; resolving both system prompt and MPS-native memory bugs finally allows reliable evaluation on standard MCP and tool-calling benchmarks. On a subset of MCPMark, the debugged model completes up to 30\% of real agentic tasks (up from the original's 0\%), while, on BFCL, it is near-perfect at single tool calls (yet fails the majority of multi-tool tests). We release the patched checkpoint, system prompt optimizer, and evaluation harnesses at https://github.com/johnhalloran321/Nanbeige4.2-3B-mps-fix.
1 Introduction
Nanbeige4.2-3B is a parameter-efficient Loop Transformer model designed for agentic tasks, but its released checkpoint requires multiple software, memory, and prompting fixes before reliable MPS evaluation. These fixes enable reproducible benchmarking despite the architecture’s increased peak attention-memory demands.
- Motivation: Nanbeige4.2-3B uses a Loop Transformer to increase effective depth and decoding compute without increasing parameter count, targeting improved agentic-task capability.Its model card reports competitive or better results than Qwen3.5-4B and Qwen3.5-9B on agentic and office-workflow benchmarks.
- Deployment and memory issues: Five initial bugs, including a silently-zeroed RoPE buffer and removed transformers cache APIs, must be fixed for the released checkpoint to run on MPS.Even at 3B parameters in bf16 on a 32 GiB shared-memory system, the Loop Transformer doubles peak attention memory and limits scalability for agentic tasks.
- Remaining deficiencies: Nanbeige4.2-3B’s trained-in tool-use system prompt is silently replaced rather than merged when a caller supplies any system message, while MPS OOM errors can permanently reduce the serving process’s usable memory budget.The memory degradation surfaced during evaluation of the debugged model on MCPMark.
- Contributions: Five initial bug fixes, an alternative prefilling algorithm, system prompt correction, and an MPS-specific OOM fix enable reproducible evaluation on standard agentic and tool-use benchmarks.Together, these changes address both deployment failures and the Loop Transformer’s memory-capacity constraints.
2 Five Initial Deployment Bugs
The unmodified Nanbeige4.2-3B checkpoint fails or silently misbehaves on Apple Silicon for five independently reproduced deployment bugs. These include broken rotary position encoding, configuration and cache incompatibilities, an MPS-only crash, and serialization failure; all five are fixed without modifying cached transformers files.
- Five Initial Deployment Bugs: Five independent bugs prevent reliable loading or execution of the unmodified checkpoint, and each was directly reproduced against it.Table 1 records each bug’s exact trigger, source line, and error or symptom against transformers==5.8.1.
- Five Initial Deployment Bugs: The dominant RoPE bug silently zeros the inv_freq buffer, removing token-order information and producing fluent-looking but positionally incoherent generation.The buffer is never repopulated before the first forward pass, so the failure can be missed without inspecting loaded values.
- Five Initial Deployment Bugs: A RoPE-type dispatch KeyError blocks construction for some valid configurations, while a removed DynamicCache API is called when forward() receives past_key_values=None.The dispatch failure occurs before device placement and is not MPS-specific; the cache mismatch occurs when calling forward() directly rather than generate().
- Five Initial Deployment Bugs: A position-IDs re-trim bug causes a hard crash specifically on MPS, and an incompatible tied-weights key convention breaks save_pretrained().The serialization issue remains even after the model is successfully patched and running.
3 The Looped-Transformer Memory Tradeoff
Nanbeige4.2-3B’s Looped Transformer reuses L layers for two effective passes, improving parameter efficiency but doubling naive prefilling memory. Chunked prefilling bounds attention-memory growth and extends feasible context, trading lower memory for additional latency.
- Looped-Transformer tradeoff: Two effective passes through L shared layers provide 2L layer-executions from L layers’ parameters, improving model quality without increasing parameter count.The passage notes that looped transformers can outperform larger non-looped models at a fixed parameter budget.
- Looped-Transformer tradeoff: Naive prefilling doubles the O(prompt_len^2) attention-memory requirement because the same prompt’s attention computation repeats for each loop.This penalty is especially severe on Apple Silicon unified memory, which is shared with the operating system and competing processes.
- Chunked prefilling: Chunked prefilling processes fixed-size prompt chunks while incrementally growing the Key-Value cache, bounding each attention-score tensor to chunk_size × running-total.The method replaces one full-prompt prefill with sequential sub-calls and produced bit-identical outputs to naive prefilling.
4 System-Prompt Regression
Nanbeige4.2-3B’s chat template replaces its trained default when callers provide system messages, making system-prompt inclusion break multi-tool outputs. The fix preserves the zero-whitespace default path while inserting caller content afterward.
- Template regression: Caller-supplied system messages replace the trained default and append two newlines, while the auto-inserted default has no separator before # Tools.This branch difference silently discards the model’s trained prompt and changes the exact rendered byte sequence.
- Template regression: Tools-plus-user requests yield correctly formatted single calls without a system message, but adding one causes multi-tool outputs to become malformed text.Re-supplying byteidentical default text still fails because the explicit branch adds exactly two characters of whitespace.
- Fix: Removing the system-message branch and inserting caller content after the auto-inserted default preserves single-tool-call formatting while retaining the caller’s system content.The repair forces the zero-extra-whitespace rendering path calibrated by the checkpoint’s tool-use training.
- Related template issue: 25% of calls reportedly emit <tool_call> with a trailing space instead of <tool_call>\n in llama.cpp, breaking tag matching.This mechanically distinct issue reinforces that tool-calling reliability is sensitive to template and whitespace details.
5 Evaluation
The combined fixes enable reliable evaluation, with the patched checkpoint completing 3/10 MCPMark Filesystem tasks while BFCL shows strong single-call and no-call recognition but weak multi-call formatting. Harness isolation is required because an MPS out-of-memory error permanently reduces the process’s usable memory budget.
- Evaluation infrastructure: One MPS out-of-memory error permanently degraded the harness process’s usable memory budget, and neither torch.mps.empty_cache() nor gc.collect() reclaimed it.Restarting the harness before each task prevents an early MCPMark OOM from cascading into later spurious OOMs.
- 5.1 MCPMark: 3/10 (30%) MCPMark Filesystem tasks were completed by the patched checkpoint with chunked prefill, all described bug fixes, and per-task server isolation.The original checkpoint cannot be evaluated because of bug 2.
- 5.1 MCPMark: 21 repetitions of a long absolute path caused one MCPMark task to time out, while other failures exceeded memory capacity during multi-turn context accumulation.The model first correctly called read_multiple_files once on the failed pattern_matching task.
- 5.2 BFCL: 100% on irrelevance and 63.3% on simple, well-specified calls demonstrate reliable no-call recognition and moderate single-call reliability on BFCL.The 150-task subset used AST/exact-match grading across single-turn tool-calling categories.
- 5.2 BFCL: One call instead of two was the dominant failure in both BFCL parallel categories, revealing a format-level limitation in multi-tool emission independent of memory capacity.This weakness affected parallel and parallel_multiple categories requiring two or more calls.
6 Conclusions and Artifacts
The released Nanbeige4.2-3B checkpoint required five deployment fixes, chunked prefilling, and additional system-prompt and MPS-memory fixes for reliable Apple Silicon agentic evaluation. The authors release the patched implementation, checkpoint, and reproduction materials.
- Deployment fixes: Five independent deployment bugs and Looped Transformer memory overhead prevented reliable Hugging Face deployment on Apple Silicon.The authors fix all five bugs and address the architecture’s memory-capacity penalty.
- Memory mitigation: More than 2× usable context width is enabled under Apple Silicon shared-memory limits through chunked prefilling.The strategy targets the memory overhead introduced by the Looped Transformer architecture.
- Agentic evaluation: Reliable agentic evaluation additionally requires resolving a system-prompt regression and an MPS memory bug.These issues otherwise block reliable evaluation after the deployment bugs are fixed.
- Artifacts: The authors release the code, patched checkpoint, and reproduction scripts for all evaluations on GitHub and Hugging Face.The repository README describes the finalized layout and contents, with repository publication noted as pending in the passage.