LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUs

• Commodity GPUs offer competitive FLOPs per dollar but have reduced device memory and much lower communication bandwidth than datacenter accelerators.Local training also avoids uploading data to third parties and can reduce monetary cost.Source: §1 · p. 1
• LLMQ is an end-to-end C++ framework for language-model pretraining and fine-tuning on commodity GPUs in a single server node.It supports bfloat16 training and FP8 training with dynamic tensor-level scaling on Ada and Blackwell architectures.Source: §1 · p. 1
• Selective recomputation reduces activation costs, ranging from recomputing non-matrix-multiplication layers to recomputing full transformer blocks.Source: §1 · p. 1
• CPU offloading reduces peak GPU memory by moving residuals and optimizer states, while multi-GPU configurations shard optimizer states and can independently shard weights and gradients.Source: §1 · p. 1
• Because recent gaming GPUs cannot communicate directly over PCIe, LLMQ fully offloads sharded parameters to CPU without increasing forward/backward communication while reducing GPU memory use.Source: §1 · p. 2
• LLMQ trains a 32B model on four RTX 4090s at 3,400 tokens/second and 51% MFU, versus 29% MFU on L40S GPUs.On the same workstation, a 14B model reaches 7,800 tokens/second and 54% MFU; a 7B model on one RTX 4090 reaches 4,300 tokens/second and 61% MFU.Source: §1 · p. 2

• Sharding distributes tensors across devices; ZeRO spreads optimizer states and potentially gradients and weights, while pipeline and context parallelism distribute layers or sequence dimensions.Different forms of parallelism can be combined into multidimensional parallelism for large-scale training [5].Source: §2 · p. 2
• Activation recomputation saves memory by recalculating selected activations during the backward pass instead of storing them.The technique can avoid squared memory costs in attention layers [8].Source: §2 · p. 2
• Offloading moves large allocations from GPU memory to slower but more abundant CPU memory [9] or NVME storage [10].Source: §2 · p. 2
• Lower-precision datatypes save memory, including eight-bit Adam moment buffers and 16-bit or eight-bit activations [13] [14].BF16 or loss scaling can address the reduced dynamic range of 16-bit formats, whereas eight-bit training requires more sophisticated techniques.Source: §2 · p. 2
• Hardware-supported low-precision formats reduce data movement and fused multiply-add costs, with tensor cores amplifying these computational advantages.Scaling along the inner matrix-multiplication dimension requires native hardware support on recent Blackwell GPUs or software emulation with overheads [22].Source: §2 · pp. 2–3

Multi-GPU and implementation

• Multi-GPU execution always shards optimizer states without increasing communication, while deterministic kernels and CPU synchronization address reproducibility and collective deadlocks.LLMQ also fuses non-matmul operations, uses custom kernels, and chunks logits and attention workspaces when they become too large.Source: §3 · p. 3, §3.1 · p. 4, §3.2 · p. 6

Activation checkpointing

• LLMQ reduces activation memory through selective transformer-block recomputation, retaining feed-forward residuals and statistics needed for efficient requantization.Keeping absmax statistics avoids another global reduction and enables quantization to fuse with the nonlinearity during recomputation.Source: §3.1 · p. 4

Reduced-precision optimizer states

• The implementation reduces optimizer memory by storing AdamW states and master parameters in BF16, using stochastic rounding for unbiased conversion.FP32 AdamW states require 8 bytes per parameter, while BF16 momentum and variance halve that storage.Source: §3.1 · p. 4

Memory reduction

• For 3B training, optimizer and parameter offloading raise the achievable batch size from 2 or 8 to 10 after residual offloading.The progression is 1.5B at batch size 2 without offloading or 12 with optimizer offloading, then 3B at batch size 8 after offloading master parameters, and batch size 10 after offloading residuals.Source: §3.1 · p. 4
• A 7B model runs with micro-batch size 16 after gradient-buffer offloading, but requires approximately 54GB of host memory.The estimate includes optimizer state, quantized weights, and offloaded residuals; further NVMe offloading would slow transfers by another order of magnitude.Source: §3.1 · p. 5

FP8 impact

• FP8 reaches about 50% speed-up for sufficiently large models, but overheads limit gains on small models and optimizer time becomes noticeable for very large models.Dynamic quantization and backward-pass transposes add overhead, while larger models become increasingly dominated by matrix multiplication.Source: §4 · p. 7
• FP8 does not primarily improve performance by enabling larger batches, because activation recomputation removes many stored activations and FP8 can require extra transpose and quantization buffers.For fully recomputed transformer blocks, FP8 may use more memory rather than less.Source: §4 · pp. 7–8

Llama-Factory comparison

• At 14B, LLMQ is twice as fast as LLama-Factory, while the 1.5B BF16 gap is smaller at 52k versus 41k tokens/second.The comparison attributes the larger-scale advantage to efficient memcpy-based communication.Source: §4 · p. 8

Communication

• Memcpy-based communication is essential on consumer GPUs without peer-to-peer PCIe, whereas on professional GPUs combining NCCL and memcpy is faster than either option alone.The comparison covers gather, scatter, and all large collectives for the 14B model.Source: §4 · p. 8, Table 5 · p. 8

DGX Spark results

• On DGX Spark, reduced precision provides substantial benefits only at 7B, while pure BF16 achieves lower MFU than the RTX 5060Ti.The Spark’s 300 GB/s unified-memory bandwidth makes smaller workloads memory-bound; the comparison also attributes lower-than-advertised performance to probable power limits.Source: Table 3 · p. 7, §4 · pp. 8–9

Scenario 1: 1.5B pre-training

• The 1.5B pretraining experiments use 10B-token runs for FP8 and BF16, plus a 30B-token FP8 run with a 3x larger batch size.Source: §5 · p. 9, Figure 2 · p. 9
• E4M3 forward and backward training closely tracks BF16, whereas E5M2 activation gradients cause minor performance degradation.Source: §5 · p. 9, Figure 2 · p. 9

Scenario 2: GSM8k

• LLama2-7B improves substantially on the narrow GSM8k domain, while Qwen2.5-14B is already strong in few-shot evaluation and gains less from fine-tuning.Source: §5 · p. 9
• FP8 training does not result in noticeable performance degradation compared to the BF16 baseline on GSM8k.Source: §5 · p. 9
• FP8 quantization-aware fine-tuning consistently benefits FP8 inference, narrowing Qwen2.5-14B’s original-model gap.Source: §5 · p. 9

• Low-precision training can break down when scaled up, so LLMQ targets compute-constrained runs that do not extend far beyond the Chinchilla-optimal ratio.Source: §6 · p. 9
• 200B-token FP8 training for the 1.5B model on four RTX 4090s would take about 5 weeks.Source: §6 · p. 10

• Table 7 records offloading configurations for residuals, optimizer moments, master parameters, parameters, and gradients.Source: §A.1 · p. 13, Table 7 · p. 14
• Table 8 lists the LLama-factory baseline configurations, with activation checkpointing enabled in all settings.Source: §A.1 · p. 13, Table 8 · p. 14

LLama2-7B

• LLama2-7B is trained for two epochs with 32,768 tokens per optimizer step, sequence length 2048, and a learning rate decaying from 2×10−5 to 25%.The schedule uses 10 warm-up steps.Source: §A.2 · p. 14

Qwen2.5-14B

• Qwen2.5-14B is trained for one epoch at batch size 96 and sequence length 512, with the learning rate decaying from 6 × 10−6 to 0.The schedule uses 20 warm-up steps.Source: §A.2 · p. 14

• L40S reaches only about 270 TFLOP s−1 versus its advertised 362 TFLOP s−1 peak, making the resulting MFU appear lower.The matrix-multiplication benchmark achieves roughly three-quarters of the specification peak.Source: §A.3 · p. 14
• 5060Ti reaches up to 108% of its supposed peak FLOP/s, so its MFU is slightly overestimated.Source: §A.3 · p. 14
• DGX Spark reaches about 70% of peak FLOP/s in a matrix-multiplication microbenchmark.Source: §A.3 · p. 14