More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training

1 Introduction

• The study addresses whether visual content changes scaling behavior and improves capabilities evaluated entirely through text, questions not isolated by changing multimodal data mixtures.Changing the mixture can also alter language supervision and resource allocation.Source: §1 · p. 1
• Matched models use image patch embeddings or content-free padding while keeping architecture, training text, and training settings constant across model sizes and compute budgets.The analysis compares loss-compute frontiers and compute-optimal allocations, then measures text prediction loss differences under image and padding conditions.Source: §1 · p. 1
• Visual content reduces multimodal loss and the relative visual anchor gap grows approximately as a power law of compute, while text-loss frontiers and compute-optimal allocations remain nearly unchanged.The comparison isolates visual content while holding language supervision and training configuration constant.Source: §1 · p. 1
• Across model sizes and training budgets, image and padding conditions achieve similar aggregate general-language performance, while spatial reasoning improves with model size and additional data in both.During largest-model training, gains are most sustained in localization, navigation, and symbolic planning; gains in spatial relations, geometry, and physical commonsense are less consistent.Source: §1 · pp. 1–2

2.1 Native Multimodal Pre-Training

• Native multimodal pre-training interleaves image patch embeddings with text in one decoder-only model trained autoregressively, with loss evaluated only at text tokens.The model is trained from scratch so both modalities share parameter capacity throughout training.Source: §2.1 · p. 2

2.2 Compute-Optimal Allocation

• At fixed compute, the analysis studies allocation between model size and token count, with data composition parameterized by the multimodal-to-text token ratio r.It estimates preferred allocations separately for each objective using smoothed training loss from the jointly trained models; the matched study fixes r = 0.3.Source: §2.2 · p. 2

2.3 Estimators of the Compute-Optimal Frontier

• IsoFLOP profiles provide the primary allocation estimates, while a training-curve envelope supplies a same-run sensitivity check.The envelope selects the lowest interpolated loss across models at each supported budget, and agreement between estimators is not independent validation.Source: §2.3 · pp. 2–3

2.4 Experimental Setup

• The nominal corpus contains 250B text-source tokens and 75B multimodal input tokens, and the model ladder ranges from A71M to A3B.The decoder-only MoE Transformer maps images directly into continuous embeddings without a pre-trained vision encoder.Source: §2.4 · p. 3

3.1 Isolating the Contribution of Visual Content

• Matched models differ in visual access: PAD replaces image embeddings with content-free padding while architecture, text, data order, token budget, and optimization settings stay fixed.Both models are trained from scratch with losses evaluated only on text tokens.Source: §3.1 · p. 3
• The comparison captures both learning from visual content and using images as context during pre-training.The anchor gap is the relative loss increase under padding; a positive gap means visual content lowers loss.Source: §3.1 · p. 3

3.2 Visual Content Preserves Text Scaling

• IMG and PAD retain nearly overlapping text-loss frontiers and compute-optimal allocations; paired text gaps range from −0.095% to +0.128%, with a mean of −0.006%.The allocation-exponent differences are small within each estimator, although the absolute exponents depend on the estimator.Source: §3.2 · p. 4, Figure 1 · p. 4, Figure 2 · p. 4

3.3 Visual Content Lowers Multimodal Loss

• IMG has lower multimodal loss than PAD at every evaluated budget, and its fitted frontier remains below PAD’s when each condition selects its own allocation.The comparison covers matched allocations as well as condition-specific loss frontiers.Source: §3.3 · p. 4, Figure 3 · p. 5, Figure 4 · p. 5

3.4 Visual Anchoring Strengthens with Compute

• Across the smallest to largest fitted multimodal-compute budgets, the gap along the IMG envelope rose from 3.35% to 8.09%, with power-law exponent γ = 0.256; fixed-compute profiles retained model-size dependence.Source: §3.4 · pp. 4–5
• The power-law fits use correlated observations from the same runs, so they do not establish training-seed reproducibility or behavior beyond the fitted range.The observed scaling trend is limited to the tested configurations.Source: §3.4 · p. 6

• The comparison uses 15 text benchmarks spanning understanding, commonsense, knowledge, mathematics, logical reasoning, and coding.The analysis uses the same matched IMG and PAD runs.Source: §4 · p. 6
• Aggregate language scores track closely under IMG and PAD across model sizes and training budgets, alongside substantial gains as models scale.Both conditions also improve rapidly early in training and more gradually later.Source: §4 · p. 6
• IMG achieves higher mean scores than PAD in coding, general understanding, and commonsense reasoning.These category gains accompany closely matched aggregate scores.Source: §4 · p. 6

5.1 Spatial Reasoning Scales with Compute

• At D = 325B, IMG rises from 15.41% to 44.25% and PAD from 15.10% to 39.87% from A71M to A3B.At A3B, increasing D from 25B to 325B raises accuracy from 22.56% to 44.25% for IMG and from 20.81% to 39.87% for PAD.Source: §5.1 · p. 6, Figure 8 · p. 7

5.3 Transfer Varies Across Task Groups

• During A3B training, mean IMG-PAD gains are 2.88% in localization, 3.02% in navigation, and 5.19% in symbolic planning.The other three groups have mean gaps within 0.6% of zero, although all six favor IMG at the final checkpoint; the distinction is persistence across training.Source: Figure 10 · p. 7, §5.3 · p. 7, Figure 9 · p. 7

5.4 Emergence of Compositional Spatial Skills

• Blocksworld accuracy rises from 3.3% to 27.6% for IMG and from 1.8% to 21.0% for PAD.Symbolic planning has the largest mean IMG-PAD gain during A3B training, approximately 5.19%; success requires a complete executable sequence that reaches the goal.Source: Figure 10 · p. 7, §5.4 · p. 7

5.2 Visual Content Adds a Measurable Gain

• The IMG-PAD gap is positive at five of six model sizes and reaches 4.38% at A3B, but is not monotonic.The gap is 3.62% at A128M and approximately 1.2% at A590M and A874M.Source: §5.2 · p. 7

5.3 Transfer Varies Across Task Groups

• The sustained-gain groups involve maintaining or composing spatial configurations, unlike groups emphasizing relation inference, geometric rules, or prior physical knowledge.This interpretation suggests more consistent transfer to organizing configurations and spatial updates, but does not establish a mechanism or strict taxonomy.Source: §5.3 · p. 7

5.4 Emergence of Compositional Spatial Skills

• For 5-12-block plans, IMG has no successes through A590M, then reaches 0.78% at A874M and 16.62% at A3B, versus 12.99% for A3B PAD.The paper calls this emergence of successful long-plan composition at larger scales, not a sharp scaling transition or acquisition of individual atomic operations.Source: §5.4 · pp. 7–8, Figure 11 · p. 8
• At A3B, IMG has fewer legal-but-unsuccessful plans than PAD (523 versus 1,121), but more precondition failures (882 versus 427).The audit identifies execution as a bottleneck; the contrast does not isolate improvements in individual operations.Source: Figure 12 · p. 8, §5.4 · p. 8

6 Conclusion

• Visual content acts as a scalable anchor: its multimodal text-prediction contribution grows with compute, while pure-text loss scaling, compute-optimal allocation, and aggregate language performance remain largely unchanged.Spatial reasoning improves with scaling in both conditions, with the most sustained visual gains in localization, navigation, and symbolic planning; execution remains a bottleneck.Source: §6 · p. 8

A Related Work

• Prior work studies native multimodal architectures, scaling laws, cross-modal interactions, and the effects of changing data mixtures.This study holds text, mixture, sequence structure, and optimization settings fixed to target image information within a common setup.Source: A Related Work · p. 11
• Related work probes visual context and visual supervision in language learning, while this study compares matched models trained with or without visual content and preserves associated text.The comparison examines both visual context's contribution to training loss and capabilities retained without visual inputs.Source: A Related Work · p. 11
• Unlike prior spatial evaluations using visual inputs or visual computation at inference, this evaluation uses text-only inputs and outputs after pre-training.The matched ablation separates capability growth with scale from visual content's additional contribution, while leaving the transfer mechanism unresolved.Source: A Related Work · p. 11

• Native multimodal pre-training uses Muon with weight decay 0.1 and gradient clipping threshold 1.0.Source: B Training settings · p. 11
• Training uses a warmup-stable learning-rate schedule, global batch size 2048, and maximum sequence length 4096.This corresponds to 8M input tokens per step.Source: B Training settings · pp. 11–12
• The training loss is computed only on text tokens, with vision tokens masked out.Source: B Training settings · p. 12

C.1 General language capability

• General language capability is evaluated on 15 distinct benchmarks spanning six task groups.The listed groups are aggregate, coding, mathematics, logical reasoning, knowledge, and commonsense reasoning; evaluation uses greedy generation.Source: §C.1 · p. 12

C.2 Spatial reasoning

• All spatial benchmarks use text-only inputs with five-shot chain-of-thought prompting.Four responses are generated per item at temperature 0.3 and top-p = 0.9; results are reported as avg@4.Source: §C.2 · pp. 12–13
• Spatial reasoning evaluation uses 15 benchmarks across six task groups, as inventoried in Table 2.The groups cover spatial relations, localization, navigation, geometry, physical commonsense, and symbolic planning.Source: §C.2 · pp. 12–13, Table 2 · p. 13
• Spatial scores weight benchmarks equally within each group and average the six group scores equally.The authors state that these groups organize this evaluation rather than define a general taxonomy of spatial or physical reasoning.Source: §C.2 · p. 13

D.1 Data Preprocessing

• Compute is calculated as C_s = 6ND_s, with D = Tk and source-specific token counts determined by the fixed data mixture.The section retains source, condition, and estimator indices; the matched study fixes r = 0.3.Source: §D.1 · p. 13, D Allocation Estimation and… · p. 13
• Losses are interpolated linearly in log compute within each model’s observed range, without extrapolation, after retaining budgets supported by at least three models.The nominal total-token budget is 325B, while loss-compute analysis uses actual selected steps.Source: §D.1 · p. 13

D.2 Estimating Compute-Optimal Allocation

• IsoFLOP allocation estimates fit a quadratic in log model size at each retained compute budget, then regress model size and token allocations against log compute.The training-curve envelope instead selects the minimizing model size at each budget and derives its token count from compute.Source: §D.2 · pp. 13–14
• Both allocation estimators reuse the same runs, so their comparison is a sensitivity check rather than independent validation.The estimators are IsoFLOP profiles and a training-curve envelope.Source: §D.2 · p. 14

D.3 Fitting Loss-Compute Frontiers

• Frontier candidates are rejected if K_s^q ≤ 0 or if any fitted-budget prediction is nonpositive; remaining candidates minimize mean squared log-loss residual.This two-stage selection is not joint least squares in log loss, and each source-condition fit is performed independently at r = 0.3.Source: §D.3 · p. 14

D.4 Fitting Visual Anchoring

• The visual anchor analysis compares image and padding losses at the same model size and compute budget before taking their relative difference.The loss-path summary evaluates padding loss at the model allocation selected by the image envelope, rather than comparing independently optimized frontiers.Source: §D.4 · p. 14
• Positive multimodal gaps are regressed in log gap against log compute, with figure percentages converted to fractions for Equation (8).The reported coefficients correspond to maximum, quadratic, and loss-path summaries, respectively.Source: §D.4 · p. 14

E General Language Evaluation Results

• Benchmark-level scores are reported for the general-language comparisons in Section 4.The supplied passage identifies the scores’ scope but gives no benchmark values or image-versus-padding outcome.Source: E General Language Evaluation Results · p. 14

F Spatial Evaluation Results

• Benchmark-level scores are reported for the spatial comparisons in Section 5.The supplied passage identifies the scores’ scope but gives no benchmark values or comparative outcome.Source: F Spatial Evaluation Results · p. 14

Benchmark-level interpretation

• Physical commonsense improves most with model size but shows a much smaller visual-content gain at A3B; geometry gains concentrate in BBH(geometric shapes), while Geometry3K remains harder.PIQA-spatial can draw on physical commonsense and language knowledge, so scale-related improvement alone does not establish a specifically visual mechanism.Source: §F.1 · pp. 14–15

Plan Length and Execution Outcomes

• Long-plan success rises from none through A590M to 0.78% at A874M and 16.62% at A3B for IMG, versus 12.99% for A3B PAD.The task requires an executable sequence that satisfies every goal relation, helping explain the limited absolute success rates.Source: §F.1 · p. 15
• At A3B, IMG has approximately 53% fewer legal-but-unsuccessful plans but 2.1 times as many precondition failures as PAD.These competing first-failure categories do not isolate improvements in individual operations; truncation is uncommon at A3B but declines non-monotonically across intermediate sizes.Source: §F.1 · p. 15

F.2 Showcases: What the Model Actually Does

• On the ten-block example, A3B IMG completes all nine target relations in 18 legal actions, while A590M IMG and A3B PAD each meet only one relation.The showcased example illustrates composition but does not establish how prevalent that outcome is.Source: §F.2 · p. 15

Counting explanation quality within a fixed case panel

• In a fixed panel of eight problems per condition, 13 of 27 accepted answers had consistent derivations, while nine had substantive errors.The manual audit supports same-problem case comparisons, not population-frequency estimates or a monotonic scaling trend in explanation reliability.Source: §F.2 · p. 16