Source-linked AI summary
Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs
Samuel G. Balter, Ethan Jerzak, Connor T. Jerzak
TL;DR
Exact multiplication in multimodal LLMs is difficult to compare across representations because paired evaluations are limited. This paper introduces a controlled benchmark and heuristic probe, finding that arithmetic load predicts performance and that strategy preferences vary with modality and operand cues.
Problem
Existing evaluations provide limited systematically paired evidence for separating multimodal arithmetic computation from representation and perception effects.
Method
The paper factorially varies digit length, digit sparsity, representation, and modality in a reproducible multiplication benchmark, then probes heuristic-specific reasoning preferences.
Results
Arithmetic load is a strong accuracy predictor across modalities, while heuristic preferences shift with operand cues and modality.
Takeaways & Limitations
Digit structure is the main computational burden in exact multiplication, with input modality providing a secondary modulation of performance and heuristic preference.
Takeaways & Limitations
The study focuses on multiplication, so its patterns may differ for addition, division, symbolic algebra, or multi-step word problems.
Abstract
from arXiv · showhide
Multimodal LLMs can accurately perceive numerical content across modalities yet fail to perform exact multi-digit multiplication when the identical underlying arithmetic problem is presented as numerals, number words, images, or in audio form. Because existing benchmarks often lack systematically paired instances across modalities, it remains difficult to compare genuine arithmetic limits within and across model families. We therefore introduce a controlled multimodal multiplication benchmark that factorially varies digit length, digit sparsity, representation (e.g., numerals vs. number words), and modality (text, rendered images, audio), with paired instances from a reproducible generator. We also define arithmetic load, C, as the product of the total and non-zero digit count as a compact, mechanistically motivated proxy for operation count. Across evaluations, accuracy falls sharply as C grows, often nearing zero by C > 100. Indeed, C remains predictive of performance across modalities and models, with R-squared often > 0.5, nearing the value from more complex measures of arithmetic load that count the number of intermediate arithmetic steps. A separate perception-versus-computation decomposition shows that multimodal degradation is primarily computational rather than perceptual: on matched-perception checks, models are near-perfect (> 99%) across modalities, even when multiplication accuracy drops. Beyond measuring when models fail, we ask which procedures they are predisposed to follow. We introduce a forced-completion loss probe that scores heuristic-specific reasoning prefixes--including columnar multiplication, distributive decomposition, and rounding/compensation. Here, decomposition is favored in both text and vision modalities; heuristic-specific LoRA adapters produce near-orthogonal updates yet degrade accuracy, indicating the base model maintains a well-tuned internal router.
1 Introduction
The paper introduces a controlled multimodal multiplication benchmark and probes which arithmetic heuristics models prefer. It finds that arithmetic load predicts accuracy, while heuristic preferences shift with operand cues and modality.
- Motivation: Exact multi-digit arithmetic remains brittle despite improvements from chain-of-thought prompting and math-focused pretraining.Performance can change between nearby two-digit multiplication instances and with surface form.
- Controlled benchmark: The benchmark factorially varies operand digit length, non-zero digit count, representation, and modality using standardized paired evaluations.The same underlying problems are tested across text, rendered images, and audio.
- Heuristic probing: A forward-pass forced-completion loss probe fingerprints preferences for columnar multiplication, distributive decomposition, and rounding/compensation.The probe evaluates heuristic-specific reasoning preambles without requiring generated traces.
- Difficulty measure: Arithmetic load, defined as total digits multiplied by non-zero digits, strongly predicts accuracy and summarizes degradation across modalities.The measure is presented as an interpretable proxy for operation count.
- Heuristic probing: Heuristic preference margins shift with operand cues and modality, indicating model-specific low-loss preferences rather than a single fixed procedure.The introduction frames these shifts as part of the paper’s strategy-preference analysis.
- Parameter-space analysis: Heuristic-specific LoRA adapters exhibit separated effective update directions, with stronger orthogonality in 235B than in 30B.The result suggests distinct parameter subspaces for the examined heuristics.
2 Background
Prior work establishes that exact arithmetic is sensitive to operand structure and presentation, while the paper frames multiplication strategies as distinct heuristic procedures with different burdens and error modes.
- Arithmetic brittleness: Exact multi-digit arithmetic accuracy drops as digit length and carry interactions grow, and small operand changes can reverse correctness.These effects can persist under deterministic decoding.
- Representation and modality: Arithmetic success is also sensitive to encoding, but prior multimodal evaluations rarely isolate elementary arithmetic under controlled modal variation.This leaves perceptual noise and underlying computational limits difficult to separate.
- Arithmetic heuristics: Multiplication can use multiple procedures whose choice depends on operand structure, experience, and context.The paper focuses on columnar multiplication, distributive decomposition, and rounding-compensation.
- Arithmetic heuristics: Columnar multiplication computes digit-level partial products with explicit carry propagation, making it systematic for unstructured or carry-heavy operands.The paper labels this heuristic OT.
- Arithmetic heuristics: Distributive decomposition rewrites multiplication as summed partial products and can simplify operands with trailing zeros or place-value structure.For example, 47 × 36 becomes 40 × 36 + 7 × 36.
- Arithmetic heuristics: Rounding-compensation uses a convenient nearby base plus a correction, which can simplify operands close to round numbers.The paper labels this heuristic RC.
- Arithmetic heuristics: These heuristics differ in partial products, carries, adjustments, and characteristic failure modes when steps are omitted or misapplied.The paper uses probing to test which procedural paths models naturally favor.
3 Methodology
The methodology independently varies operand structure, modality, and representation to measure multiplication difficulty and strategy preference. It combines arithmetic-load modeling with forced-completion heuristic probes and adapter-based parameter analyses.
- Controlled problem design: The benchmark samples multiplication pairs from dense and sparse digit templates, varying operand length and non-zero-digit arrangement without changing task format.Templates include fully dense numbers, trailing-zero structures, and non-adjacent non-zeros, producing varied carry patterns.
- Difficulty measure: Arithmetic load C is defined from total and non-zero digits and used as an interpretable proxy for operation count.The study validates C against more complex carry-aware operation counts.
- Multimodal evaluation: The paired benchmark evaluates 10,000 shared multiplication instances across text, images, and, where possible, audio, with additional alphabetic representations.The same underlying items are presented through different input channels to isolate modality effects.
- Multimodal evaluation: Correctness is modeled with modality-specific logistic regressions using arithmetic load as the predictor.The logistic formulation provides a comparable monotone fit across problem families; higher intercepts indicate stronger baseline performance and less-negative slopes indicate slower degradation.
- Heuristic fingerprinting: A forced-completion loss probe compares columnar, decomposition, and rounding/compensation continuations using matched heuristic templates and relative loss.The probe measures preference from token-level cross-entropy under the same question context, avoiding sampling and answer-formatting confounds.
- Heuristic fingerprinting: The study evaluates heuristic preferences with a 1,000-item Heuristic-Disagreement Set and adversarial traps, then trains separate heuristic LoRA adapters for parameter-space comparison.HDS items are uniquely labeled by cost-model preference, while adapters cover rounding/compensation, decomposition, columnar multiplication, and a style control.
4 Results
Arithmetic load is the dominant axis of multiplication degradation across models and modalities, while representation shifts where failure begins. Procedural probing further shows decomposition preference, but forcing heuristic-specific traces generally harms accuracy.
- 4.1 Algorithmic Load Degrades Accuracy Across Modalities: Accuracy declines monotonically with arithmetic load across modalities, approaching near-zero at the highest observed difficulty levels.Text is usually strongest, while audio and image variants tend to lag; modality effects vary by model.
- 4.1 Algorithmic Load Degrades Accuracy Across Modalities: Most model–modality fits explain roughly half or more of outcome variation, with arithmetic load R2 values commonly above 0.5.This supports arithmetic load as a compact one-dimensional summary of performance.
- 4.1 Algorithmic Load Degrades Accuracy Across Modalities: 50% accuracy thresholds cluster near C ≈50–54 for Gemini 2.5 Flash, Qwen3-VL-30B, and GPT-4o, versus C ≈74–75 for Qwen3-VL-235B.GPT-5.4 and Grok 4.20 retain relatively large thresholds in numerical text or numerical-image settings, but alphabetic-image conditions drop noticeably.
- 4.1 Algorithmic Load Degrades Accuracy Across Modalities: Modality effects more often shift intercepts or 50% thresholds than produce uniformly steeper degradation.Numerical images often track numerical text, alphabetic images are the most consistent weak point, and audio shows no uniform penalty in controlled conditions.
- 4.2 Decomposition Preference?: DD attains the lowest Δloss for both Qwen3-VL sizes in text and image modalities, indicating a cross-modal preference for distributive decomposition.For 30B, DD has the most negative text Δloss and smallest positive image Δloss; for 235B, it remains the least positive in both modalities.
- 4.3 LoRA Nudges Change Behavior, Do Not Improve Performance: Heuristic LoRAs caused 114 / 121 correctness flips, with only 1 / 4 improving and 113 / 117 degrading across the two model sizes.The results suggest the base model’s internal heuristic router is better optimized for performance than any single enforced trace distribution.
5 Limitations
The study’s conclusions are bounded by a narrow multiplication task, limited model coverage for probing, synthetic templates, controlled renderings, and potential dual-use risks.
- The benchmark focuses on multiplication, so patterns may differ for addition, division, symbolic algebra, or multi-step word problems.
- Token-level fingerprinting and LoRA nudges cover only Qwen3-VL-30B and Qwen3-VL-235B, despite broader multimodal accuracy analysis.
- Synthetic digit templates may not match the distributions of real-world mathematical problems.
- Controlled renderings exclude messy inputs such as scanned documents, handwritten equations, spreadsheet screenshots, and tool-augmented agent settings.
- The forced-completion and LoRA techniques could be repurposed to map and exploit arithmetic failure modes in deployed multimodal systems.
A Benchmark Construction Details and Evaluation Protocol
The benchmark and evaluation protocol combine controlled heuristic-targeted problem construction with operation-count proxies, deterministic decoding, uncertainty estimates, and likelihood-based probing.
- Evaluation protocol: The study used deterministic LoRA generations with temperature = 0 and a maximum budget of 2048 tokens.An attempted audio analysis on a Qwen3-omni variant encountered technical difficulties.
- Evaluation protocol: Standard errors use binomial estimates for accuracy and resolved-probe coverage, while loss averages use per-problem sample variability.Deterministic decoding at temperature = 0 means these standard errors reflect item-level variability.
- Benchmark construction: The Heuristic-Disagreement Set uses three roughly equal design-family buckets for rounding-compensation, distributive decomposition, and columnar multiplication.Items are retained when the minimum-cost heuristic is separated from the runner-up by a fixed margin; splits use 70/15/15 train/validation/test proportions.
- Operation-count proxy: C = ns + nt + ms + mt combines total digit length and non-zero digit counts as an operation-count proxy.C is monotone in total digit length and total non-zero digit count, but it does not preserve exact operation-count ordering across problems.
- Evaluation protocol: The probe compares length-normalized cross-entropy for heuristic templates under the same problem context, interpreting lower loss as stronger preference.With token-length-matched templates, −T∆ℓ(h) equals the log-likelihood ratio; otherwise, ∆ℓ is a normalized naturalness score rather than an exact decision rule.
C Forced-Completion Probe Details and Results
The forced-completion probe measures heuristic compatibility by comparing short, stylistically matched reasoning continuations against a neutral baseline.
- Probe design: The probe evaluates columnar, decomposition, rounding-compensation, and neutral-baseline continuations for the same multiplication problem.
- Scoring: Heuristic preferences are reported as length-normalized ∆loss relative to the neutral baseline.The template bank contains multiple paraphrases per heuristic and is stylistically matched, though not strictly token-length-identical.
C.2 Forced-Completion Loss Results
The supplementary results report raw forced-completion losses by heuristic template on a held-out probe split, with lower loss indicating deeper continuation preference.
- Raw cross-entropy losses are reported by heuristic template on a held-out probe split of n = 144.
- Lower forced-completion loss indicates a deeper continuation preference.
D.1 LoRA Training Examples
The paper generates heuristic-specific reasoning traces for rounding-compensation, distributive decomposition, and columnar multiplication, then probes their training effects and parameter relationships.
- Trace generation: Each heuristic uses a distinct algorithmic template for generating synthetic reasoning traces.The traces vary specific realizations while preserving consistent structure.
- Trace generation: RC applies difference of squares to symmetric near-base pairs and uses four adjustment terms for non-symmetric pairs.RC examples are programmatically selected from specified bases and offsets.
- Trace generation: DD decomposes the larger operand into tens and ones, computes two partial products, and sums them.Its examples include trailing-zero factors, easy tens-plus-ones decompositions, and generic two-digit pairs.
- Trace generation: OT simulates columnar multiplication through explicit digit products, carries, and accumulated partial products.Its examples include carry-heavy two- and three-digit pairs alongside generic multi-digit numbers.
- Adapter analysis: OT and DD LoRA updates are nearly orthogonal, with cosine similarities of ρ = 0.0726 / 0.0412 across the 30B and 235B models.Table 8 interprets this as evidence that the strategies use distinct parameter subspaces.
E.1 Adversarial Traps
Adversarial traps test whether heuristic preference signals remain stable when operand cues target known strategy weaknesses. The results show model- and heuristic-specific shifts in target support.
- Trap design: n = 30 adversarial traps compare heuristic preference signals under pressure against a held-out probe split.The trap set targets known weaknesses of specific heuristics.
- RC-targeted traps: 30B RC target support rises from 26.5% ± 0.5% to 34.5% ± 1.4% on anti-round traps.The corresponding 235B shift is smaller, from 12.4% ± 0.3% to 15.6% ± 0.3%.
- DD-targeted traps: On missing-term traps, DD support moves from 27.9% ± 0.3% to 24.1% ± 0.4% for 30B and from 15.2% ± 0.2% to 19.3% ± 0.2% for 235B.The direction of change differs by model size.
- OT-targeted traps: On OT-targeted traps, OT support rises from 25.8% ± 0.4% to 31.7% ± 1.4% for 30B and from 5.9% ± 0.3% to 12.1% ± 0.3% for 235B.The results indicate that adversarial operand cues redistribute heuristic preference mass.
Data Availability
The multimodal multiplication benchmarks are released in text, image, and audio renderings of the same items. The section also points to accompanying contrastive-probe and robustness result tables.
- Benchmark release: The benchmarks are released across text, image, and audio renderings of the same multiplication items.The dataset is hosted at the cited Hugging Face location.
- Associated analyses: Table 10 reports contrastive step-probe results on HDS test problems with n = 144.Preference rates measure how often the correct step has lower loss; loss gaps are mean incorrect minus correct loss.