Multiplication in Multimodal LLMs: Computation with Text, Image, and Audio Inputs
Abstract
Multimodal LLMs can accurately perceive numerical content across modalities yet fail to perform exact multi-digit multiplication when the identical underlying arithmetic problem is presented as numerals, number words, images, or in audio form. Because existing benchmarks often lack systematically paired instances across modalities, it remains difficult to compare genuine arithmetic limits within and across model families. We therefore introduce a controlled multimodal multiplication benchmark that factorially varies digit length, digit sparsity, representation (e.g., numerals vs. number words), and modality (text, rendered images, audio), with paired instances from a reproducible generator. We also define arithmetic load, C, as the product of the total and non-zero digit count as a compact, mechanistically motivated proxy for operation count. Across evaluations, accuracy falls sharply as C grows, often nearing zero by C > 100. Indeed, C remains predictive of performance across modalities and models, with R-squared often > 0.5, nearing the value from more complex measures of arithmetic load that count the number of intermediate arithmetic steps. A separate perception-versus-computation decomposition shows that multimodal degradation is primarily computational rather than perceptual: on matched-perception checks, models are near-perfect (> 99%) across modalities, even when multiplication accuracy drops. Beyond measuring when models fail, we ask which procedures they are predisposed to follow. We introduce a forced-completion loss probe that scores heuristic-specific reasoning prefixes--including columnar multiplication, distributive decomposition, and rounding/compensation. Here, decomposition is favored in both text and vision modalities; heuristic-specific LoRA adapters produce near-orthogonal updates yet degrade accuracy, indicating the base model maintains a well-tuned internal router.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a comprehensive examination of arithmetic performance in multimodal LLMs, introducing a novel benchmark and methodology that reveals critical insights into model behavior across different representations. The findings contribute meaningfully to the understanding of computational limitations in AI systems, particularly in the context of multimodal interactions.
Methodology
The paper introduces a controlled multimodal multiplication benchmark that systematically varies digit length, digit sparsity, representation, and modality. This approach is methodologically sound as it isolates the effects of these variables on model performance, providing a clear framework for evaluating arithmetic capabilities across different modalities. The definition of arithmetic load (C) as a predictor of performance is a novel and insightful contribution, allowing for a compact representation of computational complexity. The use of forced-completion loss probes to assess heuristic preferences is an innovative method that adds depth to the analysis of model behavior.
Results & evaluation
The experiments are robust, involving multiple model families and a variety of modalities (text, image, audio). The logistic regression analysis effectively demonstrates the relationship between arithmetic load and accuracy, with R-squared values indicating a strong predictive capability. The results reveal significant insights into how different models handle arithmetic tasks under varying conditions, highlighting the computational challenges faced by multimodal LLMs. However, the reliance on synthetic templates for problem generation may limit the generalizability of the findings to real-world scenarios.
Reproducibility
The paper provides detailed descriptions of the experimental setup, including model configurations and evaluation protocols, which enhances reproducibility. However, the absence of publicly available datasets or code repositories limits the ability of other researchers to replicate the study fully. The methodology for generating the benchmark and the specific models used are well-documented, but sharing the actual implementation would further support reproducibility.
Limitations
The study's focus on multiplication limits its applicability to other arithmetic operations, such as addition or division, which may exhibit different computational characteristics. Additionally, the model coverage is restricted to specific families, and the synthetic nature of the digit templates may not accurately reflect the complexity of real-world arithmetic problems. The controlled rendering of inputs may also overlook the challenges presented by messy real-world data.
Broader impact
This research has the potential to significantly influence the development of multimodal LLMs, particularly in applications requiring precise arithmetic capabilities. By identifying the computational limits and preferred strategies of these models, the findings can inform future training methodologies and benchmark designs. The insights gained could lead to improved performance in practical applications such as educational tools, automated reasoning systems, and AI-driven assistants that require reliable arithmetic processing. The paper presents a comprehensive examination of arithmetic performance in multimodal LLMs, introducing a novel benchmark and methodology that reveals critical insights into model behavior across different representations. The findings contribute meaningfully to the understanding of computational limitations in AI systems, particularly in the context of multimodal interactions.
Author of this paper? Send a correction.