Apertus 1.5: From Open Text Models to Open Multimodal Reasoners
Abstract
Language models are becoming infrastructure, and infrastructure built in the open, by public institutions, on documented data, is the alternative to dependence on a few providers; Apertus shows this is viable at scale. Apertus 1.5 extends the fully open Apertus family from text to multimodal reasoning. Continued pretraining of the Apertus 1.0 models, on 4T additional tokens for the 8B and 2T for the 70B, adds image and audio understanding and extends the context window fourfold to 262,144 tokens; multimodality enters through the tokenizer, the language-model head remains restricted to text, and the architecture is unchanged. Supervised finetuning, long-context finetuning, reinforcement learning with verifiable rewards, and preference alignment complete the released models. Thinking is a flag of the chat template rather than a separate model, and lifts mathematical reasoning by up to 20 points on the 8B model. The released models improve over their predecessors on nearly every benchmark; among fully open models of its class, the 8B leads on knowledge, instruction following, and cultural benchmarks, and its instruction following surpasses all compared models, including open-weight ones. On image and audio understanding, a first for the family, the models are competitive with fully open multimodal models, strongest on document understanding and long-form and lower-resource speech recognition, while specialized vision and audio models remain ahead on perception and transcription of English and other high-resource languages. As in Apertus 1.0, we train only on openly available data, respect crawler opt-outs retroactively, and exclude restrictively licensed material. We release the model weights, data preparation and training recipes, and evaluation suites under permissive licenses; openness extends to values, with the Apertus charter guiding alignment stages published alongside the models.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a comprehensive and well-executed approach to extending a text LLM to a multimodal one using discrete tokenization, resulting in a fully open model family that is competitive with other open models. The detailed methodology, extensive experiments, and high reproducibility make it a significant contribution to the field of multimodal LLMs.
Methodology
The paper proposes a robust methodology for extending a text-only LLM to a multimodal one using discrete tokenization rather than continuous encoders. By utilizing the Emu3.5 image tokenizer and WavTokenizer for audio, the authors integrate new modalities into the existing vocabulary, allowing the standard next-token prediction objective to apply to all data types. This approach simplifies the training infrastructure, as it avoids the computational overhead of running separate vision/audio encoders during the forward pass. The training recipe is carefully staged: Stage 1 focuses on unpaired multimodal data to learn representation, while Stage 2 introduces paired data to align modalities with text, restricting the loss to text tokens to prevent the model from generating non-text outputs. This design choice is well-motivated and supported by ablation studies showing that text-only targets in the paired stage yield stronger image understanding. The extension of the context window to 262k tokens is also handled systematically through a multi-step extension process with adjusted rotary position embeddings.
Results & evaluation
The experimental evaluation is extensive, covering text, image, and audio benchmarks. The models are compared against other open and open-weight models, demonstrating competitive performance, particularly in document understanding and lower-resource speech recognition. The paper includes ablations on the choice of image tokenizer (Emu3 vs. Emu3.5) and the two-stage training recipe, providing insight into the trade-offs between reconstruction quality and token efficiency. The evaluation of the thinking mode shows a significant lift in mathematical reasoning, validating the effectiveness of the post-training pipeline. However, the paper acknowledges that specialized vision and audio models still outperform Apertus 1.5 on high-resource perception and transcription tasks, which is a fair and honest assessment of the current limitations.
Reproducibility
The paper excels in reproducibility, adhering to the "fully open" philosophy. All model weights, intermediate checkpoints, data preparation pipelines, training code, and evaluation harnesses are released under permissive licenses. The detailed documentation of the data mixture, including source versions and licenses, allows for full auditability. The release of the tokenizer benchmarks and the specific hyperparameters for each training stage further enhances the ability for other researchers to reproduce the results. The use of standard frameworks like Megatron-LM, with clearly documented modifications, lowers the barrier to entry for replication.
Limitations
The primary limitation is the performance gap with specialized vision and audio models on high-resource tasks. The discrete tokenization approach, while efficient, results in a coarser representation of images, which negatively impacts OCR and fine-grained detail tasks. The audio codec, while efficient, may not capture all nuances of high-quality audio, potentially limiting performance in specialized audio tasks. The paper also notes that the 70B model was trained on fewer tokens than intended due to resource constraints, which may affect its final performance relative to the 8B model.
Broader impact
The release of Apertus 1.5 has significant broader impact by providing a fully open alternative to proprietary multimodal models. This supports research and development in AI by enabling independent study, deployment, and modification of the models. The emphasis on open data and values aligns with ethical AI development practices, promoting transparency and accountability. The models' multilingual capabilities and cultural benchmarks contribute to more inclusive AI systems. The open release of the training infrastructure and data pipelines can serve as a reference for other institutions aiming to develop open multimodal models. The paper presents a comprehensive and well-executed approach to extending a text LLM to a multimodal one using discrete tokenization, resulting in a fully open model family that is competitive with other open models. The detailed methodology, extensive experiments, and high reproducibility make it a significant contribution to the field of multimodal LLMs.
Author of this paper? Send a correction.