Multilingual Long-Form Speech Instruction Following: KIT's Submission to IWSLT 2026
Abstract
With the advent of Large Language Models, single-task and token-based multi-task models have evolved into instruction-based systems that infer task and target language implicitly from natural language prompts. This trend is reflected in IWSLT's Instruction Following Track, which this year introduced new tasks including an unknown surprise task, posing a genuine challenge against overfitting to known tasks. We present KIT's submission to the Long and Short Instruction Following tracks in the unconstrained setting. Our approach combines a general data augmentation pipeline that converts short-form corpora into long-form training data through segment concatenation, LLM-based label generation, and cross-lingual translation, yielding over 1M instances across six tasks and four languages. We further show that likelihood-based re-ranking, while highly effective for ASR, systematically degrades semantic tasks by spuriously selecting candidates generated from segmented audio processing rather than holistic long-form inference, a failure mode resolved by combining likelihood with Minimum Bayes Risk decoding.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of a novel data augmentation framework that enables effective long-form speech instruction following across multiple languages and tasks. This work significantly advances the state of the art in multilingual speech processing and provides valuable insights into the challenges and solutions for handling long-form audio data in instruction-following systems.
Methodology
The paper presents a comprehensive data augmentation framework that effectively transforms short-form speech datasets into long-form instruction-following training data. The methodology includes segment concatenation, LLM-based label generation, and cross-lingual translation, which collectively yield a significant dataset of over 1M instances across multiple languages and tasks. The empirical evaluation of different sampling strategies (fixed-probability vs. temperature-scaled) and the exploration of re-ranking strategies under realistic constraints demonstrate a thoughtful and systematic approach to addressing the challenges of long-form audio processing.
Results & evaluation
The experiments are well-structured, utilizing a variety of datasets for different tasks, including ASR, ST, SQA, SSUM, and more. The results indicate that the proposed methods yield substantial improvements in performance, particularly in ASR and SQA tasks. The paper includes a thorough analysis of the results, comparing different models and configurations, and highlights the effectiveness of the combined Likelihood+MBR re-ranking strategy. However, the paper could benefit from more detailed quantitative results for each language and task combination.
Reproducibility
The authors provide sufficient details regarding the training setup, data preparation, and evaluation metrics, which supports reproducibility. However, the lack of a publicly available code repository limits the ease with which other researchers can replicate the experiments. The authors do release their dataset, which is a positive aspect for the community.
Limitations
One limitation of the study is the potential for overfitting due to the reliance on existing datasets and the challenge of generalizing to unseen tasks, particularly the surprise task. Additionally, the performance variations across languages suggest that the model may not be equally effective for all target languages, which could hinder its applicability in multilingual settings.
Broader impact
The work has significant implications for the development of multilingual instruction-following systems, particularly in the context of long-form audio processing. The proposed methods could enhance accessibility and usability in various applications, such as automated transcription services, language translation, and interactive voice assistants. The release of the dataset and the findings from the experiments could inspire further research in the field, particularly in improving the robustness and generalization of models across diverse tasks and languages. The main contribution of this paper is the development of a novel data augmentation framework that enables effective long-form speech instruction following across multiple languages and tasks. This work significantly advances the state of the art in multilingual speech processing and provides valuable insights into the challenges and solutions for handling long-form audio data in instruction-following systems.
Author of this paper? Send a correction.