Transformer Redesign for Late Fusion of Audio-Text Features on Ultra-Low-Power Edge Hardware
Abstract
Deploying emotion recognition systems in real-world environments where devices must be small, low-power, and private remains a significant challenge. This is especially relevant for applications such as tension monitoring, conflict de-escalation, and responsive wearables, where cloud-based solutions are impractical. Multimodal emotion recognition has advanced through deep learning, but most systems remain unsuitable for deployment on ultra-constrained edge devices. Prior work typically relies on powerful hardware, lacks real-time performance, or uses unimodal input. This paper addresses that gap by presenting a hardware-aware emotion recognition system that combines acoustic and linguistic features using a late-fusion architecture optimised for Edge TPU. The design integrates a quantised transformer-based acoustic model with frozen keyword embeddings from a DSResNet-SE network, enabling real-time inference within a 1.8MB memory budget and 21-23ms latency. The pipeline ensures spectrogram alignment between training and deployment using MicroFrontend and MLTK. Evaluation on re-recorded, segmented IEMOCAP samples captured through the Coral Dev Board Micro microphone shows a 6.3% macro F1 improvement over unimodal baselines. This work demonstrates that accurate, real-time multimodal emotion inference is achievable on microcontroller-class edge platforms through task-specific fusion and hardware-guided model design.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a novel approach to multimodal emotion recognition on ultra-low-power edge devices, combining acoustic and linguistic features through a late-fusion architecture. The methodology is well-defined, and the results demonstrate a meaningful advancement in the field, although further validation and resource sharing would enhance its impact.
Methodology
The paper presents a well-structured methodology for late fusion of audio and text features tailored for ultra-low-power edge devices. The integration of a quantised transformer-based acoustic model with a lightweight keyword spotting model is innovative, particularly in the context of real-time emotion recognition. The use of MicroFrontend and MLTK for spectrogram alignment demonstrates a thoughtful approach to ensuring consistency between training and deployment, which is critical for performance in edge environments. The late-fusion architecture is a significant contribution as it allows for the combination of multimodal data while adhering to strict resource constraints.
Results & evaluation
The experimental setup is robust, utilizing the IEMOCAP dataset for evaluation, which is appropriate for emotion recognition tasks. The reported 6.3% macro F1 improvement over unimodal baselines indicates a meaningful enhancement in performance, showcasing the effectiveness of the proposed architecture. However, the paper could benefit from a more extensive evaluation across diverse datasets and real-world conditions to fully validate the model's robustness and generalizability.
Reproducibility
The paper provides a clear description of the model architecture and training procedures, which aids in reproducibility. However, the absence of a publicly accessible code repository or demo limits the ability for other researchers to replicate the results fully. Including such resources would enhance the paper's impact and utility in the research community.
Limitations
The paper acknowledges limitations regarding robustness to varied environmental conditions and speaker diversity, which are critical factors in real-world applications. The focus on a specific dataset (IEMOCAP) may also limit the generalizability of the findings. Future work is suggested to address these limitations, but they remain a significant concern in the current study.
Broader impact
The proposed system has significant implications for privacy-preserving emotion recognition in wearable technology and other edge applications. By enabling real-time processing on low-power devices, the research opens avenues for practical applications in mental health monitoring, conflict resolution, and responsive devices, contributing to the growing field of ethical AI. This paper presents a novel approach to multimodal emotion recognition on ultra-low-power edge devices, combining acoustic and linguistic features through a late-fusion architecture. The methodology is well-defined, and the results demonstrate a meaningful advancement in the field, although further validation and resource sharing would enhance its impact.
Author of this paper? Send a correction.