Pruning as Regularization: Sensitivity-Aware One-Shot Pruning in ASR
Abstract
We challenge the conventional view of neural network pruning as solely a compression technique, demonstrating that one-shot magnitude pruning serves as a powerful implicit regularizer for ASR. Using Whisper-small, we combine gradient- and Fisher-based sensitivity diagnostics with targeted, component-wise pruning. This reveals architectural asymmetries: decoder FFNs are pruning-fragile, whereas decoder self-attention and the last encoder layers contain redundancy that, when removed, improves generalization. Without fine-tuning, pruning 50% of decoder self-attention reduces WER by 2.38% absolute (20.44% relative) on LibriSpeech test-other; pruning the last four encoder layers at 50% instead yields a 1.72% absolute (14.8% relative) improvement. Gains persisted on Common Voice and TED-LIUM datasets. Beyond regularization benefits, our sensitivity-aware approach enables more aggressive one-shot compression. At 40% sparsity, where established global pruning approaches catastrophically fail, our method preserves near-baseline accuracy. This positions pruning as a first-class architectural design tool: knowing where to prune is as important as how much to prune.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a novel perspective on neural network pruning, demonstrating its potential as an implicit regularizer in ASR systems. The combination of sensitivity analysis and targeted pruning not only enhances model performance but also challenges traditional views on pruning as merely a compression technique.
Methodology
The methodology presented in this paper is robust, combining both first-order and second-order sensitivity analyses to inform pruning decisions. The authors effectively leverage these analyses to identify which components of the Whisper-small model are more sensitive to pruning, thus allowing for targeted and effective pruning that serves as an implicit regularizer. The approach of treating pruning not merely as a compression technique but as a means to improve generalization is innovative and well-justified through empirical evidence. However, the paper could benefit from a more detailed discussion on the theoretical implications of pruning as regularization.
Results & evaluation
The experiments conducted are thorough, utilizing multiple datasets (LibriSpeech, Common Voice, and TED-LIUM) to validate the effectiveness of the proposed pruning method across different acoustic conditions. The results are compelling, showing significant improvements in WER without the need for fine-tuning, which is a notable contribution to the field of ASR. The authors provide clear metrics and comparisons against standard pruning methods, demonstrating the advantages of their sensitivity-aware approach.
Reproducibility
While the paper provides a detailed description of the methodology and experimental setup, it lacks specific implementation details that would aid in reproducibility, such as hyperparameters used during pruning and the exact configurations of the Whisper-small model. Including a supplementary material or a link to a code repository would greatly enhance reproducibility.
Limitations
One limitation of the study is the focus on a single model architecture (Whisper-small), which may limit the generalizability of the findings to other architectures. Additionally, the paper does not address potential trade-offs between pruning for regularization and the risk of underfitting, especially in more complex models. The empirical results, while promising, may not be universally applicable across all ASR systems.
Broader impact
The findings of this research have significant implications for the design of neural network architectures in ASR and potentially other domains. By positioning pruning as a first-class architectural design tool, the work encourages practitioners to rethink model optimization strategies, potentially leading to more efficient and robust models. This could have broader applications in real-time speech recognition systems, where both accuracy and computational efficiency are critical. This paper presents a novel perspective on neural network pruning, demonstrating its potential as an implicit regularizer in ASR systems. The combination of sensitivity analysis and targeted pruning not only enhances model performance but also challenges traditional views on pruning as merely a compression technique.
Author of this paper? Send a correction.