A Technical Overview of Whisper: Developments in Automatic Speech Recognition
Introduced in 2022, “Whisper” represents a significant advancement in automatic speech recognition systems (ASR). Developed by a team of researchers, this system is based on an extensive and diverse dataset, consisting of 680,000 hours of multilingual and multitask supervised learning from the web. Whisper is designed to efficiently handle a range of linguistic challenges such as accents, background noises, and technical language. In addition, it enables transcription in multiple languages as well as translation of these languages into English.
The core of Whisper is an end-to-end encoder-decoder transformer, specifically designed for processing and converting audio data. Audio data is initially divided into 30-second segments and converted into log-Mel spectrograms. These are then fed into an encoder, which works in combination with a decoder. The decoder is trained to predict text transcriptions and uses special tokens to handle complex tasks such as language identification, phrase-level timestamps, multilingual transcription, and translation into English.
Compared to other ASR models that often rely on smaller and more specific audio-text training datasets, Whisper offers broader applicability. Despite its extensive and diverse training dataset, Whisper does not achieve peak performance in specialized areas such as the LibriSpeech benchmark. However, it shows impressive robustness in a variety of application scenarios, attributable to its comprehensive training base.
A significant portion of Whisper’s audio material, about one-third, is non-English. This allows the system to transcribe in the original language as well as translate into English, which has proven to be particularly effective for speech-to-text translations.
Whisper thus represents an important milestone in the development of ASR technologies and offers potential application possibilities for developers looking to integrate voice interfaces into their applications. Further information is available in the related study, the model description, and the code.