Once a solution for amateur, off-key singers on karaoke nights, Auto-Tune and other voice tuning software have progressed from the club to the recording studio but never beyond sounding artificial.
But researchers at Johns Hopkins have advanced this technology, making significant improvements to traditional voice tuning software's previous capabilities. More than just Auto-Tune on steroids, this innovation—called Diff-Pitcher—seamlessly corrects out-of-tune singing while maintaining the original vocal timbre and naturalness, expanding its possible applications beyond entertainment and the music industry and into the health care setting.
"Diff-Pitcher is a generative deep neural network that takes pitch-correction technology to a new level," said team member Jiarui Hai, a PhD student in the Whiting School of Engineering's Department of Electrical and Computer Engineering. "Its precision and control can not only help musical artists and producers but also open new possibilities in areas such as voice rehabilitation and assistive technologies."
Hai and lead researcher Mounya Elhilali, a professor in electrical and computer engineering, presented their new technology at the 2023 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics last fall.
In music, the ability of pitch-correcting software to adjust the sound of a voice or instrument to the desired pitch is crucial. Diff-Pitcher employs advanced algorithms to enhance the naturalness and quality of pitch correction, surpassing previous tools. Unlike older software that trained on pairs of corrected and original vocals, Diff-Pitcher analyzes the visual representation (spectrogram) of original vocals requiring correction. It identifies target notes, predicts needed adjustments, and transforms the corrected spectrogram into audio, achieving pitch correction in two easy steps.
According to Hai, Diff-Pitcher is notable not only for its ability to create authentic and controlled voices but also for the method used to achieve it.
"[The results sound] really natural, and unlike in older ways of fixing pitch, we can still regulate how high or low the voice goes," Hai said.
The researchers plan to continue to refine Diff-Pitcher to make voices sound even more natural and to extend the tool's capabilities beyond enhancing music production. They believe Dif-Pitcher has a wide range of potential applications, including aiding those undergoing voice rehabilitation, particularly those with vocal impediments.
"The technology could revolutionize a spectrum of speech-related disabilities," Hai said, "offering valuable support for post-laryngectomy patients and contributing to the voice rehabilitation of stroke victims."
