University of Washington researchers have developed smart headphones which use artificial intelligence to learn who the wearer is listening to so they can hear them better, while muting background noise.
The prototype system, dubbed “proactive hearing assistants,” includes headphones that proactively isolate up to four of the wearer’s conversation partners in a noisy soundscape, the University of Washington (UW) reported on its website.
The prototype headphones are powered by an AI model that detects the cadence of a conversation and another model that mutes any voices which don’t follow that pattern, along with other unwanted background noises, the university reported.
The current prototype uses off the shelf hardware – commercial over-the-ear headphones, microphones and circuitry, the developers said. It can identify conversation partners using two to four seconds of audio.
Its creators said, in future, the technology may help users of hearing aids, earbuds and smart glasses to filter their soundscapes without the need to manually direct the AI’s “attention.”
Available for download
The team presented the system on 7 November at the Conference on Empirical Methods in Natural Language Processing in Suzhou, China. The underlying code is open-source and available for download.
“Existing approaches to identifying who the wearer is listening to predominantly involve electrodes implanted in the brain to track attention,” said Professor Shyam Gollakota on the UW website.
“Our insight is that when we’re conversing with a specific group of people, our speech naturally follows a turn-taking rhythm. And we can train AI to predict and track those rhythms using only audio, without the need for implanting electrodes.”
Prof Gollakota, from UW’s Paul G. Allen School of Computer Science & Engineering, is the senior author.
The system activates when the person wearing the headphones starts speaking. One AI model starts tracking conversation participants by performing a “who spoke when” analysis and looking for low overlap in exchanges.
It forwards the result to a second model which isolates the participants and plays a clean audio version for the wearer.
Tests on 11 participants, who rated noise suppression and comprehension with and without the AI filtration, found that, overall, the group rated the filtered audio more than twice as favourably as baseline audio.
The team previously developed a smart headphone prototype that can pick a person’s audio out of a crowd when the wearer looks at the person. They also designed a prototype that creates a “sound bubble” by mutating all sounds within a set distance of the wearer.
“Everything we’ve done previously requires the user to manually select a specific speaker or a distance within which to listen, which is not great for user experience,” said lead author Mr Guilin Hu, a doctoral student in the Allen School.
“What we’ve demonstrated is a technology that’s proactive – something that infers human intent noninvasively and automatically.”
Prof Gollakota said more refinement was needed including to prevent it struggling when participants talked over one another or spoke in longer monologues. Models were tested on English, Mandarin and Japanese dialogue so rhythms of other languages might require further fine-tuning, he said.
He hopes to make the system small enough to run on a chip in an earbud or hearing aid. Co-authors include doctoral students Tuochao Chen and Malek Itani.
