2026 Theses Doctoral

Brain-Controlled Selective Hearing: Translating Neural Attention into Real-Time Perceptual Enhancement

Choudhari, Vishal

Understanding speech in environments with multiple competing talkers remains one of the most persistent challenges in auditory neuroscience and assistive hearing technology. Although modern hearing aids effectively suppress background noise, they lack access to the listener’s attentional intent and therefore amplify all sound sources indiscriminately. Auditory Attention Decoding (AAD), the extraction of a listener’s attentional focus from neural activity, offers a potential solution by enabling selective amplification of the attended talker. Despite extensive research demonstrating offline decoding feasibility, fundamental questions have remained unresolved: Can attention be decoded in realistic, dynamic listening environments? Can such decoding be implemented in real time to improve perception? And what levels of decoding accuracy and system speed are required to produce meaningful user benefit?

This dissertation addresses these questions through a series of complementary studies that move brain-controlled selective hearing from laboratory feasibility to translational validation. First, using intracranial electroencephalography (iEEG) recordings from neurosurgical participants, we investigated AAD in ecologically realistic acoustic scenes containing spatially moving, turn-taking conversations in the presence of everyday background noise. A brain-controlled system combining AAD with speaker-independent binaural speech separation was developed to preserve spatial cues while enhancing the attended talker. Importantly, the system incorporated dynamically estimated talker trajectory cues into the decoding framework, significantly improving attention decoding accuracy beyond spectro-temporal speech features alone. Behavioral and neural analyses demonstrated reliable decoding under dynamic conditions, improved intelligibility, and preservation of spatial fidelity in complex listening environments.

Second, we implemented a fully closed-loop, real-time brain-controlled hearing system using high-resolution iEEG. Neural signals were decoded to infer the attended conversation and dynamically modulate target-to-masker ratio. Across multiple experiments, the system significantly improved speech intelligibility, reduced listening effort (as indexed by pupil dilation), and was consistently preferred by participants. Critically, decoding accuracy predicted subjective benefit, establishing the first direct behavioral validation that real-time AAD can enhance perception.

Third, we examined how objective system parameters, including target-to-masker ratio, decoding accuracy, and response latency, map onto user experience in both normal-hearing and hard-of-hearing listeners. Through large-scale online psychoacoustic experiments, we quantified trade-offs between intelligibility and switching ease and showed that the benefits of AAD depend on accuracy, speed, hearing status, and scene difficulty. Notably, normal-hearing listeners expressed a preference for selective hearing comparable to that of hard-of-hearing listeners.

Together, these findings establish the neural feasibility, real-time viability, and user benefits of brain-controlled selective hearing. By integrating neuroscience, engineering, and human factors, this work provides a translational framework for the development of next-generation assistive and augmented auditory technologies that adapt to listener intent in real-world environments.