2026 Theses Doctoral

Multimodal Physiological and Behavioral Signatures of Human Decision-Making and Team Performance

Qin, Yinuo

Understanding how humans make decisions and coordinate with others in complex, dynamic environments remains a central challenge across neuroscience, cognitive science, human–computer interaction, and artificial intelligence. While prior research has extensively examined isolated decision-making and dyadic interaction, significantly less is known about how cognitive, behavioral, and physiological processes unfold in multi-human teams and human–AI collectives performing embedded decision-making tasks. This dissertation addresses this gap by investigating the multimodal physiological and behavioral signatures that underlie individual decision strategies, collective coordination, and performance outcomes in immersive interactive environments.

This work focuses on three interconnected problems. First, it examines how individuals form decisions under uncertainty, including conditions of risk and ambiguity, and how latent belief models shape observable behavior and internal physiological responses. Second, it investigates how multiple humans coordinate perception, action, and communication while jointly solving sensorimotor and decision-making tasks. Third, it explores how human teams adapt when collaborating with artificial agents, including scenarios in which AI behavior is experimentally controlled using Wizard-of-Oz paradigms. Across these domains, the dissertation seeks to identify measurable physiological markers that reveal otherwise hidden cognitive states, interaction dynamics, and performance-relevant mechanisms.

To enable systematic study of these questions, this research develops novel virtual reality (VR) environments that balance ecological validity with experimental control. Participants engaged in immersive, single- or multi-participant tasks requiring real-time decision-making, coordination, and adaptation under varying levels of difficulty and uncertainty. These environments supported the simultaneous acquisition of multimodal data streams, including pupillometry as an index of arousal and cognitive effort, electroencephalography (EEG) capturing neural dynamics, speech reflecting communication processes, and detailed behavioral measures of action and performance. The methodological framework integrates experimental design, signal processing, computational modeling, and machine learning to quantify interactions across temporal, behavioral, and neurophysiological scales.

A central contribution of this dissertation is the demonstration that physiological dynamics provide critical insight into cognitive and collective processes beyond what is observable from behavior alone. Across experimental paradigms, pupil-linked arousal and neural activity reveal latent decision variables, uncertainty processing, and strategy differences that are not fully captured by overt actions. The integration of multimodal physiological and behavioral signals establishes a framework in which internal cognitive states can be quantified, modeled, and related to performance. This approach enables the development of physiologically informed biomarkers that link neural and autonomic dynamics to decision-making and task effectiveness.

At the level of team interaction, this work advances a multimodal perspective on coordination and collective performance, demonstrating that behavioral synchrony alone provides an incomplete characterization of team effectiveness, whereas joint analysis of behavioral and physiological coupling reveals richer structures of collaboration. Predictability metrics derived from multimodal signals offer a novel lens for understanding how coordination emerges and how future performance can be anticipated, emphasizing that successful teaming arises from dynamic interactions among arousal, neural processes, and shared task engagement rather than solely from explicit communication or action alignment. Extending this framework to human–AI teams, the dissertation challenges the assumption that increasing AI capability will uniformly enhance team outcomes or reduce individual stress, showing instead that the presence of artificial agents fundamentally reshapes human cognitive load, trust calibration, decision strategies, and coordination dynamics. The findings highlight that team performance depends not only on AI competence but also on how humans perceive, interpret, and adapt to AI behavior. Purely advancing AI systems cannot by itself resolve the complexities of collaboration without a complementary understanding of human neurophysiological and behavioral responses.

This dissertation further establishes that team structure and individual cognitive strategies fundamentally shape collective dynamics. Teams composed of three members exhibit qualitatively different and more diverse interaction patterns compared to dyadic teams, reflecting increased coordination complexity and distributed dependencies. In these settings, communication assumes a distinct role within a broader network of behavioral and physiological interactions. Additionally, individuals display stable yet heterogeneous belief-dependent decision strategies under uncertainty, with physiological responses tracking subjective valuation and internal belief representations. Together, these findings contribute a unified conceptual and methodological framework demonstrating that multimodal physiological and behavioral signatures provide powerful tools for understanding, predicting, and modeling human decision-making and team performance across human-only and human–AI interactive systems.