2026 Theses Doctoral

Bridging AI and the Physical World: Embedded Sensing and Embodied Drone Agents for Intelligent Environments

Zhao, Minghui

We are seeing a massive transformation in how we process information. Since the introduction of large language models (LLMs) and multimodal large language models (MLLMs), there has been massive growth in AI's capabilities in reasoning, content generation and decision-making. These advancements, together with a surge of research in software tools and agentic systems, have boosted human productivity to an unprecedented level. These advanced models and techniques, however, are still largely disconnected from the physical world. Our everyday living environment is far from being truly intelligent. In this thesis, we present architectures, systems and methods that build upon mobile, resource-constrained platforms to enable AI that actively participates in and improves our daily lives.

We first explore the idea of mobile robotics as an alternative to static sensor deployments by using drones as intrinsic environmental sensors. We apply this idea to the challenge of environmental monitoring. Mapping 3D airflow fields is important for many HVAC, industrial, medical, and home applications. However, current approaches are expensive and time-consuming. We present Anemoi, a sub-$100 drone-based system for autonomously mapping 3D airflow fields in indoor environments. Anemoi leverages the effects of airflow on motor control signals to estimate the magnitude and direction of wind at any given point in space. We introduce an exploration algorithm for selecting optimal waypoints that minimize overall airflow estimation uncertainty. We demonstrate through microbenchmarks and real deployments that Anemoi can estimate wind speed and direction with errors up to 0.41 m/s and 25.1 degrees lower than the existing state-of-the-art and map 3D airflow fields with an average RMS error of 0.73 m/s.

Anemoi demonstrates that indoor drones can serve as highly capable autonomous agents toward embodied AI for smarter environments. To take this idea further and deploy drones as persistent physical agents, we must address two fundamental challenges: continuous operation and adaptable sensing.

Due to a drone's limited payload capacity and battery life, its continuous operation requires frequent automated docking to recharge. However, existing automated drone landing approaches often require complex hardware setup, heavy computation, or lack reliability, which make them not suitable for complex indoor environments and impractical for resource-constrained platforms. We propose Moth, a low-cost, infrared light-based solution that targets precise and efficient landing of low-resource micro-drones. Moth consists of an infrared light source at the landing station and an energy-efficient photodiode (PD) sensing platform attached to the bottom of the drone. At a cost under 83 USD, Moth achieves comparable performance to vision-based methods but at a fraction of the energy consumption and computation. Moth requires only three PDs without any complex pattern recognition models to land the drone accurately, under 10 cm of error, from up to 11.1 m away.

While Moth enables the continuous operation of these drone agents, they must also be capable of perceiving and retrieving context from their surrounding environments. To address this second challenge of adaptable sensing, we introduce LegoSENSE, a low-cost open-source and modular platform that simplifies the rapid deployment of customized sensing solutions for AI-driven Internet of Things (IoT) applications. Currently, deploying these sensing systems requires significant amount of work and time, even for experienced engineers. Built on top of the widely popular Raspberry Pi single-board computer, LegoSENSE overcomes these barriers through a multiplexed carrier board and hot-pluggable sensor modules. We designed hardware and software to automatically identify the sensor modules, load required drivers without manual user configuration, and make the data seamlessly accessible via web dashboards and APIs. LegoSENSE's "plug-and-play" functionality makes the sensor modules reusable, and allows them to be mixed and matched to serve a variety of applications and scenarios that align with the task-agnostic and flexible capabilities of modern AI. We show, through a series of user studies, that LegoSENSE enables users without engineering background to deploy a wide range of applications up to 9x faster than experienced engineers without the use of LegoSENSE.

With the barriers of continuous operation and adaptable sensing addressed, we can finally combine the mobile capabilities of drones with the agentic power of modern foundation models (FMs). Building on our prior works, we introduce EmbodiedFly, an embodied LLM agent combining a foundation model pipeline with a reconfigurable drone platform to observe, understand, and interact with the physical world. Our co-design approach features: 1) an FM orchestration framework connecting multiple large language models (LLMs), vision language models (VLMs), and an open-set object detection model; 2) a novel image segmentation technique that identifies task-relevant areas; and 3) a custom drone platform that autonomously reconfigures with appropriate sensors and actuators based on commands from the FM orchestration framework.

Through real-world deployments, we demonstrate that EmbodiedFly completes diverse physical tasks with up to 85% higher success rate compared to traditional approaches leveraging static deployments.

This dissertation provides a comprehensive framework for building next-generation intelligent environments by using drones as the agile, physical extensions of modern FMs. By replacing rigid static sensing infrastructure with autonomous, modular mobile embedded platforms, and coupling them with modern agentic AI frameworks, we establish a scalable pathway that seamlessly bridges AI digital space into embodied, physical action to actively participate and improve our daily lives.