2025 Theses Doctoral

Reducing Label Dependence in Animal Behavior Modeling: Diverse Supervision Strategies for Improved Generalization

Blau, Ari

Understanding and modeling animal behavior is a central goal in behavioral neuroscience and computational ethology. With advances in imaging and tracking technologies, the field hasentered a new era of high-dimensional spatiotemporal data. However, the ability to provide high-quality labels has not—and inherently cannot—scale at the same pace. This growing gap underscores the need for diverse forms of supervision to extract meaningful insights from increasingly complex behavioral datasets.

This dissertation presents a series of computational frameworks for representing and segmenting animal behavior, progressing from pose estimation to structured temporal modeling, with a focus on leveraging supervised, semi-supervised, and unsupervised methods.

The first part of this dissertation focuses on semi-supervised keypoint estimation in multi-animal settings. In Chapter 2, we introduce SemiMultiPose, a semi-supervised model for multi-animal pose estimation that combines supervised keypoint annotations with unsupervised losses on unlabeled video data. By leveraging the structure inherent in pose-based representations, this framework extracts meaningful intermediate features from raw video, even in the absence of dense labels. This approach enables reduced supervision requirements and improved generalization in complex behavioral settings, demonstrating the value of incorporating unlabeled frames into the training process.

Chapter 3 presents a comparative analysis of action segmentation frameworks under different supervision regimes. Action segmentation involves classifying discrete animal behaviors over time based on spatiotemporal features extracted from video. We evaluate supervised, unsupervised, and semi-supervised methods on multiple benchmark datasets, highlighting their trade-offs in performance, data efficiency, and interpretability. This study underscores the importance of temporal structure and representation learning in developing scalable, accurate models of behavior.

The third part of the dissertation, presented in Chapter 4, focuses on transformer-based behavior segmentation. We adapt vision transformers (ViTs) for behavioral video analysis by first extracting unsupervised frame-level representations from raw video using a pretrained ViT backbone. These embeddings are then used as inputs for action segmentation models, leveraging the temporal modeling tools developed in Chapter 3. We also explore combining these video-derived features with structured pose representations to improve segmentation performance. This work, part of a collaborative project, demonstrates the effectiveness of ViT backbones in segmenting behavior from both raw video and pose data.

Chapter 5 turns to the problem of encoding behavior into neural activity. Building on the representations developed in earlier chapters—continuous kinematic features from Chapter 2 and discrete behavioral states from Chapter 3—we examine how these different forms of behavioral abstraction are reflected in neural population activity. We develop and evaluate encoder models that map these distinct behavioral representations into neural space, with a focus on how the structure of the input—symbolic versus metric—shapes the geometry, predictability, and biological interpretability of the resulting neural codes.

Together, these chapters address the central challenge of extracting meaningful insights from large volumes of high-dimensional spatiotemporal data. By developing behavior modeling methods under varying levels of supervision, this work shows how structured representations can emerge even when labeled data are limited. Finally, neural encoding models provide a framework for probing how these learned representations—both continuous and discrete—are reflected in neural activity, offering a principled lens into the brain-behavior relationship.