2025 Theses Doctoral

Diversifying Data for Learning Contact-Rich Manipulation: Sensors, Kinematics, and Human Factors

He, Zhanpeng

Despite advancements in robot learning demonstrating diverse capabilities across various hardware platforms, many systems remain limited to pick-and-place style tasks, characterized by low precision and high compliance. These tasks typically do not require the policy to reason about contact between the object and its environment, which limits the generalizability of such methods to more complex settings where contact reasoning is essential. This limitation stems from two main factors: (1) kinematics – many systems use manipulators limited to simple grasping motions and hence can only obtain action data of simple interactions; and (2) sensing – their hardware platforms often lack contact sensing capabilities and fail to provide enough information in the observations. Furthermore, policies learned by these systems often exhibit low success rates, limiting their practical utility in real-world deployments. A common challenge is distribution shift—a mismatch between the training data and deployment conditions—which undermines the performance of data-driven methods in uncontrolled environments.

To address these challenges, this dissertation investigates several key aspects critical to enabling more effective data collection for contact-rich manipulation tasks, ultimately improving policy deployment performance. We begin by introducing novel policy learning pipelines to learn an extremely difficult task--delicate object in-hand rotation. Our proposed learning pipeline starts with an imitation learning step followed by an off-policy on-robot RL fine-tuning using semi-sparse rewards. To achieve this, we use a versatile dexterous robotic hand, ROAMHand3, which is equipped with a multimodal tactile sensor, SpikeATac. This shows the potential of dexterous manipulation using robots that are carefully designed by humans. Next, we explore how to directly optimize hardware configurations for task performance using reinforcement learning, enabling co-design of morphology and control. Finally, we present a framework for leveraging human assistance efficiently during policy deployment, demonstrating how minimal but strategic human interventions can significantly enhance real-world success rates.

Through these proposed methods, we argue that data for contact-rich manipulation tasks can be improved along multiple dimensions and they are as important as aspects (e.g., scene, object and task diversity) that other works emphasize. Enhanced sensing capabilities enable more robust performance under environmental perturbations by providing richer and more reliable feedback. Task-optimized kinematics -- achieved through joint design and control optimization -- expand the robot’s reachable workspace and facilitate different interaction modes with diverse objects. Finally, human-in-the-loop policies provide targeted corrections in failure-prone states, enabling the system to recover from suboptimal behaviors and adapt more effectively during deployment.