Reiter Austin David author Columbia University. Computer Science Allen Peter K. author Columbia University. Computer Science Columbia University. Computer Science originator text Articles 2010 English In this paper we present an online algorithm for robustly tracking surgical tools in dynamic environments that can assist a surgeon during in-vivo robotic surgery procedures. The next generation of in-vivo robotic surgical devices includes integrated imaging and effector platforms that need to be controlled through real-time visual feedback. Our tracking algorithm learns the appearance of the tool online to account for appearance and perspective changes. In addition, the tracker uses multiple features working together to model the object and discover new areas of the tool as it moves quickly, exits and re-enters the scene, or becomes occluded and requires recovery. The algorithm can persist through changes in lighting and pose by using a memory database, which is built online, using a series of features working together to exploit different aspects of the object being tracked. We present results using real in-vivo imaging data from a human partial nephrectomy. Robotics Piscataway, N.J. IEEE 2010 3441 3446 http://dx.doi.org/10.1109/IROS.2010.5650852