Tracking non-rigid objects using depth camera

Surface of non-rigid objects, such as leafy vegetables, meats and fishes can be completely and accurately tracked using a depth camera with the approach proposed in this article. This is exceptionally useful while interacting with these objects using a robot.  

Automating the handling of meats, vegetables, seafoods and other delicate and irregularly shaped consumables has been a challenging problem for the food processing industry. Using the recent advances in robotic gripper technology, it could be possible to use robots for sorting fruits, vegetables and fresh produce. It can also be used for cutting and slicing seafood and meat.  

However, this can only be possible if the robotic system manages to observe and localize the shape and surface of the object it intends to manipulate. This can be done using state-of-the-art computer vision techniques along with the aid of some of the latest depth sensors. One such approach is proposed here, which tracks the surface of deforming objects using an approximate CAD model.   


The term ‘tracking’ is being used in this context to mean spatio-temporal position of the visible surface of the object. The approximate CAD model of the object being tracked is assumed to be known. The food object, while being processed, will be observed using a RGB-D camera. This camera provide color and depth information about the scene. This depth data is utilized to fit the deformed CAD model, such that the entire observable surface of the object remains accountable.  

Robotic manipulation of food objects is an interesting application in the food-processing industry. If robots are tasked with cutting, slicing, chopping or deboning of food objects, it is important to enable tracking of the surface of the substance that is being manipulated. The knowledge about this tracking information allows the robot to independently plan where and how it can grasp or cut the object to bring about the desired deformation.  


Research has  shown two different examples of output from the proposed system. The results are demonstrated on a banana and a pizza. The banana was tracked using the rigid object tracking mechanism. The model used for tracking the banana was obtained from manual measurements. Despite this, the tracking was quite accurate. Pizza is a non-rigid object that was deformed by a large amount. The tracking was accurate and consistent. The model deforms and closely follows the surface of the pizza as it deforms. Once the deformed surface gets tracked, the output of the tracking algorithm can be used for many robotic applications such as cutting, grasping, squeezing or picking up any generic deformable objects. Most food objects, including leafy vegetables, meats and fishes, are inherently and extremely deformable. Among other practical applications, deformable surface tracking can also be utilized for augmented reality (AR) or robotic surgeries.  

The efficacy of the proposed method has been tested on real and simulated objects and the tracking accuracy is consistently adequate. However, some work can be done towards making the system more robust. It can also be optimized for performing faster.  









Tracking of a pizza being deformed



















By: Alexander Krupa and Francois Chaumette