Grasping in the Wild:

Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations

Intelligent manipulation benefits from the capacity to flexibly control an end-effector with high degrees of freedom (DoF) and dynamically react to the environment. However, due to the challenges of collecting effective training data and learning efficiently, most grasping algorithms today are limited to top-down movements and open-loop execution. In this work, we propose a new low-cost hardware interface for collecting grasping demonstrations by people in diverse environments. Leveraging this data, we show that it is possible to train a robust end-to-end 6DoF closed-loop grasping model with reinforcement learning that transfers to real robots. A key aspect of our grasping model is that it uses “action-view” based rendering to simulate future states with respect to different possible actions. By evaluating these states using a learned value function (Q-function), our method is able to better select corresponding actions that maximize total rewards (i.e., grasping success). Our final grasping system is able to achieve reliable 6DoF closed-loop grasping of novel objects across various scene configurations, as well as dynamic scenes with moving objects.


Latest version: arXiv:1912.04344 [cs.CV]

CAD Models for the 3D printed parts can be found: here

Sample data can be found: here (715MB). Full dataset can be found: here (75G).


1 Columbia University            2 Google


title={Grasping in the Wild: Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations},
author={ Song, Shuran and Zeng, Andy and Lee, Johnny and Funkhouser, Thomas},
journal={Robotics and Automation Letters},

Technical Summary Video (with audio)


We would like to thank Stefan Welker and Ivan Krasin for their help on designing the data collection device, Adrian Wong, Julian Salazar, and Sean Snyder for operational support, Chad Richards for helpful feedback on writing, and Ryan Hickman for managerial support. We are also grateful for financial support from Google and Amazon.


If you have any questions, please feel free to contact Shuran Song