This last week, I dug up my trustworthy Kinect for a spin. I’ve been wanting to mess around with the PCL (Point Clouds) library for some time, so I decided to give it a shot.
Installation on OSX using Homebrew is fairly straigthforward, as shown in their documentation. However, I want to make sure that I have support for the Kinect (the Xbox 360 model).
Side note on Kinect support: To get data off of your Kinect, you can use the OpenNI library (which handles Kinect “1”). OpenNI2 does exist, but it handles only the Kinect “2” and Occipital’s Structure Sensor. I’ll be using OpenNI here, because it’s supported directly by PCL. However, for standalone applications, I’d greatly recommend libfreenect (also available through Homebrew), which is fast, lightweight, and very easy to use.
So, we’ll be using Homebrew to get the following libraries: Boost, Flann, Qt4, GLEW, VTK and last but not least, OpenNI (1 and 2, just for argument’s sakes). Run:
brew update
brew install boost flann qt glew vtk openni openni2
Grab a coffee, ’cause this will take some time. Now, to install PCL, you may want to check the available options (brew options pcl), then install it with at least the following settings:
brew install pcl --with-openni --with-openni2
Great, if nothing intensely wrong happened midway, you should be golden. Now, at that point I was eager to get some Kinect-demo up an running, which led me to PCL’s openni-grabber page (go ahead, visit the page). They are kind enough to supply the C++ file and a CMakeLists.txt to compile it. It compiled fine, but crashed hard every time I tried to use it, spitting the message:
Assertion failure in +[NSUndoManager _endTopLevelGroupings], /Library/Caches/com.apple.xbs/Sources/Foundation/Foundation-1256.1/Misc.subproj/NSUndoManager.m:359
2016-01-06 12:50:51.779 openni_grabber[42988:7013866] +[NSUndoManager(NSInternal) _endTopLevelGroupings] is only safe to invoke on the main thread.
Now, I don’t know if this happens under other platforms. After wasting some days struggling with this, I realized what the heck was going on. Let’s take a look at the non-working supplied code. In the header section, we see:
|
1 2 |
#include <pcl/io/openni_grabber.h> #include <pcl/visualization/cloud_viewer.h> |
It uses the PCL’s Cloud Viewer, which has been long deprecated (meaning that this example is very out-of-date). Secondly, we see that the grabber’s functionality is based on a callback, defined here:
|
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
void cloud_cb_ (const pcl::PointCloud<pcl::PointXYZ>::ConstPtr &cloud) { if (!viewer.wasStopped()) viewer.showCloud (cloud); } void run () { pcl::Grabber* interface = new pcl::OpenNIGrabber(); boost::function<void (const pcl::PointCloud<pcl::PointXYZ>::ConstPtr&)> f = boost::bind (&SimpleOpenNIViewer::cloud_cb_, this, _1); interface->registerCallback (f); interface->start (); |
The cloud_cb_ callback function is called every time a new data packet arrives from the Kinect. This is fine, but the showCloud() command updates the display of the point cloud from within the callback, witch is what’s creating our "...is only safe to invoke on the main thread..." error. To fix that, the callback should only update the cloud’s data, while the cloud itself must be displayed with a call placed on the main thread. The fixed code I came up with looks like this:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 |
#include <boost/thread/thread.hpp> #include <pcl/common/common_headers.h> #include <pcl/io/pcd_io.h> #include <pcl/io/openni_grabber.h> #include <pcl/visualization/pcl_visualizer.h> /** pthread_mutex_t cpp wrapper */ class Mutex { public: Mutex() { pthread_mutex_init(&m_mutex, NULL); } void lock() { pthread_mutex_lock(&m_mutex); } void unlock() { pthread_mutex_unlock(&m_mutex); } class ScopedLock { public: ScopedLock(Mutex &mutex) : _mutex(mutex) { _mutex.lock(); } ~ScopedLock() { _mutex.unlock(); } private: Mutex &_mutex; }; private: pthread_mutex_t m_mutex; }; /** class to handle the drawing and openni callbacks */ class SimpleOpenNIViewer { public: boost::shared_ptr<pcl::visualization::PCLVisualizer> viewer; Mutex * mutex; SimpleOpenNIViewer(): viewer(new pcl::visualization::PCLVisualizer ("3D Viewer")), mutex(new Mutex) {} /** OpenNI callback. Each time a data pack arrives, the Mutex is locked, * and the cloud is updated. */ void cloud_cb_ (const pcl::PointCloud<pcl::PointXYZ>::ConstPtr &cloud) { mutex->lock(); viewer->updatePointCloud(cloud, "kinect_cloud"); mutex->unlock(); } void run () { // Start off by creating an empty point cloud (with support for depth (XYZ) and color (RGBA))) pcl::PointCloud<pcl::PointXYZRGBA>::Ptr kinect_cloud_ptr (new pcl::PointCloud<pcl::PointXYZRGBA>); // Add the empt cloud to the viewer. The cloud will be updated with every openni callback viewer->addPointCloud<pcl::PointXYZRGBA> (kinect_cloud_ptr, "kinect_cloud"); viewer->setPointCloudRenderingProperties (pcl::visualization::PCL_VISUALIZER_POINT_SIZE, 3, "kinect_cloud"); // Create the openni interface, and set cloud_cb_ as the callback pcl::Grabber* interface = new pcl::OpenNIGrabber(); boost::function<void (const pcl::PointCloud<pcl::PointXYZRGBA>::ConstPtr&)> f = boost::bind (&SimpleOpenNIViewer::cloud_cb_, this, _1); // Start the interface interface->registerCallback (f); interface->start (); while (!viewer->wasStopped()) { // Aquire the mutex, to make sure that the cloud isn't being updated mutex->lock(); // Redraw cloud viewer->spinOnce (20); // Release mutex mutex->unlock(); } interface->stop (); } }; int main () { SimpleOpenNIViewer v; v.run (); return 0; } |
Now, the callback cloud_cb_ only updates the cloud’s data, not touching the UI. The drawing happens in a loop inside the SimpleOpenNIViewer::run(). Since the callback and the loop are handling the same data set asynchronously, I’ve added a mutex around the critical sections to avoid any issues (the nice CPP wrapper stolen from libfreenect’s examples, thanks!). To compile it, the following CMakeLists.txt should do the trick:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
cmake_minimum_required(VERSION 2.8 FATAL_ERROR) project(openni_grabber) find_package(PCL 1.2 REQUIRED) aux_source_directory(. SRC_LIST) include_directories(${PCL_INCLUDE_DIRS}) link_directories(${PCL_LIBRARY_DIRS}) add_definitions(${PCL_DEFINITIONS}) add_executable (openni_grabber openni_viewer_simple.cpp) target_link_libraries (openni_grabber ${PCL_LIBRARIES}) |
The results are not astounding: the RGB data is slightly out of place with the depth data, but I believe this is a problem internal to PCL, since my Kinect is properly calibrated. To use only the depth data in the code above, simply replace all instances of pcl::PointXYZRGBA with pcl::PointXYZ.
’til next time!