NtKinect: Kinect V2 C++ Programming with OpenCV on Windows10

How to recognize gesture with Kinect V2

Prerequisite knowledge

• NtKinect: How to recognize human skeleton with Kinect V2

Recognize Gesture with Visual Gesture Builder file

With the Visual Gesture Builder by Microsoft, you can generate gesture classifier by machine learning. The extension of the generated file is ".gbd" or ".gba".

In Visual Gesture Builder, there is more than one project in one solution. When you build each project you get a ".bga" file, and when you build the whole solution you get a ".bgd" file. That is, the extension of the individual gesture classifier file is ".gba", and that of the file gathering multiple classifiers is ".gbd".

Gesture recognition of Visual Gesture Builder is defined in Kinect for Windows SDK 2.0 as follows.

typedef enum _GestureType GestureType;
  enum _GestureType {
  GestureType_None	= 0,
  GestureType_Discrete	= 1,
  GestureType_Continuous	= 2
};

There are two kinds of gestures: "Discrete" and "Continuous", with the former "confidence" and the latter with "progress" value added. Each value is from 0.0 to 1.0.

The discrete gesture is determined by the AdaBoost algorithm and the result is returned as a boolean value. You can change the judgement criterion by referring to the value of "confidence".

The continuous gesture is determined by the Random Forest algorithm , and the result is a progressive result in real number. This value is "progress".

If you define USE_GESTURE constant before including NtKinect.h, the functions and variables of NtKinect for gesture recognition becom effective.

You can use the classifier file ".gbd" and ".gba" generated by Visual Gesture Builder to recognize gestures with Kinect V2.

NtKinect's function for Gesture Recognition

NtKinect's member variable for Gesture Recognition

If you want to know a skeleton such as "who do the gesture?", you can judge it using the following expression.

In case of discrete gesture:
    kinect.discreteGestureTrackingId[i] == kinect.skeletonTrackingId[j] 
In case of continuous gesture:
    kinect.continuousGestureTrackingId[i] == kinect.skeletonTrackingId[j] 

How to write program

1. Start using the Visual Studio's project KinectV2_skeleton.zip of "NtKinect: How to recognize human skeleton with Kinect V2" .
2. Add some settings to the project's property.
1. Open the properties panel.

Click the project name (KinectV2 in this example) in Solution Explorer to select, right-drag and select Properties in the menu.

2. Set up to link libraries for Visual Gesture Builder.

Kinect20.VisualGestureBuilder.lib





3. Set the DLL and other file for the Visual Gesture Builder and gesture classifier file to be copied into the appropriate folder after the compilation.

"Configuration Properties" -> "Build Event" -> "Post Build Event" -> "Command Line". Add the next two lines.

xcopy "$(KINECTSDK20_DIR)Redist\VGB\x64" "$(OutDir)" /e /y /i /r if exist "$(ProjectDir)\*.gbd" ( copy "$(ProjectDir)\*.gbd" "$(OutDir)" /y )





3. Copy the gesture classifier file to the project folder (same folder as main.cpp or NtKinect.h).

In this example, I use the following ".gbd" file destributed with Kinect for Windows SDK.

$(KINECTSDK20_DIR)Tools\KinectStudio\databases\SampleDatabase.gbd

4. Change the contents of main.cpp.

First, we define USE_GESTURE constant before including "NtKinect.h".

We declare a variable "kinect" of type "NtKinect".

Call kinect.setGestureFile(string) and set the gesture classifier file.

After calling kinect.setSkeleton() function, Call kinect.setGesture() function(). When one or more gestures are recognized, they are set to kinect.discreteGesture or kinect.continuousGesture.

main.cpp

#include <iostream> #include <sstream> #define USE_GESTURE #include "NtKinect.h" using namespace std; void putText(cv::Mat& img,string s,cv::Point p) { cv::putText(img, s, p, cv::FONT_HERSHEY_SIMPLEX, 1.0, cv::Scalar(0,0,255), 2.0, CV_AA); } void doJob() { NtKinect kinect; kinect.setGestureFile(L"SampleDatabase.gbd"); //kinect.setGestureFile(L"Seated.gbd"); while (1) { kinect.setRGB(); kinect.setSkeleton(); for (auto person: kinect.skeleton) { for (auto joint: person) { if (joint.TrackingState == TrackingState_NotTracked) continue; ColorSpacePoint cp; kinect.coordinateMapper->MapCameraPointToColorSpace(joint.Position,&cp); cv::rectangle(kinect.rgbImage, cv::Rect((int)cp.X-5, (int)cp.Y-5,10,10), cv::Scalar(0,0,255),2); } } kinect.setGesture(); for (int i=0; i<kinect.discreteGesture.size(); i++) { auto g = kinect.discreteGesture[i]; putText(kinect.rgbImage,kinect.gesture2string(g.first)+" "+to_string(g.second), cv::Point(50,30+30*i)); } for (int i=0; i<kinect.continuousGesture.size(); i++) { auto g = kinect.continuousGesture[i]; putText(kinect.rgbImage,kinect.gesture2string(g.first)+" "+to_string(g.second), cv::Point(500,30+30*i)); } cv::imshow("rgb", kinect.rgbImage); auto key = cv::waitKey(1); if (key == 'q') break; } cv::destroyAllWindows(); } int main(int argc, char** argv) { try { doJob(); } catch (exception &ex) { cout << ex.what() << endl; string s; cin >> s; } return 0; }

5. When you run the program, RGB images are displayed. Exit with 'q' key.

Recognized gestures are displayed above the RGB image. The discrete gesture is displayed along with the confidence in the upper left, and the continuous gesture is displayed along with the progress to the right.








6. Please click here for this sample project KinectV2_gesture.zip

Since the above zip file may not include the latest "NtKinect.h", Download the latest version from here and replace old one with it.

7. In Kinect for Windows SDK v2.0, another gbd file is attached as a sample. Please use this to change to recognize "sitting" motion. this is the part commented out in "main.cpp" above.

$(KINECTSDK20_DIR)bin\Database\Seated.gbd

8. Let's create your own gesture classifier file using Visual Gesture Builder. (2016/10/20 added)。 (Notice) How to use Kinect Studio and Visual Gesture Builder is briefly mentioned in this article. Please refer to the manual or other documents for their detailed usage.

First of all, using Kinect Studio, we shoot a movie "including both cases where you are performing certain gestures and not doing them".

 Data type to record with "Record" panel
    o Nui Body Frame
    x Nui Body Index                <--not needed, remove check
    o Nui Depth
    o Nui IR
    o Nui Nui Sensor Telemetry
    x Nui Title Audio               <-- not needed, remove check
    x Nui Uncompressed Color        <-- not needed, remove check






Next, we generate a gesture classifier file from the video using Visual Gesture Builder. Manually, you define the part of the movie that "moves the left hand above the should" as positive, and others as negative. If you let Visual Gesture Builder do machine learning, the gesture classifier file LeftHandUp.gba is generated as a result.

By the way, the name of this file is not "LeftHandUp.gbd", but "LeftHandUp.gba".






Let's replace the gesture classifier file of the project in this article with LeftHandUp.gba generated this time.

    kinect.setGestureFile(L"LeftHandUp.gba");

Please recognize the action "Raise the left hand above the shoulder" motion as a discrete gesture and not recognize the action "Raise only the right hand on the shoulder" motion or "Raise both hands above the shoulder" motion.