In " NtKinect: How to make Kinect V2 program as DLL and using it from Unity ", it is explained how to create a DLL file that uses the basic functions of OpenCV and Kinect V2 via NtKinect, and use it from other homebrew programs and Unity.
Well, the programs that recognize face, speech and gesture with Kinect V2 require some additional DLL files corresponding to each function at execution time. Here, I will explain
how to make your own NtKinect program as a DLL, even if the program uses the Kinect V2 function which needs other DLLs and setting files,and
How to use your NtKinect DLL from other languages/environments as C#/Unity, for example.
In this article, we will explain how to create a DLL file for gesture recognition.
Let's craete a DLL library that perform gesture recognition as an example of Kinect V2 functions that actually requieres other DLLs and additional files.
It should have the following file structure. There may be some folders and files other than those listed, but ignore them now.
NtKinectDll4/NtKinectDll.sln
x64/Release/
NtKinectDll/dllmain.cpp
NtKiect.h
NtKiectDll.h
NtKiectDll.cpp
stdafx.cpp
stdafx.h
targetver.h
|
"Configuration Properties" -> "Linker" -> "General" -> "Input"
Kinect20.VisualGestureBuilder.lib
The red color part is the prototype declaration of the function added this time.
| NtKinectDll.h |
#ifdef NTKINECTDLL_EXPORTS
#define NTKINECTDLL_API __declspec(dllexport)
#else
#define NTKINECTDLL_API __declspec(dllimport)
#endif
namespace NtKinectGesture {
extern "C" {
NTKINECTDLL_API void* getKinect(void);
NTKINECTDLL_API void setGestureFile(void* ptr, wchar_t* filename);
NTKINECTDLL_API int setGestureId(void* ptr, wchar_t* name, int id); // id: non-zero
NTKINECTDLL_API void setGesture(void* ptr);
NTKINECTDLL_API int getDiscreteGesture(void* ptr, int *gid, float *confidence);
NTKINECTDLL_API int getContinuousGesture(void* ptr, int *gid, float *progress);
NTKINECTDLL_API void stopKinect(void* ptr);
}
std::unordered_map<std::string, int> gidMap;
}
|
To recognize gesture, define USE_GESTURE constant before including NtKinect.h. If you define USE_GESTURE, the program must be linked "Kinect20.VisualGestureBuilder.lib" library.
The basic policy to write NtKinectDll.cpp is as follows.
Frequent exchange of character string data between Unity and DLL will make execution performance worse. Therefore, Unity side defines "Gesture ID Number" as positive integer at the begining, and tells it to the DLL side. And when the DLL side recognizes the gesture, it notifies the gesture type by "Gesture ID Nubmer" to the Unity side.
In this way, mapping from gesture name to id number is needed in DLL side, but by using std::unordered_map<string,int> of C++14 we can efficiently implement the mapping. The ID number of such a gesture whose id number is not defined is 0.
| variable declaration in header file |
|---|
std::unordered_map<std::string, int> gidMap;
|
| the code adding string->int mapping in setGestureId() |
gidMap[s] = id; // <-- s is a string of C++ and id is a positive integer.
|
| the code referring string->int mapping in getDiscreteGestureId() |
gid[i] = gidMap[gname] // <-- returns int for gname. returns 0 if the name is not regstered.
|
For DLL functions, strings are received as wchar_t * type. When type conversion is needed, convert it into UTF8 on the DLL side.
| The code onverting UTF16 wchar_t* type name into UTF8 string type nameBuffer |
|---|
int len = WideCharToMultiByte(CP_UTIF8, NULL, name, -1, NULL, 0, NULL, NULL) + 1; // <-- Find the number of bytes of the converted characters.
char* nameBuffer = new char[len]; // <-- Allocate memory for the UTF8 string
memset(nameBuffer, '\0', len); // <-- Fill the area with NULL character
WideCharToMultiByte(CP_UTIF8, NULL, name, -1, nameBuffer, 0, NULL, NULL) + 1; // <-- convert from UTF16 to UTF8, and write it in nameBuffer
|
We want to use the recognized gesture data on the Unity side, so prepare the data area on the Unity side. On the DLL side, data to be returned to the Unity side will be written as an element of a one-dimensional array area.
In the function that returns a recognized gesture, it is useful that the number of recognized gesture is the return value. However, since the number of discrete gestures and that of continuous gestures recognized are different generally, we will return each as a different function this time. Gesture recognition itself is done simultaneously by one function call.
Since the window displayed by OpenCV is operated by another thread, it may survive and continue using the NtKinectDll.dll file, even if the Unity application terminates. This is troublesome because it is necessary to restart Unity when trying to replace NtKinectDll.dll. Therefore, we prepare a function to destroy all the windows of OpenCV, and call it when the Unity application terminates.
By comparing the trackingId of the skeleton with trackingId of gesture, you can see which skeleton do the gesture. However, this time we will not use the information of gesture's trackingId in favor of easiness of understanding. Only when sending skeleton information to Unity side, there is a meaning to use gesture trackingId.
| NtKinectDll.cpp |
#include "stdafx.h" #include <unordered_map> #include "NtKinectDll.h" #define USE_GESTURE #include "NtKinect.h" using namespace std; namespace NtKinectGesture { NTKINECTDLL_API void* getKinect(void) { NtKinect* kinect = new NtKinect; return static_cast<void*>(kinect); } NTKINECTDLL_API void setGestureFile(void* ptr, wchar_t* filename) { NtKinect *kinect = static_cast<NtKinect*>(ptr); wstring fname(filename); (*kinect).setGestureFile(fname); } NTKINECTDLL_API int setGestureId(void* ptr, wchar_t* name, int id) { int len = WideCharToMultiByte(CP_UTF8,NULL,name,-1,NULL,0,NULL,NULL) + 1; char* nameBuffer = new char[len]; memset(nameBuffer,'\0',len); WideCharToMultiByte(CP_UTF8,NULL,name,-1,nameBuffer,len,NULL,NULL); string s(nameBuffer); gidMap[s] = id; return id; } NTKINECTDLL_API void setGesture(void* ptr) { NtKinect *kinect = static_cast<NtKinect*>(ptr); (*kinect).setRGB(); (*kinect).setSkeleton(); int scale = 4; cv::Mat img((*kinect).rgbImage); cv::resize(img,img,cv::Size(img.cols/scale,img.rows/scale),0,0); for (auto person: (*kinect).skeleton) { for (auto joint: person) { if (joint.TrackingState == TrackingState_NotTracked) continue; ColorSpacePoint cp; (*kinect).coordinateMapper->MapCameraPointToColorSpace(joint.Position,&cp); cv::rectangle(img, cv::Rect((int)cp.X/scale-2, (int)cp.Y/scale-2,4,4), cv::Scalar(0,0,255),2); } } cv::imshow("rgb",img); cv::waitKey(1); (*kinect).setGesture(); } NTKINECTDLL_API int getDiscreteGesture(void* ptr, int *gid, float *confidence) { NtKinect *kinect = static_cast<NtKinect*>(ptr); for (int i=0; i<(*kinect).discreteGesture.size(); i++) { auto g = (*kinect).discreteGesture[i]; string gname = (*kinect).gesture2string(g.first); gid[i] = gidMap[gname]; confidence[i] = g.second; } return (*kinect).discreteGesture.size(); } NTKINECTDLL_API int getContinuousGesture(void* ptr, int *gid, float *progress){ NtKinect *kinect = static_cast<NtKinect*>(ptr); for (int i=0; i<(*kinect).continuousGesture.size(); i++) { auto g = (*kinect).continuousGesture[i]; string gname = (*kinect).gesture2string(g.first); gid[i] = gidMap[gname]; progress[i] = g.second; } return (*kinect).continuousGesture.size(); } NTKINECTDLL_API void stopKinect(void* ptr) { cv::destroyAllWindows(); } } |
In the above NtKinectDll.cpp codes, the character color is basically based on the following policy.
green characters: Parts related to DLL creation in Visual Studio
blue characters: Parts defining DLL functions for gesture recognition
red characters: Parts using NtKinect and OpenCV
magenta characters: Parts related to data conversion (marshaling) between C# and C++
[Caution](Oct/07/2017 added) If you encounter "dllimport ..." error when building with Visual Studio 2017 Update 2, please refer to here and deal with it to define NTKINECTDLL_EXPORTS in NtKinectDll.cpp.
Since the above zip file may not include the latest "NtKinect.h", Download the latest version from here and replace old one with it.
Let's use NtKinectDll.dll in Unity.
[Notice] $(KINECTSDK20_DIR) is set to "C:\Program Files\Microsoft SDKs\Kinect\v2.0_1409\" in my environment. Please read according to your own environment.
Let's use $(KINECTSDK20_DIR)Tools\KinectStudio\databases\SampleDatabase.gbd which we used at " NtKinect: How to recognize gesture with Kinect V2 ". Copy this file directly under Unity's project folder.
CheckNtKinectDll4/SampleDatabase.gbd
From the top menu, "Assets"-> "Create" -> "C# Script" -> Filename is "NtKinectGesture"
NtKinectGesture.cs is created in the "Assets/Scripts/" folder of the project.
The basic policy to describe NtKinectGesture.cs is as follows.
The string of Unity (C#) is managed, and the string and wstring of DLL (C++) are unmanaged. When passing Unity (C#) string as an argument to a function of DLL (C++), convert it to unmanaged UTF16 string on heap memory using Marshal.StringToHGlobalUni() function. This operation eliminates the danger that data may be moved by the garbase collector of C#. When the unmanaged data becomes unnecessary, you must call Marshal. FreeHGlobal() function to free the memory.
We want to use the recognized gesture data on the Unity side, so prepare the data area on the Unity side. We prepare an array in this example. When passing an array of C# as an argument to a DLL function, pin it using GCHandle so that the data area does not move by the gabage collector of C#. Please note that the pinned area needs to be released when it is no long needed.
GCHandle gch = GCHandle.Allocate(array on heap, GCHandleType.Pinned) // allocate unmanaged area. ... gch.AddrOfPinnedObject() // Returns the address of the beginning of the unmanaged area. ... gch.Free(); // Free unmanaged area. |
After calling setGesture(), call getDiscreteGesture() and/or getContinuousGesture().
In this example, there are 3 types of discrete gestures and 1 type of continuous gesture. 6 people may be recognized simultaneously by Kinect. Therefore, The variable gestureCount, the number of elements, is set to max(3,1) * 6 = 18.
| NtKinectGesture.cs |
using UnityEngine;
using System.Collections;
using System.Runtime.InteropServices;
using System;
public class NtKinectGesture : MonoBehaviour {
[DllImport ("NtKinectDll")] private static extern System.IntPtr getKinect();
[DllImport ("NtKinectDll")] private static extern int rightHandState(System.IntPtr kinect);
[DllImport ("NtKinectDll")] private static extern void setGestureFile(System.IntPtr ptr, System.IntPtr filename);
[DllImport ("NtKinectDll")] private static extern int setGestureId(System.IntPtr ptr, System.IntPtr name, int id);
[DllImport ("NtKinectDll")] private static extern void setGesture(System.IntPtr kinect);
[DllImport ("NtKinectDll")] private static extern int getDiscreteGesture(System.IntPtr kinect, System.IntPtr gid, System.IntPtr cnf);
[DllImport ("NtKinectDll")] private static extern int getContinuousGesture(System.IntPtr kinect, System.IntPtr gid, System.IntPtr cnf);
[DllImport ("NtKinectDll")] private static extern void stopKinect(System.IntPtr kinect);
private System.IntPtr kinect;
int gestureCount = 18;
void Start () {
kinect = getKinect();
System.IntPtr gbd = Marshal.StringToHGlobalUni("SampleDatabase.gbd"); // gbd file
System.IntPtr g1 = Marshal.StringToHGlobalUni("Steer_Left"); // discrete
System.IntPtr g2 = Marshal.StringToHGlobalUni("Steer_Right"); // discrete
System.IntPtr g3 = Marshal.StringToHGlobalUni("SteerProgress"); // continuous
System.IntPtr g4 = Marshal.StringToHGlobalUni("SteerStraight"); // discrete
setGestureFile(kinect,gbd);
setGestureId(kinect,g1,1);
setGestureId(kinect,g2,2);
setGestureId(kinect,g3,3);
setGestureId(kinect,g4,4);
Marshal.FreeHGlobal(gbd);
Marshal.FreeHGlobal(g1);
Marshal.FreeHGlobal(g2);
Marshal.FreeHGlobal(g3);
Marshal.FreeHGlobal(g4);
}
void Update () {
setGesture(kinect);
int[] gid = new int[gestureCount];
float[] cnf = new float[gestureCount];
GCHandle gch = GCHandle.Alloc(gid,GCHandleType.Pinned);
GCHandle gch2 = GCHandle.Alloc(cnf,GCHandleType.Pinned);
// discrete gesture
int n = getDiscreteGesture(kinect, gch.AddrOfPinnedObject(), gch2.AddrOfPinnedObject());
for (int i=0; i<n; i++) {
Debug.Log(i + " discrete " + gid[i] + " " + cnf[i]); // "cnf" value is "confidence"
}
// continous gesture
n = getContinuousGesture(kinect, gch.AddrOfPinnedObject(), gch2.AddrOfPinnedObject());
for (int i=0; i<n; i++) {
Debug.Log(i + " continous " + gid[i] + " " + cnf[i]); // "cnf" value is "progress"
}
gch.Free();
gch2.Free();
}
void OnApplicationQuit() {
stopKinect(kinect);
}
}
|
In the above NtKinectGesture.cs list, character color is basically based on the following policy.
green character: Declaration part for using DLL function in Unity
blue character: Parts related to the DLL's function call
magenta chracter: Parts related to data conversion (marshaling) between C# and C++
In this example, the Game screen does not change. The recognized gestures are displayed on the console panel as outputs of Debug.Log
There are four types of gestures defined in the "SampleDatabase.gbd" file.
| name | typee | ID Number |
|---|---|---|
| Steering_Left | discrete | 1 |
| Steering_Right | discrete | 2 |
| SteeringProgress | continuous | 3 |
| SteeringStraight | discrete | 4 |
0 discrete 1 0.09684379 <-- Steering_Left is recognized with confidence 0.09684379 1 discrete 4 0.06129133 <-- SteeringStraight is recognized with confidence 0.06129133 0 continuous 3 0.4763844 <-- SteeringProgress is recognized with progress 0.4763844 |
[Notice] In this case , SteeringProgress of continuous gesture recognized once, it seems to remain recognized as long as the skeleton is recognized. It remains unknown whether this situation is normal or not.
[Notice] We generates an OpenCV window in DLL to display the skeleton recognition state. Note that when the OpenCV window is focused, that is, when the Unity game window is not focused, the screen of Unity will not change. BUT the console panel will change in that situation. HOWEVER, in this example, the game screen will not change even if the unity window is focused.
In short, you only need to extract the readable characters contained in the gesture classifier file. You can also open and examine the classifier file with a text editor which can handle binary. Here is an example of using the "strings" command of Unix.
In Unix (Linux, MacOS X, cygwin) environment, there is a "strings" command to retrieve readable strings in the binary. For example, running "strings" against "SampleDatabase.gbd" on Linux shell result in the following output.
| Quoted from the output of strings command |
$ strings SampleDatabase.gbd VGBD Steer_Left Steer_Left AdaBoostTrigger Steer_Right Steer_Right AdaBoostTrigger SteerProgress SteerProgress RFRProgress SteerStraight SteerStraight AdaBoostTrigger Accuracy Level 33s?" Number Weak Classifiers at Runtime Filter Results ...(abbreviated)... |
Discrete gestures are determined by the AdaBoost algorithm and continous gestures are determined by the Random-Forest algorithm. So, in the above gbd file, you can see that
Steering_Left Steering_Right SteeringStraightas descrete gesture, and
SteeringProgressas continuous gesture.
[Notice] In the cygwin environment, the "strings" command may not be installed. See here for additional installation.