NtKinect: Kinect V2 C++ Programming with OpenCV on Windows10

How to make Kinect V2 program as DLL and use it from Unity

2016.08.25: created by
2017.10.07: revised by
Japanese English
To Table of Contents

Prerequisite knowledge

Creating Dynamic Link Library using Kinect V2

How to create and use a DLL is explained at the Microsoft official pages.

Let's create a DLL file using OpenCV and Kinect V2 via NtKinect. Visual Studio Proefssional 2017 provides a project template to create Dlls in C++.

  1. Create new project for DLL.
    2. [Before "Visual Studio 2017 Update 2 (version 15.3.5)"]
    1. Select "File" -> "New" -> "Project"-> "Visual C++" -> "Windows" -> "Win32" -> "Win32 console application"

      2. Input Name -> "OK" -> Select "Next" and go to "Application settings"

    2. In "Application settings", select "DLL" in "Application Type", check out the "Export symbols" option for "additional options". Uncheck "Security Development Lifecycle (SDL)" if it exists.
    ["Visual Studio 2017 Update 2 (version 15.3.5)" or later]
    1. Select "File" -> "New" -> "Project" -> "Visual C++" -> "Windows desktop" -> "Windows desktop wizards"

      2. Type the project name and click "OK" button.

    2. In "Windows desktop project" window, select "Dynamic Link Library (.dll)" in "Application Type", check out the "Export symbols" option for "additional options".
  2. Change the "x86" in the upper menu of the Visual Studio window to "x64" and set this project to create a 64-bit application.
  3. Write the function declarations in the header file "ProjectName.h".

    4. Examples of the declaration of variables, functions, and classes has already been inserted in the header file. Write your own declaration referring those examples. The predecessor of the function declaration "PROJECT_NAME_API" is also defined here. In this case, the predecessor is "NTKINECTDLL_API".

  4. Write function definition in "ProjectName.cpp".

    5. Examples of the definition of variables, functions, and classes has already been inserted in the c++ file. Write you own definition referring those examples. As seen in examples, you must write "PROJECT_NAME_API" ("NTKINECTDLL_API" in this case) before your function definition. This predecessor is for the export/import switching.

  5. Compile with "Application configuration" set to "Release". Just "Build" or "Rebuild". NEITHER "Run" NOR "Debug". A ".lib" file and a ".dll" file are generated in the folder "x64/Release/".

How to write program

Let's actually create a DLL library that use Kinect V2 and use it in Unity.

I will explain by an example of recognizing the state of right palm (Open, Closed, Lasso), because I think is an easiest example to understand.

  1. Create a new project for DLL.
    2. [Before "Visual Studio 2017 Update 2 (version 15.3.5)"]
    1. Select "File" -> "New" -> "Project"-> "Visual C++" -> "Windows" -> "Win32" -> "Win32 console application"。

      2. The name here is "NtKinectDll". The solution name will automatically be "NtKinectDll". OK -> Select "Next" and go to "Setting up the application"




    2. In "Application settings", select "DLL" in "Application Type", check out the "Export symbols" option for additional options. Uncheck the "Security Development Lifecycle (SDL) check" if it exists.



    ["Visual Studio 2017 Update 2 (version 15.3.5)" or later]
    1. Select "File" -> "New" -> "Project"-> "Visual C++" -> "Windows desktop" -> "Windows desktop wizards"

      2. The name here is "NtKinectDll". The solution name will automatically be "NtKinectDLL". Click OK button.




    2. In "Windwos desktop project" window, select "Dynamic Link Library (.dll)" in "Application Type", check out the "Export symbols" option for additional options.



  2. The solution explorer of Visual Studio should be displayed as follows.



  3. Change the "x86" in the upper menu of the Visual Studio window to "x64" and set this project to create a 64-bit application.



  4. Configure settings for include files and libraries from project's properties.
    1. In the Solution Explorer, right-drag over the project name and select "Properties" on the menu.



    2. Set the mode as Configuration: "All Configuration", Plathorm: "Active (x64)". When you configure in this mode, you configure both settings of "Debug" and "Release" at a time. Of courcse, you can set them separately.
    3. Add the paths of include files.

      4. "Configuration Properties" -> "C/C++" -> "General" -> "Additional include directory" 

        $(KINECTSDK20_DIR)inc
  C:\opencv\include






    4. Add the paths of libraries

      5. "Configuration Properties" -> "Linker" -> "General" -> "Additional library directory" 

        $(KINECTSDK20_DIR)Lib\x64
  C:\opencv\lib






    5. Add libraries.

      6. "Configuration Properties" -> "Linker" -> "General" -> "Input" 

        Kinect20.lib
  opencv_world330.lib






  5. (Important) Add NtKinect.h ( this site, github ) to the project's header file.

    6. Please download NtKinet.h from the above link. Place NtKinect.h in the folder where the project's source files such as dllmain.cpp are located (NtKinectDll/NtKinectDll/ in this example). Right-click on "header file" of "Solution Explorer" and select "Add" -> "Select Existing Files" -> NtKinect.h.









  6. Write the declaration in the header file. The name of the header file is "ProjectName.h", in this case, it will be "NtKinectDll.h".

    7. The part of green letter is the part related to import/export of the DLL defined since the project was created. NtKinectDll.h is a declaration for export in this project, and it becomes a declaration for import in other projects.

    The part of blue letter is the part about the functions we defined ourselves. To avoid mangling function names in C++, we declare function prototypes in extern "C" {}. This makes it possible to use this DLL in other languages. The word Mangling means that the function name is changed by the C++ compiler to a name including return value and arguments type.

    NtKinectDll.h
    #ifdef NTKINECTDLL_EXPORTS
#define NTKINECTDLL_API __declspec(dllexport)
#else
#define NTKINECTDLL_API __declspec(dllimport)
#endif

extern "C" {
  NTKINECTDLL_API void* getKinect(void);
  NTKINECTDLL_API int rightHandState(void* kinect);
}
  7. The function is described in "ProjectName.cpp". In this example, the file name is "NtKinectDll.cpp".

    8. Please note that "NTKINECTDLL_API" must be written at the beginning of function definition.

    In the DLL, the object must be allocated in the heap memory. For this reason, the void* getKinect() function instantiate a new NtKinect object on heap, and returns its pointer casting to (void *).

    When executing a function of the DLL, the pointer to the NtKinect Object is given as an argument of type (void *). We cast it to a pointer of (NtKinect *) type and use NtKinect's function via it. For example, access to a member variable "rgbImage" is described as (*kinect).rgbImage. In the rightHandState(void *) function, the image acquired by the RGB camera is first reduced to 1/16, the joint is drawn in red on it, and the image is displayed by cv::imshow(). In order to correctly display the contents of the OpenCV window, it is necessary to call cv::waitKey(1); After that, the program return the state of someone's right palm state. The meaning of the value is defined in Kinect.h as follows. 

    enum _HandState {
    HandState_Unknown= 0,
    HandState_NotTracked= 1,
    HandState_Open= 2,
    HandState_Closed= 3,
    HandState_Lasso= 4
};
    NtKinectDll.cpp
    #include "stdafx.h"
#include "NtKinectDll.h"

#include "NtKinect.h"

using namespace std;

NTKINECTDLL_API void* getKinect(void) {
  NtKinect* kinect = new NtKinect;
  return static_cast<void*>(kinect);
}

NTKINECTDLL_API int rightHandState(void* ptr) {
  NtKinect *kinect = static_cast<NtKinect*>(ptr);
  (*kinect).setRGB();
  (*kinect).setSkeleton();
  int scale = 4;
  cv::Mat img((*kinect).rgbImage);
  cv::resize(img,img,cv::Size(img.cols/scale,img.rows/scale),0,0);
  for (auto person: (*kinect).skeleton) {
    for (auto joint: person) {
      if (joint.TrackingState == TrackingState_NotTracked) continue;
      ColorSpacePoint cp;
      (*kinect).coordinateMapper->MapCameraPointToColorSpace(joint.Position,&cp);
      cv::rectangle(img, cv::Rect((int)cp.X/scale-2, (int)cp.Y/scale-2,4,4), cv::Scalar(0,0,255),2);
    }
  }
  cv::imshow("rgb",img);
  cv::waitKey(1);
  for (int i=0; i<(*kinect).skeleton.size(); i++) {
    Joint right = (*kinect).skeleton[i][JointType_HandRight];
    if (right.TrackingState == TrackingState_NotTracked) continue;
    auto state = (*kinect).handState(i,false);
    if (state.first == HandState_Open
	|| state.first == HandState_Closed
	|| state.first == HandState_Lasso ) {
      return state.first;
    }
  }
  return HandState_Unknown;
}
  8. Compile with "Application configuration" set to "Release". From the top menu, select "Build" -> "Rebuild Project". NtKinectDll.lib and NtKinectDll.dll are generated in the folder "x64/Release/".

    9. [Caution] Do just "Compile", "Build", or "Rebuild". Do not "Run" or "Debug" here. Since we have not write the program body, an meaningless error will occur if you try to "Run" or "Debug".

    [Caution2] ".dll" and ".lib" files are generated in the "x64/Release/" folder immediately under the project, that is, "(ProjectName)/x64/Release". Do not search wrong folders like "(ProjectName)/NtKinectDll/x64/Release".



    [Caution3](Oct/07/2017 added) If you encounter "dllimport ..." error when building with Visual Studio 2017 Update 2 (version 15.3.5), please refer to here and deal with it to define NTKINECTDLL_EXPORTS in NtKinectDll.cpp. (Nov/03/2017 added) This bug is fixed in Visual Studio 2017 version 15.4 released Oct/11/2017.

  9. Please click here for this sample project NtKinectDll.zip。

    10. Since the above zip file may not include the latest "NtKinect.h", Download the latest version from here and replace old one with it.

Confirm the operation of the generated DLL file

Let's create a simple project to confirm that the generated DLL file works properly.

  1. Select "File" -> "New" -> "Project" -> "Visual C++" -> "Windows" -> "Win32" -> "Win32 console application"

    2. Input the name as "CheckDLL", select "OK" -> "Next" and goto "Application settings".




  2. In "Application settings", select "Console application" in "Application Type", check "Procompiled headers" for additional options, also check "Security Development Lifecycle (SDL) check" if it exists. I think it is the default settings.



  3. Change the "x86" in the upper menu of the Visual Studio window to "x64" and set this project to create a 64-bit application.



  4. Add NtKinect.h to the project source file.

    5. Copy NtKinectDll.h to the folder where the project source files (stdafx.cpp or CheckDll.cpp) are located. After that, right-click on "Header file" in "Solution Explorer" and select "Add" -> "Select existing file" -> NtKinectDll.h.





  5. Copy the DLL files in the project's folder

    6. Copy NtKinectDll.dll and NtKinectDll.lib to the folder where the project source files (stdafx.cpp or CheckDll.cpp) are located.

  6. In the project's properties, set to link the NtKinectDll.lib library.

    7. At the "Project's properties", "Linker" -> "Input" -> "Additional dependencies file" -> add NtKinectDll.lib.









  7. Change the contents of "CheckDLL.cpp"
    8. CheckDLL.cpp
    #include "stdafx.h"
#include <iostream>
#include <sstream>

#include "NtKinectDll.h"

using namespace std;

int main() {
  void* kinect = getKinect();
  while (1) {
    int state = rightHandState(kinect);
    switch (state) {
    case 2: cout << "Open" << endl; break;
    case 3: cout << "Closed" << endl; break;
    case 4: cout << "Lasso" << endl; break;
    default: cout << "unknown" << endl;  break;
    }
  }
  return 0;
}
  8. When you run the program, the state of your right palm is displayed on the console screen as Open, Closed, Lasso, or Unknown. The right palm state can be recognized only when the skeleton can be recognized. In the rightHandState() function of the DLL file, the RGB image on which the joint positions are drawn in red is displayed in a window. So you can see if the skeleton is currently recognized.



  9. Please click here for this sample project CheckDLL.zip。

Use NtKinect DLL from Unity

Let's use NtKinectDll.dll in Unity.

  1. For more details on how to use DLL with Unity, please see the official manual .
  2. The data of Unity(C#) is managed and the data of DLL (C++) is unmanged. Managed means that the data is moved by the gabage collector. Unmanaged means that the data is not moved by the gabage collector. In order to pass data between C# and C++, it is necessary to convert the data state. To do this, the functions of System.Runtime.InteropServices.Marshal Class in C# can be used.
  3. Start a new Unity Project.



  4. Import NtKinectDll.dll to the "Assets/Plugins/x86_64/" in the project



  5. Place a "Cube" in the scene.

    6. From the menu at the top, "Game Object"-> "3D Object" -> "Cube"



  6. Create a new C# script at "Assets/Scripts/" in the project.

    7. From the menu at the top, "Assets" -> "Create" -> "C# Script" -> Filename is CubeBehaviour




    C++ pointers are treated as System.IntPtr in C#.

    CubeBehaviour.cs
    using UnityEngine;
using System.Collections;
using System.Runtime.InteropServices;

public class CubeBehaviour : MonoBehaviour {
    [DllImport ("NtKinectDll")] private static extern System.IntPtr getKinect();
    [DllImport ("NtKinectDll")] private static extern int rightHandState(System.IntPtr kinect);
    private System.IntPtr kinect;
    
    void Start () {
	kinect = getKinect();
    }
    
    void Update () {
	int state = rightHandState(kinect);
	if (state == 2) {
	    gameObject.GetComponent<Renderer>().material.color = new Color(1.0f, 0.0f, 0.0f, 1.0f);
	} else if (state == 3) {
	    gameObject.GetComponent<Renderer>().material.color = new Color(0.0f, 1.0f, 0.0f, 1.0f);
	} else if (state == 4) {
	    gameObject.GetComponent<Renderer>().material.color = new Color(0.0f, 0.0f, 1.0f, 1.0f);
	} else {
	    gameObject.GetComponent<Renderer>().material.color = new Color(1.0f, 1.0f, 1.0f, 1.0f);
	}
    }
}
  7. Add CubeBehaviour.cs to the Cube as a component.



  8. When you execute the program, the color of the cube changes according to the your right palm state as Open -> Red, Closed -> Green, Lasso -> Blue, otherwise -> White.

    9. [Notice] We generates an OpenCV window in DLL to display the skeleton recognition state. Note that when the OpenCV window is focused, that is, when the Unity game window is not focused, the screen of Unity will not change. Click on the top of Unity's window and make it focused, then try the program's behaviour.




  9. Please click here for this Unity sample project CheckNtKinectDll.zip。

http://nw.tsuda.ac.jp/