NtKinect: Kinect V2 C++ Programming with OpenCV on Windows10

How to make Kinect V2 skeleton recognition as DLL and use it from Unity

2017.07.11: created by
2017.10.07: created by
Japanese English
To Table of Contents

Prerequisite knowledge

Creating Dynamic Link Library using Skeleton Recognition with Kinect V2

In the article " NtKinect: How to make Kinect V2 program as DLL and use it from Unity ", it is explained how to create a DLL file that uses the basic functions of OpenCV and Kinect V2 via NtKinect, and use it from other programs and Unity.

In this article, we will explain how to create a DLL file for skeleton recognition.

How to write program

Let's craete a DLL library that perform face recognition as an example of Kinect V2 functions that actually requieres other DLLs and additional files.

  1. Start using the Visual Studio 2015's project KinectV2_dll.zip of "How to make Kinect V2 program as DLL and use it from Unity" .
  2. Write the declaration in the header file. The name of the header file is "ProjectName.h", which is "NtKinectDll.h" in this case.

    3. The magenta color part is the prototype declaration of the function added this time.

    NtKinectDll.h
    #ifdef NTKINECTDLL_EXPORTS
#define NTKINECTDLL_API __declspec(dllexport)
#else
#define NTKINECTDLL_API __declspec(dllimport)
#endif

extern "C" {
  NTKINECTDLL_API void* getKinect(void);
  NTKINECTDLL_API int setSkeleton(void* ptr, float* skelton, int* state, int *id);
}
  3. Write the definition in "ProjectName.cpp" which is "NtKinectDll.cpp" in this case.

    4. Add the function definition of int setSkeleton(void *, float *, int *, int *). In this function, the image acquired by the RGB camera is reduced to 1/16, and is displayed with cv::imshow(). On the image, the joints are drawn in red. Notice that calling cv::waitKey(1) is needed to display the OpenCV window properly. Since 25 joints are recognized per person, and 6 people may be recognized, so the data area should be allocated more than "sizeof(float) * 3 * 25 * 6" bytes on the caller side. Return value is the number of recognized skeletons.

    NtKinectDll.cpp
    #include "stdafx.h"
#include "NtKinectDll.h"

#include "NtKinect.h"

using namespace std;

NTKINECTDLL_API void* getKinect(void) {
  NtKinect* kinect = new NtKinect;
  return static_cast<void*>(kinect);
}

NTKINECTDLL_API int setSkeleton(void* ptr, float *skeleton, int* state, int* id) {
  NtKinect *kinect = static_cast<NtKinect*>(ptr);
  (*kinect).setRGB();
  (*kinect).setSkeleton();
  int scale = 4;
  cv::Mat img((*kinect).rgbImage);
  cv::resize(img,img,cv::Size(img.cols/scale,img.rows/scale),0,0);
  for (auto& person: (*kinect).skeleton) {
    for (auto& joint: person) {
      if (joint.TrackingState == TrackingState_NotTracked) continue;
      ColorSpacePoint cp;
      (*kinect).coordinateMapper->MapCameraPointToColorSpace(joint.Position,&cp);
      cv::rectangle(img, cv::Rect((int)cp.X/scale-2, (int)cp.Y/scale-2,4,4), cv::Scalar(0,0,255),2);
    }
  }
  cv::imshow("face",img);
  cv::waitKey(1);
  int idx=0, jt=0, st=0;
  for (auto& person: (*kinect).skeleton) {
    for (auto& joint: person) {
      skeleton[jt++] = joint.Position.X;
      skeleton[jt++] = joint.Position.Y;
      skeleton[jt++] = joint.Position.Z;
      state[st++] = joint.TrackingState;
    }
    id[idx] = (*kinect).skeletonId[idx];
    idx++;
  }
  return idx;
}
  4. Compile with "Application configuration" set to "Release". NtKinectDll.lib and NtKinectDll.dll are generated in the folder KinectV2_dll5/x64/Release/.

    5. [Caution](Oct/07/2017 added) If you encounter "dllimport ..." error when building with Visual Studio 2017 Update 2, please refer to here and deal with it to define NTKINECTDLL_EXPORTS in NtKinectDll.cpp.

  5. Please click here for this sample project NtKinectDll5.zip。

    6. Since the above zip file may not include the latest "NtKinect.h", Download the latest version from here and replace old one with it.

Use NtKinect DLL from Unity

Let's use NtKinectDll.dll in Unity.

  1. For more details on how to use DLL with Unity, please see the official manual .
  2. The data of Unity(C#) is managed and the data of DLL (C++) is unmanged. Managed means that the data is moved by the gabage collector. Unmanaged means that the data is not moved by the gabage collector. In order to pass data between C# and C++, it is necessary to convert the data state. To do this, the functions of System.Runtime.InteropServices.Marshal Class in C# can be used.
  3. For details on how to pass data between different languages, refer to the " Interop Marshaling " section of the " Interoperating with Unmanaged Code " at MSDN.
  4. Start a new Unity project.
  5. Copy NtKinectDll.dll to the project folder Assets/Plugins/x86_64/ .



  6. Place the Empty Object in the scene, and change its name to Skeleton.
  7. Create C# script under the Project's Assets/Scripts/

    8. From the top menu, "Assets"-> "Create" -> "C# Script" -> Filename is "NtKinectBehaviour"

    The pointers in C++ are treated as System.IntPtr in C#.

    NtKinectBehaviour.cs
    using UnityEngine;
using System.Collections;
using System.Runtime.InteropServices;

public class NtKinectBehaviour : MonoBehaviour {
  [DllImport ("NtKinectDll")] private static extern System.IntPtr getKinect();
  [DllImport ("NtKinectDll")] private static extern int setSkeleton(System.IntPtr kinect, System.IntPtr data, System.IntPtr state, System.IntPtr id);

  private System.IntPtr kinect;
  int bodyCount = 6;
  int jointCount = 25;
  GameObject[] obj;
  int counter;
    
  void Start () {
    kinect = getKinect();
    obj = new GameObject[bodyCount * jointCount];
    for (int i=0; i<obj.Length; i++) {
      obj[i] = GameObject.CreatePrimitive(PrimitiveType.Cube);
      obj[i].transform.position = new Vector3(0,0,-10);
      obj[i].transform.localScale = new Vector3(0.1f,0.1f,0.1f);
    }
  }
  void Update () {
    float[] data = new float[bodyCount * jointCount * 3];
    int[] state = new int[bodyCount * jointCount];
    int[] id = new int[bodyCount];
    GCHandle gch = GCHandle.Alloc(data,GCHandleType.Pinned);
    GCHandle gch2 = GCHandle.Alloc(state,GCHandleType.Pinned);
    GCHandle gch3 = GCHandle.Alloc(id,GCHandleType.Pinned);
    int n = setSkeleton(kinect,gch.AddrOfPinnedObject(),gch2.AddrOfPinnedObject(),gch3.AddrOfPinnedObject());
    gch.Free();
    gch2.Free();
    gch3.Free();
    int idx=0;
    for (int i=0; i<bodyCount; i++) {
      for (int j=0; j<jointCount; j++) {
	if (i < n) {
	  float x = data[idx++], y=data[idx++], z=data[idx++];
	  obj[i*jointCount + j].transform.position = new Vector3(x,y,z);
	  obj[i*jointCount + j].GetComponent<Renderer>().material.color = new Color(0.0f,1.0f,0.0f,1.0f);
	} else {
	  obj[i*jointCount + j].transform.position = new Vector3(0,0,-10);
	  obj[i*jointCount + j].GetComponent<Renderer>().material.color = new Color(1.0f,1.0f,1.0f,1.0f);
	}
      }
    }
  }
}
  8. Add "NtKinectBehaviour.cs" script to "NtKinect" as a Component.
  9. Change the position of Main Camera.

    10. The default position of Main Camera is (x,y,z)=(0,1,-10). The z coordinates of the recognized joints are positive , so move the Main Camera to (x,y,z)=(0,0,-1).

  10. When you run the program, green cubes are placed at the positions of recognized joints in the scene. Exit with 'q' key.

    11. [Notice] We generates an OpenCV window in DLL to display the skeleton recognition state. Note that when the OpenCV window is focused, that is, when the Unity game window is not focused, the screen of Unity will not change. Click on the top of Unity's window and make it focused, then try the program's behaviour.







  11. Please click here for this Unity sample project CheckNtKinectDll5.zip。
  12. Here is the explanation on "How to make Unity's Humanoid take the same pose as Kinect V2's skeleton recognized".

http://nw.tsuda.ac.jp/