NtKinect: Kinect V2 C++ Programming with OpenCV on Windows10

How to run Kinect V2 in a multi-thread environment

2016.08.24: created by
Japanese English
This article is for NtKinect.h version 1.6 or later
To Table of Contents

Prerequisite knowledge

Multi-thread Programming on Windows

NtKinect

To write a program that runs in multi-threading on Windows, first include "proces.h". 

#include <process.h>

The function unsigned func (LPVOID) runs in the thread is defined in the following format. 

unsigned __stdcall func (LPVOID pParam) {

  // The work this thread should do.

  _endthreadex(0);
  return 0;
}

For programs that operate in multi-threading, exclusive control must be needed in critical sections where conflicts may occur. We use Mutex for this purpose. We create a mutex as a variable that can be refered from any thread, and we decide that only the thread that acquire the right of the mutex can enter the critical section. 

Handle mutex;

int main(int argc, char** argv) {
  mutex = CreateMutex(NULL, FALSE, NULL); // create mutex
  if (GetLastError() == ERROR_ALREADY_EXISTS) throw ERROR;

  ...(abbreviated)...

}

unsigned __stdcall func (LPVOID pParam) {

  ...(abbreviated)...

  DWORD ret = WaitForSingleObject(mutex, INFINITE); // acquire the right of Mutex
  if (ret == WAIT_FAILED) throw ERROR;

  ...(abbreviated)... // critical section

  ReleaseMutex(mutex); // release the right of Mutex

  ...(abbreviated)...

  _endthreadex(0);
  return 0;
}

To start N threads in the main thead, write as follows. _beginthreadex() function is used to start a thread, and WaitForMultipleObjects() function is used to wait for all threads to finish. 

  HANDLE hThread[N ] = { 0 };

  hThread[0] = (HANDLE) _beginthreadex(NULL,0, functionName1,NULL,0,NULL); // create thread to call function1
  if (hThread[0] == 0) throw Error ;

  ...(abbreviated)... // create threads of 2〜(N-2)

  hThread[N -1] = (HANDLE) _beginthreadex(NULL,0,functionName>N,NULL,0,NULL); // create thread to call functionN
  if (hThread[N -1] == 0) throw Error ;

  ...(abbreviated)... // Job for the main thread

  DWORD ret = WaitForMultipleObjects(N ,hThread,TRUE,INFINITE); // wait for all threads to finish
  if (ret == WAIT_FAILED) throw Error ;
  return 0;
}

If you define USE_THREAD constant before including NtKinect.h, the functions and variables of NtKinect for multi-thread become effective.

NtKinect

NtKinect's functions for mult-thread

type of return value function name descriptions
void acquire() version1.6 or later.
Acquire the rights of Mutex. This function call is blocked until the mutex can be acquired.
void release() version1.6 or later.
Release the right of Multex

If you want to ensure atomic among multiple function calls, you had better to call acquire() and release() call for mutual exclusion. However, the following functions are prepared for convenience of use. It will do the following basically. 

Call acquire().
Call the Kinect's function().
Copy the results to the argument variable.
Call release()

type of return value function name descriptions
void _setRGB(cv::Mat& image ) version1.6 or later. thread-safe.
Call setRGB() and copy rgbImage to image. Same as the following operations.
    acquire();
    setRGB();
    Copy rgbImage to image.
    release();
void _setDepth(cv::Mat image, bool raw = true) version1.6 or later. thread-safe.
Call setDepth(raw) and copy depthImage to image. Same as the following operations.
    acquire();
    setDepth(raw );
    Copy depthImage to image.
    release();
void _setInfrared(cv::Mat& image ) version1.6 or later. thread-safe.
Call setInfrared() and copy infraredImage to image. Same as the following operations.
    acquire();
    setInfrared();
    Copy infraredImage toimage.
    release();
void _setBodyIndex(cv::Mat& image ) version1.6 or later. thread-safe.
Call setBodyIndex() and copy bodyIndexImage to image. Same as the following operations.
    acquire();
    setBodyIndex();
    Copy bodyIndexImage toimage.
    release();
void 
_setSkeleton(vector<vector<Joint> >& skel ,
             vector<int>& skelId ,
             vector<UINT64>& skelTrackingId )
 version1.6 or later. thread-safe.
Call setSkeleton() and copy the result to the arguments. Same as the following operations.
    acquire();
    setSkeleton();
    Copy skeleton to skel.
    Copy skeletonId to skelId.
    Copyo skeletonTrakingId to skelTrackingId.
    release();
pair<int,int> _handState(int id =0, bool isLeft =true) version1.6 or later. thread-safe.
Call handState() and return the result. Same as the following operations.
    acquire();
    auto ret = handState(id , isLeft );
    release();
    return ret;
void 
_setFace(vector<vector<PointF> >& point ,
         vector<cv::Rect>& rect ,
         vector<cv::Vec3f>& direction ,
         vector<vector<DetectionResult> >& property ,
         vector<UINT64>& trackingId ,
         bool isColorSpace =true)
 version1.6 or later. thread-safe.
Call setFace(isColorSpace ) and copy the result to arguments. Same as the following operations.
    acquire();
    setFace(isColorSpace );
    Copy facePoint to point.
    Copy faceRect to rect.
    Copy faceDirection to direction.
    Copy faceProperty to property.
    Copy faceTrackingId to trakingId.
    release();
void 
_setHDFace(vector<vector<CameraSpacePoint> >& vertices ,
           vector<UINT64>& trackingId ,
           vector<pair<int,int> >& status )
 version1.6 or later. thread-safe.
Call setHDFace() and copy the results to arguments. Same as the following operations.
    acquire();
    setHDFace();
    Copy hdfaceVertices to vertices.
    Copy hdfaceTrackingId to trackingId.
    Copy hdfaceStatus to status.
    release();
void 
_setGesture(
    vector<pair<CComPtr<IGesture>,float> >& discrete ,
    vector<pair<CComPtr<IGesture>,float> >& continuous ,
    vector<UINT64>& dTrackingId ,
    vector<INT64>& cTrackingId 
)
 version1.6 or later. thread-safe.
Call setGesture() and copy the results to arguments. Same as the following operations.
    acquire();
    setGesture();
    Copy discreteGesture to discrete.
    Copy continuousGesture to continuous.
    Copy discreteTrackingId to dTrackingId.
    Copy continuousTrackingId to cTrackingId.
    release();
string _gesture2string(const CComPtr<IGesture>& gesture ) version1.6 or later. thread-safe.
Call gesture2string(gesture).
void 
_setAudio(float& angle ,
          float& confidence ,
          UINT64& trackingId ,
          bool flag  = false)
 version1.6 or later. thread-safe.
Call setAudio(flag) and copy the results to argument. Same as the following operations.
    acquire();
    setAudio(flag);
    Copy beamAngle to angle.
    Copy beamAngleConfidence to confidence.
    Copy beamTrackingId to trackingId.
    release();
void 
_setSpeech(pair<wstring,wstring>& p )
 version1.6 or later. thread-safe.
Call setSpeech() and copy the results to arguments. Same as the following operations.
    acquire();
    setAudio(flag);
    Set pair of speechTag's copy and speechItem's copy to p.
    release();
HRESULT _MapCameraPointToColorSpace(
    CameraSpacePoint sp ,
    ColorSpacePoint *cp )		 version1.6 or later. thread-safe.
Call coordinateMapper->MapCameraPointToColorSpace(sp,cp).
HRESULT _MapCameraPointToDepthSpace(
  CameraSpacePoint sp ,
  DelpthSpacePoint *dp )		 version1.6 or later. thread-safe.
Call coordinateMapper->MapCameraPointToDepthSpace(sp,dp).
HRESULT _MapDepthPointToColorSpace(
  DepthSpacePoint dp ,
  UINT16 depth ,
  ColorSpacePoint *cp )		 version1.6 or later. thread-safe.
Call coordinateMapper->MapDepthPointToColorSpace(dp,depth,cp).
HRESULT _MapDepthPointToCameraSpace(
  DepthSpacePoint dp ,
  UINT16 depth ,
  CameraSpacePoint *sp )		 version1.6 or later. thread-safe.
Call coordinateMapper->MapDepthPointToCameraSpace(dp,depth,sp).

How to write program

  1. Start using the Visual Studio's project KinectV2_skeleton.zip of "NtKinect: How to recognize human skeleton with Kinect V2" .
  2. In order to understand this program, knowldge of the following topis is necessary.
  3. Change the contents of main.cpp as follows.
    4. main.cpp
    #include <iostream>
#include <sstream>

#define USE_THREAD
#include "NtKinect.h"

using namespace std;

NtKinect kinect;

unsigned __stdcall doJob1(LPVOID pParam) {
  cv::Mat image;
  vector<vector<Joint> > skel;
  vector<int> skelId;
  vector<UINT64> skelTrackingId;
  while (1) {
    kinect._setRGB(image);
    kinect._setSkeleton(skel,skelId,skelTrackingId);
    for (auto person: skel) {
      for (auto joint: person) {
	if (joint.TrackingState == TrackingState_NotTracked) continue;
	ColorSpacePoint cp;
	kinect._MapCameraPointToColorSpace(joint.Position,&cp);
	cv::rectangle(image,cv::Rect((int)cp.X-5,(int)cp.Y-5,10,10),cv::Scalar(0,0,255),2);
      }
    }
    cv::imshow("1", image);
    auto key = cv::waitKey(1);
    if (key == 'q') break;
  }
  cv::destroyWindow("1");
  _endthreadex(0);
  return 0;
}

void drawHand(cv::Mat& image, Joint& joint, pair<int,int>& state) {
  cv::Scalar colors[] = {
    cv::Scalar(255,0,0),  // HandState_Unknown
    cv::Scalar(0,255,0),  // HandState_NotTracked
    cv::Scalar(255,255,0), // HandState_Open
    cv::Scalar(255,0,255), // HandState_Closed
    cv::Scalar(0,255,255),  // HandState_Lass
  };
  ColorSpacePoint cp;
  kinect._MapCameraPointToColorSpace(joint.Position,&cp);
  cv::rectangle(image, cv::Rect((int)cp.X-8, (int)cp.Y-8, 16, 16), colors[state.first], 4);
}

unsigned __stdcall doJob2(LPVOID pParam) {
  cv::Mat image;
  vector<vector<Joint> > skel;
  vector<int> skelId;
  vector<UINT64> skelTrackingId;
  while (1) {
    kinect._setRGB(image);
    kinect._setSkeleton(skel,skelId,skelTrackingId);
    for (int i=0; i<skel.size(); i++) {
      Joint left = skel[i][JointType_HandLeft];
      Joint right = skel[i][JointType_HandRight];
      if (left.TrackingState != TrackingState_NotTracked) {
	auto state = kinect._handState(i,true);
	drawHand(image,left,state);
      }
      if (right.TrackingState != TrackingState_NotTracked) {
	auto state = kinect._handState(i,false);
	drawHand(image,right,state);
      }
    }
    cv::imshow("2", image);
    auto key = cv::waitKey(1);
    if (key == 'q') break;
  }
  cv::destroyWindow("2");
  _endthreadex(0);
  return 0;
}

unsigned __stdcall doJob3(LPVOID pParam) {
  cv::Mat image;
  while (1) {
    kinect._setBodyIndex(image,false);
    cv::imshow("3", image);
    auto key = cv::waitKey(1);
    if (key == 'q') break;
  }
  cv::destroyWindow("3");
  _endthreadex(0);
  return 0;
}

void doJob() {
  HANDLE hThread[3] = { 0 };
  hThread[0] = (HANDLE) _beginthreadex(NULL,0,doJob1,NULL,0,NULL);
  if (hThread[0] == 0) throw runtime_error("cannot create thread 0");
  hThread[1] = (HANDLE) _beginthreadex(NULL,0,doJob2,NULL,0,NULL);
  if (hThread[1] == 0) throw runtime_error("cannot create thread 1");
  hThread[2] = (HANDLE) _beginthreadex(NULL,0,doJob3,NULL,0,NULL);
  if (hThread[2] == 0) throw runtime_error("cannot create thread 2");
  DWORD ret = WaitForMultipleObjects(3,hThread,TRUE,INFINITE);
  if (ret == WAIT_FAILED) throw runtime_error("wait failed");
}

int main(int argc, char** argv) {
  try {
    doJob();
  } catch (exception &ex) {
    cout << ex.what() << endl;
    string s;
    cin >> s;
  }
  return 0;
}
  4. When you run the program, 4 windws are displayed.

    5. When you close all the windows except the console screen with 'q' key, the program ends and the console screen will be closed.

  5. Please clickhere for the sample project KinectV2_thread.zip.

    6. Since the above zip file may not include the latest "NtKinect.h", Download the latest version from here and replace old one with it.

How to write program (2)

  1. Start using the Visual Studio's project KinectV2_speech.zip of "NtKinect: How to recognize speech with Kinect V2 " .

    2. In this project, file necessary for speech recogniti0on have been added such as WaveFile.h, KinectAudioStream.h, KinectAudioStream.cpp, and Grammaer_jaJP.grxml. And set sapi.lib to be linked when build.

  2. In order to understand this program, knowldge of the following topis is necessary.
  3. Change the contents of main.cpp as follows.

    4. We call functions that are not thread-safe between kinect.acquire() and kinect.release() to avoid race conditions.

    main.cpp
    #include <iostream>
#include <sstream>

#define USE_SPEECH
#define USE_THREAD
#include "NtKinect.h"

using namespace std;

NtKinect kinect;

unsigned __stdcall doJob1(LPVOID pParam) {
  cv::Mat image;
  vector<vector<Joint> > skel;
  vector<int> skelId;
  vector<UINT64> skelTrackingId;
  while (1) {
    kinect._setRGB(image);
    kinect._setSkeleton(skel,skelId,skelTrackingId);
    for (auto person: skel) {
      for (auto joint: person) {
	if (joint.TrackingState == TrackingState_NotTracked) continue;
	ColorSpacePoint cp;
	kinect._MapCameraPointToColorSpace(joint.Position,&cp);
	cv::rectangle(image,cv::Rect((int)cp.X-5,(int)cp.Y-5,10,10),cv::Scalar(0,0,255),2);
      }
    }
    cv::imshow("1", image);
    auto key = cv::waitKey(1);
    if (key == 'q') break;
  }
  cv::destroyWindow("1");
  _endthreadex(0);
  return 0;
}

unsigned __stdcall doJob2(LPVOID pParam) {
  cv::Mat image;
  while (1) {
    kinect._setBodyIndex(image,false);
    cv::imshow("2", image);
    auto key = cv::waitKey(1);
    if (key == 'q') break;
  }
  cv::destroyWindow("2");
  _endthreadex(0);
  return 0;
}

unsigned __stdcall doJob3(LPVOID pParam) {
  ERROR_CHECK(CoInitializeEx(NULL, COINIT_MULTITHREADED));
  kinect.acquire();
  kinect.startSpeech();
  kinect.release();
  std::wcout.imbue(std::locale(""));
  
  pair<wstring,wstring> p;
  while (1) {
    bool ret = kinect._setSpeech(p);
    if (ret) {
      wcout << p.first << L" " << p.second << endl;
    }
    if (p.first == L"EXIT") break;
    //Sleep(1L);
  }
  kinect.acquire();
  kinect.stopSpeech();
  kinect.release();
  _endthreadex(0);
  return 0;
}

void doJob() {
  HANDLE hThread[3] = { 0 };
  hThread[0] = (HANDLE) _beginthreadex(NULL,0,doJob1,NULL,0,NULL);
  if (hThread[0] == 0) throw runtime_error("cannot create thread 0");
  hThread[1] = (HANDLE) _beginthreadex(NULL,0,doJob2,NULL,0,NULL);
  if (hThread[1] == 0) throw runtime_error("cannot create thread 1");
  hThread[2] = (HANDLE) _beginthreadex(NULL,0,doJob3,NULL,0,NULL);
  if (hThread[2] == 0) throw runtime_error("cannot create thread 2");
  DWORD ret = WaitForMultipleObjects(3,hThread,TRUE,INFINITE);
  if (ret == WAIT_FAILED) throw runtime_error("wait failed");
}

int main(int argc, char** argv) {
  try {
    doJob();
  } catch (exception &ex) {
    cout << ex.what() << endl;
    string s;
    cin >> s;
  }
  return 0;
}
  4. When you run the program, 3 windows are displayed.

    5. When all the thread ends, the program ends and the console screen will be closed.

    [Notice] It seems that the performance will be degraded if you operate speech recognition and skeleton recognition simultaneously, or speech recognition and gettin bodyIndex. In the former case, the skeleton recognition thread becomes very slow, and in the latter case the accuracy of speech recognition becomes bery poor. If you adjust the timing of action by putting Sleep(INT32) function in either thread, it may be a little better. But it is unconfirmed.

  5. Please clickhere for this sample project KinectV2_speech_thread.zip.

    6. Since the above zip file may not include the latest "NtKinect.h", Download the latest version from here and replace and use it.


http://nw.tsuda.ac.jp/