NtKinect: Kinect V2 C++ Programming with OpenCV on Windows10

How to make RGB image correspond to Depth image with Kinect V2

Prerequisite knowledge

• NtKinect: How to get RGB camera image with Kinect V2 (Fundamental Settings)
• NtKinect: How to get Depth image with Kinect V2

Getting RGB came image with Kinect V2

Kinect V2 can acquire RGB camera image with resolution of 1920 x 1080. Since OpenCV uses BGR format or BGRA format as the basis, NtKinect adopts the BGRA format.

NtKinect's member functions for RGB image

NtKinect's member variable for RGB image

Getting Depth image

Depth (distance) images can be acquired with a resolution of 512 x 424. The measurable distance range is from 500 mm to 8000 mm, but the range to recognize human beings is from 500 mm to 4500 mm.

In the Kinect20.lib IDepthFrameSource has the "get_DepthMaxReliableDistance()" and "get_DepthMaxReliableDistance()" functions, each returns 500 and 4500 respectively.

In NtKinect, the obtained Depth image is represented by UINT16 (16 bit unsigned integer) for each pixel.

NtKinect's function for Depth image

NtKinect's member variable for Depth iamge

3 types of coordinate system of Kinect V2

Since the position and resolution of each sensor is different, the data is obtained as a value expressed in the coordinate system of each sensor. When using data obtained from different sensors at the same time, it is necessary to convert the coordinates to match.

Kinect V2 has 3 coordinate systems, ColorSpace, DepthSpace, and CameraSpace. There are 3 data types ColorSpacePoint, DepthSpacePoint, and CameraSpacePoint representing coordinates in each coordinate system.

typedef struct _ColorSpacePoint {
    float X;
    float Y;
} ColorSpacePoint;

typedef struct _DepthSpacePoint {
    float X;
    float Y;
} DepthSpacePoint;

typedef struct _CameraSpacePoint {
    float X;
    float Y;
    float Z;
} CameraSpacePoint;

Coordinate systems and data types of Kinect V2

For the RGB image, Depth image, and skeleton information, the coordinate system is different. The coordinate system of the RGB image is ColorSpace, that of the Depth image is DepthSpace, and that of the skeleton information is CameraSpace.

Kinect V2's function for mapping coordinate systems

"Coordinate system conversion function" held by ICoordinateMapper class of Kinect V2 is as follows.

NtKinect's member variable for mapping coordinate system

An instance of ICoordinateMapper class used for mapping coordinate systems in Kinect V2 is held in NtKinect's member variable "coordinateMapper".

How to write program

Call kinect.setDepth() function to set depth (distance) data to kinect.depthImage. Since no argument is specified, the value of pixel is raw, that is, the distance to the object in millimeters.

By painting pixels farther than 2000 mm with blue, only images of nearby objects are displayed. Since we can not measure a value smaller than 500 mm, we treat that pixel as being more than 8000 mm.

Since RGB image and Depth image are different in resolution, one pixel of Depth image corresponds to 4 pixels of the RGB image and the 4 x 4 rectangle area is painted.

#include <iostream>
#include <sstream>

#include "NtKinect.h"

using namespace std;

void doJob() {
  NtKinect kinect;
  while (1) {
    kinect.setRGB();
    kinect.setDepth();
    for (int y=0; y<kinect.depthImage.rows; y++) {
      for (int x=0; x<kinect.depthImage.cols; x++) {
        UINT16 d = kinect.depthImage.at<UINT16>(y,x);
        DepthSpacePoint dp; dp.X=x; dp.Y=y;
        ColorSpacePoint cp;
        HRESULT hr = kinect.coordinateMapper->MapDepthPointToColorSpace(dp,d,&cp);
        if (hr != S_OK) continue;
        if (d > 2000 || d < 500) {
          int ax = (int) cp.X;
          int ay = (int) cp.Y;
          cv::rectangle(kinect.rgbImage,cv::Rect(ax-2,ay-2,4,4),cv::Scalar(255,0,0),2);
        }
      }
    }
    cv::imshow("rbg near", kinect.rgbImage);
    auto key = cv::waitKey(1);
    if (key == 'q') break;
  }
  cv::destroyAllWindows(); 
}

int main(int argc, char** argv) {
  try {
    doJob();
  } catch (exception &ex) {
    cout << ex.what() << endl;
    string s;
    cin >> s;
  }
  return 0;
}

Since the above zip file may not include the latest "NtKinect.h", Download the latest version from here and replace old one with it.