NtKinect: Kinect V2 C++ Programming with OpenCV on Windows10

How to get audio beam direction with Kinect V2

2016.07.18: created by

To Table of Contents

Prerequisite knowledge

NtKinect: How to get and record audio with Kinect V2

Getting Audio Beam Direction

If you define USE_AUDIO constant before including NtKinect.h, the functions and variables of NtKinect related to audio become effective.

Calling the setAudio() function will put the direction of the suond in the member variable "beamAngle". From the viewpoint of a person facing Kinect V2, when moving the right, it becomes a positive value, and when moving to the left, it becomes a negative value.

NtKinect

NtKinect's function for Audio

type of return value	function name	descriptions
void	setAudio(bool flag = false)	Acquire audio and save it in a file during recording Determines the direction of the speech and sets it to the variable "beamAngle". Argument flag When Calling this function with the argument set to "true" after setSkeleton() function call, the speaker's skeletonTrackingId[i] is set to the member variable "audioTrackingId".
void	drawAudioDirection(cv::Mat& image )	Draw the direction of audio beam on image.
bool	isOpenedAudio()	Returns whether or not recording is in progress.
void	openAudio(string path )	Open path as a recording file
void	closeAudio()	Close the recording file.

NtKinect

NtKinect's member variable for Audio

type	variable name	descriptions
float	beamAngle	Direction of audio (angle to the left and right)
float	beamAngleConfidence	Confidence value of beamAngle (0.0 ... 1.0)
UINT64	audioTackingId	Speaker's skeletonTrackingId

How to write program

Start using the Visual Studio's project KinectV2_audio.zip of " NtKinect: How to get and record audio with Kinect V2

WaveFile.h should already have been added to the project.

Change the contents of main.cpp as follows.

Define USE_AUDIO constant before including NtKinect.h.

When calling setAudio(), audio data is acquired by the array microphone and the direction is calculated. The result is in the "beamAngle" and "beamAngleConfidence" member variables. Draw the audio direction on cv::Mat by the drawAudioDirection() function call.

main.cpp

#include <iostream>
#include <sstream>

#define USE_AUDIO
#include "NtKinect.h"

using namespace std;

void doJob() {
  NtKinect kinect;
  cv::Mat beam;
  while (1) {
    kinect.setRGB();
    kinect.setAudio();
    cv::putText(kinect.rgbImage, "beamAngle: "+to_string(kinect.beamAngle),
                cv::Point(50, 50), cv::FONT_HERSHEY_SIMPLEX, 1.2, cv::Scalar(0, 0, 255), 1, CV_AA);
// rename CV_AA as cv::LINE_AA (in case of opencv3 and later)
    cv::putText(kinect.rgbImage, "beamAngleConfidence: "+to_string(kinect.beamAngleConfidence),
                cv::Point(50, 80), cv::FONT_HERSHEY_SIMPLEX, 1.2, cv::Scalar(0, 0, 255), 1, CV_AA);
    cv::imshow("rgb", kinect.rgbImage);
    kinect.drawAudioDirection(beam);
    cv::imshow("beam",beam);
    auto key = cv::waitKey(1);
    if (key == 'q') break;
  }
  cv::destroyAllWindows();
}

int main(int argc, char** argv) {
  try {
    doJob();
  } catch (exception &ex) {
    cout << ex.what() << endl;
    string s;
    cin >> s;
  }
  return 0;
}

When you run the program, RGB images are displayed. Exit with 'q' key.

At the upper left of the window, the direction of audio and its reliability are displayed. The audio direction is displayed as a line in another window "beam".

Please click here for this sample project KinectV2_audio2.zip。

Since the above zip file may not include the latest "NtKinect.h", Download the latest version from here and replace old one with it.

http://nw.tsuda.ac.jp/