NtKinect: Kinect V2 C++ Programming with OpenCV on Windows10

How to get audio beam direction with Kinect V2


2016.07.18: created by
Japanese English
To Table of Contents

Prerequisite knowledge


Getting Audio Beam Direction

If you define USE_AUDIO constant before including NtKinect.h, the functions and variables of NtKinect related to audio become effective.

Calling the setAudio() function will put the direction of the suond in the member variable "beamAngle". From the viewpoint of a person facing Kinect V2, when moving the right, it becomes a positive value, and when moving to the left, it becomes a negative value.

NtKinect's function for Audio

type of return value function name descriptions
void setAudio(bool flag = false)

Acquire audio and save it in a file during recording
Determines the direction of the speech and sets it to the variable "beamAngle".

Argument
flag
When Calling this function with the argument set to "true" after setSkeleton() function call, the speaker's skeletonTrackingId[i] is set to the member variable "audioTrackingId".
void drawAudioDirection(cv::Mat& image ) Draw the direction of audio beam on image.
bool isOpenedAudio() Returns whether or not recording is in progress.
void openAudio(string path ) Open path as a recording file
void closeAudio() Close the recording file.

NtKinect's member variable for Audio

type variable name descriptions
float beamAngle Direction of audio (angle to the left and right)
float beamAngleConfidence Confidence value of beamAngle (0.0 ... 1.0)
UINT64 audioTackingId Speaker's skeletonTrackingId

How to write program

  1. Start using the Visual Studio's project KinectV2_audio.zip of " NtKinect: How to get and record audio with Kinect V2
  2. " .

    WaveFile.h should already have been added to the project.

  3. Change the contents of main.cpp as follows.
  4. Define USE_AUDIO constant before including NtKinect.h.

    When calling setAudio(), audio data is acquired by the array microphone and the direction is calculated. The result is in the "beamAngle" and "beamAngleConfidence" member variables. Draw the audio direction on cv::Mat by the drawAudioDirection() function call.

    main.cpp
    #include <iostream>
    #include <sstream>
    
    #define USE_AUDIO
    #include "NtKinect.h"
    
    using namespace std;
    
    void doJob() {
      NtKinect kinect;
      cv::Mat beam;
      while (1) {
        kinect.setRGB();
        kinect.setAudio();
        cv::putText(kinect.rgbImage, "beamAngle: "+to_string(kinect.beamAngle),
                    cv::Point(50, 50), cv::FONT_HERSHEY_SIMPLEX, 1.2, cv::Scalar(0, 0, 255), 1, CV_AA);
    // rename CV_AA as cv::LINE_AA (in case of opencv3 and later)
        cv::putText(kinect.rgbImage, "beamAngleConfidence: "+to_string(kinect.beamAngleConfidence),
                    cv::Point(50, 80), cv::FONT_HERSHEY_SIMPLEX, 1.2, cv::Scalar(0, 0, 255), 1, CV_AA);
        cv::imshow("rgb", kinect.rgbImage);
        kinect.drawAudioDirection(beam);
        cv::imshow("beam",beam);
        auto key = cv::waitKey(1);
        if (key == 'q') break;
      }
      cv::destroyAllWindows();
    }
    
    int main(int argc, char** argv) {
      try {
        doJob();
      } catch (exception &ex) {
        cout << ex.what() << endl;
        string s;
        cin >> s;
      }
      return 0;
    }
    
    
  5. When you run the program, RGB images are displayed. Exit with 'q' key.
  6. At the upper left of the window, the direction of audio and its reliability are displayed. The audio direction is displayed as a line in another window "beam".




  7. Please click here for this sample project KinectV2_audio2.zip
  8. Since the above zip file may not include the latest "NtKinect.h", Download the latest version from here and replace old one with it.



http://nw.tsuda.ac.jp/