NtKinectDLL and NtUnity: Tutorial:

Use the Audio Beam Direction on Unity acquired with Kinect V2


2017.08.25: created by
Japanese English
To Table of Contents

Prerequisite knowledge


Acquire the audio beam direction with Kinect V2 and use it in Unity

  1. I will use the Unity project created in "Using NtKinectDLL in Unity (Fundamental Settings)&auot;. Download UnityDLL01.zip and expand it. Also change the name of the expanded folder to "UnityDLL04/".
  2. Create a new scene and save it under "Assets/Scenes/" as "Sample04.unity".
  3. File -> New Scene
    
    File -> Save Scene as ... -> Sample04.unity
    
  4. Create a Folder "Assets/Materials/" in the project, and create 4 Materials in it.
  5. Change the name and Albedo of each Material as follows.

    NameAlbedo
    RGBA
    Red25500255
    Green02550255
    Blue00255255
    Ground2141640255






  6. Create 3 Cylinders in Herarchy. Change each name and set Transform and Material.
  7. Above Menu -> GameObject -> 3D Object -> Cylinder
    NamePositionRotationScaleMaterial
    XYZXYZXYZ
    X-axis 000 0090 0.1100.1 Red
    Y-axis 000 000 0.1100.1 Green
    Z-axis 000 9000 0.1100.1 Blue



  8. Create a Plane in Hierarchy. Change its name as "Ground" and set the Transfrom and Material as follows.
  9. Above Menu -> GameObject -> 3D Object -> Plane
    NamePositionRotationScaleMaterial
    XYZXYZXYZ
    Ground 000 000 555 Ground



  10. Create a Sphere in Hierarchy. Change its name as "Marker" and set the Transfrom. Leave its Material as white as default.
  11. above menu -> GameObject -> 3D Object -> Sphere
    NamePositionRotationScaleMaterial
    XYZXYZXYZ
    Marker 00.10 000 10.11 White (Default-Material)



  12. Create new C# script under "Assets/Scripts/" in the Project window.
  13. from above menu -> "Assets" -> "Create" -> "C# Script" -> Rename as "Sample04"

    Sample04.cs
    using System.Collections;
    using System.Collections.Generic;
    using UnityEngine;
    using System;
    
    public class Sample04 : MonoBehaviour {
      public GameObject target;
      public float radius = 2.0f;
      NtUnity.Kinect nt;
      void Start () {
        nt = new NtUnity.Kinect();
      }
      void Update () {
        nt.setAudio(false);
        if (nt.beamAngleConfidence < 0.5) return;
        double theta = nt.beamAngle;
        double x = radius * Math.Sin(theta);
        float y = target.transform.position.y; 
        double z = radius * (-Math.Cos(theta));
        target.transform.position = new Vector3((float)x,y,(float)z);
      }
    }
    
    
  14. Place "Empty Object" in Hierarchy, and change its name to "GameController". Add "Sample04.cs" as a component to it, and set "Marker" in Hierarchy to the "Target" item.
  15. above menu -> GameObject -> Create Empty
    









  16. With the "Main Camera" selected in Hierarchy, change its Transform to Position (x,y,z) = (0,1,-3), Rotateion=(20,0,0).
  17. When executed, the Marker moves so that there is as sound source on the line connecting the origin and the Marker.
  18. Runtime Video UnityDLL02.mp4



  19. The sample project of Unity is UnityDLL04.zip
  20. By the way, the movie at the head of this web page is obtained by adding the following 4 lines to the top of the Update() function of "Sample04.cs". For the privacy, face recognition is performed and face recognition is made to hide the face.
  21.      nt.setRGB();
         nt.setSkeleton();
         nt.setFace();
         nt.imshowBlack();
    


http://nw.tsuda.ac.jp/