NtKinectDLL and NtUnity: Tutorial:

Recognize Skeleton and Face with Kinect V2 and move Unity's Humanoid Character in Real Time

2017.08.23: created by
Japanese English

Sorry, this topics does not correspond to some Humanoid Model ("Ethan" of Starndard Asset, etc). (August/25/ 2017)

At NtUnity version 1.1, "Unity-Chan" 3D Model is supported. Please read this. (2017/09/09)

To Table of Contents


Prerequisite knowledge

Move Unity's Humanoid in real time with Skeleton and Face recognition by Kinect V2

  1. I will use the Unity project created in "Using NtKinectDLL in Unity (Fundamental Settings)&auot;. Download UnityDLL01.zip and expand it. Also change the name of the expanded folder to "UnityDLL02/".
  2. Create a new scene and save it under "Assets/Scenes/" as "Sample02.unity".
    3. File -> New Scene

    File -> Save Scene as ... -> Sample02.unity
  3. Make a new folder where you put Humanoid in the Asset.
    4. Right click in the Assets -> Create -> Folder -> Rename it as "Models"
  4. Import Humanoid data into "Assets/Models/".

    5. Here we use makehuman2.zip which is created in " MakeHuman: Create a Humanoid Model to use in Unity ".

    When expanding the above zip file, there should be AsianBoy.fbx, AsianGirl.fbx, AfricanBoy.fbx, AfricanGirl.fbx, CaucasianBoy.fbx, CaucasianGirl.fbx, and textures/ under "exports/". Import them under the "Assets/Models/".




    [CAUTION]You can do the above operation through 

    Assets -> Import New Asset... -> AsianBody.fbx
    , but in this way, some textures may not be imported and the model became white. If you meet the situation, you must import all the textures in Assets/Models/Materials/ manually.




  5. Select Humanoid (fbx) in the Project window, Change the Animation Type into "Humanoid" inthe Inspector's "Rig" window, click "Apply" button while Avatar Definition is "Create From This Model". When you apply the Humanoid Bone correctly, you can see the checked "Configure" in the Inspector window. If there is no checked mark before "Configure" button, click the "Configure" button and assign some Bones manually.

    6. The following figure shows an example of converting "AsianBoy.fbx" to Humanoid. Please convert all 6 fbx to Humanoid.







  6. Drag "Assets/Models/AsianBody.fbx" into Hierarchy. With "AsianBoy" selected in Hierarchy, click "Reset" from the Transform's "Settings" in the Inspector window.






  7. Create new C# script under "Assets/Scripts/" in the Project window.

    8. from above menu -> "Assets" -> "Create" -> "C# Script" -> Rename as "Sample02"

    Sample02.cs
    using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class Sample02 : MonoBehaviour {
  public GameObject humanoid;
  public bool mirror = true;
  public bool move = true;

  NtUnity.Kinect nt;
  NtUnity.HumanoidSkeleton hs;

  void Start () {
    nt = new NtUnity.Kinect();
    hs = new NtUnity.HumanoidSkeleton(humanoid);
  }

  void Update () {
    nt.setRGB();
    nt.setSkeleton();
    nt.setFace();
    nt.imshowBlack();
    int n = nt.getSkeleton();
    if (n > 0) {
      hs.set(nt,0,mirror,move);
    }
  }

  void OnApplicationQuit() {
    nt.stopKinect();
  }
}
  8. Place an "Empty Object" in Hierarchy and change the name to "GameController". Add "Sample02.cs" as a Component of "GameController".



  9. When "GameController" is selected in the Hierarchy window, the "Sample 02 (Scripts)" Component is displayed in the Inspector window. Assign "AsianBoy" in Hierarchy to the "Humanoid" item of the component.



  10. In Hierarchy, select "Main Camera" and change Position of Transform to (x,y,z)=(0,1,-1).
  11. When executed, the recognition status of skeleton and face will be displayed in a separate window. On the Unity's game window, Humanoid moves follwoing the movement of human beings. If "mirror" is checked, the mirror image will be displayed in face-to-face format. Also, if "move" is checked, it follows the change of human position.

    12. The orientation of human face is reflected in the face direction of Humanoid.

    [NOTICE] We generates an OpenCV window in DLL to display the skeleton recognition state. Note that when the OpenCV window is focused, that is, when the Unity game window is not focused, the screen of Unity will not change. Click on the top of Unity's window and make it focused, then try the program's behaviour.

    Runtime Video UnityDLL02.mp4





    w
  12. The sample project of Unity is here UnityDLL02.zip.

    13. Since the above zip file may not include the latest "NtKinectDLL.dll" and "NtUnity.cs", Download the latest version from here and replace old one with it.

Recognize multiple skeletons and reflect on the movement of Humanoids

We will create a project to move up to 6 Humanoids simultaneously.

  1. Expand the Unity project UnityDLL02.zip created above. Rename the folder as "UnityDLL02b/".
  2. Save the scene with a new name.
    3.   File -> Save Scence as -> "Sample02b.unity"
  3. Drag and add 5 Humanoids from "Assets/Models/" into the Hierarchy.

    4. In the Hierarchy, there are 6 Humanoids now. Set each Position and Rotation as follows. With the Z coordinate of Position set to -10, when the skeleton is not recognized, it is not shown in the "Main Camera".

    Model NamePositionRotation
    XYZXYZ
    AsianBoy00-10000
    AsianGirl00-10000
    AfricanBoy00-10000
    AfricanGirl00-10000
    CaucasianBoy00-10000
    CaucasianGirl00-10000



  4. Craete new C# script "Sample02b.cs" at "Assets/Scripts/"
    5. Sample02b.cs
    using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class Sample02b : MonoBehaviour {
  public GameObject[] humanoid = new GameObject[] {null, null, null, null, null, null};
  public bool[] mirror = new bool[] {true, true, true, true, true, true};
  public bool[] move = new bool[] {true, true, true, true, true, true};

  NtUnity.Kinect nt;
  NtUnity.HumanoidSkeleton[] hs = new NtUnity.HumanoidSkeleton[] {null, null, null, null, null, null};
  
  void Start () {
    nt = new NtUnity.Kinect();
    for (int i=0; i< humanoid.Length; i++) {
      if (humanoid[i] != null) {
	hs[i] = new NtUnity.HumanoidSkeleton(humanoid[i]);
      }
    }
  }

  void Update () {
    nt.setRGB();
    nt.setSkeleton();
    nt.setFace();
    nt.imshowBlack();
    for (int i=0; i<nt.skeleton.Count; i++) {
      if (hs[i] != null) {
	hs[i].set(nt,i,mirror[i],move[i]);
      } else {
	humanoid[i].transform.position = new Vector3(0,0,-10);
      }
    }
  }

  void OnApplicationQuit() {
    nt.stopKinect();
  }
}
  5. With the "GameController" selected in the Hierarchy, Remove "Sample 02 (Script)" component in the Inspector window.



  6. Drag and drop "Assets/Scripts/Sample02b.cs" onto the "GameController" in the Hierarchy.



  7. Select 6 models for "Humanoid" in the Inspector's "Sample 03 (Script)" component, which is displayed when you select "GameObject" in the Hierarchy.



  8. Save the scene.
    9. File -> Save Scenes
  9. When executed, up to 6 people will be recognized simultaneously and Humanoid will move in the Unity's Game Window.
    10. Runtime Video UnityDLL02.mp4


  10. The Unity's sample project is here UnityDLL02b.zip。

    11. Since the above zip file may not include the latest "NtKinectDLL.dl" and "NtUnity.cs", Download the latest version from here and replace old one with it.


http://nw.tsuda.ac.jp/