OnlyKinect

Cross platform person detection and skelatal tracking for the Kinect V1 (Xbox 360 / Kinect for Windows) and Kinect V2 (Xbox One).

• up to 6 people, 23 joints in 3D space, confidence and tracking state per joint
• hand state per hand: open / closed / unknown

Pipeline

The interface follows the classes behind it. Each thing has its own object and its own functions:

object (C)	class (C#)	C++ class behind it	what lives there
OkDeviceList	KinectDeviceList	KinectHandler	get connected kinects
OkDevice	KinectDevice	KinectDevice	near mode, motor, V2 depth pipeline, serial
OkStream	KinectStream, ExternalStream	FrameStream (KinectStream, ExternalStream)	raw depth / color, photos, camera model
OkTracker	HumanoidTracker	HumanoidTracker	people, skeletons, hand states, frame rate, motor following
OkFrame	TrackerFrame	HumanoidTracker::FrameResult	everything made of ONE frame, kept together
OkSegmentation	BodySegmentation	BodyCutout	which pixel is whom, cutouts and photos of people
OkSkeletonFilter	SkeletonFilter	SkeletonPost	anchor and freeze

Functions are named okObject_Method (okDevice_SetNearMode, okStream_GetPhoto, okSegmentation_GetDepth, ...). Every object you create has a Destroy / Release; a tracker keeps its stream alive and a stream keeps the device list alive, so the order in which you destroy things does not matter.

Build

Per platform (Linux, Windows, macOS), with and without CUDA, check BUILDING.md.

Usage (C)

#include "OnlyKinect.h"

OkDeviceList* devices;  okDeviceList_Create(&devices);
okDeviceList_Detect(devices);
OkDevice* device = okDeviceList_GetDevice(devices, 0);

okDevice_SetNearMode(device, 1);                      /* optional, Kinect V1 for Windows. Default: off */

OkStream* stream;       okStream_CreateForDevice(device, &stream);
OkTracker* tracker;     okTracker_Create(stream, NULL, &tracker);     /* NULL = default settings */
okTracker_Start(tracker);

uint64_t last = 0;
while (running) {
    if (!okTracker_WaitForFrame(tracker, last, 100)) continue;
    OkFrame* frame;
    if (okTracker_AcquireFrame(tracker, &frame) != OK_SUCCESS) continue;
    last = okFrame_GetNumber(frame);

    OkSkeleton skeletons[6];
    int n = okFrame_GetSkeletons(frame, skeletons, 6);
    /* ... skeletons[i].joints[OK_JOINT_HEAD].x / .y / .z in meters, skeletons[i].leftHand.state ... */
    okFrame_Release(frame);
}
okTracker_Destroy(tracker);  okStream_Destroy(stream);  okDeviceList_Destroy(devices);

Usage (C#)

Copy bindings/csharp/OnlyKinect.cs into your project and put the native library next to your program.

using (var devices = new KinectDeviceList()) {
    devices.Detect();
    using (KinectStream stream = devices[0].CreateStream())
    using (var tracker = new HumanoidTracker(stream)) {
        tracker.Start();
        ulong last = 0;
        while (tracker.WaitForFrame(last, 100)) {
            using (TrackerFrame frame = tracker.AcquireFrame()) {
                last = frame.Number;
                foreach (OkSkeleton skeleton in frame.GetSkeletons())
                    Console.WriteLine(skeleton.personId + ": " + skeleton.leftHand.State);
            }
        }
    }
}

Classes

KinectDevice - NearMode (Kinect V1, default off; only the Kinect for Windows has it, an Xbox 360 Kinect refuses it and the call returns "not supported"; the device remembers the setting and applies it when a stream opens it), Tilt (V1 motor), DepthPipeline / ActiveDepthPipeline (V2), serial, connection state.

Stream (KinectStream, or ExternalStream for frames you push) - the pictures of the running stream: GetDepth (raw, mm), GetColor (RGB / RGBA, lined up with the depth), TakePhoto (PNG, BMP or raw bytes), GetIntrinsics, FrameCount, WaitForFrame. The color picture has the depth resolution (640x480 on a V1, 512x424 on a V2), not the V2's 1920x1080. PNG is stored without compression (no dependency): about 0.9 MB at 640x480.

Tracker - Start / Stop, EstimateSkeletons, HandStates, SetDetectionRange, SkeletonFilter, person events, and:

• FrameRate: (default 30, 0 = every frame). The sensors deliver 30 at max; frames that come too early are dropped before any work is done, which also saves CPU.
• FollowMotion (Kinect V1 motor): FloorFirst (default) keeps the floor and the feet in view and cuts off a head that does not fit; HeadFirst keeps the head in view first, tilts up, and can lose the floor

Frame (TrackerFrame) - the people, skeletons, intrinsics and segmentation of ONE processed frame, so they always belong together. Skeletons come with the tracker's skeleton filter applied.

Segmentation (BodySegmentation) - GetBodyIndex (person id per pixel, 255 = nobody), CountPixels, and cutouts of depth, color and photos. Which people a cutout keeps: PersonSelection.Whole (no cutout), PersonSelection.Everybody (every tracked person, background cleared) or PersonSelection.Of(1, 2). In C: idCount < 0, 0 or > 0. crop cuts the picture to the people (the info says where it starts in the full picture). A PNG of selected people is RGBA with a transparent background.

SkeletonFilter - the order is track -> anchor -> freeze.

• Anchor: the spine base is held at one place (only the axes you choose) and every other joint keeps its offset, so arms, legs and bending are still seen. OkAnchor.Horizontal is x and z (leave y free to keep jumping and crouching); Yaw also turns the body to face the sensor. No position = "where the person is when it takes effect". Person -1 = everybody, also people who appear later.
• Freeze: freezes parts (head, torso, each arm, each hand incl. its open/closed state, each leg, each foot) or single joints completely, after everything else is done: they keep the pose they had when frozen, the rest keeps moving. FollowTorso lets the frozen pose move and turn with the torso.
• A joint's pixel position always describes where the real joint is in the depth image, anchored or frozen.
• A filter can be attached to a tracker, or applied to any skeleton (Apply).

Hand state - Open / Closed / Unknown with a confidence for the first two (0 for Unknown). It comes from the depth picture around the hand, with the color picture as a helper (skin color learned from the face, sleeves recognised). A hand that cannot be judged (too far, merged with the body, cut off) is reported as Unknown.