How to Implement and Test Body Position Recognition with Microsoft Kinect

Since Microsoft released its Microsoft Kinect Software Development Kit (SDK) in June 2011, it is possible for everyone to implement gesture based interaction interfaces with the help of the Kinect sensor. Because I believe that this kind of new forms of human computer interaction (HCI) is emerging in the future also in the field of enterprise applications, together with a friend I started implementing a simple client which recognizes basic body positions. My friend was laughing when I created a test class as the first step and he claimed that my beloved Test Driven Development (TDD) approach would not work this time. This article shall show how easy it is to identify the user’s body position with the Microsoft Kinect and proof that my friend was wrong.


Getting the Skeleton

The Kinect SDK basically connects to available Kinect sensors and delivers tracked skeletons of up to two persons. A skeleton consists of a pre-defined set of so-called “joints” which represent skeleton points with their X, Y and Z coordinates as well as their tracking state. You will get all of this out-of-the box by writing only a couple of lines of code. Everything what you need is explained at the developer manual.
As an example: My event handler for an skeleton update looks like this:

private void Sensor_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
	using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame())
	{
		if (skeletonFrame != null && skeletonFrame.SkeletonArrayLength > 0)
		{
			if (skeletons == null || skeletons.Length 
					!= skeletonFrame.SkeletonArrayLength)
			{
				skeletons = 
					new Skeleton[skeletonFrame.SkeletonArrayLength];
			}
			else
			{
				for (int i = 0; i < skeletons.Length; i++)
				{
					skeletons[i] = null;
				}
			}

			skeletonFrame.CopySkeletonDataTo(skeletons);

			Skeleton updatedSkeleton = null;
			
			// is already tracked skeleton updated?
			if (this.trackedSkeleton != null)
			{
				for (int i = 0; i < skeletons.Length; i++)
				{
					if (skeletons[i].TrackingId == 
						this.trackedSkeleton.TrackingId)
					{
						updatedSkeleton = skeletons[i];
						break;
					}
				}
			}

			// get updated skeleton
			if (updatedSkeleton == null)
			{
				double closestX = 1;
				for (int i = 0; i < skeletons.Length; i++)
				{
					Skeleton newSkeleton = skeletons[i];
					if (newSkeleton.TrackingState 
						!= SkeletonTrackingState.NotTracked &&
						Math.Abs(newSkeleton.Position.X) < closestX)
					{
						updatedSkeleton = skeletons[i];
					}
				}
			}

			this.trackedSkeleton = updatedSkeleton;

			// handle skeleton update
			if (updatedSkeleton != null && this.sensor != null)
			{
				this.SkeletonUpdated(updatedSkeleton);
				return;
			}
		}

		this.trackedSkeleton = null;
	}
}

private void SkeletonUpdated(Skeleton skeleton)
{
	// gesture recognition goes here...
}

Writing the Test

The scenario to implement is the tracking of the right upper arm. We want to know whether the right arm is up or points to the front. The idea is to calculate the horizontal and vertical angle between the vectors of the joints ShoulderLeftShoulderRight and ShoulderRightElbowRight. We hold the data in the following model:

A class BodyRecognition could have a static method RecognizePositions(Skeleton skeleton, Body body) which calculates the angles and saves them into the model.

As announced, we write a test first because the whole thing requires a bit of mathematics and it would be difficult to verify the exact results while jumping in front of the Kinect sensor. For this purpose I created a “skeleton creator” which simply creates a new skeleton and adds some fake joints, e.g.

class SkeletonCreator
{
	public static Skeleton CreateRightUpperArmSideAndUpSkeleton()
	{
		Skeleton skeleton = new Skeleton();

		updateJoint(skeleton, JointType.ShoulderLeft, 
			0.09f, 0.1f, 0.001f);
		updateJoint(skeleton, JointType.ShoulderRight, 
			0.1f, 0.1f, 0.001f);
		updateJoint(skeleton, JointType.ElbowRight, 
			0.15f, 0.15f, 0.001f);

		return skeleton;
	}

	private static void updateJoint(Skeleton skeleton, 
		JointType jointType, float x, float y, float z)
	{
		Joint joint = skeleton.Joints[jointType];
		SkeletonPoint point = joint.Position;
		point.X = x;
		point.Y = y;
		point.Z = z;
		joint.TrackingState = JointTrackingState.Tracked;

		joint.Position = point;
		skeleton.Joints[jointType] = joint;
	}
}

Which enables our first test:

[TestClass]
public class BodyRecognitionTests
{
	[TestMethod]
	public void TestRightArmUpAndSideRecognition()
	{
		Skeleton skeleton = 
			SkeletonCreator.CreateRightUpperArmSideAndUpSkeleton();
		Body body = new Body();
		BodyRecognition.RecognizePositions(skeleton, body);

		Assert.AreEqual(45, body.RightUpperArm.VerticalAngle);
	}
}

Identify the Body Position

Calculating the angles is then very straight forward if you know how to create a vector between two points:

public static SkeletonVector CalculateVector(
	SkeletonPoint a, SkeletonPoint b)
{
	return new SkeletonVector {
		X = b.X - a.X,
		Y = b.Y - a.Y,
		Z = b.Z - a.Z
	};
}

and how to compute an angle:

private static int calculateAngle(float x1, float y1, float x2, float y2)
{
	double angle = Math.Atan2(x1, y1) - Math.Atan2(x2, y2);
	return Convert.ToInt32(angle * (180.0 / Math.PI));
}

This looks simple, and it is in fact simple, but also has tradeoffs. Every person will have different values for the angles, depending on the exact position of their skeleton joints. E.g., we realized that the angle between ShoulderCenter and ShoulderRight differs between our test users quite heavily. For our application it is still good enough since we do not work with exact numbers, but it just reacts when the user raises his arm very clearly.

Related Resources

Advertisements

Tags: , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: