Can you imagine not being able to see in three dimensions? Like millions of people around the world, I have strabismus or “cross-eye,” which has caused my brain to ignore input from my non-dominant eye. While it was long thought that strabismus could not be treated after a critical threshold of 8-12 years, recent research has shown that manipulating images shown to both eyes can be an effective treatment.

That’s why I started an IndieGoGo campaign for Diplopia, a virtual reality video game that would help strabismus and amblyopia (lazy eye) sufferers. With Diplopia, I use the Oculus Rift to deliver a different image to each eye, in order to to trick the player’s brain into strengthening their weaker one. The game itself is a 3D block-breaking game with a paddle and ball..

Using Leap Motion interaction, the game lets the player control the paddle in a natural and fluid way. In the video you see above, you can see a fuller explanation and the first version of the game. In this post, we’ll take a peek under the hood of Diplopia, which is built in Unity Pro.

(Update: Diplopia is now Vivid Vision. As of April 2015, their debut product is rolling out to eye clinics throughout the US.)

Dammit Jim!

Before we get started, I should mention that I’m a game developer, not a doctor – Diplopia has not yet been proven to help those with amblyopia, and you should consult your eye doctor before playing it. I’d encourage anyone who has questions about the game’s design to check out my site’s FAQ or contact me @jamesblaha.

Why the Leap Motion Controller?

Put simply, the Leap Motion Controller is a perfect match for the Oculus Rift and virtual reality more generally. When you are inside a virtual reality, it can be quite difficult to locate individual keys on your keyboard, and using a mouse is limiting when you want to interact and move in three dimensions.

The Leap Motion Controller allows people to see where their hands are in-game, opening up new and exciting ways of interacting with their virtual environment. It provides a more immersive experience which also makes it easier for our player to interact with the UI in virtual reality. By feeling engaged with the game world and immersed in the interactions, my hope is that Diplopia will make retraining lazy eyes fun.

image

In Diplopia, you can manipulate the contrast of game elements such as the bricks, ball, and paddle in order to force your brain into integrating the two images. By showing only some of the game elements to each eye, Diplopia forces the player to incorporate information coming from both in order to win.

Integrating Diplopia and the Leap Motion Controller

To get started, I downloaded M1 Interactive’s Leap Motion Unity package and imported it into my project. With this, I was I was able to get up and running extremely quickly!

Since my game was written in JavaScript and the library files are written in C#, I had to put the C# files in the Plugins folder so they would be compiled first. Then I made a LeapControl.js to access the Leap frame data and control my palm/finger GameObjects.

The first thing I added was the following code:

function Update () {
	var leapFrame = leapScript.GetFrame();

	if(leapFrame != null) {
		//leap is sending us data
		if(leapFrame.Hands.Count > 0) {
			//there is at least one hand visible
			//do stuff with the hand here
var palmPosition = Vector3(leapFrame.Hands[0].PalmPosition.x,leapFrame.Hands[0].
PalmPosition.y,leapFrame.Hands[0].
PalmPosition.z);
Debug.Log(palmPosition);

}
}
}

Just for a moment, go back and take another look at the first video. In this version, I wasn’t using any kind of input smoothing – just PalmPosition and TipPosition.  Now take a look at the updated gameplay video below, where I use position smoothing. You can see a dramatic improvement in the input:

The new version uses StabilizedPalmPosition and StabilizedTipPosition. As a result, it’s much more stable, with no lost frames or flickering fingertips. You can see the code I use to move the palm/finger GameObjects and the paddle below. I also use the Leap Motion Controller to select menu items, by casting a ray from the camera through the palm to detect if it overlaps with a selectable menu item from the player’s perspective.

function Update () {
	//where leapScript is the C# leap lib script
	var leapFrame = leapScript.GetFrame();

	if(leapFrame != null) {
		if(leapFrame.Hands.Count > 0) {
			//update the palm game object
			var palmPosition = Vector3(leapFrame.Hands[0].StabilizedPalmPosition.x,leapFrame.Hands[0].
			StabilizedPalmPosition.y,leapFrame.Hands[0].StabilizedPalmPosition.z);

			//if we have a hand render the GUI gameobject for the palm and set its position
			if(!transform.renderer.enabled) transform.renderer.enabled = true;
			transform.position.x = palmPosition.x*leapXratio;
			transform.position.y = ((palmPosition.y-leapYoffset)*leapYratio) + palmYoffset;
			transform.position.z = transform.parent.gameObject.transform.position.z - ((palmPosition.
			z-leapZoffset)*leapZratio) + palmZoffset;

			//lock the paddle to the palm
			paddle.transform.position.x = palmPosition.x*paddle.GetComponent(PaddleControl).leapXratio;
			paddle.transform.position.y = (palmPosition.y-paddle.GetComponent(PaddleControl).
			leapYoffset)*paddle.GetComponent(PaddleControl).leapYratio;

			//update the finger GUI gameobjects
			var fingerCounter = 0;
			while(fingerCounter < 5) {
				//update the position of the fingertip
				var thisFinger : GameObject = fingers[fingerCounter];
				if(leapFrame.Hands[0].Fingers.Count > fingerCounter) {
					thisFinger.transform.position.x = leapFrame.Hands[0].Fingers[fingerCounter].
					StabilizedTipPosition.x*leapXratio;
					thisFinger.transform.position.y = ((leapFrame.Hands[0].Fingers[fingerCounter
					].StabilizedTipPosition.y-leapYoffset)*leapYratio) + palmYoffset;
					thisFinger.transform.position.z = transform.parent.gameObject.transform.
					position.z - ((leapFrame.Hands[0].Fingers[fingerCounter].
					StabilizedTipPosition.z-leapZoffset)*leapZratio) + palmZoffset;
					thisFinger.renderer.enabled = true;
				} else {
					//turn off the finger ui gameobject
					thisFinger.renderer.enabled = false;
				}

				fingerCounter++;
			}

        			//draw a ray from the camera through the palm, ignoring the palm gameobject
        			var length : float = 50.0f;
			var hit : RaycastHit;
			var rayDirection = (transform.position - cameraTransform.position).normalized;
			var rayStart = cameraTransform.position + rayDirection;
			var guiLayerMask : int = 1 >> 0;

        			//Debug.DrawRay(rayStart, rayDirection * length, Color.red);
        			LookAtScript.leapHit = false;
        			if(Physics.Raycast(rayStart, rayDirection, hit, length, guiLayerMask)) {

				if(hit.transform.gameObject.tag == "UI") {
					LookAtScript.leapHit = true;
					LookAtScript.leapHitObj = hit;
				}
			}
		} else {
			//turn off the palm UI gameobject
			if(transform.renderer.enabled) transform.renderer.enabled = false;
		}
		//turn on the leap GUI notification
		GuiScript.leapEnabled = true;
	} else {
		//turn off the leap GUI notification
		if(transform.renderer.enabled) transform.renderer.enabled = false;
		GuiScript.leapEnabled = false;
	}
}

Overall, it was quite easy to get started using Leap Motion in Unity. Any Oculus Rift developer should seriously consider supporting the Leap Motion Controller to add the depth and user experience it can provide for your game. So far, I have only implemented the basic functionality for the Leap Motion interface I want in the final game. So much is possible when you’re able to get full 3D gesture control in a fully 3D simulated environment, and I can’t wait to see what else will become possible with the Leap Motion and Oculus Rift together.

I’d like to hear your thoughts about the possibilities of gaming and new interface technologies. By making therapeutic actions fun and entertaining, game developers can help people address a wide variety of conditions. What sorts of uses can you imagine with the Leap Motion Controller?

James Blaha has been coding for the Leap Motion Controller since early 2013 on LeapGamer and LeapWebApp, where he hosts a number of small demos, experiments, apps, and games. You can follow James’  work with Diplopia on his website.

Recommended Posts