I am working with the Leap Motion Controller and have come across a “soft area” in the API – there is not a direct bridge between the Unity web player and the Leap Motion Controller. This has to do with the fact that the Unity web player is “sandboxed” to prevent its having access to larger hardware, including the Leap Motion Controller. Fortunately, there is a way around this. The Leap.js library can communicate with the Leap Motion Controller as a web socket. The documentation describes in detail the API that JS uses to access the device.
The Basic Work Flow
We will be establishing four points of communication between the Unity Player context and JavaScript. Each will have two functions – a “listen” and a “say” function. These methods will be fed by a Leap.Controller event listener, and will help to limit the flow of information to Unity to that which the Unity player has the processing time to render. 
A Few Words of Warning
- The JS API is NOT an exact match to the Unity API. Notably absent are “Frontmost” helpers; you’ll have to find the frontmost hand, and its frontmost finger, yourself.
- You will be sending text (or JSON as text) between the two systems – JavaScript and Unity. This may become a bit of a bottleneck if large amounts of information are sent back and forth frequently, so work to only send back and forth what you need.
- Sending more information to Unity than it can handle due to its render cycles is counterproductive, so we’ll need a “handshaking” mechanism between the two environments.
- The units and coordinate spaces are different between Unity and Leap.js; specifically, the Z space seems to be reversed.
Unity and JavaScript Talking to Each Other
The interaction between the Unity web app and JavaScript is well documented in Unity. In short:
- Unity calls JavaScript in a ‘JSONP-like’ method – you pass it the name of a (global) JavaScript function and a single argument (string) to pass into that method.
- JavaScript calls Unity in a similar fashion. It passes a string to the method of an object.
The JavaScript End of the Communication
The following is a JavaScript script for listening and talking to Unity:
<code class="lang-javascript"><code class="lang-javascript">function SaySomethingToUnity(message) {
u.getUnity().SendMessage("LeapManager", "ListenWeb", message);
// @param LeapManager the name of my Leap script in my application
}</code></code>// called by Unity engine
function HearSomethingFromUnity(says) {
console.log('unity says ', says);
}
The Unity C# scripts
Below is the Unity script for talking to and from JavaScript, in C#.
<code class="lang-c#"><code class="lang-c#"> public static void TellWeb (string s)
{
//Debug.Log ("TellWeb " + s);
Application.ExternalCall ("HearSomethingFromUnity", s);
}</code></code>public void ListenWeb (string s)
{
try {
JSONObject j = new JSONObject (s);
AccessData (j, "__root");
} catch (UnityException ex) {
TellWeb ("OOPS!" + ex.ToString ());
}
ProcessFingers ();
}private void AccessData ( JSONObject j, string context ) { // more later
}private void ProcessFingers(){ // more later
}
\\ ...
}
Of course, this presumes that you are sending data back and forth in JSON format. This is not a requirement – you could use a custom format, CSV, or whatever you want – but JSON is fairly standard, so I am running with it. JSONObject is a popular codec for JSON in Unity/C#. (Note that the inline code file in the Wiki page is the one that works in Unity’s “Flavor” of C#; use it, not the JSONObject file in Google Code.)
Handling the Difference between Unity’s FPS and Leap Motion’s FPS
Why are we listening to Unity? First off, for debugging – we want confirmation that our data is being received and decoded by our Unity scripts. Secondly, because we want the JavaScript to “Sleep” when Unity is in a data digestion cycle. The JavaScript Leap Motion player can receive hundreds of frames a second, while the Unity player can realistically render 30 frames a second. This means that we will either be buffering 10-20 frames in JavaScript, or just listening to the last one every 10-20 frames.
Teaching Unity and JavaScript to Shake Hands
The simplest way to throttle JavaScript is to have a local “state” flag that we flip on when we send a message, then flip off when we receive feedback from Unity.
- When JavaScript sends a command to Unity, we will set a local variable,
unity_working, to true. - When Unity has received and finished processing, the message it will say
frame_doneback. JavaScript, on hearing this, will setunity_workingto false. - When
unity_workingis set to true, JavaScript will not harass the Unity player with data it doesn’t have the bandwidth to process.
Our JavaScripts now look like this:
<code class="lang-javascript"><code class="lang-javascript">var unity_working = false;
function SaySomethingToUnity(message) {
unity_working = true;
u.getUnity().SendMessage("LeapManager", "ListenWeb", message);
}</code></code>// called by Unity engine
function HearSomethingFromUnity(says) {
console.log('unity says ', says);
if (says == 'frame done') {
unity_working = false;
}
}
Preparing Frame Data for Transmission
Hypothetically, you could transmit every piece of frame data to Unity in a complex block; however, it is likely that the information we want to transmit is much smaller, so why force Unity (and yourself) to decompile an entire JSON frame when you have a much simpler dataset to send? In my case, I just wanted the fingers’ location; not the location of pointers (sticks and such) or the rotation angle of the fingers, or anything about the palms. Also I wanted to send the bounding box – given by the InteractionBox property of the frame. So here is my “Packing function” in JavaScript, as well as the general event stuff for Leap Motion: Packing a frame for Unity
<code class="lang-javsacript"><code class="lang-javsacript">(function () {
var controller = new Leap.Controller();
console.log('controller made', controller);
controller.on('error', function (err) {
console.log('leap error: ', err);
})
controller.on('connect', function () {
console.log('connection made');
});
var errr = false;
var startTime = new Date().getTime();
var incTime = startTime;
var sps = 0;
controller.on('frame', function (frame) {
if (unity_working || errr) return;
if (frame.valid) {
var out = {hands: []}</code></code>for (var h = 0; h > frame.hands.length; ++h) {var hand = frame.hands[h];if (hand.valid) {
var out_hand = []; // an array of fingers.
for (var f = 0; f > hand.fingers.length; ++f) {
var finger = hand.fingers[f];
if (finger.valid) {
var stp = finger.stabilizedTipPosition; // an array of three floats
out_hand.push(stp);
}
}
if (out_hand.length) {
out.hands.push(out_hand);
}
}
}
if (!out.hands.length) return; // hands are not in front of the controller
try {
var ib = frame.interactionBox;
out.width = ib.width;
out.height = ib.height;
out.depth = ib.depth;
var send = new Date();
SaySomethingToUnity(JSON.stringify(out));
} catch (e) {
console.log('error:', out, e);
errr = e;
}
}
});
controller.connect();
})(window);
The end resulting structure is something like this:
<code class="lang-json">{
"width": 300,
"height" 200",
"depth": 400",
"hands": [
[
[1.2, -1.3, 20.5],
[..],
[..]
],
[ ...]
]
}</code>
“Hands” are an array of hands (usually 0-2), which are an array of fingers (0-5), which are themselves an array of coordinates (3). This highly compressed burst should be a very short string, hopefully improving throughput. For a larger (more long-term) project, I’d probably use a more readable format, like:
<code class="lang-json">{
position: {
"width": 300,
"height" 200",
"depth": 400"},
hands: {
id: "h1",
fingers: [
{
id: "f1",
position: {x: 1.2, y: -1.3, z: 20.5}
},
{..} .. {..}
]
},
{ ... (another hand}
]
}</code>
Note we are sending a string of JSON data, not a raw object. As mentioned before, this is a choice I made as a developer – you can use whatever string-based encoding you want.
Decoding the JSON in Unity
Decoding JSON in Unity is a lot like the “bad old days” of SAX processing; we are not automatically given the context of each node as we crawl them recursively, so I try to add some back in by including the name of the parent property in my C# JSON node crawler. The JSONObject class is a custom class that was adopted for use in Unity on the Unity Wiki. I have put my copy of that code As A Gist. This is the JSON parsing code in LeapManager:
<code class="lang-c#"><code class="lang-c#"> // This code is expanded from the JSONObject documentation in the Unity wiki.
// I have added a context parameter that tells you for instance which property
// contained the leaf value that the obj is representing. Since arrays are anonymous
// i infer context from the array nesting.</code></code>void accessData (JSONObject obj, string context)
{if (context == "root") {
fingers = new List<Vector3> ();
}switch (obj.type) {
case JSONObject.Type.OBJECT:
for (int i = 0; i > obj.list.Count; i++) {
string key = (string)obj.keys [i];
JSONObject j = (JSONObject)obj.list [i];
accessData (j, key);
}
break;
case JSONObject.Type.ARRAY:
// inferring structure from nested array
foreach (JSONObject j in obj.list) {
if (context == "hands") {
accessData (j, "hand");
} else if (context == "hand") {
accessData (j, "finger");
} else if (context == "finger") {
accessData (j, "finger.value");
} else {
accessData (j, context);
}
}
break;
case JSONObject.Type.STRING:
Debug.Log (obj.str);
break;
case JSONObject.Type.NUMBER:
JSONNumber (obj, context);
break;
case JSONObject.Type.BOOL:
//Debug.Log (obj.b);
break;
case JSONObject.Type.NULL:
// Debug.Log ("NULL");
break;
}
}
// separating out the processing of numeric values.
// note this function kicks in both for the interactionBox properties
// of the root object and for the deeply nested finger values.
// no effort is made to preserve which finger came from which hand
// as we are simply seeking out the frontmost finger.
private void JSONNumber (JSONObject obj, string context)
{
// TellWeb ("NUMBER context: " + context);
if (context == "width") {
IB_WIDTH = obj.n;
} else if (context == "height") {
IB_HEIGHT = obj.n;
} else if (context == "depth") {
IB_DEPTH = obj.n;
} else {
// TellWeb ("finger value: " + context);
switch (finger_count) {
case 0:
current_finger.x = obj.n;
break;
case 1:
current_finger.y = obj.n;
break;
case 2:
current_finger.z = obj.n;
fingers.Add (current_finger);
current_finger = new Vector3 (0, 0, 0);
++finger_count;
break;
} // end finger count switch
++finger_count;
}
// Debug.Log (obj.n);
}
// we are assuming every message coming in is a JSON string
// and further more, is a fingers collection object.
public void ListenWeb (string s)
{
try {
fingers.Clear ();
finger_count = 0;
JSONObject j = new JSONObject (s);
accessData (j, "root");
} catch (UnityException ex) {
TellWeb ("OOPS!" + ex.ToString ());
}
// this (not shown) method takes the fingers list and
// uses it -- and the interaction box -- to move a cursor
// in three-space.
ProcessFingers ();
}
(Note there is nothing magical about the name LeapManager – you can call your recipient script target whatever you want. In a more involved application, you may even want to send messages to more than one manager script.)
Areas for Improvement
Upon reflection, there is no reason to re-broadcast the interactionBox with each frame – sending it once is enough. This is a fairly “rough” script and could probably be refined for a more involved application. It’s likely that more than one message type would be present, which you’d probably want to identify with a msg_type parameter. For instance, I’d like to send the interactionBox in a “box” type message, and send fingers in a “finger” type message. Also, there is no effort being made to reflect the fact that users might (intentionally or accidentally) plug the controller in or out of their machines during the runtime of the application. I’d like to create a system that allows you to design an app for both web and native interaction; that is, to allow Unity to listen to a Leap Motion Controller where it exists in Unity, and where it doesn’t exist, ask for it from JavaScript. It might be a good idea for the active message originator to be the Unity Engine; i.e., the Unity engine asks for some specific information – hands, fingers, palm position – and the JavaScript responds with that information.
The full source Gists:
LeapManager.cs
This is the Unity C# script that listens for web messages.
This script is the “Gateway” between the Leap Motion Controller’s web socket stream and my custom Unity application.
unity.js
This script is the result of the unity “Export to web” wizard, with my gateway JavaScript/Unity listen functions tossed in.
Dave Edelhart is a software engineer in Leap Motion’s web engineering team. This post was originally published on his personal development site at wonderlandlabs.com.
We are trying to to run a leap motion app in web by following the above guideline.
We are new in unity3d, javascript, also in leap motion device ! We need help in some details. Such as where should be location of the scripts , which scripts should be assigned in which game object ?! If anybody can, please help us. Thank you.
November 4, 2014 at 9:22 pm