Using Three.js and the Leap Motion Controller, Ukrainian developer Roman Liutikov was able to create a rigged hand that runs in your browser. Check out the demo with your device or watch the video, then hear about how Roman was able to apply rigged geometry to the Leap Motion JavaScript API.

The demo showed in the video above is actually the initial version, which is slightly different from the current one.

API

In this section I’ll describe the valuable data for current case only; for the full API, check the official docs.

Leap Motion works in a snapshot manner, which means it sends a block (frame) of data with all info about the current state of the scene in a particular moment of time. Here’s the data required for a hand model with an armature.

{
hands: [
{
direction: [0, 0, 0],
palmPosition: [0, 0, 0]
}
],
pointables: [
{
direction: [0, 0, 0]
}
]
}

There is, of course, a lot more output data, but it’s fairly enough to implement rigged manipulation.

The hands array includes objects with data which describes each detected hand. The direction array describes the direction unit vector, which points from the palm position toward the fingers. The palmPosition array is the center position of the palm in x, y, z format. The pointables array is the list of the Pointable objects. The direction array of the Pointable object describes the direction (as unit vector) in which the finger or tool is pointing.

Hand rigging

This is how the armature of the hand should look.

And here are vertex groups assigned to appropriate bones.

(If you’re wondering how to rig mesh in Blender, check out my blog post about rigging and skeletal animation.)

Preparing the scene

  1. Export the model and make sure the required export options are checked: skinning, bones, and skeletal animation.
  2. Set up a basic Three.js scene with model loader code from my rigging article.
  3. Grab the latest Leap.js client lib from its repo and include it in your html.

Setting up and passing Leap Motion data

You might know about the Leap Motion Controller object, which is used to manually connect to the device, but this is not necessary when using the frame loop, as it will setup the controller and connect by itself.

var leap = new Leap.Controller({host: 'localhost', port: 6437});leap.connect();

Run frame loop. Leap.loop(); function passes a frame of data to the callback function 60 times per second using requestAnimationFrame();. Add this function call to the very end of the model load function.

Leap.loop(function (frame) {
animate(frame, hand); // pass frame and hand model
});

The core function called in the callback includes extracted and structured data that describes the position of the hand and fingers in 3D space, as well as position updating functions.

function animate (frame, handMesh) {
  if (frame.hands.length > 0) { // do stuff if at least one hand is detected
    var leapHand = frame.hands[0], // grab the first hand
        leapFingers = frame.pointables, // grab fingers
        handObj, fingersObj;

    // grab, structure and apply hand position data
    handObj = {
      position: {
        z: -leapHand.palmPosition[0]/4,
        y: leapHand.palmPosition[1]/6-30,
        x: -leapHand.palmPosition[2]/4+10
      },
      rotation: {
        z: leapHand.palmNormal[2],
        y: leapHand.palmNormal[0],
        x: -Math.atan2(leapHand.palmNormal[0], leapHand.palmNormal[1]) + Math.PI
      },
      update: function() {
        var VectorDir = new THREE.Vector3(leapHand.direction[0], -leapHand.direction[1]+.6, leapHand.direction[2]); // define direction vector

        handMesh.lookAt(VectorDir.add(handMesh.position)); // setup view
        handMesh.position = this.position; // apply position
        handMesh.bones[1].rotation.set(this.rotation.x, this.rotation.y, this.rotation.z); // apply rotation
      }
    };

    // grab, structure and apply fingers position data
    fingersObj = {
      update: function (boneNum, fingerNum, isThumb) {
        var bone = handMesh.bones[boneNum], // define main bone
            phalanges = [handMesh.bones[boneNum+1], handMesh.bones[boneNum+2]], // define phalanges bones
            finger = leapFingers[fingerNum], // grab finger
            dir = finger.direction; // grab direction

        // if current finger is thumb, use only one additional phalange
        if (!!isThumb) {
          phalanges = [handMesh.bones[boneNum+1]];
        }

        // make sure fingers won't go into weird position
        for (var i = 0, length = dir.length; i < length; i++) {
          if (dir[i] >= .1) {
            dir[i] = .1;
          }
        }

        bone.rotation.set(0, -dir[0], -dir[1]); // apply rotation to the main bone

        // apply rotation to additional phalanges
        for (var i = 0, length = phalanges.length; i < length; i++) {
          var phalange = phalanges[i];

          phalange.rotation.set(0, 0, -dir[1]);
        }
      },
      // define each finger and update its position
      // passing main bone number and finger number
      fingers: {
        pinky: function() {
          fingersObj.update(3, 3);
        },
        ring: function() {
          fingersObj.update(7, 1);
        },
        mid: function() {
          fingersObj.update(11, 0);
        },
        index: function() {
          fingersObj.update(15, 2);
        },
        thumb: function() {
          fingersObj.update(19, 4, true);
        }
      },
      // update all fingers function
      updateAll: function() {
        var fingers = this.fingers;

        for (var finger in fingers) {
          fingers[finger]();
        }
      }
    };

    handObj.update(); // update hand postion

    // update fingers position if there are all five fingers is detected
    if (leapFingers.length == 5) {
      fingersObj.updateAll();
    }
  }
}

Basically, it’s easy to set up and run something with Leap Motion, but when the goal is to achieve the best possible results, all the pitfalls immediately go up. For example, I’ve used magical Math.atan2 for one of the hand rotation axis, instead of palmNormal value. As it turned out, there’s no nicely represented pitch, roll, and yaw values. Instead, you need to calculate some manually – check this Leap demo to see what’s wrong with the rotation data. Also, I’ve tweaked almost all data to make the model behave nicely on the screen.

One of the most important things to remember when building the armature for the model in Blender (or other software) for Three.js: do not move/rotate the armature, but always align it using the bone’s head/tail position. (This is true for Three.js r60, but it seems like in r56 it wasn’t required).

Roman Liutikov is a front-end web developer from Ukraine. Check out his GitHub profile at