Visual feedback is hugely important when it comes to motion control – since users can feel lost or frustrated when they’re not sure how their actions are affecting an application. Virtual hands can make it much easier for users to identify what’s happening onscreen. Thanks to the new v2 tracking, we’ve been able to create persistent rigged hands for LeapJS that reflect how your hands look and behave in the real world.

Beyond that, we can use other v2 tracking features – like confidence and pinch strength – to start interacting with web apps in a much more intuitive and natural way. From having it grab onto DOM elements or a div on the page, to crumpling up a banner ad, to having it work in a more interactive way so that the page becomes a piece of fabric that you can sculpt, you can create all sorts of amazing experiences with the rigged hand plugin.

Using the rigged hand in your web apps

You can drop in support for a 3D visual hand for any JavaScript app with only a few lines of code:

(window.controller = new Leap.Controller)
  .use('riggedHand')
  .connect()

Of course, you can go much further than the simple vanilla version. We’ve created a number of settings that let you customize the look, feel, and behavior of your hand.

Movement scaling. positionScale lets you customize the hand’s movement to be scaled independently of its size. With a value of 2, the hand will cover twice the distance on screen as it does in the real world.

Size scaling. The scale attribute lets you change the size of the hand within the scene. Combined with positionScale, you can have a small hand that moves rapidly across the screen – amplifying the user’s intentions.

Coloring. boneColors lets you set the color of each bone based on certain behaviors (e.g. pinching), which is useful for reinforcing key interactions or helping the user identify where their hand lies in 3D space. We’ve also included a dotsMode debugging tool, which shows pink dots at the raw position of every joint. (Note that in the current version, the dots are offset from the skinned mesh – this is something we’re working on.)

Below you can see an advanced example that colors the thumb and forefinger blue while pinching, along with a few other options:

(window.controller = new Leap.Controller)
  # handHold and handEntry are dependencies of this plugin, available to the controller through leap-plugins.js
  # By default rigged-hand will use these, but we can call explicitly to provide configuration:
  .use('handHold', {})
  .use('handEntry', {})
  .use('riggedHand', {
    parent: scene

    # this is called on every animationFrame 
    renderFn: ->
      renderer.render(scene, camera)
      # Here we update the camera controls for clicking and rotating
      controls.update()

    # These options are merged with the material options hash
    # Any valid Three.js material options are valid here.
    materialOptions: {
      wireframe: true
    }
    geometryOptions: {}

    # This will show pink dots at the raw position of every leap joint.
    # they will be slightly offset from the rig shape, due to it having slightly different proportions.
    dotsMode: true

    # Sets the default position offset from the parent/scene. Default of new THREE.Vector3(0,-10,0)
    offset: new THREE.Vector3(0,0,0)

    # sets the scale of the mesh in the scene.  The default scale works with a camera of distance ~15.
    scale: 1.5

    # Allows hand movement to be scaled independently of hand size.
    # With a value of 2, the hand will cover twice the distance on screen as it does in the world.
    positionScale: 2

    # allows 2d text to be attached to joint positions on the screen
    # Labels are by default white text with a black dropshadow
    # this method is called for every bone on each frame
    # boneMeshes are named Finger_xx, where the first digit is the finger number, and the second the bone, 0 indexed.
    boneLabels: (boneMesh, leapHand)->
      return boneMesh.name

    # allows individual bones to be colorized
    # Here we turn thumb and index finger blue while pinching
    # this method is called for every bone on each frame
    # should return an object with hue, saturation, and an optional lightness ranging from 0 to 1
    # http://threejs.org/docs/#Reference/Math/Color [setHSL]
    boneColors: (boneMesh, leapHand)->
      if (boneMesh.name.indexOf('Finger_0') == 0) || (boneMesh.name.indexOf('Finger_1') == 0)
        return {
          hue: 0.6,
          saturation: leapHand.pinchStrength
        }

    # This will add a warning message to the page on browsers which do not support WebGL or do not have it enabled.
    # By default, this will be used unless a `parent` scene is passed in.
    # This uses @mrdoob's Detector.js
    # Chrome, Firefox, Safari Developer mode, and IE11 all support WebGL.  http://caniuse.com/webgl
    checkWebGL: true

  })
  .connect()

Note that Three.js is loaded to create the scene, while LeapJS plugins are loaded to allow Hand Hold and Hand Entry functionality. In the rigged hand documentation, you’ll also find details about setting scope objects, events, scene position, and screen position.

Creating your own rigged hand

Rigging a hand is tricky business that requires both 3D modelling and JavaScript expertise. But if you’re feeling adventurous, we’ve made it possible to follow in our footsteps with our wiki article Creating Rigged Hands. It covers the process of bringing the model out of a MAYA file or other 3D rendering program and into the Three.js environment.

What’s next?

Visual experimentation. We’re experimenting with different forms of visual feedback to show when your hand is approaching the edge of the screen. Maybe the edge of your hand turns a different color, or starts to fade away. UX design is constantly evolving as we explore new ways to make web interactions truly intuitive.

Physics engines. We’d love to have the rigged hand interact with various physics engines, so that you could use it to realistically push against an object. This becomes complicated when you’re working with a mesh that has thousands of facets on it, so stay tuned.

Custom shaders. With custom shaders, you can make your hand transparent with glowing edges, or make it look cartoony. (My most recent work in this area can be found in this pull request, which includes a zebra-style hand.)

Optimization. The rigged hand plugin includes both left and right hand models, and as a result is fairly large. With some work, we could mirror the hand models to cut the size in half.

We’d love to get your thoughts on this project – what kinds of hands would you like to see? How can you imagine using them in your apps? As always, we take your feedback into account as we decide how to allocate our time and resources. You can also become directly involved with the rigged hand project. Be sure to push your changes and document your pull requests!

Peter is a software engineer at Leap Motion who loves to practice web development and bring the power of LeapJS to everyone.

Twitter