Leap Motion Path is a Second Story lab experiment exploring the use cases for the Leap Motion Controller in the digital animation field. Our objective was to create a tool to capture 3D motion that could be used within an animation platform to control digital content.

One of our goals was to record an animation without the assistance of a keyboard or mouse. To achieve this, we needed a way for the animator to know where their hand was in relation to a recording canvas. We accomplished this by creating an inset frame that gives the animator space to interact with the Leap Motion Controller and gain spatial reference to where they are inside the computer screen. Once they’re ready, they can enter the frame and start recording. Later, in the animation software, they can remove the entry and exit points from the frame by cropping the animation recording.

lmpath1

To better understand the uses for this in animation, take a look at Bret Victor’s animation example in the video below. (Incidentally, the idea for Leap Motion Path came from this video.) He animates a leaf falling from a tree and pans the scene. While he animates in 2 dimensions, Victor uses recursive organic motion capture using the touch surface to create life-like animations.

While the whole video is great, skip ahead to 29:30 to see this technique in action:

What’s cool about the Leap Motion Controller is you can now easily record 3D paths. Using a similar replay mechanism, an animator could play back the animation and control different parameters with a single finger. By capturing that initial 3D path – compared to if it were manually done with keyframes – the Leap Motion Controller saves a lot of time. During this experiment, we encountered another interesting use case.

Leap Motion provides a lot of data about the geometry of the hand. If you capture the position of the animator’s wrist and the tip of the index finger and draw a line between those points, you end up with a vector that indicates the direction of the hand. (Alternatively, you could also use the hand direction property to project a constant length for the finger.) If you capture this vector over time, you see that it produces a beautiful ribbon. The animator can record this ribbon and use it in other animations.

lmpath2

As we developed Leap Motion Path, several stand-alone libraries came to be. We’ve hosted one called Path.js here on Github (shown below). To provide some additional context: we wanted to capture a 3D position over time and then animate along the path we recorded. If we were to animate along the finite points Leap Motion gives us, the animation would be choppy because the resolution of these points wouldn’t resemble the actual path we drew with our hand. To combat this, we needed to interpolate a line or a curve between these points to give a finer resolution so we could animate at any speed. Path.js takes a collection of timestamped points and creates a linear interpolation between them. This allows Leap Motion Path to export an animation in vector format, allowing the animator to scale and stretch the animation as desired.


var Point = function (x, y, z, t, prevPoint) {

  this.x             = x || 0;
  this.y             = y || 0;
  this.z             = z || 0;
  this.t             = t || 0;
  this.previousPoint = prevPoint || null;
};

var Path = function (data) {

  this.pools = [];

  var poolsize = Math.floor ( data.length / 10 );
  
  for (var i = 0; i < poolsize; i++) {
    this.pools.push( [] );
  }

  // find min & max time
  var minTime = 0;
  var maxTime = 0;
  var point;
  for (var j in data) {
    point = data[j];
    if (point.t > maxTime) {
      maxTime = point.t; 
    }
    if (point.t < minTime) {
      minTime = point.t; 
    }
  }

  this.maxTime = maxTime;
  this.deltaT = Math.ceil( maxTime / poolsize );
  
  var previousPoint;

  for (var k in data) {
    point = data[k];

    if ( previousPoint ) {
      point.previousPoint = previousPoint;
    }

    if (point.t == 0) {

      this.pools[0].push(point);

    } else if (point.t == this.maxTime) {

      this.pools[this.pools.length - 1].push(point);

    } else {

      this.pools[ Math.floor( (point.t) / this.deltaT ) ].push(point);
    }
    previousPoint = point;
  }
  

};

Path.prototype.getXYZ = function (t) {

  if (typeof t !== "number" || t > this.maxTime) {
    console.log("error - Path.prototype.getXYZ - t specified is invalid");
    return null;
  }
  
  var poolIndex = 0;

  if (t == 0 ) {

    poolIndex = 0;

  } else if (t == this.maxTime) {

    poolIndex = this.pools.length - 1;

  } else {

    poolIndex = Math.floor( t / this.deltaT );
  }

  var x = null, y = null, z = null;

  do {

    var pool = this.pools[poolIndex];
    
    if (!pool) {
      break;
    }

    for (var i in pool) {
      
      var point = pool[i];

      if (point.t > t) {

        x = ( (point.x - point.previousPoint.x) / (point.t - point.previousPoint.t)) * (t - point.previousPoint.t) + point.previousPoint.x;
        y = ( (point.y - point.previousPoint.y) / (point.t - point.previousPoint.t)) * (t - point.previousPoint.t) + point.previousPoint.y;
        z = ( (point.z - point.previousPoint.z) / (point.t - point.previousPoint.t)) * (t - point.previousPoint.t) + point.previousPoint.z;
        break;

      } else if (point.t == t) {

        x = point.x;
        y = point.y;
        z = point.z;
        break;
      }
    }
    poolIndex ++;

  } while(x == null && y == null & z == null);
  
  return {x : Math.round(x), y : Math.round(y), z : Math.round(z)};

};

With more development, Leap Motion Path could be integrated into a standard digital animation workflow giving animators one more tool to create beautiful and lifelike work. Moving forward, to improve the motion-capture experience, we would need to re-write the recording mechanism as a plugin to an animation platform, enabling the animator to record and review all in one application. We look forward to integrating Leap Motion Path into our own animation workflow at Second Story.

Dimitrii Pokrovskii is an interactive developer at Second Story – an innovation center that creates new interactive experiences for brands and institutions across a variety of digital channels. This post originally appeared on their blog.