Mischa Spiegelmock is a software engineer and architect with a variety of specializations, including security, web applications, and real-time media. One of the earliest members of our developer program, he’s published a book about Leap Motion development essentials. You can follow Mischa’s latest work on GitHub.

Like many people, as soon as I saw the first demonstration video of the Leap Motion Controller, I knew I had to play with one. I pre-ordered a developer kit, and finally got my hands on one in September 2012.

Rev. 3 Leap Motion Controller

When I started playing with the SDK and visualizer, the finger tracking accuracy was nowhere near what it is today. The API, on the other hand, looked magical. My fear was that it would be primitive and leave a lot of work up to the application developers, but that was quickly put to rest.

Released in November 2012, 0.6.5 was one of the first versions of the SDK available to the public, and it contained support for C++, Java, Python and C#. It also allowed developers to easily read in finger, tool and hand position, rotation, velocity, and to set up event callbacks. The developers promised configuration, a skeleton model, and actual gesture recognition in later versions.

The original visualizer application. Check out the infrared lights.

Early experiments

October 2012: My first project was to emulate a mouse with your finger to control your cursor. I tried to get fancy and turn it into a two-handed interface – with one hand controlling the cursor location by pointing and the Screen intersection API, and the other hand making a pinching motion to perform a click.

This was when I ran into my first conundrum in setting up a gesture interface. I’d never even considered the problem before, but I had to decide which hand would be used for position and which would be for clicking. Seemed obvious at first – my right hand is more stable and accurate for pointing, but what about left-handed people? And how do you even know which hand is the left and which is the right?

Perhaps, I thought, one could measure the locations and lengths of the first and fifth fingers to try to determine which hand is which, although I’m not even sure such information was provided by the API at the time. I fudged it, and used the palm X coordinates to decide, though this would fall apart if the user crossed their arms.

Insights and developments

When I started coming to grips with these sorts of questions, it started to dawn on me that the Leap Motion Controller was not just a fancy mouse, but really an entirely new mode of interacting with software. The Leap Motion developers knew this from day one, which is why their focus has always been on precise measurements of a user’s finger and hand movements, instead of complex gesture recognition.

Humans prefer to manipulate representations of objects in a simulated environment using the same motions that would be used in the real world, much more than they want to learn and remember complex and arbitrary motions that have no connection to the task at hand. Witness the success of the multitouch interface pioneered in iOS; even people who could never get the hang of double-clicking can immediately start dragging, pinching, and sliding interface elements around – as though they were physical objects. No training necessary, since they’ve been doing the same thing their entire lives.

After my first foray into gesture-capable software, subsequent versions of the SDK were released almost faster than I could keep up. Every week the accuracy got better, new features were added, improvements in the documentation and sample code were added. It was exciting to watch such an ambitious project take shape and begin to mature over time.

Creating leapmidi

January 2013: For my next project, I decided to kick things up a notch and create a MIDI controller – that is, software that could be used to interface with any sort of digital music instrument, effect, or parameter. I started a library project named leapmidi, which was designed to translate hand motions (defined as a leapmidi::Gesture) into MIDI control messages, and a Mac OS X application,leapmidix, to handle the OS-specific tasks of setting up a virtual MIDI device, transmitting the messages, and dealing with threading.

(Side note: the Leap Gesture class didn’t exist at the time I created my Gesture class, so now I look like a dick for naming it the same.)

My MIDI controller was straightforward enough for my application. I wanted to control faders in the Ableton Live DAW – using the fingertip Y coordinates to control the channel gain, and the number of fingers extended to select the track. Throw in a little palm-X coordinate-based crossfader and palm Z depth for tempo control, and you’ve got yourself a nice little live performance setup:

I posted a video of my project around the time that interest in the Leap Motion Controller was starting to build. More and more developers were getting their hands on the devices and putting out all sorts of demos. I set up a little IRC channel (freenode: #leapmotion) where people experimenting with the device can get help with questions about the SDK and hardware. Even some Leap Motion employees started chilling in there, dropping interesting and useful knowledge periodically.

Writing a Book

February 2013: Apparently, someone at a publishing company saw the work that I was doing early on, and reached out to ask if I’d like to write up a book guide to using the SDK. I said sure, how hard could it be?

Doing research for the book was fun and not a problem at all. The people at Leap Motion were incredibly easy to work with – answering my questions personally, and even taking the time to meet with me at their HQ. They clarified a lot of things for me, like the natural spatial interaction paradigm they’re pushing over gesture recognition.

My visit to the Leap Motion HQ in San Francisco

The hardest part was trying to document and cover an API that was constantly changing from week to week, combined with a tight deadline and not much insight into what future versions of the API would look like. No doubt much of that future was a mystery to the Leap Motion developers as well.  Today, there are all sorts of fancy methods and features like the Interaction Box API, which normalizes spatial coordinates for you – so you don’t have to work out appropriate position coordinate ranges by trial and error as I did.

With not a lot to work with, I decided to take the time to write up my experiences integrating the controller API with real-world, practical applications – since many of the capabilities of the API were already quite nicely documented in the SDK docs. I wrote about creating a multithreaded MIDI controller, moving around and resizing windows with finger motions, using the Leap.js JavaScript library with a simple HTML5 canvas drawing surface, and a brief look at combining Leap.js with Three.js and WebGL for Leap-driven 3D web interfaces, which quite frankly is the awesomesauce. I also did my best to weave in the pile of C++ and gestural interaction tips I’d picked up in my research, hopefully into a somewhat coherent document.

New developments

September 2013: As my writing project was finishing up, I got to meet the Leap Motion team again at a well-attended meetup at 111 Minna – a fine establishment that I am pretty sure is in a perpetual state of an open bar paid for by one SF SoMa startup or another. There were cool demos, my favorite being a large robot arm that mimicked your hand motions flawlessly.

However, the most impressive thing that I saw by far though was the preview of the beta of the next generation driver and recognition software. It looked basically magical. The accuracy appeared to be flawless, the tracked motions smooth, no more disappearing fingers. They even showed it off by crossing their arms and having it track the left and right hands properly! More than a year since my first experiments with Leap Motion began, I’m definitely curious to see what the future has in store.