When developers first fire up the Leap Motion SDK, their first instinct is often to think about how their device can be used in the place of a mouse or touchscreen. But as we’ve learned through user testing, mapping legacy interactions like mouse clicks or touchscreen taps to air pokes often results in unhappy users. Unlike a mouse or touchscreen, the Leap Motion Controller doesn’t provide tactile feedback or a neutral resting position, so this interaction can feel unsatisfying.

In this post, I’d like to share a few of the insights we’ve learned from internal prototyping, and all the cool experiences we’ve seen from creative people like you. Let’s think beyond the idea of “touch at a distance” and take a look at what it means when your hand is the interface.

Three questions you should ask

Instead of asking how we can use the the Leap Motion Controller to replace the mouse, we need to ask three questions:

1. All technology aside, how would I control my computer with my hands? Close your eyes and imagine a new UI beyond the 2D screen for the use-case you have in mind. This takes us a step away from the mouse mindset.

2. What can my hands do? Your hands can roll, grab, wave, and pinch. They have fingers that can easily trace paths in 3D space. What kind of experience would you like to develop with your hand as the interface?

3. How can I affect the UI with my hand? The mouse is designed to let you quickly and easily interact with a single pixel on a 2D plane. And it does a brilliant job, because it combines precision and ease-of-use with absolute unambiguity. But what if you could affect the UI with your entire hand and its many degrees of freedom? (Daniel Plemmons’ post on boundary-breaking menus is a great way to start grappling with this.)

Brainstorming interactions

Now you’re ready to brainstorm high-level interactions. Think broadly and sketch out your ideas, keeping these questions in mind. Try to think of multiple interactions for each task, so you have many options to evaluate against. Don’t be afraid to write down every idea, no matter how crazy!

Once you’ve got lots of potential hand interactions in mind, pick out the ones that can be readily accomplished with our API. Maybe it requires combining different types of data (like hand.yaw() and hand.palmNormal). Then prototype some interactions. If there’s a usability problem with a certain interaction, don’t just ditch the idea. Many problems can be fixed using additional hand data or better visual feedback! If you’re really stuck, post about it in the developer category on the forums.

I’d love to know – have you ever reached into a Leap Motion experience that took you beyond the mouse?

Nancy is a Leap Motion UX designer. She believes that well-designed digital experiences can make everyone feel like superheroes.