At SmartThings, we think a lot about the way we interact with the everyday objects around us, and – more importantly – how they can react to us. By adding intelligence to the previously dumb, inanimate things in our homes, we can not only teach the locks, lights, thermostats, and other devices we live with to memorize our unique habits and routines, but also offer people security, peace of mind, and savings by letting them control all of these things from their phones.

Recently, we integrated Leap Motion interaction with our open platform, hoping to unlock a new world of possibilities in which home automation devices didn’t just respond automatically to our preferences and presence, but also to our hand movements. While this is an early experiment, it offers a hint of what a touchlessly interactive world might look like.

I felt like the simplest way to get started would be to create an in-browser Leap Motion and SmartThings integration. (If you have any compatible smart devices, you can take our live web demo for a spin after registering for free as a SmartThings developer.) SmartThings lets you create URL “endpoints” that control associated devices in your home. With the LeapJS examples, it was easy to see gestures in the browser and to send ajax messages to a SmartThings endpoint.

image

I based the SmartThings endpoint mapping on the code from our endpoint tutorial. As a shortcut, you could simply generate URLs to control associated devices by using the hosted example from the tutorial. Effectively, after authorizing specific devices, you can send a command by calling the endpoint URL with a path like “/switches/:id/:command”  where you replaced “:id” with a switch ID and “:command” with “on”, “off”, or “toggle.”

My initial concept was to create a controllable “carousel” of SmartThings devices. The carousel is based on a 3D Webkit demo with a rotating ring of content options. Similarly, I wanted to have our integration rotate to show the currently selected SmartThings device.

image

To spin the carousel, I experimented with swiping left and right and using hand position. Both were viable options, but given the number of connected devices I wanted to control, I decided to look for a shortcut to control any given device in the carousel.

Inside of the Leap.loop handler, my code for handling swipes looked something like this:

//swiping to rotate the carousel
for (var i = 0; i < frame.gestures.length; i++)
{
    var g = frame.gestures[i]; //get the gesture
    if (g.type === 'swipe' && g.state === 'stop') {
        if (g.direction[0] < 0) {
            onLeftSwipe();
        }
        else {
            onRightSwipe();
        }
    }
    //...
}

Ultimately, I settled on using a click-tap gesture to select the device. The number of fingers used in the gesture selects the device by number. So, clicking with three fingers picks the third device. This opened up the added benefit of being able to choose a device without looking at the screen. To make the “no-look” device selection viable, I added in a little audio feedback effect to indicate the selection registered.

var g = gesture = frame.gestures[i];
//look for keyTap gesture
if (g.type == 'keyTap' && g.state == 'stop')
{
    var numberOfFingers = fingers[0]; //single hand (could use two)
    goTo(numberOfFingers); //go the the desired device
    playSound();
}

Picking logical gestures to operate lights, locks, and a giant fan was the next challenge. I experimented again with swipe motions, and found that position worked for dimming lights – raising or lowering your hand accurately changes the intensity. While both swiping and raising or lowering my hand were viable, circular gestures turned out to be more consistent – and perhaps a better match to the desired commands.

For any gesture, we can determine the direction of the circle (clockwise or counter-clockwise) with the following line (a bit of a shortcut):

Var isClockwise = (g.normal[2] <= 0);

We can then determine that progress around the circle and the radius of the circle are greater than a set threshold to reduce false positives with the following lines:

if(g.progress.toFixed(2) > 0.9 && g.radius.toFixed(1) > 30)

var g = gesture = frame.gestures[i];
if (g.type === 'circle' && g.state === 'stop') {

    var isClockwise = (g.normal[2] <= 0);
    console.log("isClockwise", isClockwise);

    if(g.progress.toFixed(2) > 0.9 && g.radius.toFixed(1) > 30)
    {
        if(isClockwise){
            turnOn();
        }
        else{
            turnOff();
        }
    }
    playMagicSound();
}

image

These circular gestures were also nice in that they were more distinguishable from other motions. I also added some useful information by associating a clockwise motion with “on” or “lock,” and a counter-clockwise motion with “off” or “unlock.” The ability to see the distance traveled around the circle and where the motion stops is built into the Leap Motion API. This gives two interesting possibilities for lighting: one could dim to a desired level, based on the distance traveled around the circle (likely on the second time around); or in the case of multi-colored lights such as the Phillips hue, one could correlate the distance traveled around the circle with the color on the wheel at that point (say starting from red).

When it comes to circular gestures, there is some room for discussion about which direction best represents their use. Depending on how your front door is configured, the locking motion could be clockwise or counter-clockwise. If you look at older, circular dimmer switches, making them brighter is likely to be a counter-clockwise motion – similar to unscrewing something. Only time will tell which new gestures will become commonplace. Just as pinch-to-zoom has become so widely used, I suspect that new “standards” will emerge as developers in the Leap Motion community continue to give new meaning to human gestures and connect them with actions to control the Internet of Things.

Of course, powering lights, fans, and electronics is just a starting point. We’re looking forward to continuing our work with Leap Motion to create new and exciting ways to interact with the world around us. Where do you imagine touchless interaction being used in five years? Let us know in the comments.

Daniel Kleinman is Maker-in-Residence at SmartThings. His preliminary work with the Leap Motion Controller takes a step towards integrating Leap Motion interaction with the Internet of Things.

Recommended Posts