Reach Out Your Hand and Let There Be Light

Like many people, I love to ride my bicycle to get to work. However, as you can imagine, mornings in Switzerland can get rather chilly, and I needed a way to check the temperature outside – so I bought a little weather station called Netatmo. Around the same time, I discovered some fancy new bulbs from Philips called Hue, which consist of three LEDs that can emit light covering nearly the entire visible spectrum.

What do these two things have to do with each other? Nothing at first, but I started thinking about connecting the two devices – so that the light would indicate the level of outside temperature at the blink of an eye. It worked marvelously, and I started to explore other ways to use the bulb.

The Leap Motion Controller seemed like an intriguing way to control the lights, so I created a modular framework using Scala and the Leap Motion Java API. After thinking about what a modular system would look like, and what kind of features it should have, I decided that it should be easy to extend, configurable, contain a message system synchronizing the color settings from various connectors, and include a remote start/stop option for the system. In the video below, you can see the installation in action; watch the full video to see my talk at GDG DevFest 2013:

Today, we’ll take a look at how I created my Leap Motion-enabled Hue lights. I’ve also made the project available on GitHub for everyone to view and build upon.

Building a modular system with Akka

Easy to extend. To connect to the different APIs, I needed an interface (connector) for each API. These connectors implement the component APIs to fetch values and do the messaging to control the bulbs.

“core” should contain all commonly used configurations, classes, etc.
“runtime” contains all functionality to start and run the system
“control” will start/stop the system remotely, initializing the components

Configurable. For the various API hosts, usernames, passwords, etc. a configuration is required. In order to have a hierarchical configure system, I chose the “typesafe config”.

Color control message system. As I mentioned earlier, I opted for a messaging system. This utilizes a message bus to transport Leap Motion input and synchronizes them with the timing capabilities of the LED hardware. The akka framework also uses typesafe config for its configuration.

To get this feature running, I used Akka, a toolkit toolkit and runtime for building event-driven applications. While explaining this system falls beyond the scope of this post, Akka has some great documentation to help you get started with your own apps. Naturally, I used their latest Scala doc.

Remote start/stop. Akka has a nice feature called remoting. With Akka remote, you can connect to any other running Akka instance on a different JVM.

Bringing everything together

bbh_blockdiagramm

The Leap Motion connector

For the Leap Motion Controller, I needed to define which gestures and movements would cause the Hue to react. Here are the ones that I used:

hand X axis to change the whiteness of the color
hand Y axis to change the brightness of the color
cycle clockwise or counterclockwise to change the color
swipe to switch to the next bulb
key tap to switch a bulb on/off

Actor implementation for the Leap Motion listener

Before the listener implementation can begin, you need to create an Akka actor to send the processed results from the device/listener to the connector for the bulbs. To allow numerous configurable ways to react to the controller, I used an abstract actor layer, which starts an actor. This actor sends the messages to the Hue connector, or later to other connectors. This actor is then passed into the Leap Motion listener as a constructor argument.

Leap Motion listener class implementation. The listener, which is added to the Leap Motion Controller class, contains the implementation for the callbacks (onInit, onConnect, onFrame, etc.) Using the actor initially passed into it, the listener is able to process the events from the Leap Motion Controller class and send the messages via the messaging system to the Hue connector, which finally sets the color for the bulbs.

Gesture and movement controls

Circle

The Hue system for colors expects values ranging from 0 to 65535, allowing 16-bit color codes. This means there is no start and endpoint – any calculated value greater than 65535 will be set to 1, and any calculated value less than 0 is set to 65535. To change the color in both directions, it’s necessary to detect the clockwise/counterclockwise direction of the circle gesture. The API circle.pointable().direction().angleTo(circle.normal()) function is used and its result is compared to the result of Math.pi/4.

case TYPE_CIRCLE => if(state.on) { val previousCircle = new CircleGesture(controller.frame(1).gesture(gesture.id)) val circle = new CircleGesture(gesture) val circleAmount = circle.progress.toInt if (previousCircle.progress.toInt != circleAmount) { val hue = if (circle.pointable().direction().angleTo(circle.normal()) <= Math.PI/4) Clockwise(state.hue, circleAmount).newHue else CounterClockwise(state.hue, circleAmount).newHue val newState = state.copy(hue = hue, sat=255, bri=255) bulbStates += ((newState.bulbId, newState)) actor ! HueHandlerActor.SendBulbState(newState) } } /***************************************/ case class Clockwise(oldHue: Int, circleAmount: Int) extends CircleDirection { def newHue = rangeCheck(oldHue + (circleAmount * 1000)) } case class CounterClockwise(oldHue: Int, circleAmount: Int) extends CircleDirection { def newHue = rangeCheck(oldHue - (circleAmount * 1000)) }

In order to calculate the new color value, the amount of circles which are being drawn is important. This value is returned by the function circle.progress.toInt. To clearly distinguish the color being set, the saturation and brightness are both set to maximum value when drawing circles.

To cover the whole color spectrum in an efficient way, I’ve used increase/decrease steps set to 1000. This leaves us with 65 steps in a 0–65535 range of values. Any smaller would result in changes too small to see and increasingly long gesture input time to reach a color.

def rangeCheck(hue: Int) = { hue match { case x if(x > 65535) => 1 case x if(x < 0) => 65535 case x => x } }

Swipe

The swipe gesture changes the bulb being controlled. To prevent unintended swipes, the first check is whether the gesture state is “stop.” The next check is what axis the swipe was on and whether its absolute direction value is greater than 0.75 and whether it was drawn with one finger.

With this, the new active bulb blinks twice to indicate it’s now active. Then the color is reset to its previous state. If the bulb was off, then the bulb is activated.

if(Math.abs(swipe.direction.getX) > 0.75 && controller.frame.hands.leftmost.fingers.size == 1) { val direction = if(swipe.direction.getX > 0) SwipeRight else SwipeLeft /***************************************/ trait SwipeDirection { def swipe: Int => Int } case object SwipeLeft extends SwipeDirection { def swipe = { index:Int => index - 1 } } case object SwipeRight extends SwipeDirection { def swipe = { index: Int => index + 1 } }

KeyTap

The key tap has a very simple implementation, as it only sends the “state off” message for the active bulb.

Hand Position

Depending on your hand position, the saturation and brightness is set, with the x axis used for saturation and the y axis for brightness. These parameters range from 0 to 255, so that the recognition area of the Leap Motion Controller has to be scaled down to align with these ranges. From a usability point of view, I use the recognition edges of the device to prevent range overflow of range, so that there’s a gesture position where saturation or brightness no longer change.

To keep saturation and brightness settings separate from the other gestures, values are changed only if no other gesture was detected. Plus, in changing the setting mode or fix saturation and brightness, the number of fingers extended is detected by the Leap Motion Controller. The aforementioned values are changed only the device detects five fingers and the motion up/down or left/right.

If less than 5 fingers are detected (e.g. you make a fist), the values are fixed. This behavior proved to be the best way to get the hand out of the recognition area of the controller without it being detected as a movement.

if(gestures.size == 0 && state.on && controller.frame.hands.leftmost.fingers.size == 5) { val direction = controller.frame().hands().leftmost() def rangeCheck(value: Int) = { value match { case value if(value < 0 ) => 0 case value if(value > 255) => 255 case value => value } } val sat = rangeCheck(direction.stabilizedPalmPosition().getX.toInt + 127) val bri = rangeCheck(direction.stabilizedPalmPosition().getY.toInt - 50) val newState = state.copy(sat=sat, bri=bri) bulbStates += ((bulbsUnderCtrl.head, newState)) actor ! HueHandlerActor.SendBulbState(newState) Thread.sleep(500) }

bbh_messageflow

“Bogie Bridge HUE” control

Finally, the function of Bogie Bridge Control (BBC) is to start/stop the runtime and and the connectors. At this time, BBC isn’t fully implemented, and currently it’s only possible to stop a running system. While this project remains a work in progress, I hope that you’ll find some useful elements in my code and approach to use in your own hardware integrations.

Stefan Lauer is the Senior Software Developer at HolidayCheck AG, a German-language travel and booking portal. He is a Scala enthusiast and develops web services to display content on his company’s new platform. In his daily business, he uses a wide range of technologies, including Scala, MongoDB, Neo4j, Akka, elasticsearch, Jersey, WebSockets, and Tomcat. You can follow him on www.caseclass.de.