The Internet of Things is different things to different people. In recent posts on Leap Motion’s Developer Labs, we’ve seen how the Leap Motion device can open up new interaction models in smart environments. To exist within the Internet of Things – to network with other devices in a smart ecosystem on a machine-to-machine (M2M) basis, rather than being just another human-machine interface (HMI) – that’s where it gets really interesting.
The sticky point is that if a system gets its data from direct human interaction, or its outputs go to direct human interaction, then it isn’t M2M, but HMI – the telling point being that the data does not get out beyond the machine to which the Leap Motion Controller is attached. There’s only the one “machine,” the system comprised of the computer with an attached peripheral.
But we’re talking about a device that generates data. If we move up one layer in the hierarchy, by looking instead at the data which is generated by the machine, and make that data available in the cloud, then the difference between HMI and M2M becomes irrelevant. By opening up the data to the cloud, the Leap Motion Controller and its attached computer is now clearly a M2M device and supports all of evolutionary, revolutionary and emergent behavior. You’re now talking about the integration of data clouds in ways that go much further than just HMI.
Recently, I created a live demonstration of this approach using LabVIEW and a Lego NXT robot, which you can see in the video below:
Unlike other LabVIEW robot demonstrations shown on YouTube, this one demonstrates that you can drive the robot from a remote location – or from different computers each with a Leap Motion Controller.
It’s all done with the Data Distribution Service (DDS), an Internet of Things protocol and set of standards maintained by the OMG (Object Management Group). The purpose of DDS is to enable simple, anonymous, real-time data clouds in a platform- and transport-agnostic manner.
This means that a DataWriter, publishing on a DDS Topic from a 32bit Windows XP box, will match and send data to a 64bit VxWorks RTOS DataReader running on PPC, or to a 32bit Android application running on an Arm tablet, or to a Linux application running on something else. The connections might be over UDP or shared memory, serial, fiber, or 40Gb Infiniband – or satellite.
RTI Connext DDS supports data interchange between about 80 different platforms (a platform is a specific set of OS, CPU architecture, compiler and language). DDS doesn’t force you to use C++ on Intel-based Windows, MacOS, or Linux. Each node within the system, or overall system-of-systems, can be whichever platform best suits for its purpose. Java-based HMIs interface cleanly with C++ complex-event-processors or C-based sensor firmware. All without a single server sitting someplace soaking up resources and adding latency to the real-time data streams.
Inside the data flow
In the video, I’m publishing Leap Motion frame data via DDS to the cloud (from a Windows laptop, and from a MacBook Pro). The data is subscribed to by Java applications running on other computers (the Java apps can run on any of the three machines seen in the video, the third being an Arm v6 Raspberry Pi running Wheezy, a Debian variant of Linux). The data flow starts with a C++ bridge that reads the raw Leap Motion frames at the rate they’re generated, and converts the data into three defined Types (one each for Pointables, Hands and Gestures).
From there, the instances are published on the corresponding Topics; I’m using Leap::Pointable, Leap::Hand and Leap::Gesture. The nature of DDS is that if there is nothing subscribing to the data, it isn’t sent. In this demo, only the Leap::Hand instances are published written to the wire, as that’s what the Java applications are subscribing to. Additionally, the Java instances are subscribing using a time-based filter quality-of-service setting. Even though the Frame data is provided to the C++ bridge at whatever rate the Leap is running at, the data is throttled to 30 fps max on the writer side by the middleware, as, again, that is what the subscriber has contracted for.
In one of the applications (on the Raspberry Pi), the hand’s orientation data is mediated into a different format (SteeringControlType) and republished on a different Topic (SteeringControl). This new data is subscribed to by a LabVIEW instance, converted into a NXT command string, and sent out via Bluetooth to the robot. Because of DDS’s anonymous discovery protocols, I don’t need to set up addressing (no socket programming needed!). I can move this specific application to any Java-enabled computer on the subnet without a recompile or change of configuration (no IP address changes). RTI Connext DDS supports C, C++, C# .NET, Java, Ada and Lua, and Python bindings exist. You aren’t limited to a single language, even on a single machine.
In the video, there are also three instances of the Shapes demo, a GUI-based interoperability and Quality of Service demonstration tool, which has a well-defined set of Topics and Types. We often use it at OMG meetings to prove that different vendors’ implementations of DDS are interoperable (a.k.a. software “plug-fests”). I’ve written a second application that subscribes to Leap::Hand, and publishes to the Shapes topics Triangle and Circle based on each data instance seen. The triangles turn to indicate the direction of travel commanded, and also indicate speed by size (faster = larger), while the Circles trace the approximate path of the robot, based on dead-reckoning (turn rate, duration, and travel rate, duration). It’s almost like playing “Snake” using a real robot.
All of that is done without writing one line of socket code. And I’m not limited to “a second computer.” DDS doesn’t use a server, has automatic peer discovery protocols, is massively scalable, and is supported on just about any platform you’re liable to be using. And, because the data resides in a shared data space, you can do things like this.
I’m on record as saying that developers building applications that use the Leap Motion as a 1:1 interaction device, within the IoT space, are insufficiently ambitious. That’s not to say that there aren’t lots of great ways to use the Leap Motion Controller in smart homes and hardware integrations. But if you’re only thinking about 1:1, about how to use one Leap Motion device with one laptop, you aren’t thinking about where the technology will take you. You’re thinking about how to solve your current problem. It’s important to start thinking now about problems that are in the distance; implementation choices made now should not prevent future solutions to the wider requirements.
Multi-user Leap Motion-enabled games are on the horizon. Have you considered how you will solve for that requirement, as it continues to scale upwards? With a DDS-enabled application, you won’t need a server, and players in a single subnet (for example, on an aircraft) can join or leave in an ad-hoc fashion. Each player application simply publishes its state on a known Topic, and every other player’s instance sees the state and reacts accordingly, in its own frame of reference. If you scale up to “cloud based” games, discovery does need a cloud based server for discovery purposes. Once discovery is completed, however, data should be point-to-point.
Release the data to the cloud. Set it free. And see what you (and others) can do with it.
Start building with DDS
If you’d like to experiment with RTI Connext DDS and Leap Motion Controller, you can visit the RTI Community at community.rti.com. I’m starting a thread over on the forums that will describe how to get started, what libraries you need, example bridge code, and so on to make an initial “hello world” type application using the Shapes demo.
Hope to see you there!