Building Applications for Simultaneous Deployment on Web and Native Platforms; Or: How I Learned to Stop Worrying and Love Node-Webkit

TVO’s IdeaShaker Innovation Lab explores emerging technologies that have the potential to disrupt learning. They’re the creators of the free educational app Caterpillar Count, which was originally built with web technologies.

Web apps are great – people on a variety of platforms can access them instantly, there’s no need for users to upgrade to the latest version, and it’s easier to reach casual users who don’t want to install a native application. While the Airspace Store includes links to web apps, for Leap Motion developers who want their apps to be part of the native Airspace Home experience, native apps are the way to go. This leaves web developers in a bit of a dilemma. Should I build Leap Motion apps for the web, or start working a different platform altogether? We chose the third option – build and convert.

We took our app Caterpillar Count, created in Flash CS6 and HTML5 canvas – with open-source JavaScript libraries, CreateJS, and RequireJS – and packaged it to Windows(exe) and Mac(dmg) native applications with Node-Webkit. In this post, we’d like to share what we learned and how we built the application. Here’s what we’ll cover:

Getting started

Developing your web-to-native app

Getting prepared HTML5 assets from Flash CS6
Export graphic and audio files from Flash CS6 with Toolkit For CreateJS
Create sprite sheet using Flash CS6 Sprite Sheet Generator

JavaScript and the Leap Motion Controller

RequireJS
CreateJS
LeapJS
Packaging with Node-Webkit

Getting started

Caterpillar Count was originally created with Flash using AS2, and you can play the original Flash game on our website. Technically, our project was to convert the Flash game to HTML5. From there, we packaged it into Windows and Mac native applications.

In this post, we’ll assume you have some understanding of HTML5, CSS, and JavaScript. Understanding Flash is an asset, but is not essential, especially for those who already have an existing Flash game.

We used Flash CS6, Node-Webkit ver. 0.6.2, and the following JavaScript libraries:

Leap.js provides the functionality and object structure of the Leap API to assist developers who are working with the Leap Motion Controller in a browser environment.

Create JS simplifies development for HTML5 using Canvas and Audio. It uses terminology similar to Flash – eg. “Stage” and “DisplayObject” – and enables preloading of audio and image files with visual feedback.

Require JS aids in writing modular JS, which makes it easier to manage JavaScript projects. It allows modules to list dependencies and grabs JS on the fly based on dependencies.

Developing your web-to-native app

This guide application is broadly divided into 3 sections: Flash, HTML5, and the user with the Leap Motion Controller.

Getting prepared HTML5 assets from Flash CS6

You can create HTML5-ready assets from Flash CS6 that include vector graphics, MP3 audio files, and a sprite sheet from a Flash CS6 timeline animation. We used Toolkit for CreateJS for exporting vector graphic and audio files, and the Sprite Sheet Generator for creating sprite sheets, which were then used for animated HTML5 content.

Export graphic and audio files from Flash CS6 with Toolkit For CreateJS

Toolkit for CreateJS is a free extension in Flash CS6 that lets you export reusable vector symbols and MP3-format audio for your CreateJS projects. You can download it from Adobe’s official site. Once you’ve installed it to Flash CS6, you can add to the panel window below:

Create sprite sheet using Flash CS6 Sprite Sheet Generator

A sprite sheet is a bitmap image file that contains several smaller graphics in a tiled grid arrangement. Flash CS6 has a new feature called the Sprite Sheet Generator that lets you generate a sprite sheet from a MovieClip’s timeline animation. We used several sprite sheet animations to create Caterpillar Count.

Creating a sprite sheet is really easy. Right-click on any MovieClip with a timeline animation you want to publish, and select ‘Generate Sprite Sheet’ from the menu.

From here, there are actually two files generated. If you open the directory, you’ll find xxx.js and xxx.png files. In the .js file, you can see the data used to define the animation frames, and the animation has been flattened into a single .png file.

The Sprite Sheet Generator in Flash CS6 supports several data formats for output, including JSON, cocos2d, and easeljs. Since our project was built with CreateJS, exporting easeljs was quite handy for us.

JavaScript and the Leap Motion Controller

RequireJS

Game developers are used to a particular application structure, where each class is in a separate file, class files are organized into packages/namespaces and clearly list their dependencies (i.e. import), and the Document (e.g. FLA) links to a single Document Class (e.g. Main.as). With RequireJS, we were able to build our game in a way that satisfies all of these criteria.

As you can see above, our file structure shows that our classes are separated into two packages – tvokids.caterpillarcount and tvokids.common. Within tvokids.caterpillarcount, our classes are further separated into engine, overlays, sprites, and views. This helps us organize each file by its function; for example, engine only contains classes directed related to gameplay.

Unlike the traditional method of declaring multiple script tags for each file, only one script tag is used with RequireJS.

&lt;script src=”scripts/require.js”
data-main=”scripts/tvokids/caterpillarcount/Main”&gt;&lt;/script&gt;

This tag serves two purposes: it points to the location of RequireJS itself, and provides RequireJS with the location of our document class, Main.js. (Incidentally, the “js” in Main.js is missing, since the “js” extension for all files in RequireJS is implied.)

Our Main.js file only contains two method calls: requirejs.config( … ), and requirejs( … ). The config method is used to declare paths and libraries. By default, RequireJS attempts to load dependencies from the location of the HTML document. Since all of our JavaScript files are located under scripts/tvokids/, we would add this as our base path. Libraries are declared under “shim”. This allows libraries to be declared as dependencies of modules without interfering with the scope of the library.

requirejs.config({
    baseUrl: “scripts/tvokids/caterpillarcount”,
    shim : {
        leap: { exports: “Leap” },
        […]
    },
    paths: {
        common: “../common”,
        leap: “../../lib/leap.min”
        [...]
    }
}

The require method is where you initialize your application. Just like any other module, it lists dependencies and performs actions.

requirejs(
    [ “Config”, “Images”, “common/Button” ],
    function (Config, Images, Button) {
        // Main application logic;
        // add listeners, init views, etc.
    }
);

All other modules within RequireJS follow the same syntax shown above, but use the method “define” in place of “requirejs”. The first argument, an Array, lists the js files that your module depends upon. (As we mentioned earlier, the js extension is implied, and should not be included.) The second argument is a function; in the case of module definitions, this function should return the object you’re defining. The parameters Config, Images, and Button shown above coincide with the declared dependencies, and receive the object that each js file returns. The order is important, as the order of parameters should match the order of the dependencies.

CreateJS

CreateJS is an excellent and versatile set of libraries: PreloadJS, SoundJS, EaselJS and TweenJS. Our application was built as described by a vast number of online tutorials that are already available from the community at large. However, we did run into a number of caveats when combining CreateJS with Node-Webkit and our other libraries. These are some key issues that you can avoid on your web-to-native journey.

First, it’s worth mentioning that CreateJS and RequireJS work together beautifully when using the “shim” method described earlier.

When using PreloadJS, you can load assets using either XML HTTP Requests (XHR) or DOM methods (tag loading). Unfortunately, XHR loading did not seem to play well with Node-Webkit. A quick work-around for this problem is to force tag-loading in PreloadJS by passing false into LoadQueue:

var queue = new createjs.LoadQueue( false );

Another problem we uncovered seems to lie with HTML5 Audio, where audio files delivered by the server without certain headers (eg. Content-Range) will cause the audio to play incorrectly. This problem is solved by newer versions of the SoundJS library (0.4.1). Unfortunately, the newest version of SoundJS seemed to cause Node-Webkit to crash to desktop after prolonged play on Mac OSX. This is a problem we have yet to solve elegantly; our Node-Webkit release still uses SoundJS 0.3.0, while any version to be delivered by a regular web server would have to use SoundJS 0.4.1.

LeapJS

The LeapJS library is very easy to use, and can be summed up in only a few lines of code:

Leap.loop(function (frame) {
console.log(frame);
});

Most applications will find everything they require within the frame object shown above. By examining the object through the console, you should have a clear view of what types of data can be accessed from the device.

Alternatively, Leap.Controller can be instantiated with listeners added explicitly to the instance:

var leapController = new Leap.Controller();
leapController.on(“frame”, function () { … });
leapController.on(“deviceConnect”, function () { … });
leapController.on(“deviceDisconnect”, function () { … });

Either method will allow your app to access data such as gestures, finger and palm position, orientation, velocity, etc. from the frame object.

Packaging with Node-Webkit

You’re almost home free! At this point, you can create Mac OSX, Windows, and Linux desktop standalone apps from HTML and Javascript using Node-Webkit. This is a new way of writing native applications with all web technologies, created and developed at the Intel Open Source Technology Centre.

The steps to actually packaging your app are very well documented at node-webkit developer Roger Wang’s GitHub page. (Be sure to use Node-Webkit version 0.6.2+ to avoid getting false positives from virus scans.)

The steps are fairly straightforward – basically anything that will work using a webkit browser should work with node webkit. Put all of the files in a folder and make sure all of your links are relative links that point to the files in that folder (or a subfolder). Then you just need a package.json configuration file – there’s a simple version of one on Roger’s page.

Before you submit your app to the Airspace Store, be sure to follow Leap Motion’s review guidelines – including making sure that your application goes full-screen automatically when it starts, and enabling the Esc key to close the application. Beyond these strictly native interaction elements, you should also be sure that your app offers a great Leap Motion-enabled experience.

So that’s it! In this tutorial, we covered the development workflow from Flash to HTML5, added the Leap Motion Controller, and packaged it to OSX/Windows native applications. We hope that you can benefit from our experience in developing your own apps. If you have any questions or feedback, please post your comments below. We’d love to hear from you.