Hi, we’re Sam and Kartik – a computer science at the University of Bristol in England, and a computational physics student at the University of Waterloo in Canada. This past Valentine’s Day, we decided to embrace the engineer’s stereotype and spend it at the University of Pennsylvania, coding up a cool creation as part of the PennApps hackathon.

After a very hectic 36 hours of coding we presented our hack – Leapouts, a collaborative 3D model builder, controlled with Leap Motion interaction, that runs inside a Google Hangout. We’ve made a demo available for you to try at leapouts.com. But how did we build it over a weekend without ever having used the Leap Motion API, Firebase, or Three.js before?

We ♥ hackathons

One of the best things about hackathons is that they give you an opportunity to work with tech you haven’t used before, or focus on the part of a project that is not normally your speciality. In past internships and hackathons, we’ve both worked on very backend and algorithm-heavy products, but at PennApps we decided to change that. Part of a hackathon is working outside your comfort zone, so we decided to try a hardware project and ended up making an all front-end graphics-focused Javascript hack using Leap Motion Controllers.

So you might be thinking, "how did a student from England and a student from Canada end up getting to know each other?" We met because Kartik is bad at setting up Facebook events. Last summer we both interned at companies in the Bay Area. On a Friday near the beginning of the summer, Kartik meant to post a dinner event to the Waterloo interns Facebook group, but accidentally posted it to the humongous 5000-person group containing all the Bay Area interns.

Sam was one of the 60-or-so people that ended up coming to what was meant to be a 10-person dinner. After that, we became good friends through a shared appreciation for the finer things in life – like hackathons and action movies with lots of explosions.

From the jaws of defeat

As is usually the case at hackathons, Leapouts was not our first idea. Originally, we were working separately on different projects, but encountered various issues. At this point it was about 6AM on Saturday, and after drowning our sorrows in coffee and donuts, we decided to make Leapouts. This gave all the other teams a 10-hour headstart, so to catch up we had to use some (what can only be described as) ‘hacky’ coding methods. But, then again, that’s the whole point of a hackathon.

Our initial goal with Leapouts was to make a collaborative 3D model builder, and in 26 hours, we more or less achieved that. By the time the presentations rolled around, we had added a unique spin on the 3D model builder. Instead of just having a shared 3D plane to build on, we embedded Leapouts into Google Hangouts. Everyone in the Hangout gets automatic access to the model builder without any tricky authentication or extra logins during a video chat.

When users load up a Hangout, at first they see a blank plane, which can be moved around with the Leap Motion Controller. The spacebar cycles through available 3D shapes, and ‘n’ adds them to the scene. Pointing at a shape highlights it, indicating that it’s selected. It can then be moved around and resized in the scene using the Leap Motion Controller.

Building Leapouts

The tech:

• Three.js for the scene
• LeapJS for control
• Firebase for synchronisation and monitoring state change
• ThreeLeapControls for a simplified interface between the Leap Motion Controller and Three.js objects

One of the core problems our hack tackled was cross-machine scene synchronisation. Unfortunately, very few 3D modelling libraries are built with this use-case in mind, so we had to resort to some dirty hacks. Not to mention, it was our first time using Three.js!

Whenever an object in Three.js is created, it’s assigned a unique ID. We intercepted that action, and upon object creation propagated a message to Firebase telling it to update our data model with that new shape’s information. Then, Firebase triggered events in all other users’ browsers, which automatically created similar objects. The issue here is that those new objects have their own IDs assigned by Three.js, which made tracking object movements between machines quite difficult – since the same object could have multiple IDs.

Our solution overwrote the object ID of all objects but the original. The various types of object manipulation were handled in a similar way. Whenever a user selected, resized, or moved an object, the resulting changes were forwarded to Firebase, which then triggered update events for all other users.

Another issue we had to deal with was collisions – two users selecting the same object at once. We partially remedied this by not letting users select objects that had already been explicitly highlighted for selection, but the difficult part was when two users selected the same object before the selection of one could be propagated. Our solution was to make the changes to the object that was modified last to be the definitive properties associated with that object.

We used the following function to create a shape and deliver changes to Firebase. Here, pressing ‘n’ would execute the generateRectangle shape function, which passes the param.shape action to Firebase along with the new object’s properties to renderShape function.

var generateRectangle = function(param, uuid) {
  var geometry = new THREE.CubeGeometry(param.len, param.width, param.height);
  var rendered = generateObject(param.x, param.y, param.z, geometry, param.color);

  param.shape = "generateRectangle";
  renderShape(rendered, param, uuid);
}

The function renderShape then checks if the uuid of the new shape should be overwritten and passes the coordinates for the size and rotation to Firebase.

var renderShape = function(shapeObject, params, uuid) {

  // if shape id exists, overwrite it with synced id
  if(typeof uuid !== 'undefined' || uuid != null) {
      shapeObject['object']['uuid'] = uuid;
  }

  // insert shape on screen
  scene.add(shapeObject['object']);
  objects.push(shapeObject['object']);
  objectsControls.push(shapeObject['objectControls']);
  renderer.render(scene, camera);

  // if the shape id is not defined, get default id
  // and sync across all sessions
  if(typeof uuid == undefined || uuid == null) {
    var uuid = shapeObject['object']['uuid'];
    var fbref = new Firebase('https://leapmotion.firebaseio.com/'+ window.hangoutID +'/' + uuid);

    var sobj = shapeObject['object']['scale'];
    var robj = shapeObject['object']['rotation'];

    // get shape's size and position coordinates and
    // and propagate changes to firebase
    params.scale = {x: sobj.x, y: sobj.y, z: sobj.z};
    params.rotation = {x: robj.x, y: robj.y, z: robj.z};
    params.uuid = uuid;

    fbref.set(params); // update firebase
  }

}

Similarly, we placed observers in each session that would listen for any changes to the object and get the corresponding action from Firebase and apply changes locally. Since all this was done with websockets, the changes propagated within milliseconds, and the whole experience seemed realtime!

var watchdog = new Firebase('https://leap.firebaseio.com/' + window.hangoutID);

/*
 When any attendee creates a new shape
*/
watchdog.on('child_added', function(snapshot) {
  var uuid  = snapshot.name()
  var param = snapshot.val();
  var shape = param.shape;

  for(var i in objects) {
    // check if the new object already exists
    // in the current scene
    if(objects[i]['uuid'] == uuid) {
      return;
    }
  }

  // render the shape with overwritten id
  shapes[shape](param, uuid)
});


/*
 When any attendee removes an existing shape
*/
watchdog.on('child_removed', function(snapshot) {
  var uuid  = snapshot.name();

  for(var i in objects) {
    // if the shape exists in our scene
    // delete it
    if (objects[i]['uuid'] == uuid) {
      scene.remove(objects[i]);
    }
  }
});

/*
 When any attendee modifies an existing shape
*/
watchdog.on('child_changed', function(snapshot) {
  var uuid  = snapshot.name()
  var param = snapshot.val();

  // go through all scene objects
  for(var i=0; i<objects.length; i++) {
    // check if the ith object is the one
    // that got modified
    if(objects[i]['uuid'] == uuid) {

      // update it's position and render new scene
      objects[i]['position'].set(param.x, param.y, param.z);
      renderer.render(scene, camera);

      // now apply the new changes to local
      // object data model
      objects[i].scale = param.scale;
      objects[i].rotation = param.rotation;
    }
  }
});

This is an example of the data model that was synced across instances and stored in Firebase

{
   "z":-138.02564000885235,
   "color":"#CFCFCF",
   "y":146.9915181057351,
   "len":19.2813565954566,
   "x":50.05251270515234,
   "height":26.35056875878945,
   "shape":"generateCube",
   "rotation":{
      "z":0,
      "y":0.43024888935094685,
      "x":0
   },
   "width":23.807244678027928,
   "scale":{
      "z":3.338711500000005,
      "y":3.338711500000005,
      "x":3.338711500000005
   },
   "uuid":"D9E253BC-4FED-4E18-A82A-CE52C32285A3"
}

Potential use cases

One of the main areas we envision software like this being used is in architecture. Currently, collaboration tools in architecture are rather terrible. Across a team, things like revision control become incredibly messy very quickly, since architectural models typically aren’t distributed across multiple files like code is. Our tools would enable an architectural team to all simultaneously work on a shared model. We don’t sync camera views, so although the model is shared, every member of the team can view it from independent angles.

The second area where our software could be greatly useful is gaming; in fact this was one of our original stretch goals for the hack. We were thinking of building either a simple Lego-style scene editor, or a board game like chess or checkers. Almost any game that involves real-time multiplayer within an interactive 3D scene could work.

The future

If you have the opportunity to go to a hackathon, and have never been to one before, then go. It will be a fantastic experience because of the people you’ll meet and the new tech you’ll get to work with.

You can check out a demo version of Leapouts at leapouts.com. Just click the ‘Start Hangout’ button and invite your friends to get started!

You can follow Kartik and Sam on Twitter @therealkartik and @sambodanis. Check out more of their work on Github: kartiktalwar and sambodanis.