The toughest part of my Leap Motion guitar project was the nature of the mapping – deciding what does what. To get started, I decided to approach the problem from a design perspective. As in any design task, there are many correct answers, and yet also a lot more wrong answers lurking in the deep. Every decision needs to be well thought-out, with a clear picture of the final outcome in mind. There are pitfalls everywhere.

Thinking about motion control means imagining what’s possible within three dimensions of movement, as the interactive space above the Leap Motion Controller has three axes – width (X), height (Y), and depth (Z). The first thing that came to my mind was mapping each to a distinct MIDI parameter. This formed the basis of my first prototype.

The beauty of MIDI is that many programs (such as FL Studio, which I use in my videos) allow you to control practically anything in real time. It’s both fascinating and terrifying at the same time to have so many options. When designing the mapping of a musical interface like this, there are a lot more factors to take into account than you may think.

Guitar movement design challenges

• Certain movements feel natural while playing a guitar. You might even be performing them involuntarily on your own when you get carried away playing alone in your room! Try to pay attention and identify what feels natural to you when you play certain things.
• Other movements might make playing difficult, as they affect the way you have to grasp the neck of the guitar.
• Sustaining an odd position over time might be straining.
• Some movements can be ergonomically impossible while playing certain things.
• Some movements might just look goofy.
• With some movements, we have more control than with others. With better control, you can express yourself more fluidly and prioritize significant parameters accordingly.
• When moving in the 3D space over the Leap Motion Controller, you’ll never move purely in a single direction. It’s inevitable that you’ll also move along another axis, which is sometimes undesirable. Be sure to avoid mapping undesired modulation parameters to degrees of freedom that can’t be isolated away from other movements.
• The most interesting thing you can do with this app is to intentionally move in more than one axis at a time. When altering two parameters through a diagonal movement, your imperfect trajectory can sometimes manifest interesting subtleties that breathe more life into your sound. Pay attention to the diagonal movement possibilities that arise from the mappings you define.

• Since the sound properties of an acoustic instrument are complexly interwoven together by the laws of physics, often a change in one property inevitably brings about a change in another – hitting a drum louder also raises its pitch, playing a saxophone louder gives its sound a harsher texture with more overtones. If you seek to design a mapping that feels more “organic,” that feels more like playing a “real instrument,” don’t go for a one-to-one mapping between variables and sound properties. Try relating several qualities of the sound to a single degree of freedom. On the other hand, if you’re seeking a more synthetic, dubstep-like sound, then maybe you should try the opposite. (This brilliant paper offers a wealth of insights into this idea.)

Once you have a working prototype, play around, find out what works well and what doesn’t, and go back to rethinking the mapping. Keep iterating through redesign and prototype evaluation until you have something that works nicely with your unique playing style. These are early days of growth and experimentation, with a lot of exciting new ground for artists to explore and uncover.

Experimenting with dimensional mapping

In this section, we’ll look at the main effects that I mapped to each axis in the video at the top of this post – including some initial design problems and creative solutions.

Y Axis

The first thing I discovered is that the Y position of the guitar worked very well for setting overdrive level: it was something meant to be gradually changed over time and that wouldn’t cause any strain from holding different positions for long periods of time. Control over this dimension is somewhat clumsy. I could never use it to control a wah wah-like effect, but that’s alright for the use I want to give it.

I later added greater richness to this degree of freedom by mapping it to the relative mix between two different effects platforms running in parallel – one producing a channel with heavy distortion plus some delay, and the other producing a cleaner sound plus some chorus, delay, and LFO effects. Holding the guitar all the way down, only the clean channel was audible; as I moved the guitar up, the distorted channel would start to gradually push the other away, until it took over entirely when holding the guitar all the way up.

Z Axis

This axis worked well for setting an oscillation speed for an LFO effect. While thrusting the guitar forward made playing a little hard, and I couldn’t hold this position for long, it worked well for the interaction that I wanted – to add some texture to a few longer sustained notes for an occasional psychedelic effect. This only affected the clean channel, not the distorted one, so pushing the guitar forward was progressively more noticeable the further down I held the guitar.

X Axis

With the X axis, I wanted to shift the guitar’s pitch, like when bending a string. This was the most exciting feature for me. Some people say that no two guitarists bend strings exactly alike – it’s an action full of subtleties and it yields a lot of expressiveness. Since the X axis is the dimension along which I can move with the most control, this modulation is well deserving of VIP treatment. I even sometimes find myself naturally turning forward a little bit when bending a string, so this mapping even matches my natural movements. In theory, it was a perfect fit!

But there was a problem. During a song, you only occasionally want to slide through intermediate notes. Most of the time, you want to be playing proper notes that are well in tune. As it’s near-impossible to keep the guitar perfectly rigid in the X dimension while playing, I was inevitably detuning all over the place. The solution I found was splitting the sensing area into two halves and only bending the pitch past a given threshold – where X = 0. You can see how this works in the diagram below, or see it live.

On the left side of that threshold, I created a neutral area where the pitch bending amount is always 0. I later pushed things a bit further and assigned two different modalities for changing the pitch – raise it up two semitones (like when you bend a string) and lowering it down a whole octave. To make both options available in the same space, I split the area past the X = 0 threshold ( the pitch-bending area) into two along the Z = 0 line, sending two different mutually exclusive MIDI messages – one sliding pitch up two semitones and the other sliding pitch down an octave.

Creating visual feedback for musical performance

Faders and knobs can tell you a lot through their physical position. As physical constructs, they naturally offer intuitive affordances – i.e. their form tells you about their state and function. An invisible air interface like the Leap Motion Controller, as cool and fluid as it may be, suffers an important disadvantage – it has no affordances at all. If the interaction gives no clues about what’s being detected, it will keep you guessing.

This can leave you in a lot less in control of your output. It’s important to know, for instance, where you stand in relation to the limits of the sensor’s field of view, and if you’ve fallen out of it. There is, of course, an audible feedback to all our actions – but once you’re hearing it, the damage is already done. Creating a visual representation can help you stay one step ahead of potential flukes and work within the device’s unique capabilities.

For my JavaScript app, I kept the GUI simple, sticking to a basic 2D visualization. I thought of building something cool in 3D, but soon came to the conclusion that for it to feel as real-time and as fluid possible, I couldn’t afford to waste processing power on complex graphics. So the GUI visualization is simple but very thoughtfully designed. Everything has a meaning. The tracking circle changes size and color according to the guitar’s position, indicating when it’s close to the limits of the field of view, or when it has crossed a meaningful threshold.

In one of the variations of my app, in which the guitar’s rate of movement is being tracked, I display a trail of the guitar’s previous positions, as well as a red bar on the side that fills up as stronger movement is being detected. This offers a clear impression of how much movement is being recognized by the sensor. You can instantly understand it through a fleeting glimpse without needing to devote a lasting gaze. This allows you to redirect your attention back to your instrument without breaking the flow.

Doubly invisible: thinking and rethinking your mapping

Remember that good mapping can make the difference between a richly expressive extension to a musical instrument and a merely amusing but unusable gimmick! The objective I’m after is to make our natural inclination – that we have to move while expressing ourselves musically – come to life as a dimension of our music. This is not about making musicians juggle with an additional instrument just to show off coordination skills. By multitasking and forcing our cognitive load to the limit of our abilities, we step further away from art and closer to a circus act.

Ideal mappings are easy to learn and feel effortless to play, as artistic expression can only be achieved when we are proficiently in control of what we do. Counterintuitively, it is often said that good design by its nature goes unnoticed, as it blends seamlessly with its context. Any well-designed tool should allow you to carry out a task without getting in your way, so that eventually you don’t even need to think consciously about it.

The Leap Motion Controller is already an invisible interface – you can tweak sound in without touching anything – but to be truly useful, it must become twice-invisible. Keeping that in mind, I invite you to try new mappings that may enrich your playing experience. Let me know about your experiments on my Leap Motion guitar forum thread.