I have a recurring dream that starts as a nightmare but turns into something else altogether.

Imagine the stage of a monumental concert hall. The auditorium is packed, and as the audience notices you, thousands of conversations turn into a deep, imposing silence that sends a chill down your spine. Spotlights on a majestic grand piano. You take your seat, and the thought hits you – I don’t know how to play the piano!

Terrified, you look up again at the audience. They’re all still sitting there, thousands of them, surely expecting an awe-inspiring performance from a world-class musician. Your heart feels like ice water in your chest. You glance around, looking for an escape. But instead of running, you put your fingers down on the keyboard and press a few keys at random.

To your surprise, the piano produces a very pleasant and perfectly harmonic sound. Notes fill the air. Your heartbeat slows, and a crazy idea flashes in your head – What if I just stayed and tried to improvise something?

Flying feels natural in your dreams

I’ve heard that many people often dream about flying. They start by running down the street, extend their arms and then, all of a sudden, their feet leave the ground. Controlling the flight is purely instinctive. Something similar happens in my concert. My hands carelessly fly up and down the keyboard, improvising a melody as it forms in my head. I only need to focus on the flow of the music, the patterns, the strokes.

There’s something strange and magical about the experience. Sometimes it’s not clear whether I’m playing the piano, or it’s playing me, or we’re just one music-making entity. Then, as with all great dreams, everything slides away, and I wake up disappointed. I return to my keyboard and try to improvise in the same way, but it doesn’t work. The spell is broken. And then, one day, I heard about Leap Motion.

Harmony = musical theory + math

I wanted to build an instrument that worked like the magic piano, where you would just wave your hands up and down, and the music would just pour out of it, following your lead. It was immediately obvious that the instrument couldn’t rely on simply playing notes depending on the position of your fingers, like traditional instruments do. My instrument had to be intelligent – interpreting gestural input to define the underlying structure of the music.

Digging further, I determined that the timing of the note events, how loud they’re played and the overarching chord may be defined by the performer using some basic – almost instinctive – triggers. Typically, these would be finger motions and gestures, as well as velocity. However, the actual note to be played each time would be decided by the algorithm, depending on a number of parameters.

My project, the Motion Harp, is smart enough to understand the foundations of music theory and applies this knowledge to determine which notes will sound pleasant to the human ear when played together. This works because harmony, like many musical concepts, can be described mathematically. The Motion Harp traces your finger movements using the Leap Motion Controller and plays harmonic notes, trying to respect your input whenever possible (i.e. using your finger positions to determine the notes to be played), but correcting them when the notes to play are inconsistent with the current chord.

In this way, your music is well-sculpted, ensuring that the outcome is always going to sound pleasant and harmonic. The bottom line is that, regardless of what you do or how you play the Motion Harp, it’s always going to sound nice. In that sense, it works just like the magic piano of my dreams.

Becoming Freddie Mercury’s pianist

Perhaps the most fun way to start using the Motion Harp is in play-along mode. Also known as reverse karaoke, this lets you play the instrumental accompaniment of a famous song. In the featured video, you can see me playing the piano and strings together with an a capella version of Queen’s The Show Must Go On. (The effect is best achieved when you use an a capella, or vocals-only, version of a song. I’d love to see someone take a crack at Under Pressure.) Here’s an extended version of that video:

It’s absolutely thrilling to feel like Freddie Mercury’s pianist. This mode requires some previous preparation because the algorithm needs to know in advance the chords (and their timing) in the song. I can imagine that a more advanced future version might be able to detect this information in real time as the song is played. For instance, here’s the configuration file that I created in Motion Harp for The Show Must Go On:

#!/usr/bin/env python
#
# mp3 file of song vocals ripped from this video posted by Rhye cl: https://www.youtube.com/watch?v=2GGFdY3Sopc
#
songfile='queen.show.mp3'
strictharmony_left=1.0
strictharmony_right=1.0
timedelay=1.0 # Time (seconds) to wait for the external song before chord sequence starts
songvolume=0.8
bpm=60
#time1=30.97 # Chord 1, "Empty sp-aces" cambio a Amin
#time2=3*60.+56.618 # Chord 74, "...on with the show"... cambio a Dmi5- (E min)
#bpm=(74.-1.)*4./((time2-time1)/60.)
#bartimes=[x*60./bpm*4. for x in range(len(chords2play))]

tuneshift=2 # Shift all notes by this many half-tones

chords2play= ['A min', 'A min', 'A min', 'F ', 'F ', 'D min7', 'D min7', 'E sus4', 'E ', 'D min', 'D min7', 'A min', 'A min', 'F ', 'F ', 'D min7', 'D min7', 'E sus4', 'E ', 'D min', 'A min', 'A min', 'F ', 'F ', 'D min7', 'D min7', 'E sus4', 'E ', 'D min', 'D min7', 'A min', 'A min', 'B min', 'B min', 'G ', 'G ', 'E min7', 'E min6', 'F#sus4', 'F#', 'E min', 'E min7', 'B min', 'B min', 'G ', 'G ', 'E min7', 'E min6', 'F#sus4', 'F#', 'E min', 'E min7', 'D min', 'A min', 'A min', 'F ', 'F ', 'D min7', 'D min7', 'E sus4', 'E ', 'D min', 'D min7', 'A min', 'A min', 'F ', 'F ', 'D min7', 'D min7', 'E sus4', 'E ', 'D min', 'D min7', 'D#', 'F ', 'D min', 'G min', 'D#', 'F ', 'D min', 'G min', 'G 9', 'C ', 'A min', 'A min', 'F ', 'F ', 'D min7', 'D min7', 'E sus4', 'E ', 'D min', 'A min', 'A min', 'F ', 'F ', 'D min7', 'D min7', 'E sus4', 'E ', 'D min', 'D min7', 'D min', 'A min',
             'A min','A min',
             'A min','A min','A min','A min','A min']

bartimes=[0.0,2.533916,5.259754,8.145974,11.074259,13.904522,15.305486,16.767951, 18.191854,19.613805,21.043787,22.39159,25.296849,28.1851,31.042223,33.899382, 35.40739,36.749832,38.452647,39.67957,42.38289,46.788196,48.158804,52.425072, 53.884473,56.173917,56.697277,58.222756,59.539141,60.981626,62.324943,65.357807, 68.082607,70.926903,73.718021,76.771667,79.47994,81.418602,82.415361,84.391472, 85.286992,87.269014,88.095119,90.853685,93.691893,96.580328,99.495682,100.914349, 102.355663,103.900849,105.393979,107.254566,109.367109,110.926674,115.22507, 116.726141,119.477252,122.370006,124.660673,125.326094,127.279175,128.218932, 130.08445,130.889897,133.746921,136.620955,140.065846,141.529712,143.097537, 145.837099,147.440506,148.735401,150.288588,151.69717,153.329244,154.572879, 155.825705,157.272017,158.693455,160.219647,161.675917,163.05136,164.529705, 165.892459,170.104829,174.441056,177.287831,180.174833,182.090808,183.024531, 184.830904,186.016432,188.748107,191.607597,194.37595,197.26714,200.136571, 201.674509,203.164508,204.704994,206.54176,208.752472,211.394736,214.151376, 217.067148,219.815456,221.204235,222.480935,222.918065,223.226339,223.495976, 223.788555]

Instant gratification and the future of music

The future of technology in music, and probably art in general, is to set us free from the tedious, repetitive work that it takes to become a skilled performer. It takes many years of dedication to learn the intricacies of the technique required to play an instrument or paint a picture. I believe that in the future, truly creative minds will be able to produce inspiring music even if they can’t play an instrument, as technology will let them translate their ideas into tangible artwork. The Motion Harp is a step in this direction. Unlike conventional music, it doesn’t take any formal training to play and it creates a rewarding experience from the very first moment.

Because of its instant gratification nature, Motion Harp is perfect for introducing small children to music. Since designing it, I’ve had children as young as three years old experiment with it and enjoy the experience. After a couple of hours, a four-year-old was able to understand the concepts of note and chord.

More recently, I learned that a music teacher who works with children with fine motor-skills disabilities found it a useful tool for her classroom. These children have severe difficulties, or are completely unable, to play a conventional instrument, but with Motion Harp they can now feel the joy of playing their own music. I’m very much looking forward to hearing more about their experience and sharing it with you!

Open source for everyone

I’m a strong believer in open source. My coding skills, just like my music knowledge, are rather limited. It would make me very happy to see this project grow and expand with the contributions of others who share my excitement about this concept. That’s why I’ve published the Motion Harp source code under GNU license on GitHub.

It still needs a proper UI – since all the configuration and run options are currently hardwired in the code, you need to manually edit the file to make changes. Everything is well-commented, and requirements and instructions are provided in the comments at the beginning of the code. I expect to have a GUI implemented in the next few weeks, and hopefully, that will be the first of many commits to come in the near future. Have fun and be creative!