When Craig Winslow and Justin Kuzma paint interactive digital media directly onto the physical world, your senses feast in unexpected ways. Last fall, they delivered Growth, an immersive forest of trees you could manipulate and command with your hands in the air. Their most recent Leap Motion installation, ZX, went up this February in Vermont, where the team installed the 10-foot geometric structure in a snowy courtyard at Champlain College.

ZX combines projection mapping with Leap Motion technology to explore the boundaries of color and sound in 3D space. A strong tone welcomes you as you hover your hands over the Controller. Bells chime as you wiggle your fingers. The higher you lift your hands, the brighter the color. Reaching your fingers forward makes for richer saturation, while panning your hand left or right adjusts hue. Hues pan left and right in the stereo speakers, so that your hands begin to define the limits of an intangible field of view.

For the sonic side of things, Craig and Justin enlisted the sound design talent of Nicholas Giordani. Inspired by the theremin, Nicholas mapped sounds to gestures, as well as created a harmonic scale controlled by the height of your hand. “Because we aren’t limited by physical interaction, like turning a knob or pressing a button, we have a lot more room for sonic exploration,” he says. “It’s almost like we’re creating a new instrument.”

The finished project makes you feel like you’re conducting an orchestra and a light show simultaneously – a blending of the senses that reflects the experience of synesthesia. At the same time, the combination of light and sound creates a surreal experience, where the interactive space takes on an almost tactile reality.

For the ZX crew, this first iteration is just the tip of the iceberg. “One of the interests for us in adding a microphone is that we can visualize sound from the user as color as well. Combined with kinetic hand movements as color and sound, we could have someone could sing a note that translates to a color, while creating a harmony with their hands,” Craig told us.

The team is also hoping to take the installation into the education space to teach children about the color spectrum dynamically in 3D, both in large scale settings – museums and exploratoriums – and smaller ones – the classroom and the home.

What do you think about 3D motion control as an exploratory engine for color and sound? Tweet us at @LeapMotion or share your thoughts on Facebook.