As the 3D Jam approaches, developers around the globe are already getting a headstart on their projects. Zach Kinstner, the creator behind Hovercast and Firework Factory, has been sharing his latest project through a series of videos – a virtual reality guitar! We caught up with Zach this week to talk about his design process, especially the guitar’s unique combination of visual depth cues.

Stage 1: Building the Guitar

In this first video, Zach dives quickly into Unity, having set up the MIDI guitar sounds and attached them to some simple visual “strings.” He adds a visual indicator to the fingertips to help show the user where the hand is in 3D space.

What’s the importance of these visual indicators?

Visual indicators are very helpful for understanding where your hands are in virtual space. They can provide a good sense of depth, show which parts of your hand are being used for input, and help you discover which objects in the scene are interactive. (I wrote about this in more detail in my Hovercast blog post.)

For the guitar project, there’s a “strum zone,” which is basically a 3D rectangle. To strum the guitar, your finger has to be within that zone. In that scenario, I thought it was important – necessary, really – to give the user a strong sense of depth. The visual indicators are helpful for keeping your strumming finger inside of the zone when you want to hit the strings, and outside of the zone when you don’t.

These twinkles are the first in his visual clue experiments.

Why not just allow the hands at any depth to interact with the strings?

Without some depth restrictions, the guitar strings would make noise with just about any hand movement. While I’m not looking for total realism, I do want the strumming to feel somewhat natural. I play “real” guitar, so I’m using that experience as a general guideline. Sometimes you want to strum downward, or to strum upward, or to hit individual strings, or to move your hand without hitting the strings at all. The “strum zone” is a good way to achieve those goals.

Stage 2: Chord Selection

In his second update video, Zach added the ability to select chords using input from the user’s left hand.

What’s the thinking behind the chord selection design?

Working with 3D input devices like the Leap Motion, I try to find the simplest ways to accomplish a task, and also to utilize the strengths of the device. I have to consider which movements, poses, and gestures will work most reliably with the hardware, and how easily they can be learned by new users. I’ve done a significant amount of exploration with the “hover” interaction, and found that it works well, so I decided to try it for the chord selectors.

The layout of the chord selectors should seem familiar to guitar players. The selectors are arranged in a grid to match the first five notes of the first three strings. Essentially, you’re selecting the bass note of the chord, and the other strings update to form the full chord. I may also add the ability to switch between major and minor chord formations – possibly using the orientation of your hand or the “spread” of your fingers.

Stage 3: Additional Depth Cues for VR

When it comes to visual indicators in VR, there’s a delicate balance between being distracting and being easy to overlook. In the third video, Zach has added a few new types of visual indicators that hit this crucial balance.

Are you concerned that visual indicators on the edge of the screen risk causing distraction in the periphery of your vision?

I see it the other way – placing them at the edges of vision helps avoid distraction. My first attempt at a depth indicator (in the second video) placed graphics on the front and back sides of the “strum zone.” This worked well for slow, deliberate tests, but not as well in an actual guitar-playing scenario. The indicators were too subtle to see clearly when moving quickly, and making them brighter or bolder meant more clutter in front of the strings and selectors.

You made several “layers” of indicators in this update video. Red/yellow/green ones to show depth and whether the hands are in the “strum” space. White highlight blocks to help show you where your chord hand was in space. And light grey blocks to show on the sides where the strings lined up.

That’s a somewhat complicated combo, yet it seems to work beautifully. What was your thought process here?

Thanks! So far, I agree – I think I’m on the right track with this latest concept. Wrapping the visual indicators around the “strum zone” allows them to be easier to see – or maybe, easier to perceive – without obstructing your view of the main interactions.

My goal is for the user to be aware of these visual indicators, and find them helpful, without really paying attention to them. For example, you might be focused on your chord selections, but with the flash of red near the edge of your vision, you immediately know when, and where, you have moved outside the “strum zone”.

Of course, the design of these indicators is still in an early phase. As I refine them, I anticipate that certain colors or shapes or sizes will be more effective than others, and that each “layer” will retain distinct visual cues. All of these elements need to be balanced properly to make this “complicated combo” work well.

For the latest on Zach’s VR Guitar, be sure to watch his 3D Jam project thread on our community forums. What do you think of the demo so far? Let us know in the comments!