Ever wanted to set a course for Farpoint Station, punch your robot buddy, and push a spaceship into overdrive? Make it so. Designed as an experimental research project for hybrid interfaces, our VR Cockpit demo is now available for Windows (64-bit only) on our Developer Gallery.

The demo puts you in control using a combination of Leap Motion interaction and a fully integrated Hands On Throttle and Stick (HOTAS) control system. (Even if you don’t have a HOTAS handy, you can still use a keyboard or gamepad.) In this post, we’ll take a look at the design process behind VR Cockpit.

(Want to see more videos like this one? Subscribe to our Twitch channel!)

Designing with Hybrid Interfaces

The VR Cockpit is an ongoing research project that we created to investigate design patterns for hybrid interfaces – combining physical hardware with motion controlled interactions. As a pre-production demo, it also explores a variety of other concepts including:

  • the use of narrative context to aid motion control usability
  • visual and audio feedback mechanisms for motion control interfaces
  • the use of image passthrough effects to blur the border between virtual and augmented reality experiences

Why a space simulator? Besides the obvious cool factor, the space genre gives us an excellent sandbox for exploring these concepts. Bringing motion control into a space sim cockpit creates an opportunity to unlock new kinds of interactions that can dramatically deepen the fantasy of commanding of a futuristic, faster-than-light space vessel.


Leap Motion + HOTAS Gamepad

Flight and space simulator games already make heavy use of complex physical controls such as HOTAS rigs to increase controllability, realism, and immersion. However, these hardware controls are also cumbersome in traditional virtual reality. We designed the VR Cockpit to work with one of several devices alongside the Leap Motion Controller:

  • Saitek X52/Saitek X52 HOTAS (recommended!),
  • the XBOX 360 gamepad,
  • the XBOX ONE gamepad, or
  • keyboard controls.

Our team has developed a novel approach to integrating these controls that seamlessly blends real world and virtual reality interactions using Leap Motion’s image passthrough technology. After a quick calibration, both your real world hands and the HOTAS or gamepad controls are visible inside the cockpit. This allows us to:

  • enable easy switching between virtual and physical controls
  • give players a better sense of their surroundings without removing them from their virtual environment
  • reduce the sense of isolation that bothers many new virtual reality users


Visual and Set Design

In real life, spaceship cockpits are enormously complicated, involving years (or even decades) of research. In designing the VR Cockpit, we needed to balance ergonomics and hardware tracking range with the visual layout of the console. As part of our VR set design, the team rapidly designed different geometric models in Maya and exported them to Unity (shown below).

With this approach, we were able to quickly experience and iterate different console styles. We ultimately chose a design that was inspired by helicopters and old-school DC3 cockpits. The consoles are curved to reflect how your arms naturally swing in a radius.


Different styles developed during the set design process.

Interaction and UX Design

This is the real heart of the VR Cockpit project, and one that we’re still exploring. While you can get a fuller understanding of how this works in our feature video, here’s a quick overview.

Motion controls are used for all of the secondary ship systems, such as navigation, while HOTAS controls are responsible for the ship’s primary controls, such as ship thrust and rotation. There are six main control systems inside the Cockpit:

  • Joystick: Control the direction of your spaceship using a HOTAS joystick or gamepad.
  • Throttle: Control the speed of your spaceship using a HOTAS throttle or gamepad shoulder buttons.
  • Power System: Reach up towards the Power Systems icon to make the controls appear for distributing power between your ship’s systems. You can use the sliders associated with each of the ship’s Power Systems to set the power allocated to various subsystems.
  • Navigation System: Reach to your left towards the Navigation system icon to activate the navigation system menu. Using this menu, you can select between various destinations for your ship’s hyperdrive to take you.
  • Hyperspace Jump Button: After you’ve selected a destination using the navigation system, the Hyperspace Jump Button will appear in the front and center in the cockpit. Activate the hyperspace button by using both hands to press down on it – and hang on!
  • Communication System: Reach to your right towards the Communication system icon to activate the Communication System menu. Using this menu, you can listen to messages from various communication channels.

Combining gaze and reach. The first thing you might notice is that the menus don’t appear all the time – otherwise they’d block your view of the endless void of space. By looking left or right and reaching out, you can instantly trigger the menu. The menus themselves were modelled in Maya before being exported to Unity and infused with Mecanim animations.

runwayAudio and visual feedback. Buzzes, whirring, and other audio elements are absolutely crucial to designing useful VR interfaces, especially in the absence of haptic response. These work alongside visual feedback elements, such as the “runway light” hover effects on either side of each button. These subtly compress as your hand approaches the button.

Powerful physical motions. There’s also a lot of fun territory in broad physical interactions. From swatting your helper robot to pushing into hyperspace with both hands, these have a lot more physicality than interacting with floating menus.

robot-smackdownNarrative learning. For first-time users, the interfaces and hyperspace sequence can be pretty complicated. Fortunately, the helper robot is more than just an easy target for abuse – he also talks! We created a flow system built on the new abstract state capabilities that Unity recently introduced in Mecanim. This means that instead of breaking the narrative using tutorial text or bullet points, the robot will give you tips based on what’s currently happening. (Though if you’ve ever played Ocarina of Time, you might relish the chance to wallop your helpful flying companion.)

VR Cockpit explores a number of interactions made possible by combined physical and motion control technologies, but there’s an enormous wealth of design space that remains unexplored. We’re looking forward to digging deeper into this space, and seeing how others can expand on different possible combinations.