Virtual reality. Augmented reality. Mixed, hyper, modulated, mediated, diminished reality. All of these flavours are really just entry points into a vast world of possibilities where we can navigate between our physical world and limitless virtual spaces. These technologies contain immense raw potential, and have advanced to the stage where they are pretty good, and pretty accessible. But in terms of what they enable, we’ve only seen a sliver of what’s possible.
Virtual, augmented, mixed, hyper, modulated, mediated, diminished reality. These are just entry points into a vast world of possibilities where we navigate between our physical world and limitless virtual spaces. Click To TweetThe cultural expectations for the technology, popularised by Hollywood, present our future as cities filled with holographic signage and characters, escapist VR sex-pods, or specially equipped ‘holodeck’ rooms that are used for entertainment or simulation. We can have relationships with virtual beings, or might give ourselves over completely to virtuality and upload our souls to the network.
Our actual future will be much stranger, subtler and more complex. These technologies will have a more profound impact on the way we interact with our environments and each other. We will spend our days effortlessly slipping between realities, connecting with others who may be physically or virtually present, dialling up and down our level of immersion. Creation will be fast, collaborative and inexpensive. Where it used to take years of hard labour and valuable resources to build a cathedral, we will be able to build and share environments in moments, giving birth to impossible new forms of architecture. We will warp time and space, bending or breaking the rules of physical reality to suit our needs. All of this will be normal and obvious to us.
VR isn't about features, it's about the kinds of experiences those features enable. Click To TweetThe current generation of VR is great for transporting you to far-off lands, trading up your physical environment for separate virtual worlds. Some let you exist in other times and places. Some put you in a blank canvas and encourage you to create. The latest VR devices to be announced have a host of new features: high resolution screens, untethered capabilities, inside-out tracking. Headsets are getting more comfortable, costs are coming down. These make VR more accessible and portable, which will undoubtedly help reach larger audiences. But VR isn’t about features, it’s about the kinds of experiences those features enable. We’re on a course toward a new set of possibilities that are tantalisingly close, and will open up huge new areas of VR for exploration – a whole new category of experience. It’s a space we’ve been exploring, that we call…
M I R R O R W O R L D S
Mirrorworlds are alternative dimensions of reality, layered over the physical world.* Rather than completely removing you from your environment, they are parallel to reality, transforming your surroundings into refracted versions of themselves. Think Frodo when he puts on the One Ring, or the Upside Down in Stranger Things. These realities are linked to our own, and have strange new properties. Limited in some ways. Superpowered in others.
Physical obstacles like cabinets and chairs become mountains that can be moved to affect a weather system. Mirrorworlds transform aspects of the physical world into a new experience, rather than augmenting or replacing it.
Mirrorworlds immerse you without removing you from the space. You are still present, but on a different plane of reality. You will be able to see and engage with other people in your environment, walk around, sit down on a chair. But you can also shoot fireballs, summon complex 3D models, or tear down your walls to look out on a Martian sunrise. Mirrorworlds re-contextualise your space. They change its meaning and purpose, integrating with our daily lives while radically increasing the possibilities for a space.
Social Context
From command-lines to mobile interfaces, tech companies have made huge advances in making complex technology accessible to the masses. However, our relationship with technology is still largely based around an interaction between a human and a computer. When a person looks down at their smartphone, they are immediately disconnected from their social context – a phenomenon widely complained about by parents, friends and romantic partners around the world.
VR is perhaps the ultimate example of technological isolation, where our link with the physical environment is almost totally severed. This is fine if you’re alone in your bedroom, but can be a big limiting factor for its adoption in almost any other situation. People feel embarrassed, insecure or just unwilling to put themselves in such a profoundly vulnerable situation.
Our relationship with technology is still largely based around an interaction between a human and a computer. Mirrorworlds form new connections with the world around you. Click To TweetMirrorworlds don’t break social convention in the same way that conventional VR (or even mobile) does. Rather than cutting you off from the world, they form a new connection to it. At a basic level, we may just be aware of other people’s presence by making out their shape. In time, devices will be able to recognise these shapes as people, and replace them with avatars. In both cases, your social context is preserved. You will be able to stay engaged with the people and environment around you. In fact, we could say that Mirrorworlds move us away from human-computer interfaces, and towards human-environment interfaces, with technology as a mediating filter on our perception.
Truly Mobile
This awareness also allows VR to be not just portable, but truly mobile. Currently, both tethered and untethered devices still require you to stay in a clear, relatively small area. Even then people often move around tentatively, worried about stubbing their toes, walking into a wall, or stepping on a cat. In Mirrorworlds, we will be able to walk out of the door, down the stairs, get on the train, all in VR.
This requires a big shift in thinking about how virtual environments are designed. In today’s VR, developers and designers build 3D models of rooms, landscapes and dungeons, and drop us into them. We then have to find ways to move around them. This is fine with small environments, but to be able to move around larger spaces, we either have to climb into a virtual vehicle, or invent new ways like flying or teleporting. VR is intuitive and compelling because it matches our physical movement to the virtual world; asking users to learn an additional set of controls just to be able to move around around could be confusing and alienating to many.
These kinds of environments give the developer a lot of control, but they can also feel isolated and self-contained. Mirrorworld environments are not predefined 3D models, they are procedural. They incorporate elements from your physical environment and transform them. Developers building Mirrorworlds will think in a different way. Turn the floor to water. Remove the ceiling. Change furniture into mountains. Make them snow capped if over 6ft tall. Apply these rules, and the whole world is reinvented.
As well as transforming our environments, Mirrorworlds can also transform the physical objects within it. We can pick up a pencil, and use it as a magic wand. We can turn our tables into touchscreens. We can access virtual control panels for our connected IoT devices. We will obviously want to use our hands, but we will also use our bodies, our voices. In some cases we might want specialist controllers.
Over time, more and more of the physical world becomes available to us. But unlike AR, the creators of Mirrorworlds can choose how much they bring into their experiences. Mirrorworlds aren’t additive, they’re transformative. They will be able to selectively draw from the physical world – to simplify, focus, or completely restructure reality. It will be up to developers and users to decide how deep they want to go.
A Design Framework for New Realities
The emergence of Mirrorworlds will give rise to new types of spatial and social relationships. We will need to figure out how to present spaces that can be shared by physically and virtually present people, invisible audiences, and virtual assistants. We will collide worlds, meshing together boardrooms that are separated by thousands of miles into a continuous space. We will need to establish a design language to understand who is visible to who, which objects are shared, and which are private.
A consultant uses overlaid scan data to advise a surgeon in a remote operating theatre. The virtual and physical scenes intersect, and physical tools and objects can be used in the virtual world.
Unlike AR, the creators of Mirrorworlds can choose how much they bring into their experiences. Mirrorworlds aren’t additive, they’re transformative. Click To TweetWe may also need to consider what an application is. Should we continue combining virtual tools, environments, and objects together into isolated worlds? Or should we allow users to bring tools and objects with them between worlds? Should we combine tools made by different developers?
We are being faced with what feels like limitless possibility, but over time rules and standards will emerge. A shared set of principles that start to feel intuitive, maybe even inevitable. Some of these rules might be migrated from mobile/desktop. Some might be drawn from the physical world. And some might be entirely new, native to the medium of immersive media. Conversely, these new limitations will allow more to happen. The structure of rules and conventions will act like a scaffold, allowing us to reach further in our colonisation of virtual space.
But we must be careful that our structure is built on the right principles. As with pioneering any new territory, the opportunities for exploitation are rife, and there are many interested parties with different priorities. Should we be locked into a single ecosystem? Do we have to sacrifice privacy for convenience? Can we turn consumers of these experiences into producers? How can we elevate people without compromising them?
AR and VR are often presented to the public as separate, even competing technologies. Ultimately though, devices will be able to span the entire continuum, from AR to VR and all of the rich shades of reality in between. In this future, we will be constantly moving between worlds, shifting between perspectives, changing the rules of reality to suit our purposes. We will be able to fluidly and intuitively navigate, build and modify our environments, creating spaces where physically present people and objects intersect seamlessly with their virtual counterparts. We will look back on the current era and try to remember what it was like being trapped in one place, in one body, obsessed with devices and squinting at our tiny screens.
AR and VR are often presented to the public as separate, even competing technologies. Ultimately though, devices will be able to span the entire continuum. Click To TweetThis future is closer than you might think. It’s largely possible on today’s hardware, and now the limitations are less about technical constraints, and more in our ability to conceptualise, structure and prioritise the aspects of the world we want to build. That’s the brief we’ve been working on at Leap Motion Design Research. As we continue to build this framework, we’ll be exploring all facets of virtuality, from its materials to its grammar and spatial logic. We are working to carve out a robust, believable and honest vision of a world elevated by technology, with people (and their hands) at the centre.
* David Gelernter explored the connection between the virtual and physical world in his seminal book from 1993, also called ‘Mirror Worlds’. Our use of the term is distinct from Gelernter’s, but shares some interesting parallels.
