At Leap Motion, we’re always looking to advance our interactions in ways that push our hardware and software. As one of the lead engineers on Project North Star, I believe that augmented reality can be a truly compelling platform for human-computer interaction. While AR’s true potential comes from dissolving the barriers between humans and computers, I also believe that it can help improve our abilities in the real world. As we augment our reality, we augment ourselves.
With its best-in-class field-of-view, refresh rate, and resolution, the North Star headset has proven to be an exceptional platform for representing high-speed motions with small objects. So when David asked me to put together a quick demo to show off its ability to interact with spatial environments, I knew just what to build. That’s right – table tennis.
As we augment our reality, we augment ourselves. Click To TweetWith this demo, we have the magic of Leap Motion hand tracking combined with a handheld paddle controller. The virtual ball soars through the air and bounces on the real table. And, of course, an AI opponent to challenge you.
While augmented reality table tennis is a lot of fun, it also demonstrates a key concept that’s largely unexplored in mixed reality right now – artificial skills training for real-world scenarios. In VR, we can shape the experience to optimize learning a task or behavior. AR elevates this potential with familiar real-world environments, allowing us to contextualize learned skills. By overlaying virtual indicators and heuristics onto the user’s view, we can even help them develop a deeper intuition of the system.
What if you could see the future? Using parabolic equations of motion, our table tennis demo easily predicts where the ball will go. By showing this prediction in the form of a trajectory, we now have a superpower – without changing how the game itself behaves! By keeping the physics simulation authentic, the hand-eye coordination and muscle memory built up while training in AR can transfer directly to the real world.
Our AI opponent takes our training to the next level. Under the hood, the AI paddle uses the same action/reaction logic to calculate the reflected trajectory of the ball as your paddle. In other words, its motions must be physically correct to play the game. To this end, we implemented a kinematic formulation of the Cubic Bezier Curve to naturally drive the AI’s paddle to the correct position and velocity at the correct time.
A visualization of the Bezier Curves driving the opponent paddle.
What if you could see the future? #AR plus #AI can give you that ability – to some extent. Click To TweetThis special type of motion path allows us to specify that the paddle should hit the ball just after it’s reached the apex of its parabolic trajectory. A high level of physical accuracy is important, because it provides an example (or role model) for the user to learn from when starting to play the game. We didn’t want our users to become discouraged or frustrated if their opponent were to teleport to the ball or use some other unfair trickery to score a point.
The realism and physical reproducibility of this demo were built with the intent that the user should grow in their understanding of the system by interacting with it. As a medium, AR has the potential to improve how we learn about and interact with the real world. Simulations like this have the unique ability to adjust their difficulty downward to accommodate novices and upward to challenge experts in a whole new way – appealing to players at all skill levels.
Eventually, as AR systems become more advanced and lifelike, we will be able to practice against “impossibly difficult” artificial opponents and use that intuition in the real world like never before. Current and near-future professions may be aided by advanced AR training systems that allow us to casually achieve levels of skill that previously required months of determined practice.
With the advent of centaur chess and other collaborations between humans and AI, we’re just barely scratching the surface of what might be possible. We now have access to abilities that our ancestors could have scarcely imagined. Though the age of swords and their utility has long since been relegated to the distant past, it is my belief that the greatest swordsman of all time has not yet been born.