Update: This example has been deprecated, as the Leap Motion Interaction Engine makes these types of interactions much easier to design! Learn more in our blog post Introducing the Leap Motion Interaction Engine.

The opposable thumb gives human beings a hand up in grabbing and holding objects. At Leap Motion, we want to achieve the same grabbing ability in the virtual world, and with our V2 tracking beta we have the ability to detect these more complex interactions. We’re now working on standard ways to grab and release virtual objects; and to demonstrate this, we’ve developed a simple ragdoll game in the Unity game engine where you can grab and throw ragdolls around a room. The whole thing is available through our examples gallery.

Grabbing small objects – the stabilizing thumb

There are many ways to grab onto a small physical object, but the most natural ways use the thumb. We can pinch an object between the tip of our thumb and one of our fingertips, or we can hold an object between the thumb and palm. In almost all cases when grabbing small objects, the thumb tip is close to a finger’s joints or base. With this in mind, we’ve developed a simple method that compares the thumb tip to the joints on the other digits. If the thumb becomes close enough to any of the other joints, we start grabbing.

void UpdatePinch(Frame frame) {
    bool trigger_pinch = false;
    Hand hand = frame.Hands[handIndex];

    // Thumb tip is the pinch position.
    Vector3 thumb_tip = hand.Fingers[0].TipPosition.ToUnityScaled();

    // Check thumb tip distance to joints on all other fingers.
    // If it's close enough, start pinching.
    for (int i = 1; i < NUM_FINGERS && !trigger_pinch; ++i) {
      Finger finger = hand.Fingers[i];

      for (int j = 0; j < NUM_JOINTS && !trigger_pinch; ++j) {
        Vector3 joint_position = finger.JointPosition((Finger.FingerJoint)(j)).ToUnityScaled();
        Vector3 distance = thumb_tip - joint_position;
        if (distance.magnitude < THUMB_TRIGGER_DISTANCE)
          trigger_pinch = true;

    // Only change state if it's different.
    if (trigger_pinch && !pinching_)

Grabbing – so magnetic!

When interacting with a virtual world, we can do magical things. When a hand makes a grabbing motion, we can assume the user’s intention and turn it into reality. In Ragdoll Thrower, when a grab is detected, we find the closest rigid body (e.g. ragdoll joint). If that rigid body is within a specified distance, we assume the hand was grabbing for that object, and then we pair that object with the grabbing hand. This means the user can grab the ragdoll effortlessly. You only need to reach in a general direction and grab, and the closest ragdoll joint will zoom into your hand.

  void OnPinch(Vector3 pinch_position) {
    // ...
    // Check if we pinched a movable object and grab the closest one.
    Collider[] close_things = Physics.OverlapSphere(pinch_position, PINCH_DISTANCE, layer_mask_);
    Vector3 distance = new Vector3(PINCH_DISTANCE, 0.0f, 0.0f);
    for (int j = 0; j < close_things.Length; ++j) {
      Vector3 new_distance = pinch_position - close_things[j].transform.position;
      if (close_things[j].rigidbody != null && new_distance.magnitude < distance.magnitude) {
        grabbed_object_ = close_things[j];
        distance = new_distance;

Holding on – rubber bands

In Ragdoll Thrower, we accelerate the selected joint toward the grab position, so that you can move the ragdoll around. The effect is similar to attaching the joint to your hand with a rubber band or a spring. This is great for flinging or twirling a ragdoll around a room, and even (with a little practice) sling-shotting them. In other applications you may want to try a different type of grabbing physics.

  void Update() {
    // ...
    // Accelerate what we are grabbing toward the pinch.
    if (grabbed_ != null) {
      Vector3 distance = pinch_position - grabbed_.transform.position;
      grabbed_object_.rigidbody.AddForce(SPRING_CONSTANT * distance);

Releasing and throwing – keep it going

If you decide to release a grabbed object, you simply separate your thumb’s tip from the rest of your hand. When it exceeds the specified distance threshold, we release the object. Since we’ve accelerated the ragdoll and it now has its own velocity, we simply stop interacting with it and allow Unity physics to take over the now-projectile ragdoll.

Gamification – om nom nom

We also added a simple goal to Ragdoll Thrower: get the ragdolls through the robot’s mouth. When you succeed:


When designing the scene, it’s also important to choose borders that match the range of the Leap Motion Controller. The walls in Ragdoll Thrower are within the range on the left and right sides, and the robot mouth in the back is just within range. This was deliberate – we wanted to make it easier to fling the ragdolls through than to place them through, so people would get comfortable with a throwing motion.

Future development – threshold issues

There are a few places where this simple grabbing method could use some work. When someone tries to grab an object, but doesn’t first fully release past the threshold, a new grab doesn’t occur. Having multiple grabbing thresholds could solve this problem but it’s a little more complicated and would take some tweaking to get the most natural behavior.

Final thoughts

Simple interactions like grabbing and throwing are game dynamics that you could build an entire game around. As a fan of Black and White, I’d love to see a deity game where you interact positively or negatively with your worshippers. You could also create a slingshot game where you pull back and release objects at the end of a rubber band. What sorts of games can you imagine being created with these two building blocks? What would you like to see built on top of this demo? Let me know in the comments.

Epilogue: Matt on Twitch!

In March 2015, Matt joined us on our Twitch channel to dive into building intelligent 3D interaction schemes, translating sound into physics simulations, and how to “use the whole buffalo” in user interface design:

For cutting-edge projects and demos, tune in every Tuesday at 5pm PT to twitch.tv/leapmotiondeveloper. To make sure you never miss an episode, enter your email address to subscribe to up-to-the-minute Twitch updates:


Everything Matt does is waves. He has a background in music software, and in his free time develops interactive art using physics simulations and audio. If he’s not writing software, Matt is biking around San Francisco while eating a slice of pizza. Check out more of his work at tytel.org.

LinkedIn Vimeo