augmented-hand-series

“It’s a box. You put your hand in it. You see your hand with an extra finger.”—Visitor, 7

The hand is a critical interface to the world, tied into mind and identity. Click To TweetThe “Augmented Hand Series” (by Golan Levin, Chris Sugrue, and Kyle McDonald) is a real-time interactive software system that presents playful, dreamlike, and uncanny transformations of its visitors’ hands. It consists of a box into which the visitor inserts their hand, and a screen which displays their ‘reimagined’ hand—for example, with an extra finger, or with fingers that move autonomously. Critically, the project’s transformations operate within the logical space of the hand itself, which is to say: the artwork performs “hand-aware” visualizations that alter the deep structure of how the hand appears. An earlier version of this post appeared on flong.com.

The Augmented Hand Series is a real-time interactive software system that presents playful, dreamlike, and uncanny transformations of its visitors’ hands. Originally conceived in 2004, the project was developed at the Frank-Ratchye STUDIO for Creative Inquiry in 2013-2014 through a commission from the Cinekid Festival of children’s media.

The installation consists of a box into which the visitor inserts their hand, and a touchscreen interface which displays their ‘reimagined’ hand, altered by various dynamic and structural transformations. In the version shown here, in its premiere at the 2014 Cinekid Festival in Amsterdam, the kiosk is accompanied by a large rear-projection. The touchscreen allows participants to select among the different transformations.

Critically, the project’s morphological transformations operate within the logical space of the hand itself. That is to say: the artwork performs “hand-aware” visualizations that alter the deep structure of how the hand appears—unlike, say, a funhouse mirror, which simply distorts the entire field of view.


Visitor interactions

The system uses the real-time posture of the participant’s real hand as the moment-to-moment baseline for its transformations. Participants are free to use either of their hands, and the system works properly even with visitors who wiggle their fingers, or who move and turn their hand—within certain limits. The software may produce unpredictable glitches if the visitor’s hand differs significantly from a flat palm-down or palm-up pose. Currently, the system’s behavior for postures like fists (in which many of the fingers are occluded) is undefined, as is multi-hand interaction.

Developing an interaction that can work with a very diverse public is always a challenge. The Augmented Hand Series accommodates a wide range of hand sizes, from children (of about 5 years old) through adults, as well as a very broad range of skin colors. The system also performs robustly with hands that have jewelry, nail polish, tattoos, birthmarks, wrinkles, arthritic swelling, and/or unusually long, short, wide or slender digits. Nevertheless, the system’s behavior for individuals with more or fewer than five fingers is presently undefined. There are many more kinds of hands to support, and doing so remains an area of active research for the project.


Visitor interactions at the Cinekid Festival (continued)

About twenty different transformations or scenes have been developed. Some of these perform structural edits to the hand’s archetypal form; others endow the hand with new dimensions of plasticity; and others imbue the hand with a kind of autonomy, whose resulting behavior is a dynamic negotiation between visitor and algorithm. The videos here present live demonstrations of ten of these scenes:

  • Plus One: The hand obtains an additional finger.
  • Minus One: The hand has one finger omitted.
  • Variable Finger Length: The fingers’ length changes over time.
  • Meandering Fingers: The fingers take on a life of their own.
  • Procrustes: All fingers are made the same length.
  • Lissajous: The palm is warped in a periodic way.
  • Breathing Palm: The palm inflates and deflates.
  • Vulcan Salute: The third and fourth fingers are cleaved.
  • Angular Exaggeration: Finger adduction and abduction angles are amplified.
  • Springers: Finger movements are exaggerated by bouncy simulated physics.
Minus One: The hand has one finger omitted.   Plus One: The hand obtains an additional finger.
Angular Exaggeration: Finger angles are amplified.   Vulcan Salute: The third and fourth fingers are cleaved.

Many more transformations are planned. The sketches below depict some of the current scenes, as well as some as-yet-unfinished scenes conceived during the project’s proposal phase.

sketches

sketches

About

The hand is a critical interface to the world, allowing the use of tools, the intimate sense of touch, and a vast range of communicative gestures. Yet we frequently take our hands for granted, thinking with them, or through them, but hardly ever about them. Our investigation takes a position of exploration and wonder. Can real-time alterations of the hand’s appearance bring about a new perception of the body as a plastic, variable, unstable medium? Can such an interaction instill feelings of defamiliarization, prompt a heightened awareness of our own bodies, or incite a reexamination of our physical identities? Can we provoke simple wonder about the fact that we have any control at all over such a complex structure as the hand?

Neo (Matrix), Legolas (LOTR), and Captain Willard (Apocalypse Now), staring at their hands.
Neo, Legolas, and Willard, in dissociative states, look to their hands for cues to ground their perception of reality.

We know that the interrelations of hand, mind and identity are far from simple. Persons with alien hand syndrome, for example, have hands which move independently of their conscious will, as if they belonged to another person. By contrast, amputees suffering from phantom limb syndrome continue to feel their missing hand as if it were still there; their discomfort is sometimes relieved with a mirror box, which uses the virtual image of their intact hand to trick the mind and retrain the brain. Within this framework, the Augmented Hand Series can be understood as an instrument for probing or muddling embodied cognition, using a ‘direct manipulation’ interface and the suspension of disbelief to further problematize the mind-body problem. We see evidence of our instrument’s powers in the actions of young visitors who, uncertain whether to believe their eyes, peek into the box to double-check what is (not) really happening to their hand.

hands_seuss_disney_gould_665x230

The Augmented Hand Series is positioned to prompt, like Dr. Seuss (“I wish I had eleven, too!“) an empathetic acceptance of difference, and a recognition that there are many ways to be. Or could have been. In his essay “Eight Little Piggies“, the eminent biologist Stephen Jay Gould considers why we are shaped the way we are, and concludes—based on fascinating fossil evidence from some of the first land animals, some 400 million years ago—that there is nothing special or inevitable, nor even necessarily evolutionarily optimal about having five fingers on each hand. As Walt Disney discovered nearly a century ago, five fingers may not even be ideal for expressive gestural communication. From this perspective, the Augmented Hand Series is a work of participatory, transhuman biology-fiction that allows for the first-person exploration of these concepts.

Enraptured visitors

A wish to elicit and address wonder—our own, and others’—has been the deep common bond between Kyle, Chris and myself as we developed this project. Ultimately, as collaborators, our underlying personal motivations for developing the project are each quite different, but Kyle’s, which arise from his experiences as a lucid dreamer, are perhaps the most poetic:


Kyle discusses some of his motivations as a project collaborator.

A longer discussion of some of the ideas motivating the Augmented Hands Project can be seen in Golan’s presentation at the 2013 Eyeo Festival, beginning at 22’18”.

Recent Progress

Since this post was originally published last year, the team has created several new scenes, including fractal hands (by student Zach Rispoli), swapped thumbs, double thumbs, fewer knuckles, and extra knuckles. Here are some of these more recent experiments, straight from the depths of the uncanny valley. –Ed.

swapped-thumb

Screen Shot 2015-02-03 at 11.58.53 AM

Screen Shot 2015-02-03 at 11.59.23 AM

extra-knuckles

Press

Artist Biographies

Golan Levin explores the intersection of abstract communication and interactivity. Blending equal measures of the whimsical, the provocative, and the sublime in a wide variety of media, Levin applies creative twists to digital technologies that highlight our relationship with machines, expand the vocabulary of human action, and awaken participants to their own potential as creative actors. At Carnegie Mellon University, he is Associate Professor of Electronic Art and serves as Director of the Frank-Ratchye STUDIO for Creative Inquiry, a laboratory dedicated to supporting atypical, anti-disciplinary and inter-institutional research projects across the arts, science, technology and culture.

Chris Sugrue is an artist and engineer who develops interactive installations, audio-visual performances and experimental interfaces. Her works experiment with technology in playful and curious ways and investigate topics such as artificial life, gestural performance and optical illusions. She has exhibited internationally in such festivals and galleries as Ars Electronica, Sónar Festival, Pixel Gallery, Medialab-Prado, Matadero Madrid, and La Noche En Blanco Madrid. She teaches new media arts at The Parsons School of Design in Paris.

Kyle McDonald works with sounds and codes, exploring translation, contextualization, and similarity. With a background in philosophy and computer science, he strives to integrate intricate processes and structures with accessible, playful realizations that often have a do-it-yourself, open-source aesthetic. He enjoys creatively subverting networked communication and computation, exploring glitch and embedded biases, and extending these concepts to reversal of everything from personal identity to work habits. Kyle is a member of F.A.T. Lab, community manager for openFrameworks, and an adjunct professor at the NYU ITP.

Credits & Acknowledgments

cc_by_nc_sa_88x31

logos of sponsoring & supporting organizations

The Augmented Hand Series was commissioned by the Cinekid Festival, Amsterdam, October 2014, with support from the Mondriaan Fund for visual art. It was developed at the Frank-Ratchye STUDIO for Creative Inquiry at Carnegie Mellon University with additional support from the Pennsylvania Council on the Arts and the Frank-Ratchye Fund for Art @ the Frontier.

The Augmented Hand Series could not have been possible without several open-source C++ addons generously contributed by others in the openFrameworks community: ofxPuppet by Zach Lieberman, based on Ryan Schmidt’s implementation of As-Rigid-As-Possible Shape Manipulation by Takeo Igarashi et al.; ofxLeapMotion by Theo Watson, with updates by Dan Wilcox; ofxCv, ofxLibdc, and ofxTiming by Kyle McDonald; ofxCvMin and ofxRay by Elliot Woods; and the ofxButterfly code for mesh subdivision, by Bryce Summers. Adam Carlucci‘s helpful tutorial on using the Accelerate Framework in openFrameworks was also essential to achieving satisfactory frame rates.

The artists wish to specially thank Paulien Dresscher, Theo Watson, and the Eyeo Festival for their encouragement; and to Dan Wilcox, Bryce Summers, Erica Lazrus, Zachary Rispoli, Elliot Woods, Simon Sarginson, and Caitlin R. Boyle for their assistance in realizing the project. We extend additional thanks to the entire openFrameworks community; the staff of the Frank-Ratchye STUDIO for Creative Inquiry; Golan Levin’s Electronic Media Studio students, who served as beta-testers; Tim Hoogesteger of Cinekid; Rick Barraza and Ben Lower of Microsoft; Christian Schaller and Hannes Hofmann of Metrilus GmbH; Dr. Roland Goecke of University of Canberra; and Doug Carmean and Chris Rojas of Intel.

Golan is an an artist and educator living in Pittsburgh. He teaches at Carnegie Mellon University, where he also directs the STUDIO for Creative Inquiry, an interdisciplinary arts-research center.

Facebook Twitter Flickr YouTube Vimeo 

Recommended Posts