This talk, entitled “Finding (and Making) your Happy Place” took place at SFHTML5 on January 23, 2014.

Coffee

I’m Isaac, and I’m an experience engineer at Leap Motion. I work exploring a newly discovered relationship with our hands. I’m intensely interested in communing with digital nature, and the tools that I use are Three.js, Web Audio API, and most of all, Leap Motion. Most of what I have done can be found at cabbibo.com where I have been working on constructing my own personal Ice Kingdom.

Unlike our previous speakers, I have no idea what I’m doing. When they started talking about state machines, I was totally lost, and my understanding of equivalence rigorously stops at two equals signs. Don’t worry though, there is a reason you’ve been asked to sit here and listen to me for the next thirty minutes. It’s because I’m addicted… to coffee.

I love coffee. Every day it’s the first thing I consume, and many days it’s the last. I don’t consume it because it’s the hardest drug I use. I don’t consume it because it makes me feel like a mental triathlete, and I don’t even consume it because it tastes like the delectable blood of an angel. No, I love coffee because I love milk. I love that milk is a different consistency than coffee, I love that it is differently colored than coffee, and I love that when I delicately combine the two, it creates the most beautiful pattern I have ever seen.

It’s actually to the point of obsession. I never get hot coffee. It makes the milk dissipate too quickly, and isn’t as likely to come in a clear glass. Regular iced coffee isn’t even enough. I need cold brew, condensed, New Orleans-style iced coffee. Only its viscosity will satiate me.

At the coffee shop that I go to, they make sure that they never put the milk in themselves and always give me the container to pour it drop by drop. People can’t tell if I’m super conscious about my milk consumption, or just straight-up weird. The latter is obviously the true statement, but it’s mostly just because I can’t get over the mystical nature of chaos.

It’s kinda funny how this turbulence brings me peace. It’s such a lonely joy after all, its haunting dance into nothingness is somehow melancholy in my mind. I think it could be because, to me, it’s a firm reminder of the unshakable arrow of time or vacuum of space, which are in many ways terrifying. But it’s just so damn beautiful that it can do nothing but calm me.

Maybe it’s because I can’t unmix the milk and coffee. Maybe it’s because when I mix the two the today, they will look completely different than they will tomorrow. That one magical tendril that I witnessed on the first drop might be replaced with a dendrite appearing only moments before the completion of the sepia solution. Maybe it’s because any other individual touching that glass will create a totally different pattern. They will never see the same turbulence that I do. Maybe it’s because each and every experience is only a single moment in a single branch of infinite possibilities, unique and miraculous as it is.

Or maybe it’s just the fact that after it’s all mixed, I get to take that first sweet sip and feel the delicious nectar of caffeine flowing through my drug-addled veins.

Nature

I hope it’s the turbulence though. This would be such a more interesting talk if it wasn’t predicated on a deep physical addiction.

Either way, that moment when I watch the milk mix, I find myself in a happy place. For that moment, rent is not important, that giant Keep Calm and Carry On poster that is now the centerpiece of our living room doesn’t bother me so much, and I forget how late I am to my nine o’clock standup. I find the turbulence in that cup beautiful, and that beauty makes me happy. Not in an urgent way, in an important way.

I want that peace all the time. I want to go to my happy place 10, 11, 12 times a day, and I’ve tried doing that with coffee before. It was a mistake.

Luckily as artists, as I believe each and every one of us are, we can create that happy place, and we can go to it any time we want. As magicians, every time a silly client tells us to use lime green to make the text ‘pop’ more, we can just load up a web page, and experience peace – experience turbulence.

Here’s the thing about turbulence though. To get the detail that you see in coffee you need infinite computing power. To get each precise flume and dexterous wrinkle, you need to get all the way to the atomic level. We can get close, but we can’t be exact.

Sure you have scientists doing brilliant work on subjects like Rayleigh–Bénard convection, but even in their brilliance, they fall short of exact replication. The butterfly effect, as we have all learned from the brilliant scientist Ashton Kutcher, is quite a powerful one indeed. Floating point error is painfully inaccurate when compared with natures uncertainty, Planck’s length.

In the end, if I believe that turbulence is the most beautiful art, nature is the most beautiful artist. It can not only create turbulence, but the snow-dappled mountains, the powerful rushing rivers, and every heart-wrenching sunset. Nature is awesome.

Nature is so awesome, in fact, that I have seriously considered giving up computers to become a gardener. They are the artists, in my mind, who have chosen the ‘truest’ medium. Anyone who has spent time in the bonsai garden over by Lake Merritt will certainly agree. The twists are more elegant than any Bézier curve could hope to convey, and the sheer detail of the turtled bark makes even the most processor-intensive physics simulation look like a crayon drawing.

Any one of us could spend our entire life trying to create that turtled bark and still fall dramatically short of our goal. Hell, all of us in this room together could combine our superpowers, attempt to conjure a tree, and nature’s shaders would triumph without competition.

Physical Art Scene

If we were physical artists on the other hand, we could use the physical nature of our work to introduce some of the beauty of chaos. Back before I understood that computers gave you superpowers, I used to just spray paint on vegetable oil, splatter it on a canvas, and call that texture Art. And it was, in many ways more than what I create now.

At an even higher level I tried to sample the texture of an elephant’s hide or toad’s skin. Sampling nature in this way was many times beautiful. It felt profound to watch the flow of the Voronoi cells of a dragonfly’s wings slowly unfurl into existence as I painstakingly etched them into metal. But even this was a step away from the original. Even this was a simulacra. The further away I got from milk mixing with the coffee, the more I gave me creative power, and nature less. By doing this, I had to exert my own will upon the art that I was creating, and that meant that nature was doing less work, and I was doing more.

Since I’m lazy, and since we’ve established that nature is a much better artist than I am, creating in this way is ridiculous. I was trying to say that the thing that I could posit and execute was better than trillions of years of artistic honing. Generations upon generations of evolutionary pruning. A masterpiece older than time itself.

The closest I could get is mimicry, or possibly borrowing. I could not understand how the paint mixed with the oil, but I could see that when I haphazardly threw them on the canvas, it created something that felt important. I could not make the elephant itself, but I could trace a picture of its skin, taking from nature what I could not create myself.

In this way, to be good artists, to be good creators, we did not need to be good makers. Rather, we need to be good thieves. The Voronoi cells, the elephant’s texture and the swirling paint mixture were after all the creation of nature, and for me to harness it in my work, and call it my own, was to steal it from the world itself. This may seem like a cardinal crime, but anything less is preposterous.

Digital

What we do is even another step away from nature. Typically when creating a program, the functions we create and the tasks we accomplish have nothing to do with the logarithmic structure of ferns or the Fibonacci arrangement of flower petals.

It’s all about paragraph tags and hrefs and target=“_blank.” We are the ones that place these tags on the page. We are the ones who create functions and methods, and objects that use objects, and as we build up this shoddy Tower of Babel, we tend forget how much more complex an ecosystem is, and how much less energy it would take to destroy our work than that of nature.

Even when we create recursive functions, and use math as the language of nature, adding one more layer of complexity can easily crash our computer, limiting our detail to quite childish standards. Let’s not even begin to talk about the fact that we are limited to two-dimensional space, and that no matter how many DPI your screen is, it is still crude compared to even the smallest grain of sand under the examination of our retinas. In this way our medium is quite pathetic, by any measure of detail.

Even if we are to sample nature as a background or texture, we are limited by pixels, memory, and rendering power. Instead of the rugged shape of an elephant’s ear, we have to make spheres and cubes and make sure that the subdivisions are only of a certain amount, otherwise our rendering time will drop below the already pathetic 60 frames per second.

Even Alterqualia’s and Mr.Doob’s bump mapped face has dramatically less detailed than any of the ones in this crowd. This is not to say it is any less amazing. It’s just to say whenever we do anything in our medium, We will fall short. When we try and mimic movement, when we try and replicate geometry, and especially when we try and create turbulence.

I hope this is disheartening. It means that we have a long way to go. Our struggle may not be comfortable, but if we can somehow emerge victorious, what a victory it will be. Our medium is young. But if we can help it grow, we can use it to create, it means we have accomplished something beautiful, and that accomplishment should make us happy.

Thievery

As well, there are certain things that are part of our medium that no other medium can claim to possess. There are certain parts of our medium that give us a fighting chance.

First off, in our medium, stealing in most cases, not destructive. Because replication comes for free, sharing and utilizing others’ code is widely accepted and expected. Coding is just socially acceptable thievery. This is not to say you should go out willy-nilly and jack IP, just that sharing is caring.

You can see communities arise from this sharing, and it makes everyone involved more mighty. The best example of this that I know is Three.js. Because of Mr.doob’s fervent dedication to creating a useful library, many other power players of the WebGL community have joined in making a tool that lets even someone like me, who has no idea what they are doing, create something.

It’s more than just that though. Others have let Mr.doob call their work his own (with of course the necessary credits in the source)! In this way, alteredqualia, bhouston, zz85, gero3, west langley, and many, many others have added to the community with selfless dedication. With each additional commit comes more complexity, more substance, and more code to steal.

It may be difficult to steal directly from nature, but in this way we can create complex enough networks of thievery, that the stealing itself becomes its own organic being. For the demo that I will show you, I have stolen so much work from so many people, that there is no way that I could said to have created it myself. I have only selected a mix of others’ works and put it together in a way that I have not seen before.

All work we do is not our own. It is that of our community. Even in the unlikely chance that you have built something using no libraries, the work that the V8 engine does in order to compile your code, and even more the work that Intel or NVIDIA has done to render your program, trumps any contribution that you could make to the overall success of your application.

Any time you think you have made something, you are wrong. You have channeled the organic power of communities, corporations, and people in order to manifest a creation of humanity.

I can’t explain exactly why, but the miracle of these different components coming together every time I refresh my browser makes me think of the mixing of milk and coffee that I love so much.

Suddenly my messy room, the deadlines coming up, and the fact that rent was due two days ago doesn’t matter so much, and I start drifting in mental space, marveling at the smallness of my own accomplishment as I consider how magnificent our species is. I forget the urgent, and remember the important.

Variables

It doesn’t stop at the channeling of humans though. Our medium has something else that is spectacular: variables.

Although I will never be able to make a face that is better than the one nature can make, there is something magical I can do. I can change the skin color, I can even the color of the lights that shine upon the skin. And I get to do this all with a single keystroke.

In this way, I can create an object that maybe has less detail, but has more “self-freedom” than that which is created by nature. Although this goes against letting nature do the work, there is something special about variables. They can be controlled by another manifestation of nature: data.

In my mind, data is our most fierce weapon in the creation of that which is organic. Because it is so expansive, it allows us to pass through one piece of nature to another. Although this manifestation usually tends to be a one-to-one correlation – say placing population data on a globe – there is nothing to say that we can’t recklessly link two uncorrelated phenomena. Imagine a face whose features are distorted by the number of endangered animals per geographic unit. (I’m sorry I didn’t take the time to make this, but I hope that you can concede it is at least possible.)

In this way, we can begin to create works which utilize data to create an interface between the natural world and ours. By figuring out ways to get “real-life” data we get to make “real-life” applications. “Real-life” Art.

There are so many sources of “real-life” data, but to me, two are the most important. What I hear, and what I see. I am not yet able to rig my eyes and ears up to a thunderbolt cable, but in the meantime, there are two ways to access these senses: the Web Audio API and Leap Motion.

The Web Audio API is great because any time I want to link audio frequencies to parts of an application, all I need to do is create a stream, create an analyzer, and call an analyze function on the update loop. Thats all it takes to get access to natural data.

The Leap Motion Controller is great for a similar reason. All I need to do is create a controller, connect it, and now I have access to something see, something as fluid and authentic as the movement of my own hands.

Because of this simple access to data, I get to start doing strange things with code. I get to start saying, “OK, make a face. But also deform it based on the sound of my voice. Oh, and also, make it so that when I see my hand turn, the colors change!”

Suddenly something that I stole comes to life by funneling data from one place to another. Although I tend to fetishize the aesthetic, this fluid data is more than just a visual nicety. I have seen interfaces that feel more playful and organic than any I have experienced pre-Leap Motion and pre-Web Audio API.

I have seen things created that were never before possible. I myself learned to 3D model and made a mug overnight. It’s because all I had to do was reach my hand into space, and use the senses that I love so much.

Because many minds more brilliant than mine did find a way to mine the data, I get to lovingly thieve their work through robust and simple APIs. They make the data easy to thieve, and as creators that is the kindest thing we could ask for.

Additionally, because of their nuanced connection with natural data, programming with Web Audio API and Leap Motion becomes a pleasure rather than a task. Although mimicking the binary click of a mouse may be less than ideal, new interfaces that deal with flow and movement and the full spectrum of human capability become possible.

Because I can make the frequency of my voice control the scroll rate, or use the position of my hand to control the camera movement, I can readily change one piece of data to another, and this flow, in my mind, is miraculous thievery.

It’s because in our medium there are no units. Creating a hundred-foot sculpture only means multiplying scale by 100. Linking gravitational potential to note frequency only costs a single line of code. If I want to change from XYZ to RGB, there’s absolutely nothing stopping me. And that is where WebGL steps in.

WebGL

For a meet-up on WebGL you might think I would mention the technology earlier than two-thirds of the way into the speech, but I hope that everything up to this point will still apply to your creation even if you never end up using webGL.

That being said, you should use WebGL. You should use it because 3D is ridiculously awesome. You should use it because you can make an entire presentation and it doesn’t take that long. But the biggest reason to me that you should use it is because shaders.

Now I only learned that shaders were the truth and the light about two months ago, so please take the following with a grain of salt. I am feverishly studying how they work and what they can do, but that being said, it’s only been two months. This should give you a lot of hope, though, because if I can learn anything about shaders in two months, it means that you can too!

In the end, WebGL shaders do two things: XYZ and RGB. XYZ is referring to a vertex shader which positions every single vertex in a geometry. RGB refers to a fragment shader, which colors every pixel in the object. Together the two can deform and color any shape until it looks nothing like the original.

We could do a full speech just about deforming and coloring a cube in interesting ways, and I’m sure that if you searched somewhere on the internet, you could find exactly that speech. But we’ve got to remember, what we’re here for is the coffee.

To get there, we need to do a few tricks. Because we can sample textures in vertex shaders we can easily exchange XYZ with RGB, basically placing particles based on the RGB value of a texture. I’m going to start with just a picture, where we take the RGB values of the picture and place them around based on the value of those three objects. So R becomes X, G becomes Y, and B becomes Z. Here we can see the pixels and colors turned to positions, and see although there are these deep reds, the yellows actually have value that contain R, G, and B. So it reveals something fundamental about this picture that we may not have known before.

We can take this a step further and sample a texture that we have created with a fragment shader. In this way, we can color each texture of a shader, as we’ve done in the middle, giving values from -1 to 1 for R, G , and B, and placing each particle as it corresponds to its pixel in three-dimensional space.

Then, by chaining vertex and fragment shaders together, we can create one fragment shader that talks about position, and one fragment shader that talks about velocity. Once we link all this through, we can do something spectacular: a physics simulation done entirely on the GPU.

What is so special about this is that all of the calculations happen in parallel. In this gravity demo, every single particle is enacting gravitational force on every single other particle. We’re talking about millions and millions of calculations every frame. We’re talking instead of hundreds of particles, you have thousands, even millions.

There are a few brilliant examples of doing GPU particles including Mr.doobs ‘sporel’ demo, zz85’s sparks.js, and Soulwire’s WebGL GPU Particles. The tasks that they are doing are all different, but if you closely examine them all you will learn dramatically more about GPU particles than I can hope to explain.

However, with their knowledge aptly stolen, I could move onto the holy grail.

Coffee.

Turbulence

To make my turbulence I used something called Curl Noise. It is not actually fluid dynamics, and I’ve been told by a few math PhDs that now that I can’t even call it mimicry. It is smoke and mirrors, but then again, we are magicians. Now the most beautiful example I had ever seen of curl noise was from the astute creator Eddie Lee in his demo Artifacts. Eddie is a straight up mastermind, and if you ever play Kyoto or Lotus, I’m sure that you will understand why I believe so. He elegantly navigates the issues I have discussed with digital creation, both by creating authentically organic shaders, or buying wholeheartedly into the fact that his creation cannot be natural, and making it playfully simple.

Additionally, he’s awesome, because he let me see how he implemented curl noise. He showed me how to mimic my turbulence. My happiness. I didn’t just steal code from him, he gave it to me, free of charge, and out of kindness.

After hectically slapping together his shader code into zz85’s GPU particle simulator, I refreshed my page, and nothing happened. I did it again, and again nothing appeared. After multiple more hours of the frustration that can only be known by someone who has tripped on a shoelace moments before finishing a marathon, I finally asked one of my other idols, Reza Ali, for help. Being yet another brilliant mind, and another gracious soul, Reza sent over his Curl Noise code, which made me realize that I had used an equals instead of a plus equals for the most vital part of the equation.

I told you I know nothing about equalities.

But with that bug fixed, I crossed my fingers, refreshed my browser, and:

There it was. My very first turbulent system. And by mine I mean Mr.doob, Alteredqualia, zz85, Eddie Lee, Reza Ali, and probably many of the people sitting in this room right now.

Sure it was no milk and coffee, and I now know that by using more optimized code I could use 10 times more particles, but it was a start, and that alone made me happy.

I really think that this could have been enough for me. If I had committed my changes, shut down my computer, and went to sleep, it would have been more than enough creation to gratify my gluttonous desire for the important.

But If I stopped, it would just be a frail imitation of nature. It would be dots algorithmically moving around the screen, and as happy as I was with those small white squares, it is after this point of initial implementation that our medium begins to shine.

This piece of code is in my engine forever. Now I get to play. Now I get to create my happy place. Now I finally get to explore in a way that was never possible in the physical world.

That sunset, though heartwrenching, only happens once a day. It will never be the beautiful turquoise and magnetas that I can so lucidly imagine it to be. Although I can watch the glowing ball from far away, I can never reach out and touch it.

Yes, we are stuck in simulation. We are perpetrators of imitation. But in that imitation, in that mimicry, comes a new truth. It is a “hyper-reality” we create. The sun not only can set in the east, but sing to us if we only will it. The stars are no longer a diamond curtain, but a playground.

We can explore and experience that which was not possible. Nature may be a brilliant artist, but if we choose to create with its infinite power, we have to take on the shackles of physicality.

Now, with WebGL, with access directly to the senses, we can begin to cast those shackles off. I am no longer limited to my flesh self, and I come to you as Cabbibo, the ice mandala of the northern kingdom, floating in the darkness.

Nature may want to keep me here on this earth, harnessed to my decrepit human form, but I refuse that requirement.

I will create a universe where I can where I can fly through the stars with the flick of my wrist. Watch eons unfold with each harmonious Ohm. I will reject common sense to make the impossible possible. I will create.