How to use the Leap Motion controller to manipulate an image in real time
1 Hey I’m Travis with team Vuo. In the following tutorials we’ll explore how Vuo and the Leap Motion controller can be used together as a powerful medium for creativity.
2 Let’s begin by defining what we’re working with. I’ll start with what I can hold — the Leap Motion Controller. It’s a physical device that easily fits in my hand. It plugs into a computer via usb port and functions as a controller.
3 To offer a comparison, a mouse is also an example of a controller. It plugs in via usb, and when I use my hand to move the mouse in a circular motion, that physical movement is translated into digital movement, as my cursor moves in circles on my screen. I can use the mouse buttons to give myself further choices about what I’m selecting. We can use a mouse navigate the web, to draw, we can play games, etc…
4 In similar fashion, with exponentially more options, the Leap Motion Controller gives you control in certain digital environments by detecting the physical motion of your hand when it’s positioned above the device, like so. (Demo Display-Hand). This is Vuo using input from the Leap Motion Controller to represent the center of my hand and my fingertips in Vuo’s 3 dimensional grid system. Crazy cool huh?
5 Two prominent differences between a mouse and the Leap Motion Controller are: first, a mouse is limited to flat 2 dimensional space while the Leap Motion Device offers control in 3 dimensional space; second, a mouse is limited to a small number of buttons on top for different inputs while the Leap Motion Controller can sense a variety of hand gestures for inputs.
6 The Leap Motion Controller needs an environment that can harness its capabilities. This is where Vuo comes in. Vuo is a brand new, visual environment for creating. It has a built-in 3 dimensional grid system to receive controlled input, and it allows users to assemble intuitive building blocks to create compositions. A composition is a broad term that could mean anything from a screensaver, to a program or an app, to a visual display that interacts with live music at a concert. At this moment, December 13th 2013, Vuo is in early beta, and not all of this functionality is released yet, but we’re refining the details now.
7 So let’s start learning how to utilize the Leap Motion controller within Vuo. Let’s take a glimpse at a simple example composition, then we’ll build it together, step by step. To open up an example composition related to Leap Motion, we’ll select “File”, “Open Example,” and scroll down to “vuo.leap” And I’ll choose Twirl Image.
8 When we open it up, our composition appears before us. This arrangement of building blocks is all it takes to produce Twirl Image. If you’ve never used Vuo before, I’ll do a very brief introduction, and if you find yourself lost, it may be helpful to watch some of our tutorials on the basics of Vuo, which you can find on our support page at vuo.org.
9 Here we go — this area with the white background is our canvas, these colored boxes upon it are Vuo’s building blocks, called nodes. Each node performs its own function or task. We assemble a composition by selecting nodes from the Node Library, here, and putting them on the canvas. Information flows from the left of the node to the right, and we disseminate information between nodes by linking them together with cables. We craft our inputs to produce what we want for our output, with our final output being our composition. Then we set our composition in motion by pressing “run” here. Let’s run this and see what it creates.
10 This composition presents us with an image, and allows us to use the Leap Motion Controller to manipulate the image with a twirl effect. It’s as if the image were displayed before me directly above the Leap Motion Controller. I can move my hand about the image, and where it engages the the plane of the image, a twirl effect takes place. We can also control the degree of the effect by how close we are to the plane, and how curved my hand is. The farther I pull my hand away from the image, the greater the angle of the effect, and the more I open my hand, the larger the radius of the effect. So here we see the Leap Motion controller utilizing Vuo’s 3 dimensional grid system to manipulate an image in real time.
11 Okay, let’s stop the composition. Now I’ll do a quick overview of how these nodes work together to produce the Twirl Image composition, then we’ll build it together. First off, you’ll notice that there are several different colors of nodes on the canvas. In Vuo, each node begins as grey, and you can assign colors to help organize your composition as you’d like. I’ve set this composition up so that the orange nodes are the ones that fire off events, they set things in motion. All the cyan nodes deal specifically with the Leap Motion device, all the green nodes deal specifically with the image and the twirl effect, and the two magenta nodes deal with assigning values within Vuo’s 3 dimensional grid system.
12 Let’s walk through what’s happening here. I’ll start with the section related to the image. When we click “Run,” Fire On Start fires off an event. When Get Image receives the event, it retrieves this specified image from my computer. The image is sent onward to Twirl Image, which has the capacity to create a twirl effect on the image, depending on how it is controlled. Twirl Image then sends the image onto Render Image to Window, which displays the image on a window.
13 Now if we take a closer look at the Twirl Image node, the inputs that create the twirl effect: the center of the twirl, the angle of the twirl, and the radius of the twirl, are 3 independent values, all of which we can simultaneously originate using the Leap Motion device.
14 So let’s look at our progression of Leap Motion nodes and see how we do that. Our first Leap Motion node is called Receive Leap Frame. It receives all the information that’s put out by the Leap motion device, which is actually a few hundred frames of data per second. An average monitor like mine can only render 60 frames per second, so Fire on Display Refresh and Hold Value work together to accept a new frame only when the window is ready to begin rendering a new one, which makes our composition significantly more efficient. (We’ll go over this in more detail soon.)
15 Next, that information is sent down to Get Frame Values which breaks up the available data into three different categories, from which we choose the data that has specifically to do with hands. Next that information goes to Get Item From List and we choose the first item, which, as explained in our tool tip equals the first hand that is sensed by the Leap Motion Device.
16 Next we progress down to Get Hand values, which gives us several different forms of output from our hand. We’ll go through some of these in more detail, but quickly, here’s what’s going on: the location of the palm along the z axis is being identified, then that value is being scaled into a value between negative 180 and positive 180. Then the new value is used to control the angle of the twirl. The Leap Motion controller also takes a look at the curvature from our palm to our fingers and constructs an imaginary sphere from that shape. We’re using the radius of that sphere to control the radius of our twirl, and the location of the center of the sphere is used to pinpoint the center our twirl. And that’s our composition.