How to use the Leap Motion controller to manipulate an image in real time
Transcript
1 Now let’s delete the Twirl Image, Leap Motion example composition and build it together, step by step. To start off, I’m going to work on inserting our image into the composition. To do this, we need to generate an event in order to set the process in motion, so I’m going to go to the node library and type in “start.” And we’ll drag Fire on Start onto the canvas. Now, when we start the composition, Fire on Start will send an event outward from its “started” port, which will tell the other nodes that it’s linked to, that it’s time to perform their function.
2 Next we’ll retrieve an image. I’ll go to the library and type in image, now all the nodes immediately relative to dealing with images appear in the node library. The first thing we need to do is get a specific image into our composition. In the future, we’ll be able to just drag and drop an image on the canvas to create the proper node, but Vuo is still in early beta and we need to create it ourselves. I’ll scroll down and here we see Get Image. I’ll click on it to read our tooltip, which says: “Loads or downloads an image from a URL.” Great, I’ll drag an drop Get Image onto the canvas.
3 Now I’m going to link a cable from the “started” port of Fire on start, which fires our event, and I’m going to link it to the the refresh port of Get Image. When Get Image receives the event, it will track down an image from a URL that I paste in here, and output the image, plus the original event from Fire on Start, here. So let’s get a URL for an image. Vuo utilizes relative file paths, so when my composition is saved to a folder, in this case the desktop, I can reference other files in that folder by their file names. So all we need to do is copy the file name here, then take it and past it into our image URL port. It’s important to note that we’re only sending one event through the Get Image node, because that’s all it takes. Upon starting the composition, Fire on Start will send one event to Get Image, and it will retrieve the image and hold on to it as long as the composition is running. If we were to send a stream of events through Get Image, it would retrieve the same image over and over again, and make our composition significantly less efficient.
4 Now that we have our image on the canvas, the next thing we want to do is run it through a twirl effect. If we go back to our node library, we see the node called Twirl Image. We’ll drag and drop that onto the canvas here. Then we’ll link our outputted image from Get Image to our Image input port of Twirl Image. Twirl Image receives an image here, then it receives values into its input ports which control the location of the center of the twirl as well as the angle and the radius of the twirl . We’ll come back to these later, for now, let’s put our current image on a window. I’ll go back to the node library and we see Render Image to Window. Excellent. We’ll drag and drop that here, and we’ll link our outputted “image” from Twirl image to our “image” input port on Render Image to Window.
5 Now that we have our image flowing to a window, let’ go ahead and try our composition out as-is and see if it works. I’ll click run, and here’s our window. This is an image I took on the Oregon Coast couple weeks ago, and the strong lines work well to demonstrate the twirl effect. Let’s take a look at what’s happening with the twirl. The center of the twirl is occurring at (0,0) which refers to the x and y axis in Vuo’s coordinate system. Vuo’s coordinate system is set up so that, no matter how the window is shaped, that for the x axis, the center equals 0, the right edge equals the value of 1 and the left edge equals the value of negative 1.
6 The Y axis is setup similarly, with 0 being the center, except it automatically adjusts the Y values in order to maintain proportion with the x axis. So the center of our twirl, is the absolute center of the window. Next we have the angle. The twirl effect is capable of implementing a full twirl of 360 degrees, and this one is currently set to 135. Lastly, we have the radius. Using the same distance of measurement as dictated by the x axis, with the distance between the origin and one side being one total unit, the radius of our twirl is currently set to a value of .5 total units. Excellent.
7 Now let’s work on having the Leap Motion device control the inputs of Twirl Image. The first node I’m going to put on the canvas is Fire On Display Refresh. I’m choosing this node because we’ll be manipulating our image in real time, and, as the tooltip explains, this node will fire an event the moment the window is ready to begin to receive a new frame, which will be the optimal flow rate for our composition.
8 Next I’m going to choose Receive Leap Frame. This node is responsible for receiving each frame of data from the Leap Motion device. As we said earlier, it puts out about a few hundred frames per second, and my hardware can only handle about 60 frames per second, so the next node I’ll put on the canvas is Hold Value. You’ll notice there’s multiple options for the Hold Value node, this is also because Vuo is in early beta, and they’ll soon be consolidated into one node, until then, I’ll chose the one that says “LeapFrame” at the end.
9 Hold Value can be used in different ways, and in this context it will work perfectly to regulate our flow of events. I’ll utilize the output of events from Fire on Display Refresh and connect it to the refresh port of Hold Value. Then I’ll take the output of frames from Receive Leap Frame and connect it to the “newValue” port of Hold Value. Now, the information that flows out of Receive Leap Frame will hit the “newValue” port and stop and be held. These curved bars on the back sides of the input ports symbolize event walls, which stop the flow of events and information.
10 When Hold Value receives an event in its refresh port here, it takes a look at it’s 2 input ports. If an event has never entered “newValue,” it will pass along the information present at “initialValue,” which in our case would be nothing, but if an event has ever entered the “newValue” port, it will always pass the most recently inputted information found there through its “heldValue” output port. So in our case, every time a new frame is ready to be rendered, Hold Value will output the most recent frame of data from Receive Leap Frame. Cool.
11 Now that we’ve regulated our flow of events, let’s start refining the information coming out of the Leap Motion device. The next node I’ll choose is Get Frame Values. Get Frame Values separates the information from the Leap Motion device into a few different outputs. The first one is a unique identification number that comes with each frame. The second is information pertaining to a hand, and the last is pointables, which refers to information about specific fingers or tools. For this composition, we want the information that deals with hands. The Leap Motion device has the capacity to track multiple hands, and we just want to track one hand, so the next node we’ll put on the canvas is Get Item from List.
12 Get Item from List is a general node that allows us to pick what we want from a list. In this context, if we set our “which” input to the value of one, it will choose the first hand that is detected, and ignore any other hands that might make an appearance. Now that we’ve selected our hand, the next node I’ll put on the canvas is Get Hand Values. We’ll connect our chosen hand to the “hand” input port.
13 Get Hand Values converts the information about our hand into a series of different values that we can use for creative input. If we look over at Twirl Image, we can see that some of our values can immediately be used as controlling input for the twirl effect. We discussed earlier how the Leap Motion device also produces data pertaining to a sphere that it creates by observing the curvature of the palm through the fingers. The radius of that sphere will work nicely for the radius of our Twirl effect, so I’ll link those together. And the center of that sphere will work perfectly for controlling the center point of our twirl .
14 You’ll notice that when I connected to the “center point” a type converter automatically popped up to let us know that we sent in values for the x, y, and z axis, but since we’re working with the center of a 2 dimensional image, the value for the z axis is being discarded. So let’s go ahead and use the value of the z axis to control the angle of the twirl. We don’t have a value coming out that isolates the z axis, so the next node I’m going to put on the canvas is Get 3D Point Values.
15 I’ll take a centrally located point, the “palm position,” and link it to the input for Get 3D Point Values. Now we can isolate the z axis of the palm position. But remember, in Vuo’s 3D grid system, we’re mainly working between the values of negative one and positive one, and with our angle input, we’re working with values between 0 and 360. So we need to scale our numbers to offer a useful level of input.
16 So the last node we’ll put on the canvas is Scale. This is a cool node that works perfectly for a scenario like this one. The way it works is we input our value here, then we establish the range of numbers we’ll accept for our input, for us it will be between negative one and positive one. Then we set a range for the numbers want to output. For today, let’s set the range of the angle of our twirl between zero and one-eighty. Great. Now, the value of the z axis, will output a value be zero and 180, and we can link that scaled value to our angle input port.
17 And there we have it! We walked through how to get an image and run it through an effect, and we walked through how to take input from the Leap Motion device, and refine it to get specific values to be used to creatively control our effect. Let’s run this an make sure it works. All right. We’re creating. Notice how all of our inputs are working simultaneously to control our Twirl. If you’d like to learn more about Vuo development or our plans for Vuo, you can check out vuo.org. And we hope that you’ll consider becoming part of the Vuo community, and supporting Vuo by purchasing a subscription. Thanks so much for checking out Vuo, and we look forward to seeing what you create.
