Making music visuals — Part 2

Building a video mixer


Here’s the setup that Steve and I often use when doing visuals for other people’s music performances. Since the setup I described earlier, with MIDI messages controlling visuals, requires quite a bit of advance planning and coordinating with the musicians, what we end up doing more often is to program the visuals to react to the Behringer controls.

What we do is, as we listen to the music, we interpret what we’re hearing into visuals by moving the Behringer controls — the faders and knobs and stuff. We listen really carefully to what the musicians are doing, and try to imitate them. It’s kind of a call and response, improvising visuals to follow and sometimes even anticipate the musicians, who may be improvising themselves.

A thing about music is that it gives a sense of motion. When I listen to music in the car, a lot of times I feel like the scenery going by kind of goes with the music. You know what I mean? Like, it feels right to our brains to see motion while we hear music. So when making music visuals, I often like to start by bringing in some motion.

One of the easiest ways to bring in motion is by playing videos, as shown on the dome. But how do you play videos improvisationally, where you can react live to changes in the music? We use a setup called a video mixer, which fades between different video clips and adds effects to them.

The first thing you’ll need for a video mixer is some video clips, around 15 seconds to a few minutes each. Since we’re projecting on a dome, we have to be aware that we can’t just pick out any old video clip because it’s gonna be warped on the dome. Unless you actually want it warped, you should either pick videos that are specifically made for the dome using a fisheye lens, or you should pick videos that are abstract enough that the warping doesn’t really matter. We’ll take the latter approach, with abstract videos. Here’s one I made of some shadows of a sculpture.

Get ready, because we’re gonna build a composition. We’ll start out the video mixer by just having Vuo play back this one video that I just showed. (By the way, since I don’t have the hand-mouth coordination to talk and build compositions at the same time, these are recorded screen captures. Some of them are a bit choppy because my computer doesn’t like to do graphics while I’m taking a screen capture, so I apologize for that, but we’ll demo the composition again on the dome after I walk you through it.)

We just built this composition that consists of a Play Movie node and a Render Image to Window node. How does this actually work? If you were to play a movie from a filmstrip, what you’d actually be seeing is a series of images shown rapidly in sequence, which gives the illusion of continuous motion — that’s how movies work. When you play a video on a computer, it’s pretty much the same idea. In Vuo, the Play Movie node sends a rapid series of images to the Render Image to Window node, which displays them in a window. You can see that here as I’m pausing and unpausing the stream of images.

Vuo plays back the video as a rectangle, but we need it to go onto a round dome. We don’t want the corners of the video to come down and shine right in people’s eyes — that would be bad. So we’ll use the Vignette Image node to round out the image. I’m adding the node here and connecting it up so it’s in the middle of that stream of images. Now we’ve got our vignette, we’ll just make it bigger and a little sharper. We’re vignetted now and we’ll fit on the dome.

As you may have noticed, to start the movie playing, we had to command-click on the Play port. Watch this: that is a command click and then it starts playing. The Play Movie node waits until it gets its cue to start playing; it doesn’t start right away. We can make it play automatically when the Run button is pressed — like that — by adding a Fire on Start node connected to the Play input.

Now we’ve got a video player; let’s turn it into a video mixer. We’re gonna add a second movie, and we’re gonna turn the movie images into layers that can be layered on top of each other, so we can control their opacity — which is like transparency — and blend them together. I’m turning the movies into layers, and then we have to turn the layers into an image, and since we’re on the dome we vignette it. Let’s see how that looks. Oops, it’s a rectangle, so we need to make it a square.

We can control the mixing with the mouse like this, or it would be more ergonomic if we could control it with the Behringer faders. Remember, the Behringer is that control box with the different faders or sliders on it. To demonstrate it in here I have a virtual version of it that runs in software. We’re setting up fader #1 to control the opacity of the first movie and fader #2 to control the opacity of the second. So there’s the virtual BCF and we’re turning up the opacity of the first one and the opacity of the second one.

Now we can fade between the two videos, but that gets old pretty fast, so let’s add some effects to make it more interesting. We type “image filter” into the search box. That gives us a list of effects to choose from. “Pixellate” is a nice simple one, so we’ll start with that. We’re going to insert that into our stream of images.

We’ll set up fader #3 on the Behringer to control the pixel size. Tip on making visuals: we like to make effects range from nothing to mega extreme. That gives you a big range to work with and lets you turn off the effect completely. For one section of the music we could turn on pixellate a little bit, and for a different section, maybe something really distorted, we could turn the pixellation all the way up.

Now let’s add another effect. Looking back through our image filters here, Twirl Image looks kinda cool on a dome; let’s go with that. We’ll set it up — put it into the image stream, and set it up so fader #4 on the Beheringer controls the amount of twirl. Once again we’ve set up the range of the effect to go from nothing to pretty extreme. That’s our twirl.

Now let’s see that on the dome. At this point we have a simple video mixer that takes 2 videos, and it blends one on top of the other, and adds some effects.

Probably what I’d do next in real life is add some more videos to this since we just have the two. Instead of having them hardcoded in those URL inputs in the composition, what we could do is make Vuo grab the list of all the movies in a folder, and set up Behringer controls to pick between them. Or, what I haven’t mentioned up to this point, is that there are a number of VJ software applications that implement video mixers — so they have the logic of picking videos and applying effects built in. So another option would be to use one of those applications that takes Vuo compositions as input, as plugins, and build up the visuals from there. In any case, we’ll leave the video mixer here because I want to move on and show you more ways of making visuals.