Rock Band Robot

Warning: Extreme geekiness ahead.

So, I’ve picked up a new idea for a project, and it’s kind of taken over my brain.

At some point in the middle of the night, I got this great idea. I was going to build a robot that was capable of playing the drums in Rock Band (I know, but stick with me here). I was going to have a camera pointed at the TV screen, and write a program that would analyze the image to figure out where the notes where, and then trigger some sort of mechanism that would hit each drum at the right time. I had it all worked out.

When I woke up the next day and thought about it more, I realized that it was probably a little more complicated than I initially thought. But I figured I’d still poke around and see what I could figure out.

The first thing I found was that it’s really easy to write an application and interface with a webcam. In no time I had a little application that was capturing images from my dinky little webcam. I was halfway there! Moments later, I discovered multiple reasons why a webcam couldn’t be used. It was simply too damn slow at capturing images. I was getting just a few images per second, where I was hoping for something more like 30-50.

After doing some websearching, I found some smart people who made a robot that could play Guitar Hero. I noticed that they simply split off the video signal from the game (the yellow cable), and did direct video processing. Reading about video signal processing scared me. Video signals aren’t just frames of animation like film. There are keyframes and compression and lots of bits I didn’t understand.

However, I found a really cool programming library that does all the work for me. Basically, you feed in a video stream (webcam, VCR, digital video camera, etc), and it gives you each frame. It even allows you to use a video file as a video stream. So, I hit up YouTube and downloaded a video of someone playing Rock Band drums. I was able to load that up, get each frame of animation as a picture, and could then program against it.

The programming to analyze the frame is actually a lot easier than it might sound. The notes all fall down in the screen in a very regular pattern. Each color notes always follows the same path. So, on each frame of animation, you simply look at a particular point on the screen. I picked a small rectangle, and find the average color across that rectangle. If the average color is black’ish, then there is no note there. If the average color is red (when looking on the Red note track), then there is a note there.

Within an hour or two, I had a program that was able to dynamically “read” the video, and surprisingly accurately detect notes. It was only a proof of concept, but I’m fully convinced that I could do all the software necessary to complete my little project. No problem.

So I started thinking about the hardware side. How would I make something that would bang on the drums? Actually, I can think of a lot of ways. Some sort of motor with a stick on it that bangs down. Or a solenoid suspended over the drum that shoots down at it (like a doorbell, but instead of hitting a chime, hitting the drum). A really cool idea, however, came from a workboy. He suggested dropping a small ball-bearing from some height. I thought that sounded interesting. We talked a bit more about it to see what we could come up with.

The best idea I can think of is a hollow tube suspended directly over each drum. At the top of a tube is a big funnel filled with 1/4 inch steel ball bearings. The inner diameter of the tube is just over 1/4 inch, so the balls line up single-file. Without any intervention, of course, the balls would flow through the pipe freely. The plan requires them to be dropped one at a time (possibly in rapid succession). So, something needs to get in the way of the balls, holding up the train, and releasing them one at a time.

An idea I can up with was cutting a vertical slot in the side of the tube, allowing access to the stack of balls. Into the slot would be inserted the edge of a bicycle gear, oriented vertically. The idea is that the first ball would get stuck on one tooth of the gear. The next ball would sit on top of the first ball, and the second tooth of the gear would nestle in between the two. So, as the gear spins, a single ball is dropped, and the next ball is held in place. Using something like a stepper motor (a motor that moves in very exact steps) to drive the gear, and I really think this could have a chance at working.

And if it didn’t work, it wouldn’t be a big deal. I think there are lots of ways to create a simulated drum hit. I’m not too worried about that.

So, the next big question, and one that I think is interesting in its own right, is how do I interface hardware and software? How do I send a signal from a computer to do something in the physical world (no matter what that thing is… turn on a light, drive a motor, flip a switch, whatever). If I was able to do that, I’m pretty sure I could take over the world. Well, maybe not

Unfortunately, this is where it gets a little hairy. I’ve made circuits by hooking batteries up to light bulbs, but that’s really about the extent of my knowledge.

Lots of websearching later, and I found that there are hobby robot makers that do exactly this kind of stuff. In fact, they sell little kits and parts that make it all really easy. You hook up a little controller board via USB to your computer. Then you hook up various motors and devices to that controller board (motors, etc), and it becomes super easy to drive the motors. The problem is those kits are outside of the budget which I’ve set aside for this project (which is about $10).

So, that’s about where I am. I’ve found some online electronics places that sell little solenoids and stepper motors for just a couple dollars, which is pretty cool. If I could just figure out how to make ‘em go.