After cheesing the Jump Rope challenge in Super Mario Odyssey with a Python script and a microcontroller and finding myself at the top of the leaderboard for a few weeks, the next logical step was to see if the Volleyball Challenge could also be cheesed. Volleyball is a lot more complex than jump rope, with a lot more variables involved, so this was obviously going to be a much more difficult challenge. First, though, let’s take a look at some other methods for cheesing the Volleyball challenge. If you’re just playing the game casually and all you care about is clearing the challenge, the best pro tip for that is, as I’m sure everyone’s seen, to play the game in two player mode, and exclusively use Cappy, while Mario just stands off to the side. The problem with using Mario for the Volleyball challenge is that his acceleration curve is so slow, it’s really hard to get him to the ball in time.
In FEZ PEZ’s video on the Talkatoo glitch, he proposed that you could try to freeze Mario in place above the volleyball snail and rally the ball back that way. This strat works for a little a while, but unfortunately, Mario is just one little guy and the ball has such variance in its angle and such that he simply can’t cover the entire space above the net. So far as I know, no one has successfully gotten a high score with this method. I also found this video on Twitter of someone using some advanced Cappy throws to theoretically cheese the Volleyball.
And this technique seems to have some promise, but unfortunately if we wanted to automate the process, short of building a robot, we don’t have any way that I know of to programmatically emulate motion controls on the Switch. So then, the plan is to use a similar technique to the Jump Rope bot—use the Switch Fightstick library and a Teensy 2.0++ to emulate a Switch controller, and have that microcontroller communicate with a Python script which is using the video feed from the game to determine what button input to send back to the microcontroller. For Jump Rope, this was pretty easy—all we really cared about was the timing, so my solution was to look at the score display as a way to synchronize the timing, and all we really had to do was send a signal to press the jump button with the appropriate timing.
Volleyball, on the other hand, is a much more complex beast. You’ve got to somehow track the position of the player and the ball, then move the player into position to hit the ball, and do it all in real time without ever screwing up for hours on end. So, let’s tackle each issue one by one. First off, just like when playing casually, Mario moves too slowly to get to the ball in time, so our bot is going to be controlling Cappy. So then, the two things we have to track are Cappy and the ball. Like with the Jump Rope Bot, we’re going to be using the OpenCV library in Python for image processing. There are lots of different ways to track objects in OpenCV. You could use dedicated motion trackers or some methods for finding circles in an image, but pretty early on, even before I made the Jump Rope bot, I figured out a fast, simple, and reliable method for tracking these objects. First, let’s crop the video, then split it into its red, blue, and green components.
Notice anything? Cappy is by far the least red part of the image—the sand and the ball are both red, so when we look at the red channel, Cappy sticks out as a dark spot. That Luigi costume’s finally paying off, huh? Similarly, on the green channel, the ball sticks out as a dark spot. So, to track them, we just look for darkest spots in the red and green channels. What if the ball isn’t in the court, you might ask? In that case the darkest part in the green channel ends up being Cappy’s shadow, which isn’t really a problem. Also, sometimes it picks up the ball’s shadow instead of the ball itself, but that’s also not really a problem, because so long as Cappy follows one or the other, they’ll converge when the ball reaches the ground.
Now, for communicating between the script and the microcontroller—I pretty much glossed over that detail in the Jump Rope video, but we can go into a bit more detail now. Basically, I use a USB to UART adapter to send data from the computer to the Teensy. Mario Odyssey has such simple controls, I was able to fit everything you need in just one byte. And now, we can play Mario Odyssey with a keyboard, or, more importantly, with a Python script. So theoretically, putting these two pieces together should be pretty straightforward. Just track Cappy, track the ball, and make Cappy move in the balls general direction. Hopefully it should hit it, right? Well, if we try that… okay, okay, looks like it works pretty well. It’s getting a half-decent score. And… wait, Cappy, where are you going? Cappy, come back! Okay… looks like there’s a bit more tweaking to be done. Although Cappy is much more responsive than Mario, he’s also a lot bouncier. Every time he hits the ball, he gets sent flying off in some direction.
If he gets sent off the court, then, well, he’s lost forever. In fact, I even cheated a bit in that previous video, manually steering Cappy back into the court whenever he tried to leave. So, to try to correct for that, we can extend the bounds of the area we look for Cappy in, and add some extra rules that aggressively guide Cappy back onto the court whenever he strays. These changes help, but there’s still a problem. Cappy sort of flies around all over the place, without really looking like he’s heading directly towards the ball. You can see how he likes to sort of orbit around Mario. Sooo, to try to combat that, I finally went and added full analogue control, so Cappy should theoretically head, more or less, directly towards the ball. There’s just one more problem, though. Sometimes, Cappy will find himself on one side of the court, while the ball goes towards the other. The natural instinct for a human player is to return to the center of the court whenever they successfully hit the ball to avoid this problem.
So how do we get Volleybot to know when it hits the ball? For that, we can turn to the Jump Rope bot! The score display changes when and only when Cappy connects with the ball, so we watch for that to change, and when it does, we reset Cappy to the center of the court. And that almost works. Buuut… there’s some unavoidable (so far as I know) latency inherent in this setup, between the capture card, the script, the microcontroller… there’s some delay between the script deciding what to do and it actually seeing and processing the results of its own actions.
For the most part this hasn’t really mattered much, but watch this. Cappy starts in the center of the court like we told him to, and the ball enters the court, so Cappy starts to move towards it. But then he overshoots and has to course correct. He ends up travelling farther than he actually had to to reach the ball and eventually drops it. Soooo, rather than having him reset to the center of the court, let’s just have him reset to the left side, near where the ball enters. Problem solved. And FINALLY, we’re ready to see the Volleybot in action. I was so excited to show off my creation to the world as soon as possible, for the first time ever I set up a YouTube livestream to see how far it could go. We had about 90 people concurrently at the peak all sitting in the chat, talking about video games and stuff and watching the number go up. And oh boy did that number go up. Although I was worried it would break at any moment, it seemed to hold steady well up into the thousands and my confidence grew.
But then, after about four hours and over seven thousand successful volleys— “…tried to maintain media silence on this game before I finished it. Oh what? Oh no, what happened?” —Cappy dropped the ball! Literally. I mean, look at that, he doesn’t even try! What’s up with that? Well, upon looking at the frame-by-frame replay, it’s pretty apparent what happened. For a few frames, he starts to head towards the ball, but then gives up and returns to his home position. Something in the score display detection code is misfiring in a way that didn’t affect the Jump Rope bot. If we take a closer look at the score display itself, and specifically if we process it in the same way the script does by reducing it to just two colors, we see that there’s actually a good deal of noise in the signal.
Now, the script is tolerant of some noise, but apparently, for whatever reason, around 7138 the noise got so bad that it just so happened to trigger the reset behavior. So, even though I was perfectly happy with a score of over seven thousand—it was number 10 on the global leaderboard—knowing that I could make it better was enough to keep me motivated. So, I spent probably more time than I should have making a bunch of changes, some of which probably helped and some of which probably didn’t—most importantly, I added some noise correction to the score display detector—and I started another livestream to see how far this new and improved version could go. I also added a bunch of debugging views to the stream, partly because of a viewer request and partly so that if it did eventually fail we would know what happened.
And—oh come on, it didn’t even get as far? Well, at least it’s clear what happened this time… watch closely, the script lags! You can see the video feed stutter for just a moment, but it’s enough to screw everything up. At this point, it was midnight, I was tired, I didn’t want to write any more code, but I still wanted that number to keep getting higher. So, I said forget debug information, and forget video quality standards, we’re going to 2006-era webcam Let’s Play, aww yeah.
“There. This is a watchable stream.” I closed everything on my computer except for Python running at REALTIME priority, set up this stupid webcam, didn’t even bother changing any of the settings to make it watchable, then went to bed. I had some pretty weird dreams, but while I was sleeping the script kept running, and when I woke up it was pretty darn successful. I finally went into the webcam settings to try to make something watchable, buuut… it didn’t last much longer after that. With a score of 18,395, I was number four on the leaderboard, just short of the number three slot, which is a pretty decent score if I do say so myself.
And, well, this is the end of the road for now. With no debug information to go off of, my patience for trying new things and then sitting around for 9 hours to see if the bot sets a new record or not only lasts for so long. Maybe I’ll come back to Volleybot at some point in the future, but for now, I’d say I’m happy. That being said, if anyone has any ideas for improving Volleybot itself or any ideas of other cool projects I could work on in the future, please let me know in the comments section.
As for the code, it’s not available right now, but maybe if I find the time I’ll clean it up a bit and release it. And, of course, if you want to see more consider subscribing and following me on Twitter. I can’t guarantee that everything I upload to this channel will be high quality entertainment, but if there’s one thing you should know about me by now, it’s that I like to see the numbers go up. That’s all I’ve got for now, see you hopefully not too far in the future.
As found on Youtube