This AI Bot Is Trying To Guess What Is Being Depicted In Popular Memes, And Here Are 30 Of Its Funniest Attempts

You could say the roots of the project began to sprout when Neumann was still studying. "Throughout college, I took some courses related to machine learning, and I had a lot of assignments related to image recognition," he told Emerging Tech Brew.

"For example, one assignment was to make and train a neural network that can differentiate [between] pictures of cats and dogs. But you can pass any image you want through a neural network, and I always found it super funny to pass through memes or any random image and watch the NN [neural network] try to classify it as a dog or cat."

In other assignments, he was dealing with more advanced neural networks like ResNeXt. "In doing the same thing, the results were rarely correct, but you could start to see how the neural network thinks, and I thought that was super interesting," he recalled.

"In some cases, you could identify what the machine was looking at and why it made its prediction (for example, the stacked cheese in this prediction really does look like the texture of a pineapple IMO)."

After graduating, Neumann began working in the embedded world, so he didn't get a chance to play with machine learning anymore. That is until one seemingly uneventful evening.

"I thought that making the NNGM Twitter bot would be a good chance to both mess with NNs again and learn how to use the Twitter API, so I got to work on it that weekend." And the rest is browsing history.

ResNeXt is considered to be a simple, highly modularized network architecture for image classification. "[I chose ResNeXt for my side project because it] was trained on the ImageNet data set, a huge assortment of labeled images with hundreds of classes," the computer engineer explained.

"Each year, ImageNet has a competition among researchers to see who can make the most accurate NN. ResNeXt was the 2016 winner, having a better top-5 error percentage, beating even that of humans. When I saw that the Python library PyTorch had a pre-trained model for ResNeXt, and I wouldn't have to code it up and train it myself, it was an easy decision."

Neumann said it was a while before the bot picked up. "I would say that the first tweet to really take off was ... in October 2021, a picture of a plushie riding on top of a tank. The plushie is a character from the Touhou series, a video-game series with a cult following."

"There's a sizable Touhou fan community on Twitter, and that tweet found its way in there and blew up. After that, growth was very burst-y. Every now and then, the bot would produce a particularly funny prediction, which would get spread around, and the [account] would get a bunch of new followers all at once."

When the account was still small, the number of submissions Neumann received for his tests was low enough to the point where he could look in his DMs and download each image manually. But that eventually changed and as the computer engineer started getting more and more pictures, he wrote a script to take some load off of him. And that wasn't the end of it, too.

"Eventually there were too many submissions in the DMs for even my script to handle. Once that happened, I shut down submissions for a while so I could set up a website dedicated to submissions," he said. "By the time I got around to finishing the website, the account had exploded. I don’t remember how long I had the website up—maybe just a few weeks—but the amount of submissions I was getting was simply unmanageable. Even today, I still have a backlog of about 8,000 images [that] I have to look through and moderate."

Initially and ironically, Neumann hoped that the account "would show just how good neural networks can be."

"Sure, it has a hard time getting the predictions right, but that's kind of to be expected when I'm feeding it totally nonsensical memes," he said. "What I love about the account is that sometimes you can see how the bot thinks, you can sometimes see why it made the prediction it did. And that, to me, is a little bit spooky."

Running the bot, Neumann's favorite moments are seeing people root for it and get excited when it makes a (somewhat) good prediction.

"It's cute watching people personify a neural network, and celebrate it when it succeeds and chastise it when it fails. I understand that the NNGM Twitter bot may not be much more than a meme page with a twist, but watching people get excited over a robot predicting nonsensical memes really makes my day sometimes," he said.