Tetris Friends AI

Some of you might know the Facebook game called "Tetris Friends". It's basically just a flash version of tetris, but with some added features like the ability to "hold" a block, and to see multiple pieces in advance.

About a year ago I started to wonder if I could make an AI for it, just like I did for my own implementation of tetris. It would have to look at the rendered pixels of the webpage containing the flash object, use those to determine the current state of the game, and then send keystrokes to the web browser to provide new input.

This was quite a challenge, but I did eventually manage to pull it off. I just never made a blog post about it because in and of itself it didn't seem worthy of one. Though looking back at it now I feel like it is, so here you go. :P Consider this more as a documentation of this project for myself than as a newsflash.

Here's a video showing the screen-reading part of the program. It's calibrated so that it knows the position of the top left corner of the tetris field. It can then read colors of pixels at strategic points to determine what blocks are where. This was the hardest part of it all, since this game isn't as "8-bit" as my implementation, making the color-to-piecetype conversion relatively hard.

The rest of the challenge was just adapting my previously-made AI to work with this. This was -- apart from some timing issues -- rather easy. Below is a video of the final program working, and an explanation of the code.

Now the technical side of things:

  • libX11 is used to retrieve the mouse pointer position. This is for calibrating the board position.
  • xdo is used to send keystrokes to the browser window.
  • ncurses is used in the first video example to display the internal representation of the board in a nice way.

This pretty much makes the code Unix-only, although I imagine you could easily use win32 libraries to do the same on windows, or Cocoa libraries for OS X.

Disclaimer about the code: It's really messy. There's some pieces of commented-out-code, which I've left in there for educational and/or historical purposes. I wrote this code as a quick hack more than a year ago, and have not run it since. This means that it might not even run in your (or my current) environment, and it also means that I'm quite ashamed of my coding-style in this program. I've since had lots of classes in both C and C++, and computer science in general, so looking back at this is a bit painful. Use it as an example of bad style :)

Nevertheless, here it is.