You might know the Facebook game called “Tetris Friends”. It’s basically just a flash version of tetris but with some added features like the ability to “hold” a block and to see multiple pieces in advance.

About a year ago I started to wonder if I could make an AI for it, just like I did for my own implementation of Tetris. It would have to look at the rendered pixels of the webpage containing the flash object, use those to determine the current state of the game, and then send keystrokes to the web browser to provide new input.

Here’s a video showing the screen-reading part of the program. It’s calibrated so that it knows the position of the top left corner of the tetris field. It can then read colors of pixels at strategic points to determine what blocks are where. This was the hardest part of it all since this game isn’t as “8-bit” as my implementation, making the color-to-piecetype conversion relatively hard.

The rest of the challenge was just adapting my previously-made AI to work with this. This was – apart from some timing issues – rather easy. Below is a video of the final program working, and an explanation of the code.

Now the technical side of things:

  • libX11 is used to retrieve the mouse pointer position. This is for calibrating the board position.
  • xdo is used to send keystrokes to the browser window.
  • ncurses is used in the first video example to display the internal representation of the board in a nice way.

This makes the code X11-specific, although I imagine you could easily use win32 libraries to do the same on windows, or Cocoa libraries for OS X.

Disclaimer about the code: It’s really messy. This is mainly served for “historical” purposes.

Nevertheless, here it is.