The Reversi opponent that taught itself
What it is
Anders at Witzora, a small independent studio in Stockholm, is building an app for playing Reversi, the same game as the classic Othello. It is called Reversi Queen and has not been released to the stores yet. One of the opponents in the app is called Juno, and Juno is an AI model that learned the game entirely on its own.
It was given the rules and nothing else. Nobody has shown it a single move played by a human. Everything it knows it learned by playing against itself, around the clock, on a laptop, until it beat most strong players.
What plays is a small neural network: just under two million numbers that were adjusted a little after every round. It looks at the board and answers with two things: which move feels best, and who is ahead. Before it chooses, it tries a few hundred moves ahead in its head. The finished network is a file smaller than a photo, and it works in the app without an internet connection.
Why we built it
The app already has strong bots called Zenith. They are built on minimax, the classic way to make a computer play board games: look a few moves ahead and judge the position by hand-written rules, for example that corners are worth a lot. The strongest Zenith beats 95 percent of all people. But that is pure mathematics, not AI.
Juno is the experiment of taking the bots one step further: an opponent that nobody programmed, one that taught itself. Earlier attempts in the cloud became too expensive and were built on the wrong method. This time everything ran locally on a MacBook Pro, and running it cost no more than the electricity. The question was whether it could beat the minimax bots. The answer was yes.
How it learns, in short
Picture a beginner who plays two thousand games against themselves, looks back at them and asks: which moves led to a win? Then they adjust their gut feeling a touch and play two thousand more. We call each such round a generation, and it takes about ten minutes.
One night gives more than a hundred generations, and by morning it has played more games than a person manages in a lifetime.
The journey
The training went on for just under three weeks, in three runs.
- 10 SeptemberThe first network starts from random numbers and loses every game.
- 13 SeptemberAfter four days it is strong enough for the app and becomes the first Juno. A network twice the size is tried, but never catches up.
- 17 SeptemberA tournament between the versions shows something unexpected: the networks have become different rather than better. The newest beats the one before, which beats the one before that, but the oldest beats the newest. Like rock, paper, scissors.
- 18 SeptemberWe start again from scratch with everything we have learned built in: the network also gets to meet older versions of itself, and some games start from positions taken from human tournaments.
- 22 SeptemberWith a new, reliable way of measuring, the best version so far is chosen for the app. It is called generation 567 and still plays there.
- 26 SeptemberJuno stops taking help from an exact solver at the end of the game and becomes a trained network through and through, just as strong as before.
- 28 SeptemberThe restart has drawn level with 567 but not gone past it. We stop after 3.8 million games, happy with the network in the app.
How good it became
Stronger than Zenith, and in the opening and the middle game better than the world champions. An early version of Juno won 378 of 400 games against the strongest Zenith. The network in the app today wins roughly every third game against Egaroucid, one of the world's strongest Reversi programs, at its level 7.
We also compared Juno's moves with the humans' in thirty years of World Championship games. The judge was Egaroucid, which worked out how many discs each move cost compared with the best one. In the opening and the middle game Juno loses less than half as much as the World Championship players. In the endgame they are level.
What it is like to play against
Strength is one thing. What makes Juno worth meeting is how it wins. There are three things you come to recognise, and all three can be measured.
It looks as if it is losing. In Reversi, having many discs in the middle of the game is a disadvantage: every disc you own is a disc the opponent can flip. Juno has worked that out for itself and takes it further than people do. Against a player who always takes as many discs as possible, Juno often has only a handful after thirty moves, and still wins by a wide margin.
It takes your moves away. Juno chooses quiet moves that do not open new squares for the opponent. Against a player who likes to take discs, the opponent often has only two or three moves to choose from towards the end of the middle game, and all of them are bad: give away a corner or open an edge.
It brings the game home at the end. The position it has built with few discs and many moves of its own, it turns into a clear win at the end. Zenith plays differently: it values corners and discs by a formula and looks six moves ahead, strong but mechanical. Juno plays for things that pay off thirty moves later.
What the training looked like
While it trained we could follow one of the games live, together with the curves showing how it developed.
Questions and answers
Which AI model do you use?
None off the shelf. It is not ChatGPT, Claude or Gemini, but a small network of our own that started from random numbers and can do only one thing: play Reversi. A language model has hundreds of billions of numbers and has read nearly everything ever written. This one has a couple of million and has never seen a text. The code for the workshop was written together with Claude, Anthropic's AI assistant, but the one that plays is entirely our own.
Is it really learning?
Yes, in the most literal sense. Think of the little robot car that has to find its way through a maze: at first it bumps into the walls, after many attempts it gets through without a fault, and nobody has shown it the way. Juno is the same thing, with a board instead of a floor. It is called reinforcement learning.
We can even see the bumps. A corner in Reversi can never be flipped. After two thousand games the network took the corner only one time in four when it could. Four thousand games later it took the corner four times out of five. And then something finer: it stopped taking corners so often, because a strong player knows that the corner is usually still there and that something else is worth more right now. The rule was traded for judgement, and nobody told it to.
Isn't it the same thing as ChatGPT, only worse?
The difference speaks for Juno. ChatGPT is trained on text that people have written and at bottom imitates them. Juno got only the rules and one loss after another, and out of that grew strategy that nobody had told it about. If ChatGPT is a reader, Juno is an explorer. And ChatGPT cannot play Reversi: it breaks the rules after a few moves.
Does it think?
That is a philosophical question nobody can settle. Whether it learned, on the other hand, is not philosophy. The first version lost every game. The version in the app beats most strong players. In between there are only rules and results.
Has it never seen human games?
The network in the app has never seen anything human at all. In the restart, some games began from a position taken from a human tournament, but the network was never told what the human played there. The archive contributed questions, never answers.
Why did you stop training?
Because it was no longer getting better, measured in a way we trust. The restart tried several improvements, but no version went clearly past the network in the app, and a larger network did not win more games. The next step would have taken far more computing power. Our trained AI model Juno is plenty: it beats Zenith and has a style of play that is more fun to meet.
Part 2 tells how: How the network learns Reversi, about what the computer sees, what a neural network is, the search, the training loop and the road to the phone.