How the network learns Reversi
Part 1 told what we built: a Reversi opponent that learned everything by itself on a laptop. Reversi is the same game as the classic Othello. This is how, in four blocks. First the network: what the computer sees and what the "brain" actually is. Then the training: what the network does when it thinks, and how one round works. Then what we learned along the way. Last, how the network moves into a phone. No mathematics.
1. What the computer sees
The network never sees a board with black and white discs. It sees four grids of ones and zeros, always from the point of view of the player to move: my discs, the opponent's discs, the squares where a move is allowed, and a helper plane of nothing but ones that makes it easier to sense the edge of the board. Colour does not matter. The same network plays black and white, because all it knows is "mine" and "the opponent's".
Two answers come back out of the network. One is a probability for every square: how likely it is that a strong player moves exactly there. We call it the policy, the gut feeling. The other is a single number between minus one and plus one: how well the player to move is doing. It is called the value, the assessment of the position.
2. What a neural network is
A neural network is a system for calculating, loosely inspired by the brain, that finds patterns in data on its own. Instead of rules that someone has written, it learns by guessing, measuring how wrong the guess was and turning its inner connections a little in the right direction. The points that calculate sit in layers, and every connection between two points has a dial, a weight, that says how much the one should affect the other.
A very simple example
Picture the small network above. It gets two numbers about a position, say how many corners I have and how many moves I can choose between, and has to guess what share of the discs will be mine when the game is over. The right answer is 57. The dials are set at random, and the first guess is 30. Now four things happen, and they are the same four things that happen in our network hundreds of thousands of times a day.
- The error is measured. The guess 30 is 27 too low.
- The blame is passed backwards. For every dial it is worked out whether it pulled the guess down or up, and by how much.
- The dials are turned. Every dial is turned a small step in the direction that raises the guess. Only a small step, so that the network does not overshoot.
- The next round. The guess becomes perhaps 44, then 51, 55, 57.
With a single position the network has only learned one answer by heart. So the same thing is done with thousands of positions in turn, and the dials end up where the error is smallest across all of them at once. Then they are set right for positions the network has never seen too, and that is the whole point.
Our Reversi network works in exactly the same way, only larger: 1.8 million dials instead of 28. It is built as a tower of layers. The first layer looks at every square together with its neighbours, and every layer after that sees a little more of the board, until the tower sees all of it. At the top it splits into two heads, one for the gut feeling and one for the assessment of the position.
3. Thinking ahead: the search
The recipe we follow is called AlphaZero and comes from DeepMind. Its predecessor AlphaGo beat the world's best Go player in 2016, but had first been allowed to study human games. The year after, the same team showed that this was not needed: only the rules, and play against itself. AlphaZero used about 5,000 specially built chips. We used a laptop, and Reversi is a smaller game than chess.
The most important idea in the recipe is the search. A network that only follows its gut feeling makes silly mistakes, because it does not calculate. So the search tries moves in its head before it chooses: the network points out the moves it believes in, the search plays one of them, asks the network how things look now, and carries the answer back. After 200 such trial moves, the move that has been visited most often is played. Moves that look good get more visits, but untried moves get a chance too.
4. The training loop
The training is a loop that repeats around the clock, one round per generation.
- Play. The network plays 2,000 games against itself, with the search. A little randomness in the first moves makes the games differ.
- Remember. Every position is saved with two right answers: what the search thought of the moves, and how the game ended. Old games are overwritten, because they were played by a weaker network.
- Learn. The network guesses on positions from the memory and turns the dials towards the right answer, just as in the simple example.
- Duel. The new version meets the best one so far. If it takes more than 55 percent of the points, it becomes the new best and plays the next round's games. We tried lowering the bar so that fresher networks would play more often, but both times the network got worse.
- Measure. Last, it plays a few games that are only used for the curves, among them games against Egaroucid.
5. What it learns first
A corner in Reversi can never be flipped, so it is the safest square on the board. The square diagonally next to a corner is the classic beginner's mistake, because it leaves the corner open to the opponent. Nobody has told the network any of this. We can measure when it works it out.
| After about | Takes the corner when it can | What it means |
|---|---|---|
| 2,000 games | 27% | Chaos, barely better than chance |
| 10,000 games | 86% | The corners discovered: the beginner's rule in its purest form |
| 200,000 games | 54% | Judgement instead of a rule |
The corners in an hour. After a few thousand games the network takes the corner nearly every time and has stopped giving them away. It has learned the most important rule of the board without anyone mentioning that corners exist.
Then it stops taking corners. It looks like a step backwards and is the opposite. "Always take the corner" is a beginner's rule. A strong player knows that the corner is usually still there on the next move and that something else is nearly always worth more right now. The table shows where the network stopped being a beginner.
6. Measuring right was the hardest part
The network, the search and the rules were a couple of days' work. The hard part was knowing whether the network was getting better. In self-play everything is measured against something that moves: the duel compares the new version with last week's best, which is itself on the move.
The most dangerous mistake was to believe that strength is one number. We chose networks for the app in a chain: the new one beats the one before, which beat the one before that. When we finally let the two ends meet, strength turned out to go in a circle, like rock, paper, scissors. The oldest version beat the newest. Every version had become sharp against its closest relatives and narrower at the same time.
The solution was to measure against someone outside the family, in a way that leaves chance out. We took 300 openings from human tournament games and had every network play them with both colours against Egaroucid. Since nothing is drawn at random, two networks meet exactly the same games and can be compared game by game. That measure held all the way, and it is the one we used to choose the network that sits in the app.
7. Human positions, and how it ended
The circle is the weakness of self-play: the network always meets itself, and positions its own family never builds it has hardly seen. The cure was real games. Since 1977 the French Othello Federation has collected tournament games in an archive called WTHOR, 137,548 games, free for everyone.
We used it in a way that may sound backwards: we took the positions and threw away the moves. Some training games began from a position in a human game, and from there the network played on against itself. It was never told what the human played. An earlier attempt to teach a network to imitate human moves had shown why: a network that imitates can never become better than its model. The archive contributed questions, never answers.
With the human positions, older versions as opponents and a larger memory, we started again from scratch on 18 September. The restart trained for ten days and tried five improvements, one at a time and with the yardstick decided in advance. Every improvement moved something, but usually one part of the game at the expense of another. The best version ended up just above the network in the app, but within the margin of error.
We also tried a larger network. It learned the games better but did not win more of them. That suggests it was the games that set the limit, not the size of the network, and more games take more computing power than a laptop has. That is where we stopped, with the network in the app as the best we have built.
8. From workshop to phone
The finished network is shrunk to a format that phones can run, 3.6 megabytes, smaller than a photo. Before the file is approved, the same positions are fed through the original and the copy, and the answers have to match. Test positions travel with the file, so that the app can make the same check on every phone.
Juno in the app is only network and search, for the whole game. To begin with, an exact solver worked out the last twelve squares. Since 26 September Juno instead searches four times deeper there, and wins just as often. The wins are narrower in discs, because the network has learned to win, not to win big.
| Juno level | How the move is chosen |
|---|---|
| Easy | The gut feeling, with a lot of randomness |
| Medium | The gut feeling, with a little randomness and no search |
| Hard | A short search around the network |
| Pro | A longer search, and deepest at the end: the most the network can do |
The code was written together with Claude, Anthropic's AI assistant, and in a fixed order. No search was built before the rules were proven free of errors, and no network was trained before the search had played a thousand games without a rules error. A network trained on a board with a rules error learns the error, and nobody sees it in the curves.
9. Glossary
- Generation
- One round of the training: 2,000 games, one learning pass, one duel. The number on a network is the number of rounds.
- Policy
- The network's gut feeling for the next move, one probability per square.
- Value
- The network's assessment of the position, from −1 to +1 for the player to move.
- Search
- The trial moves around the network, with the grand name Monte Carlo tree search. The teacher in the recipe.
- Simulation
- One trial move in the search.
- Duel
- The new version against the best one so far. Above 55 percent, it takes over.
- Egaroucid
- An open, world-class Reversi program by Takuto Yamana. Our outside answer key.
- Juno, Zenith
- Juno is the network as an opponent in the app. Zenith is the app's minimax bots, which calculate instead of feel.