Our first ML opponent sucked and AlphaGo helped us find out why
The plan behind our AI opponent for our 2D Pokémon-style game seemed simple enough:
- Build a neural network
- Feed it every value the game client sees
- Train it to fight
- Let it play against itself and watch it get better
Well, we’re here because it didn’t quite go that way. But let’s start at the beginning.
The Game We’re currently building a 2D multiplayer Pokémon-style battle game, you can fight other players or go up against an AI opponent. It features a variety of fighters, attacks, and status effects.
The Network For the architecture, we wanted to keep things simple. The full game state gets encoded as a flat vector, around 115 values in the actual game, trimmed down to 101 in the version on GitHub for simplicity. That vector feeds into a small feedforward network: two hidden layers of 256 neurons with ReLU activation, followed by a 128-neuron layer with linear activation, and finally an output layer of 6 neurons, one Q-value per possible action: moves 1–4, and switching to either of your two remaining fighters.
Small by modern standards, but our problem doesn’t need GPT-scale; it just needs to learn not to use a fire attack on a water-type and in the best case not to lose.
The Training Now for where our thinking didn’t go far enough. We split training into two phases.
Phase 1: Teach the model how to fight We started simple: let the agent battle a bot that picks actions completely at random. The idea was to give it an easy first opponent, something it could actually learn to beat before facing anything smarter, just to figure out some basic fighting, no real strategy.
Below you can see the win rate plotted over 5,000 episodes of training:
The results were encouraging. The moving average trend is upward, indicating the agent is genuinely learning, not just getting lucky. It’s making better decisions than random, and improving over time. Phase 1 worked.
Phase 2 went sideways. The idea made sense to us at the time: pair two players of equal skill and let them compete — they’ll push each other to improve. Chess works that way. We wanted to give our agent a real opponent instead of a random one, something that would force it to develop an actual strategy rather than just exploit obvious mistakes. So we set up two independent agents, A and B, and let them fight each other. Every 10 episodes we’d sync their weights into a single shared network using the Bellman equation to update Q-values.
Seemed reasonable. But It wasn’t.
If you look at the plot, you might be a bit confused, we certainly were. The win rate converges around 50% almost immediately, especially compared to Phase 1, which used the exact same scale.
But why?
That was exactly the question we asked ourselves. And as it turns out, AlphaGo, DeepMind’s Go-playing AI, ran into a similar problem when training against itself. The difference is in how they solved it:
Instead of letting a model play against an identical copy of itself, DeepMind trained AlphaGo against a pool of past versions of itself. This one difference changes everything. There are two fundamental problems with our approach that we didn’t think through.
Problem 1 Moving Targets In our setup, both agents update their weights simultaneously, then immediately play each other again. There’s no fixed reference point. Every time agent A improves, agent B improves by exactly the same amount in response, and vice versa. Neither one ever gets to consolidate what it learned against a stable opponent, so no real strategy ever forms. They just chase each other in circles.
AlphaGo’s pool approach solves this directly. If you’re training against a pool of N past versions, any single update only changes 1/N of your opponents. The training signal stays consistent long enough for the model to actually learn something before the ground shifts again.
Problem 2 The Math Doesn’t Work This one is more subtle but more damning. Because exactly one agent wins each episode, the combined win rate of both agents always sums to 100%. Since both agents share the same network and update through the same backpropagation pass, every gradient update that helps one agent is immediately reflected in its opponent. The network is always winning in aggregate and always losing by the same margin. The net weight update approaches zero. Training stalls. That’s not a bug. That’s our setup being fundamentally broken. You might think our 10-episode sync interval helps, and it does, slightly. Instead of the symmetry being perfect every episode, you’re effectively averaging the weight updates over 10 episodes before syncing. But across 5,000 episodes with 500 syncs total, the average weight change per 10-episode window is small enough that the dampening effect is nearly negligible. The symmetry problem doesn’t disappear, it just gets slightly blurred.
What did we do?
Honestly? Nothing yet.
We’re still early in our ML journey and the fix isn’t as simple as just copying what AlphaGo did. Before we can implement any alternative training strategy, whether that’s a frozen opponent pool or something else entirely, we need to build a proper checkpointing and serialization system to save and load model states. That’s the real next step. As for which training approach we’ll use, we haven’t decided yet. The AlphaGo pool method is the obvious answer, but we’re curious whether there’s a more interesting approach worth exploring first, which may be even a better fit for our Goal. We’ll write about it when we get there.