Boost AI Productivity with Tight, Verifiable Feedback Loops
"‘Move 37’: a play so unconventional that professional commentators initially thought it was a mistake."
This article is part of the "move fast and brace things" series.
Ten years ago, Google's AI research division, DeepMind, defeated the world's best human Go player with their AI program, AlphaGo:
In 2016, over 200 million people watched AlphaGo face world-champion Go player Lee Sae Dol in Seoul. The match was defined by ‘Move 37’ in Game 2, a play so unconventional that professional commentators initially thought it was a mistake. But it proved to be decisive. One hundred or so moves later, the stone was in exactly the right position for AlphaGo to win the game. It was a display of incredible foresight and the AI system’s ability to go beyond mimicking human experts and find entirely new strategies.
Demis Hassabis and Demis Hassabis
RLVF: The Secret Sauce of AlphaGo
The most interesting (and relevant) part of that story is how the AI was trained. It was essentially a technique now known as Reinforcement Learning through Verifiable Feedback (RLVF). The idea is to provide the AI with verifiable feedback (e.g., did the game end in a win or loss) and then let it run lots of simulations to try different approaches to learn what works.
In fact, the DeepMind team doubled down on that exact approach when it trained AlphaGo's successors (emphasis added):
After AlphaGo, we built AlphaGo Zero, which learned the game from completely random play and became arguably the strongest player in history. Then we generalized the system further with AlphaZero, which taught itself from scratch to master any 2-player perfect information game, including Go, chess, and shogi. Beginning with no prior knowledge other than the rules of the game, AlphaZero was able to learn to master chess in a matter of hours, and beat not only the top human players but the best specialised chess programs at the time, like Stockfish. And even though chess had been so heavily analysed with the aid of these programs, just as with Go, AlphaZero was still able to come up with interesting new strategies.
RLVF for Mere Mortals
We are not in the business of training AI models.
We are in the business of using AI models.
The great thing about RLVF is that we can apply that same concept that proved so useful in model training to our day-to-day model use.
What does that look like?
It requires a paradigm shift from algorithm-based instructions to outcome-based instructions.
Rather than tell the AI in painstaking detail how you want it to solve a problem, you tell the AI what the final outcome should be.
Admittedly, certain problems lend themselves to this type of solution more than others. The key is that you need an outcome that the AI can independently verify is correct. If you have that, then you can let the AI iterate on multiple solutions until it finds one that works every time.
We'll explore practical examples of this technique in future articles.
*All text in this article written by a 100%-certified free-range human.
