We live in a world where after a decade of algorithm development, we're still seeing heavy use of tweaked variants of PPO, an algorithm proposed around 2016. What I'd like to claim is that the reason we are still using PPO 10 years later is not because there aren't better algorithms out there but because we do not know how to evaluate algorithms. I mean this in two senses: we do not have consistent statistical procedures, which I think is well observed (see 1 and 2 for further discussion), but more importantly, we do not have ranking procedures with which we can evaluating algorithms. What I mean by a ranking procedure is an agreed on practice by which we go from observed algorithmic performance to consensus about the relative rankings of algorithms. While we have a million benchmarks to rank algorithms, we do not know how to convert results on the benchmarks into clear claims about algorithms. Consequently, knowledge does not accrete and algorithm practices remain unchanged.
At the root of this issue is this: every algorithmic paper in RL, whether it states so explicitly or no, is a claim about generalization. Fundamentally, no one cares about standard RL benchmarks like Atari or DMC or GSM-8K. The only reason we care about results on these benchmarks is the unspoken promise that the results will transfer to some to-be-specified problem. We will call this the hidden challenge problem: some future problem that we care about and would like RL to solve. We don't care about improvements on basic benchmarks, we care that observed improvements transfer to a hidden challenge problem (or perhaps that the results will somehow transfer to all POMDPs). Unlike other fields with specific aims, RL has no obvious grand challenge problems that we all agree is important and so we have no real way to check "yes, this improvement on Atari also transfers to the problems we care about." There simply is no such standard "problem we care about" that we could possibly specify. This is in contrast to fields like computer vision where the training domains like Imagenet-1k were both of their own interest and broadly similar enough to other interesting problems that they served as obvious proxies.
So what happens when someone reads an RL paper and looks at the empirical results section? Functionally, you simply don't know what to do with it. You know that what you care about is generalization to the hidden challenge problem but the paper, by evaluating on standard benchmarks, provides only the weakest of evidence. Of course, you could do the work for the authors and check the results on some other benchmark, but there are 10000 other RL papers competing for your time. So you extract the core idea and move on, leaving the empirical results in the dustbin.
This failure to extract any empirical beliefs about algorithmic performance is the core thing keeping PPO alive. For PPO, we have widespread cultural knowledge that it has already been used to solve multiple problems. It has the evidence behind it to support claims of generalization: successes in games, robotics, LLMs, etc. Why on earth would you try out a new algorithm, or a new set of tricks, when you have no real belief about its likelihood of working based on the empirical results in the paper? Just use PPO, you know it'll work. To create such a deep belief about other algorithms, if pursued through the same route by which we got to PPO, will take a similar length of time and body of results.
Of course, it's not like there have been zero widespread, agreed upon algorithmic tricks that have managed to jump this hurdle and make it into the standard toolbox. Hey, there's GRPO lol. There's the two-hot trick from distributional RL (and distributional RL in general), advantage filtering (1 2), prioritized replay (sometimes), MCTS etc. What distinguishes each of these and allows them to jump the generalization hurdle is that each of them was paired with some capability breakthrough. It's fully possible that outside of their particular challenge problem, each of those is in general a regression wrt the next challenge problem, but they still leap into the toolbox by using that large-scale success to overcome disbelief. But this is a very large ask to make for each algorithmic advance and one might hope that there is a simpler path.
So how do we get from here to a world where RL algorithmic knowledge actually accrues across papers? One version is to do what worked for computer vision and adopt a, or a set of, challenge problems that we actually care about and that remain unsolved. I am not hopeful here, both because RL is intended to be a generic tool that "solves all problems" but also because, unlike CV, we simply probably don't have such cleanly formalized, important problems. The obvious alternative is large scale benchmarking. If we can't say what structure connects our benchmarks and the problems that we care about, at least we can evaluate on a tremendous number of benchmarks and select for changes that are an improvement over the prior best result according to some ranking rule. It's no guarantee over generalization to the hidden challenge problem but it feels more reliable than an evaluation on Atari.
There is of course a natural objection to such a solution: computational cost. As 2 has astutely pointed out, even 50 seeds can be not enough to make a statistically significant claim in RL. This is why I really like projects like PufferLib, which speed up all the environments through efficient C implementations and thereby reduce the computational cost. This at least makes benchmarking across many environments possible in principle for the average paper. Unfortunately, this has two two failures (1) these benchmarks are of course proxy problems that may again not matter for our unstated challenge problem and (2) it privileges algorithms that are bottlenecked by environment steps rather than gradient steps but it's better than nothing. Pufferlib is a pretty good solution for on-policy methods and figuring out how to scale benchmarking for off-policy methods is left as an exercise to the reader.
So that's my theory of why RL is stuck and a very brief suggestion about how we can fix it. What I hope you take away though, more than anything, is that until we explicitly recognize that every paper is making a claim about generalization, and figure out how to make that evaluation process more explicit and easier, RL papers are going to keep winding up in the bin instead of successfully building on each other.
P.S. I'm picking on ALE a lot here, but that's not intended to minimize how impactful it has been in spurring lots of great work. Kudos to the authors.