LW - We might be missing some key feature of AI takeoff; it'll probably seem like "we could've seen this coming" by Lukas Gloor

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We might be missing some key feature of AI takeoff; it'll probably seem like "we could've seen this coming", published by Lukas Gloor on May 10, 2024 on LessWrong.
Predicting the future is hard, so it's no surprise that we occasionally miss important developments.
However, several times recently, in the contexts of Covid forecasting and AI progress, I noticed that I missed some crucial feature of a development I was interested in getting right, and it felt to me like I could've seen it coming if only I had tried a little harder. (Some others probably did better, but I could imagine that I wasn't the only one who got things wrong.)
Maybe this is hindsight bias, but if there's something to it, I want to distill the nature of the mistake.
First, here are the examples that prompted me to take notice:
Predicting the course of the Covid pandemic:
I didn't foresee the contribution from sociological factors (e.g., "people not wanting to get hospitalized" - Zvi called it "
the control system").
As a result, I overpredicted the difference between countries with a lockdown policy vs ones without.
(Note that this isn't necessarily an update against the cost-effectiveness of lockdowns because the update goes both ways: lockdowns saved fewer lives than I would've predicted naively, but costs to the economy were also lower compared to the counterfactual where people already social-distanced more than expected of their own accord since they were reading the news about crowded hospitals and knew close contacts who were sick with virus.)
Predicting AI progress:
Not foreseeing that we'd get an Overton window shift in AI risk awareness.
Many EAs were arguably un(der)prepared for the possibility of a "chat-gpt moment," where people who weren't paying attention to AI progress previously got to experience a visceral sense of where AI capabilities progress is rapidly heading. As a result, it is now significantly easier to make significant policy asks to combat AI risks.
Not foreseeing wide deployment of early-stage "general" AI and the possible irrelevance of AI boxing.
Early discussions of AI risk used to involve this whole step about whether a superhuman AI system could escape and gain access to the internet. No one (to my knowledge?) highlighted that the future might well go as follows:
"There'll be gradual progress on increasingly helpful AI tools. Companies will roll these out for profit and connect them to the internet. There'll be discussions about how these systems will eventually become dangerous, and safety-concerned groups might even set up testing protocols ("safety evals").
Still, it'll be challenging to build regulatory or political mechanisms around these safety protocols so that, when they sound the alarm at a specific lab that the systems are becoming seriously dangerous, this will successfully trigger a slowdown and change the model release culture from 'release by default' to one where new models are air-gapped and where the leading labs implement the strongest forms of information security."
If we had understood the above possibility earlier, the case for AI risks would have seemed slightly more robust, and (more importantly) we could've started sooner with the preparatory work that ensures that safety evals aren't just handled company-by-company in different ways, but that they are centralized and connected to a trigger for appropriate slowdown measures, industry-wide or worldwide.
Concerning these examples, it seems to me that:
1. It should've been possible to either foresee these developments or at least highlight the scenario that happened as one that could happen/is explicitly worth paying attention to.
2. The failure mode at play involves forecasting well on some narrow metrics but not paying attention to changes in the world brought about by the exact initial thin...