Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.
This is: Six AI Risk/Strategy Ideas, published by Wei Dai on the AI Alignment Forum.
AI risk ideas are piling up in my head (and in my notebook) faster than I can write them down as full posts, so I'm going to condense multiple posts into one again. I may expand some or all of these into full posts in the future. References to prior art are also welcome as I haven't done an extensive search myself yet.
The "search engine" model of AGI development
The current OpenAI/DeepMind model of AGI development (i.e., fund research using only investor / parent company money, without making significant profits) isn't likely to be sustainable, assuming a soft takeoff, but the "search engine" model very well could be. In the "search engine" model, a company (and eventually the AGI itself) funds AGI research and development by selling AI services, while keeping its technology secret. At some point it achieves DSA either by accumulating a big enough lead in AGI technology and other resources to win an open war against the rest of the world, or by being able to simultaneously subvert a large fraction of all cognition done on Earth (i.e., all the AI services that it is offering), causing that cognition to suddenly optimize for its own interests. (This was inspired by / a reply to Daniel Kokotajlo's Soft takeoff can still lead to decisive strategic advantage.)
Coordination as an AGI service
As a refinement of the above, to build a more impregnable monopoly via network effects, the AGI company could offer "coordination as a service", where it promises that any company that hires its AGI as CEO will efficiently coordinate in some fair way with all other companies that also hire its AGI as CEO. See also my AGI will drastically increase economies of scale.
Multiple simultaneous DSAs under CAIS
Suppose CAIS turns out to be a better model than AGI. Many AI services may be natural monopolies and have a large market share for its niche. Suppose many high level AI services all use one particular low level AI service, that lower level AI service (or rather the humans or higher level AI services that have write access to it) could achieve a decisive strategic advantage by subverting the service in a way that causes a large fraction of all cognition on Earth (i.e., all the higher level services that depend on it) to start optimizing for its own interests. Multiple different lower level services could simultaneously have this option. (This was inspired by a comment from ryan_b.)
Logical vs physical risk aversion
Some types of risks may be more concerning than others because they are "logical risks" or highly correlated between Everett branches. Suppose Omega appears and says he is appearing in all Everett branches where some version of you exists and offering you the same choice: If you choose option A he will destroy the universe if the trillionth digit of pi equals the trillionth digit of e, and if you choose option B he will destroy the universe if a quantum RNG returns 0 when generating a random digit. It seems to me that option B is better because it ensures that there's no risk of all Everett branches being wiped out. See The Moral Status of Independent Identical Copies for my intuitions behind this. (How much more risk should we accept under option B until we're indifferent between the two options?)
More realistic examples of logical risks:
AI safety requires solving metaphilosophy.
AI safety requires very difficult global coordination.
Dangerous synthetic biology is easy.
Examples of physical risk:
global nuclear war
natural pandemic
asteroid strike
AI safety doesn't require very difficult global coordination but we fail to achieve sufficient coordination anyway for idiosyncratic reasons.
Combining oracles with human imitations
It seems very plausible that oracles/predictors and human imitations (whic...
view more