Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.
This is: Mitigating x-risk through modularity, published by Toby Newberry on the AI Alignment Forum.
Write a Review
0. Abstract and main claims
[Skip this section if you want a minimally repetitive reading experience :)]
This post discusses an approach to x-risk management called ‘mitigation through modularity’. Roughly speaking, the approach involves decorrelating risks, as opposed to minimising them; its slogan would be "don’t put all your eggs in one basket", rather than "don’t drop the basket full of eggs".
As you might suspect (given it can be summarised by a well-known proverb), this kind of thinking recurs in lots of places: diversification in finance, redundancy in software development, etc. To some extent, it has also already been applied to x-risk - most notably in the context of space settlement. But my impression is that its application to x-risk so far leaves quite a bit of value on the table (in particular, space settlement isn’t the only kind of strategy this approach points to). Below, I draw together what’s been said about the approach already, present a new frame through which this work could usefully be extended, and make a number of more speculative suggestions. My central claims are:
1. In principle, the strategy discussed below (‘mitigation through modularity’) is a robust way of achieving existential security.[1]
2. In practice, it is unlikely to be effective against:
unaligned AI; and
some other risks.[2]
3. It is likely to be (at least somewhat) effective against:
asteroid or comet impact;
supervolcanic eruption;
nuclear war;
climate change;
other environmental damage;
naturally arising pandemics;
engineered pandemics; and
some other risks.[3]
4. Near-term space settlement, as one possible implementation of this strategy, is not sufficient for existential security, and unlikely to be cost-effective.
5. One promising reframing of the strategy is to take a ‘risk-first, rather than a ‘proposal-first’, approach. By this, I mean taking a specific risk (e.g. climate change), and then thinking of ways to decorrelate this across human civilisation - rather than taking a specific proposal (e.g. Martian settlement), and then thinking of the ways it might reduce risk.
1. Mitigation through modularity
Superficially, certain species of butterfly appear to be quite bad at survival. They live in small, isolated groups (around different meadows, for example). They are relatively poor flyers, meaning reliable inter-group travel is not an option. In addition, the groups are individually vulnerable, such that each one can be obliterated by a passing squall, a disease affecting host plants, or just an unfortunate role of the Darwinian die.[4] And yet, the butterflies persist.
We can explain their surprising resilience through the use of metapopulation models. In such a model, the total butterfly population is divided into numerous subpopulations, representing the isolated groups. For any given subpopulation, there is some risk of local extinction (e.g. squall). If the subpopulations were fully disconnected from one another, this process would eventually lead to the species’ global extinction, as each group of butterflies meets with its private doom. In practice, however, the metapopulation achieves global stability. While the butterflies cannot routinely travel between different meadows, there is nonetheless a small amount of exchange between subpopulations: every so often, a butterfly will be blown from one meadow to the next, or will happen to fly an unusually long distance in one direction. As a result, when one subpopulation becomes extinct, the area it previously occupied will be resettled by accidental pioneers of this sort. Moreover, the rate at which such resettlement events occur more than balances the rate of local extinction events. Even though each subpopulat...
view more