Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.
This is: A personal take on longtermist AI governance, published by lukeprog on the Effective Altruism Forum.
Several months ago, I summarized Open Philanthropy's work on AI governance (which I lead) for a general audience here. In this post, I elaborate my thinking on AI governance in more detail for people who are familiar with effective altruism, longtermism, existential risk, and related topics. Without that context, much of what I say below may be hard to understand, and easy to misunderstand.
These are my personal views, and don't necessarily reflect Open Phil's views, or the views of other individuals at Open Phil.
In this post, I
briefly recap the key points of my previous post,
explain what I see as the key bottlenecks in the space, and
share my current opinions about how people sympathetic to longtermism and/or AI existential risk mitigation can best contribute today.
Open Phil's AI governance work so far (a recap)
First, some key points from my previous post:
In practice, Open Phil's grantmaking in Potential Risks from Advanced Artificial Intelligence is split in two:
One part is our grantmaking in "AI alignment," defined here as "the problem of creating AI systems that will reliably do what their designers want them to do even when AI systems become much more capable than their designers across a broad range of tasks."[1]
The second part, which I lead, is our grantmaking in "AI governance," defined here as "local and global norms, policies, laws, processes, politics, and institutions (not just governments) that will affect social outcomes from the development and deployment of AI systems."
Our AI focus area is part of our longtermism-motivated portfolio of grants,[2] and we focus on AI alignment and AI governance grantmaking that seems especially helpful from a longtermist perspective. On the governance side, I sometimes refer to this longtermism-motivated subset of work as "transformative AI governance" for relative concreteness, but a more precise concept for this subset of work is "longtermist AI governance."[3]
It's difficult to know which “intermediate goals” we could pursue that, if achieved, would clearly increase the odds of eventual good outcomes from transformative AI (from a longtermist perspective). As such, our grantmaking so far tends to focus on:
.research that can help clarify how AI technologies may develop over time, and which intermediate goals are worth pursuing.
.research and advocacy aimed at the few intermediate goals we've come to think are clearly worth pursuing, such as particular lines of AI alignment research, and creating greater awareness of the difficulty of achieving high assurance in the safety and security of increasingly complex and capable AI systems.
.broad field-building activities, for example scholarships, career advice for people interested in the space, professional networks, etc.[4]
.training, advice, and other support for actors with plausible future impact on transformative AI outcomes, as always with a focus on work that seems most helpful from a longtermist perspective.
Key bottlenecks
Since our AI governance grantmaking began in ~2015,[5] we have struggled to find high-ROI grantmaking opportunities that would allow us to move grant money into this space as quickly as we'd like to.[6] As I see it, there are three key bottlenecks to our AI governance grantmaking.
Bottleneck #1: There are very few longtermism-sympathetic people in the world,[7] and even fewer with the specific interests, skills, and experience to contribute to longtermist AI governance issues.
As a result, the vast majority of our AI governance grantmaking has supported work by people who are (as far as I know) not sympathetic to longtermism (and may have never heard of it). However, it's been difficult to find high-ROI grantmaking opportunities of this...
view more