The Nonlinear Library: EA Forum
Education
EA - Survey on the acceleration risks of our new RFPs to study LLM capabilities by Ajeya
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Survey on the acceleration risks of our new RFPs to study LLM capabilities, published by Ajeya on November 14, 2023 on The Effective Altruism Forum.My team at Open Philanthropy just launched two requests for proposals:Proposals tocreate benchmarks measuring how wellLLM agents (likeAutoGPT) perform on difficult real-world tasks, similar torecent work by ARC Evals.[1]Proposals tostudy and/or forecast the near-term real-world capabilities and impacts of LLMs and systems built from LLMs more broadly.I think creating a shared scientific understanding of where LLMs are at has large benefits, but it can also accelerate AI capabilities: for example, it might demonstrate possible commercial use cases and spark more investment, or it might allow researchers to more effectively iterate on architectures or training processes. Other things being equal, I think acceleration is harmful becausewe're not ready for very powerful AI systems - but I believe the benefits outweigh these costs in expectation, and think better measurements of LLM capabilities are net-positive and important.To get a sense for whether acting on this belief by launching these two RFPs would constitute falling prey tothe unilateralist's curse, I senta survey about whether funding this work would be net-positive or net-negative to 47 relatively senior people who have been full-time working on AI x-risk reduction for multiple years and have likely thought about the risks and benefits of sharing information about AI capabilities.Out of the 47 people who received the survey, 30 people (64%) responded. Of those, 25 out of 30 said they were "Positive" or "Lean positive" on the RFP, and only 1 person said they were "Lean negative," with no one saying they were "Negative." The remaining four people said they had "No idea," meaning that 29 out of 30 respondents (97%) would not vote to stop the RFPs from happening. With that said, many respondents (~37%) felt torn about the question or considered it complicated.The rest of this post provides more detail onthe information that the survey-takers received andthe survey results (including sharing answers from those respondents who gave permission to share).The information that was sent to the survey-takersThe survey-takers received the below email, which links to aone-pager on the risks and benefits of these RFPs, and afour-pager (written in late July and early August) about the sorts of projects I expected to fund. After the survey, the latter document evolved into the public-facing RFPs here and here.Subject: [by Sep 8] Survey on whether measuring AI capabilities is harmfulHi,I want to launch a request for proposals asking researchers to produce better measurements of the real-world capabilities of systems composed out of LLMs (similar to the recent work done byARC evals).I expect this work to shorten timelines to superhuman AI, but I think the harm from this is outweighed by the benefits of convincing people of short timelines (if that's true) and enabling a regime of precautions gated to capabilities. Seethis 1-pager for more discussion. You can also skim myproject description (~4 pages) to get a better idea of the kinds of grants we might fund, though it's not essential reading (especially if you're broadly familiar with ARC evals).Please fill outthis short survey on whether you think this project is net-positive or net-negative by EOD Fri Sep 8.I'm sending this survey to a large number of relatively senior people who have been full-time working on AI x-risk reduction for multiple years and have likely thought about the risks and benefits of sharing information about AI capabilities. The primary intention of this survey is to check whether going ahead with this RFP would constitute falling prey to the unilateralist's curse (i.e., to check ...
Create your
podcast in
minutes
It is Free