The Nonlinear Library: EA Forum
Education
EA - Incorporating and visualizing uncertainty in cost effectiveness analyses: A walkthrough using GiveWell's estimates for StrongMinds by Jamie Elsey
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Incorporating and visualizing uncertainty in cost effectiveness analyses: A walkthrough using GiveWell's estimates for StrongMinds, published by Jamie Elsey on November 7, 2023 on The Effective Altruism Forum.A common first step towards incorporating uncertainty into a cost effectiveness analysis (CEA) is to express not just a point estimate (i.e., a single number) for an input to the CEA, but to provide some indicator of uncertainty around that estimate. This might be termed an optimistic vs. pessimistic scenario, or as the lower and upper bounds of some confidence or uncertainty interval. A CEA is then performed by combining all of the optimistic inputs to create an optimistic final output, and allof the pessimistic inputs to create a final pessimistic output. I refer to this as an 'interval-based approach'. This can be contrasted with a fuller 'probabilistic approach', in which uncertainty is defined through the use of probabilistic distributions of values, which represent the range of possibilities we believe the different inputs can take. While many people know that a probabilistic approach circumvents shortcomings of an interval-based approach, they may not know where to even begin interms of what different distributions are possible, or the kinds of values they denote.I hope to address this in the current post and the accompanying application. Concretely, I aim to:Show how performing a CEA just using an interval-based approach can lead to a substantial overestimation of the uncertainty implied by one's initial inputs, and how using a probabilistic approach can correct this while also enabling additional insights and assessmentsIntroduce a new tool I have developed - called Distributr - that allows users to get more familiar and comfortable with a range of different distributions and the kinds of values they implyUse this tool to help generate a probabilistic approximation of the inputs GiveWell used in their assessment of Strongminds,[1] and perform a fuller probabilistic assessment based upon these inputsShow how this can be done without needing to code, using Distributr and a simple spreadsheetI ultimately hope to help the reader to feel more capable and confident in the possibility of incorporating uncertainty into their own cost effectiveness analyses.Propagating uncertainty and the value of moving beyond an interval-based approachCost effectiveness analysis involves coming up with a model of how various different factors come together to determine both how effective some intervention is, and the costs of its delivery. For example, when we think about distributing bed nets for malaria prevention, we might consider how the cost of delivery can vary across different regions, how the effects of bed net delivery will depend on the likelihood that people use the bed nets for their intended purpose, and the probability thatrecipients will install the bed nets properly. These and other factors all come together to produce an estimate of the cost effectiveness of an intervention, which will depend on the values we ascribe to the various inputs.One way that a researcher might seek to express uncertainty in these inputs is by placing reasonable upper and lower bounds on their estimates for each of them. The researcher might then seek to propagate this uncertainty in the inputs into the anticipated uncertainty in the outputs by performing the same cost effectiveness calculations as they did on their point estimates on their upper bounds, and on their lower bounds, thereby producing corresponding upper and lower bounds on the final cost effectiveness.An example of an interval-based approach is GiveWell'sassessment of Happier Lives Institute's CEA for StrongMinds. The purpose of this post is not to provide an independent evaluation of Strongminds, nor i...
Create your
podcast in
minutes
It is Free