Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.
This is: Coherent decisions imply consistent utilities , published by Eliezer Yudkowsky on the AI Alignment Forum.
(Written for Arbital in 2017.)
Introduction to the introduction: Why expected utility?
So we're talking about how to make good decisions, or the idea of 'bounded rationality', or what sufficiently advanced Artificial Intelligences might be like; and somebody starts dragging up the concepts of 'expected utility' or 'utility functions'.
And before we even ask what those are, we might first ask, Why?
There's a mathematical formalism, 'expected utility', that some people invented to talk about making decisions. This formalism is very academically popular, and appears in all the textbooks.
But so what? Why is that necessarily the best way of making decisions under every kind of circumstance? Why would an Artificial Intelligence care what's academically popular? Maybe there's some better way of thinking about rational agency? Heck, why is this formalism popular in the first place?
We can ask the same kinds of questions about probability theory:
Okay, we have this mathematical formalism in which the chance that X happens, aka
P
X
, plus the chance that X doesn't happen, aka
P
¬
X
, must be represented in a way that makes the two quantities sum to unity:
P
X
P
¬
X
1
That formalism for probability has some neat mathematical properties. But so what? Why should the best way of reasoning about a messy, uncertain world have neat properties? Why shouldn't an agent reason about 'how likely is that' using something completely unlike probabilities? How do you know a sufficiently advanced Artificial Intelligence would reason in probabilities? You haven't seen an AI, so what do you think you know and how do you think you know it?
That entirely reasonable question is what this introduction tries to answer. There are, indeed, excellent reasons beyond academic habit and mathematical convenience for why we would by default invoke 'expected utility' and 'probability theory' to think about good human decisions, talk about rational agency, or reason about sufficiently advanced AIs.
The broad form of the answer seems easier to show than to tell, so we'll just plunge straight in.
Why not circular preferences?
De gustibus non est disputandum, goes the proverb; matters of taste cannot be disputed. If I like onions on my pizza and you like pineapple, it's not that one of us is right and one of us is wrong. We just prefer different pizza toppings.
Well, but suppose I declare to you that I simultaneously:
Prefer onions to pineapple on my pizza.
Prefer pineapple to mushrooms on my pizza.
Prefer mushrooms to onions on my pizza.
If we use
P
to denote my pizza preferences, with
X
P
Y
denoting that I prefer X to Y, then I am declaring:
onions
P
pineapple
P
mushrooms
P
onions
That sounds strange, to be sure. But is there anything wrong with that? Can we disputandum it?
We used the math symbol
which denotes an ordering. If we ask whether
P
can be an ordering, it naughtily violates the standard transitivity axiom
x
y
y
z
⟹
x
z
Okay, so then maybe we shouldn't have used the symbol
P
or called it an ordering. Why is that necessarily bad?
We can try to imagine each pizza as having a numerical score denoting how much I like it. In that case, there's no way we could assign consistent numbers
x
y
z
to those three pizza toppings such that
x
y
z
x
So maybe I don't assign numbers to my pizza. Why is that so awful?
Are there any grounds besides "we like a certain mathematical formalism and your choices don't fit into our math," on which to criticize my three simultaneous preferences?
(Feel free to try to answer this yourself before continuing...)
Click here to reveal and continue:
Suppose I tell you that I prefer pineapple to mushrooms on my pizza. Suppose you're about to give me a slice of mushroom pizza; but by...
view more