Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio.
This is: Urges vs. Goals: The analogy to anticipation and belief , published by AnnaSalamon on the AI Alignment Forum.
Partially in response to: The curse of identity
Related to: Humans are not automatically strategic, That other kind of status, Approving reinforces low-effort behaviors.
Joe studies long hours, and often prides himself on how driven he is to make something of himself. But in the actual moments of his studying, Joe often looks out the window, doodles, or drags his eyes over the text while his mind wanders. Someone sent him a link to which college majors lead to the greatest lifetime earnings, and he didn't get around to reading that either. Shall we say that Joe doesn't really care about making something of himself?
The Inuit may not have 47 words for snow, but Less Wrongers do have at least two words for belief. We find it necessary to distinguish between:
Anticipations, what we actually expect to see happen;
Professed beliefs, the set of things we tell ourselves we “believe”, based partly on deliberate/verbal thought.
This distinction helps explain how an atheistic rationalist can still get spooked in a haunted house; how someone can “believe” they’re good at chess while avoiding games that might threaten that belief [1]; and why Eliezer had to actually crash a car before he viscerally understood what his physics books tried to tell him about stopping distance going up with the square of driving speed. (I helped Anna revise this - EY.)
A lot of our community technique goes into either (1) dealing with "beliefs" being an evolutionarily recent system, such that our "beliefs" often end up far screwier than our actual anticipations; or (2) trying to get our anticipations to align with more evidence-informed beliefs.
And analogously - this analogy is arguably obvious, but it's deep, useful, and easy to overlook in its implications - there seem to be two major kinds of wanting:
Urges: concrete emotional pulls, produced in System 1's perceptual / autonomic processes
(my urge to drink the steaming hot cocoa in front of me; my urge to avoid embarrassment by having something to add to my accomplishments log)
Goals: things we tell ourselves we’re aiming at, within deliberate/verbal thought and planning
(I have a goal to exercise three times a week; I have a goal to reduce existential risk)
Implication 1: You can import a lot of technique for "checking for screwy beliefs" into "checking for screwy goals".
Urges, like anticipations, are relatively perceptual-level and automatic. They're harder to reshape and they're also harder to completely screw up. In contrast, the flexible, recent "goals" system can easily acquire goals that are wildly detached from what we actually do, wildly detached from any positive consequences, or both. Some techniques you can port straight over from "checking for screwy beliefs" to "checking for screwy goals" include:
The fundamental:
"What's the positive consequence?" This is the equivalent of "What's the evidence?" for beliefs. All the other cases involve not asking it, or not asking hard enough.
The Hansonian:
Goals as clothes / goals as tribal affiliation: “We are people who have free software (/ communism / rationality / whatever) as our goal”. Before you install Linux, do you think "What's the positive consequence of installing Linux?" or does it just seem like the sort of thing a free-software-supporter would do? (EY says: What positive consequence is achieved by marching in an Occupy Wall Street march? Can you remember anyone stating one, throughout the whole affair - "if we march, X will happen because of Y"?)
Goals as a signal of one’s value as an ally: Sheila insists that she wants to get a job. We inspect her situation and she's not trying very hard to get a job. But she's in debt to a lot of her friends and is borrowing more to li...
view more