From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Fri Jul 11 2003 - 10:51:19 MDT
Robin Hanson wrote:
>
> [FYI, there are many papers/books in philosophy and psychology, and
> fewer evolutionary psychology, on self-deception. And classic
> literature has many things to say about it. (I'm co-hosting a small
> invitation-only interdisciplinary conference on the subject here in
> October.) This is exactly the sort of topic that extropians shouldn't
> try to reinvent before surveying the existing literature. RH]
Keywords? Names? Paper titles? Google fodder would be appreciated. I
expect my current knowledge to substantially exceed the state of the art
in philosophy, but I am interested in any bodies of experimental evidence
from psychology, or any work in evolutionary psychology.
I'll toss in Latane-Darley, Zimbardo, and Milgram to get the pot started,
though those are simple errors of moral self-image rather than complex
self-deceptions. Incidentally, would you say that the Milgram experiment
reveals that people "really" want to go along with authority? I wouldn't
read it that way at all.
> There are many possible mechanisms of self-deception, one of which is as
> you describe.
Yes. It was a deliberate eleemosynary oversimplification of a complex
system. I've managed to introspectively catch rationalization mechanisms
in four major classes so far:
A: Deliberate rationalization.
B: Political rationalization supported by emotional mechanisms (of the
type I described).
C: Emergent outcomes of the interface between the pleasure-pain
anticipation system and deliberation, possibly subject to evolutionary
optimization for adaptive rationalization biases.
D: Emergent outcomes of support recruitment mechanisms, again possible
subject to evolutionary optimization of hereditary adaptively relevant
information.
A&B are well-known, C&D rather less so.
> The whole system of calculating actions from goals and
> beliefs has many entry points for motivational bias.
Not to blame you personally, but I think the idea of "motivational biases"
is one of those amazingly oversimplified ideas in the philosophy of
cognitive psychology. I was familiar with the concept from old papers,
but I'm surprised it's still around today.
There are emotional biases which reinforce or disinforce specific thoughts
based on how the emotions bind to the mental imagery in question -
thoughts which are painful or pleasurable in complex ways, causing people
to flinch away, or flinch toward them.
There are biases which affect perceived truth, acting as support or
antisupport.
Evolutionary selection pressures will act on heritable variations that
correlate with the outcome of adaptively relevant cognitive processes in
arbitrary other ways.
The idea that people's high-level, consciously held motivations have the
ability to manipulate their internal information is naive. People do not
have that degree of reflective access; they are not seed AIs. It is an
extreme oversimplification of what is actually going on. In fact, it is
simply false. You cannot deduce or attribute internal motivations in that
way, and the real process is not one of intelligent planning at all
(internal or external), but rather executing adaptations.
> Some of these
> points are focused more on beliefs, others more on goals. The system
> also has many rich layers of protection from situations that might to
> remove such bias. We are much better at spotting self-deception in
> others than in ourselves, because we have ways of avoiding looking at
> the relevant evidence, and rationalizing it away when others point it out.
"We" have ways? Say rather that our cognitive processes contain
tendencies to do so, some emergent, some adaptive, some that started as
emergent but have now been evolutionary fixed.
> There are two classic ways to determine what people "really" want. One
> is based on "happiness," the other on informed choice. In your example,
> the happiness metric asks if people are happier when they get
> status/power versus when they actually do good for the tribe, without
> getting such status/power. The informed choice metric asks whether
> people would choose status/power or good for the tribe if they were
> briefly and privately informed, via enough evidence to typically be
> persuasive to a neutral observer, that this is actually what they are
> choosing between. (I say briefly so that they can quickly forget the
> conversation every happened and revert to the state where they actually
> believe they are doing good for the tribe.)
Why do you think the informed choice metric runs this way? I would guess
the opposite - that most people, asked to make a deliberate choice between
status/power and the good of the tribe, would either choose the good of
the tribe, or feel guilty about not doing so (implying that their
renormalized volition would move in the direction of choosing the good of
the tribe).
Also, if you are talking about a major upheaval in the belief system there
is no such thing as a "briefly informed" choice - you have to extrapolate
major changes within the person's volition, including reactions to many
different changes and compounded choices about those reactions. Seeing
evolution's puppet strings for the first time would be a wrenching
upheaval if ever there was one.
I would reject both metrics as adequate theories of volition or even
instantaneous want, though the informed choice metric comes closer.
> My reading of human behavior in most of the contexts in which
> self-deception is an issue is that most people are happier with the
> status/power type option, versus the doing good for the tribe type
> option, and that this is what they usually actually choose when briefly
> and privately informed. I agree that most people do believe that they
> want to do good for the tribe. My claim is that this belief is
> relatively isolated and ineffectual; it is allowed to influence what
> people say and some actions that influence social perceptions, but is
> otherwise little used.
What is the justification for taking such a dark view of things? Why make
this claim? It is the sort of thing I would be very wary of if it
appeared in my own mind, for reasons that are probably obvious to you as
well. The folk picture of people struggling between their high moral
aspirations and their inner demons is, as far as I can tell, pretty much
correct. High moral aspirations motivate some people strongly and some
people weakly - there is wide variation, certainly in final outcomes,
probably even in innate strength - but they are far from the epiphenomena
you seem to be describing.
People die saving unrelated children. Is that a lie? It seems to me that
the folk psychology of what goes through a person's head in that situation
is probably essentially correct. In many/most cases it will have nothing
whatsoever to do with personal reputation, regardless of what the
evolutionary selection pressure may derive from. Individual organisms are
adaptation-executers, not fitness-maximizers. Evolutionary selection
pressures do not necessarily translate directly into cognitively
represented goals. For linguistic political organisms there may be
specific selection pressures *against* translating a selection pressure
into a cognitively represented goal.
My own claim is that if you asked people what they cared about, what
mattered most to them, what kind of person they wanted to be, they would
say that the altruism is the most important part of them. Since that is
what they tell me, why should I - or any friend of theirs - contradict
them? The deliberative system may sometimes be weak, but it is ultimately
in charge - or at least is the center I look to, to determine how to find
the "person" I want to "help".
When I look at a person, I tend to see the center of that person in the
kind of final choices that control their words, or raises their hand, and
so on. It's not a perfect definition, but it's where I'd start looking.
The one who takes control of the vocal cords and says to me "I want to be
a better person"; that is the volition, that is the person I am talking to
at that moment, and if I want to know which parts of the rest of the mind
are worth talking to, I'll begin by asking that voice.
> Consider that today most people around here would say that the think the
> world is their tribe, but they give almost no money to help poor people
> in Africa, even when they believe that such aid would make the world a
> better place overall.
Then why am I, who know more of my own motives, who see more of puppet
strings, more effectively altruistic and not less? Was I born with an
unusually large helping of altruistic emotions? Why do I need to add this
extra postulate, when being born with an unusually large helping of
intelligence, into an era with an unancestral knowledge of evolutionary
psychology, seems like a quite sufficient explanation? (Though I should
expect that people vary more widely from me than I expect - the consensus
bias.) How does your view of human psychology even *allow for* genuine
dedicated altruists?
The person I am talking to is the process that says "The world is my
tribe." The cognitive mechanisms that produced that vocal output, that
vocal decision, are where I begin to construct the definition of volition.
Why should I begin anywhere else?
My father has a doctorate in physics and is an Orthodox Jew. He somehow
manages not to see any conflict between the Big Bang and Darwin, and the
Hexameron. Human thoughts are too complicated, too fragile, and too weak
for us to *expect* self-consistency or draw conclusions from its lack. My
father is neither "not really an Orthodox Jew", nor "not really a
physicist". He is simply grossly inconsistent.
(Also, as has been described on this mailing list in the past, money sent
to Africa appears to be incrementally futile. Pick a different example?)
> When would you say that a corporation that consistently continues to
> pollute, even though its PR denies it, "really wants" to not pollute?
I wouldn't *use* the term "want" for a corporation. I consider it a term
of art in Friendly AI with respect to constructing an interpretation of
someone's volition, and this term is not applicable to corporations, or
would require tremendous reworking in order to be applicable. Also the
above does not seem to be a good example of self-deception, just simple
deliberate lying.
-- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Fri Jul 11 2003 - 11:05:54 MDT