From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Fri Jul 11 2003 - 00:50:41 MDT
Robin Hanson wrote:
> At 10:02 PM 7/7/2003 -0400, Eliezer S. Yudkowsky wrote:
>
>> I am objecting to your phraseology here because it seems to
>> preemptively settle the issue by identifying people's built-in
>> emotional reinforcers as their real wants, while dismissing their
>> cognitively held hopes and aspirations and personal philosophy as a
>> foreign force interfering with their true selves. One could just as
>> easily view the system from the opposite perspective.
>
> I do agree that this is a subtle question, whose answer is not
> immediately obvious. The topic of self-deception can be a conceptual
> morass, as our usual anchors are not as available. Nevertheless, I do
> want to argue for the claim you find questionable.
>
> If people have contradictory beliefs, how can we say which ones are the
> "real" beliefs? By reference to the basic schema of self-deception,
> in which the real beliefs tend to determine less visible actions with
> more fundamental consequences, and the false beliefs tend to determine
> what we tell others and ourselves about ourselves, and the most
> socially visible actions with the least fundamental consequences.
When you deal with a human, you are dealing with at least three
intertwined goal-ish systems.
There's the hardware wiring that determines which thoughts and experiences
are pleasurably reinforced, or painful. This wiring is by no means
simple; it has more modes of action than just "pleasure" or "pain",
despite that usual division. Still, it overall tends to shift people's
thoughts in a goal-ish way.
There are people's declaratively held purposes and goals.
And there is evolution, which is also a goal-ish system, and which acts to
select among {heriditary information in the initial conditions of the
other two systems} based solely on the correlation between {the ultimate
outcome of the interaction of thought and emotion} and the number of
surviving grandchildren.
If you go around determining real beliefs by the *outcomes* of people's
actions, you run the risk of confusing evolutionary motives with cognitive
ones, the classic mistake in evolutionary psychology. Every atom of
complex hereditary information in humans was constructed by evolution
based on the sole and only criterion of reproductive success, yet the
explicit wish for children is only one among hundreds of explicitly
represented cognitive reinforcers, woven by memes and philosophies into
individually unique personal philosophies.
> People may want to produce art to gain social approval, wealth, mates,
> etc., but want to be thought of as doing it just for the art. People
> may want to advocate positions that make them seem clever and
> compassionate, and get them social accepted by the right folks, but
> want to be thought of as wanting only to tell the truth. People may
> want to be unfair when serving as a neutral judge, but want to thought
> of as fair. These examples should be familiar to everyone; is there
> anyone here to whom these are "news"?
Aren't these instances of the classic error?
People have emotional hardware and cognitive representations leading them
to be devoted to art for its own sake, because people who, in the past,
possessed the hardware for art, or fell into the philosophical attractor
for personal philosophies exalting art as the result of their innate
tendencies and biases, gained social approval, wealth, and mates. And
even these are proximal goals, don't forget; all evolution cared about was
the grandchildren.
People have emotions leading them to honestly advocate positions that
people applaud as clever and compassionate, because their ancestors who
did the same were socially accepted by the right folks. They say they're
just telling the truth, and they *are* just telling the truth, because in
the evolutionary arms race of liars and lie-detectors, it's easier and
more reliable to deceive the phenotype you're building than to have your
phenotype deliberately deceive other genomes' phenotypes. People think
they're honest and they are, but what think is the truth is output by
biased reasoning hardware that was constructed according to the sole and
only criterion of the number of surviving grandchildren to which that
hardware's outputs led.
Over and over, people seize power "for the good of the community". Are
they being self-deceptive? No, they are being evolutionary deceived. The
universality of this motif suggests to me that it is the result of
selection over evolutionary time among imperfectly deceptive social
organisms who argued linguistically about each other's motives in adaptive
political contexts. There is a warp applied between the selection of
goals and the selection of subgoals, because both operations are carried
out by evolutionarily constructed, emotionally influenced brainware. I
hypothesize (this is not a generally accepted truth in evolutionary
psychology, as far as I know) the following evolutionarily constructed
template of cognitive, emotional, and socioenvironmental interactions,
which was statistically likely to lead to reproduction in the ancestral
environment, and which today has become a famous motif in large-scale
human events.
1. People (emotional hardware) care about the community they live in.
(See "Any Animal Whatever" by Flack and de Waal for an argument that
"community concern can be observed in primates.)
2. People (emotional hardware) react to perceived abuses of power by the
tribal chief with righteous indignation. Note that the evolutionary force
underlying the rise of this emotional perception rests on the possibility
of installing a less abusive tribal chief *or* the possibility of
achieving higher social status in the post-revolutionary order, *not* the
factual degree of the tribal chief's abuses, nor the degree to which
opposing the tribal chief benefits the *community*.
3. People who are reacting to perceived abuses of power by the tribal
chief with righteous indignation, and see the opportunity to personally
overthrow the tribal chief or do so with a clique of friends, are biased
to see this as a *good subgoal* of serving the community. This is the
first appearance of an explicit rationalization warp in the hypothesis.
So what's a rationalization warp? Okay, as all good decision theorists
know, you get the desirability D(a) of an action A by summing over the
utility times the probability, U(x)P(x|a), for all x of interest (i.e.,
that the system can afford to compute). What I'm saying is that in this
case, the adaptive bias is being applied to the computation p(x|a) rather
than U(x). It's not that people "really want" to take over the tribe.
They really want to promote the good of the community, and they really
believe that they can do so by taking over the tribe. It would not even
be accurate to say that the people are being deceived about their "real
motives"; they are being deceived about which means correspond to which
ends. In other words, the rationalization warp looks like this:
Evolutionary end, i.e., subgoal of reproduction: Y. (Status, power...)
Cognitively held end which is socially acceptable: X. (Good of the tribe.)
So evolution is applying a bias to the computation of p(x|a) such that
people find A to appear very plausible as a subgoal of X, given that it is
*actually* a subgoal of Y. In other words, p(x|a) will be computed as
higher than it should be, given that p(y|a) is *in fact* high.
This is not being carried out by a deliberative process that evaluates
p(y|a) before evaluating p(x|a), any more than people stop to think about
their number of surviving grandchildren before eating tasty berries or
tasty candy bars. There may be emotional ties from statistical correlates
of p(y|a) to the evaluation of p(x|a), and so on, but there's not a
deliberate deception involved. It's an emergent outcome of evolutionary
selection on hereditary information that biases p(x|a) and correlates to
reproductive success after *everything*, the whole deal, is finished.
The end result of this is that acting on the evolutionary biased
computation U(x)p(x|a) tends to actually maximize Y instead of X,
providing that all goes as planned and you are in the ancestral
environment. Stalin, as far as I know, did not have a spectacularly large
number of surviving grandchildren - was no great success from an
evolutionary standpoint. Had he just been in one tribe, though, less
educated and literate, with less grand plans, he probably would have done
better from a reproductive standpoint. The point is that this is the
machinery that sent Stalin bad. And Robespierre. And... so on. See here
the roots of the human species; it is still better than not caring at all.
Step 4, of course, is another rationalization warp to believing that,
having taken power, people are then emotionally and cognitively biased to
find plausible the proposition that power must be concentrated into their
own hands, again for the good of the community.
That this template has been carried out successfully so frequently in our
own, literate, suspicious times, to say nothing of a tribe of
non-timebinding hunter-gatherers, suggests that it has been operating as a
successful strategy over evolutionary time. Evolution constructs
phenotypes to believe that they are acting for the good of the tribe,
because that is what wins public support.
But the point is that people *really do want* to help others, to create
art, to be compassionate. It's the whole reason why we find the
evolutionary puppet strings so horrifying once we become aware of them;
evolution didn't plan for that, any more than it planned for the
introduction of contraceptives. Ultimately DNA is just an
information-theoretic history of *who did in fact* reproduce, not a plan
to make organisms reproduce in the future.
We genuinely *are* altruists. Tainted by shadow, perhaps, but true
altruists nonetheless. Evolution, acting purely on the criterion of
covariance of hereditary information in dynamic cognitive processes with
reproductive success, managed to construct people who truly care about
each other, as ends in themselves, and not as means.
I have often thought that the rationalization warp is the greatest mixed
blessing in the entire history of the universe.
-- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Fri Jul 11 2003 - 01:03:14 MDT