From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Fri Jul 11 2003 - 22:27:19 MDT
Robin Hanson wrote:
> On 7/11/2003, Eliezer Yudkowsky wrote:
>
>>> ... many papers/books in philosophy and psychology, and fewer in
>>> evolutionary psychology, on self-deception. ...
>>
>> Keywords? Names? Paper titles? Google fodder would be appreciated.
>> I expect my current knowledge to substantially exceed the state of the
>> art in philosophy, but I am interested in any bodies of experimental
>> evidence from psychology, or any work in evolutionary psychology.
>
> Most of what academia has produced isn't yet on the web.
Most of it isn't, but enough of it usually is to get a quick snapshot of a
novel subfield, provided you're already familiar with most of the nearby
fields. To check my initial expectations about what the field would look
like, I Googled and used:
http://cogprints.ecs.soton.ac.uk/archive/00000295/00/MELE.html
http://www.philosophy.stir.ac.uk/cnw/self-deception.htm
Note Mele's commonsense emphasis that one need not have an explicit
cognitive representation of a self-deception goal in order to become
self-deceived (section 3). He lists mechanisms such as negative
misinterpretation, positive misinterpretation, selective
focusing/attending, and selective evidence-gathering, and concludes "In
none of the examples offered does the person hold the true belief that ~p
and then intentionally bring it about that he or she believes that p. Yet,
assuming that my hypothetical agents acquire relevant false beliefs in the
ways described, these are garden-variety instances of self-deception."
I am making a similar objection to your claims about what people *really
want*. It's a startling claim and it sets off people's cheater-detectors,
but I don't think it's true. The gross inconsistency between beliefs and
actions does not indicate that "what people believe they want" is not
"what they really want", it means that people are grossly inconsistent in
ways that have been subject to natural selection on heritable variations.
> In phil see
> Mele, in psych see Paulhaus, in evol psych see Trivers (all cited in
> http://hanson.gmu.edu/deceive.pdf). (Some experts coming to my
> conference are Robert Frank, Jay Hamilton, Dennis Krebs, Robert Kurzban,
> Al Mele, Harold Sackeim, Robert Trivers, Bill von Hippel.) (I expect you
> over-estimate your philosophy abilities, but then most people do.)
Without an understanding of how general intelligence works and how it
evolved, nothing in philosophy makes any sense; in particular, without a
functional decomposition of intelligence, philosophical modeling carves
the mind at the wrong joints and then breaks down. And that is why I
expect my knowledge to exceed the state of the art in philosophy. I can
talk about emotional reinforcement mechanisms for real experiences and
imagined experiences; declarative beliefs with emotional valencies;
deliberative decision-making; how past experience and emotion influences
which subjective future experiences are visualized during deliberation;
impulses that override deliberation; the pressure of consistency with
abstract beliefs considered as a special case of emotional pressure... and
so on. But my theory of *volition* is a constructive one - it does not
correspond to any natural category in the mind; there is no hidden
register where you can look up what people "really want". Yet most
theoreticians, faced with a problem like this, would start with "want" as
a simple fact, and try to construct more detailed accounts of cognitive
processes as complex facts phrased in terms of "wanting", *exactly the
wrong approach*.
"Studying adaptive gross inconsistencies" seems like a fine way to phrase
the problem - it encourages looking for specific cognitive mechanisms and
specific pressures exerted at specific points in the interaction of those
mechanisms. It even has the nice overtone of "Surprise! Incompetence!"
that helped the field of heuristics and biases become popular. But when
you start telling me that people do not really want what they say they
want, I become worried for several reasons.
One, you're assuming that wanting-ness is a simple natural category, which
distracts attention away from the task of arriving at a functional
decomposition of decision-making into a surprisingly weird evolutionary
layer-cake with human icing on top.
Two, you're setting off people's cheater-detectors in a way that invokes
an implicit theory of mind that I think is oversimplified, false-to-fact,
and carves the mind at the wrong joints; just as the idea that
self-deception starts with believing ~p and follows a deliberate plan to
fool oneself is oversimplified, false-to-fact, and carves the mind at the
wrong joints.
Three, you're making a preemptive philosophical judgment about the nature
of true personal identity and the construction of volition that (a) may
ultimately be a matter for personal choice (b) has serious consequences
(c) I strongly disagree with.
Four: back when the field of game theory was getting started, people
discovered the Prisoner's Dilemma, and then went around announcing that
there was no possible rational reason to cooperate and that anyone who did
so was being merely sentimental. They congratulated themselves on being
so tough-minded. Later, they realized the problem was iterative and
invented the concept of reciprocal cooperation. But *meanwhile*, game
theorists who had *not* heard of this result would *actually defect on the
iterated PD*. They would defect on the Prisoner's Dilemma! (See, for
example, the Flood-Dresher experiment.) Congratulating yourself on a
tough-minded view of human nature can render you *incompetent* on problems
that the merest hunter-gatherer knows the solution to - it is *not*,
historically speaking, a reliable heuristic. Now you plan to tell people
that nobody really cares about altruism, and that all idealism is a lie
and a patina. I think this is just as damaging as telling people that the
only possible rational action is to defect on the (iterated!) Prisoner's
Dilemma. Of course I also think that "all idealism is a lie" will turn
out to be just as *wrong* as "it is rational to defect on the (iterated)
Prisoner's Dilemma" - if it were true, I would tell you to go ahead and
report it no matter the consequences. Why? Because I am a true idealist.
And so are you, and so probably are most people to greater or lesser
degrees.
Five, there's no good reason to mess with points 1-4 - they are totally
extraneous to the real substance of your theory. You can declare yourself
to be studying "adaptive gross inconsistencies in moral belief and real
actions", and get the benefit of intersection with both evolutionary
psychology and experimental psychology, without ever needing to take a
stance about what people "really" "want", or presuming a particular
functional decomposition of the mechanisms involved. Modularize away
those contrarian points...
>> Also, if you are talking about a major upheaval in the belief system
>> there is no such thing as a "briefly informed" choice - you have to
>> extrapolate major changes within the person's volition, including
>> reactions to many different changes and compounded choices about those
>> reactions.
>
> I limited my claim to the sorts of things we've seen. We've seen how
> people behave if briefly informed. We haven't seen your ideal of fully
> informed post-upheaval choice.
When have we seen how people behave if briefly informed? How can you
briefly inform someone of something they don't believe to be true?
>> I would reject both metrics as adequate theories of volition or even
>> instantaneous want, though the informed choice metric comes closer.
>
> The question is what you would put in their place. There are huge and
> long traditions in economics and philosophy, among other places,
> discussing how to describe what people "really" want. This is another
> of those areas that you shouldn't try to reinvent before surveying what
> has been done.
I am mostly sympathetic to arguments of the type "read the literature,
dammit", though the amount of literature reading that is required does
vary from person to person. But I have to say that I was not surprised by
my brief survey - it looks pretty much like what I expected. Is there any
particular area or result of which you worry I am ignorant?
>> The folk picture of people struggling between their high moral
>> aspirations and their inner demons is, as far as I can tell, pretty
>> much correct.
>
> The folk picture that most people are much more concerned about
> appearing moral than being moral is also pretty much correct.
I am not sure what you mean by "more concerned" in this context. People
are concerned about being moral. People are concerned about appearing
moral. The mechanisms are different but intertwined. What constitutes
"concern", and how do you determine whether "being moral" or "appearing
moral" receives "greater concern"? My guess is that whether people are
"more concerned" will depend on how concern is operationally defined and
the context in which the concern is tested... meaning that researchers,
depending on their starting prejudices, could easily construct an
experiment in which people are shown to be more concerned with being moral
than appearing moral, or vice versa. For example, if I present people
with an explicit choice *apparent to the subject* of being moral or
appearing moral, I would guess that they would choose being moral, as they
would be unable to do otherwise and retain their self-respect. On the
other hand, the more distant the dilemma from conscious perception in
those terms, the more I would expect evolution's puppet strings to have
opportunities to take over. There are different mechanisms interacting
here, and by manipulating the context I could determine which mechanism
would appear to win, but the most important truth is that there are
*different* mechanisms involved!
>> People die saving unrelated children. Is that a lie?
>
> That is a pretty rare phenomena. It is rarer still in situations where
> people know they will die, and know they cannot not gain social approval
> from doing so.
In general and speaking very broadly, the less adaptive something is, the
rarer it will end up being, after all the selection on all the heritable
variations has been taken into account. But it happens; it is something
that the human mind can really do, given its configuration. The
mechanisms of true altruism are, in fact, there, operating as
independently executing adaptations, and can be exposed given the right
context.
>> My own claim is that if you asked people what they cared about, what
>> mattered most to them, what kind of person they wanted to be, they
>> would say that the altruism is the most important part of them. Since
>> that is what they tell me, why should I - or any friend of theirs -
>> contradict them? The deliberative system may sometimes be weak, but
>> it is ultimately in charge - or at least is the center I look to, to
>> determine how to find the "person" I want to "help".
>
> This is the heart of our dispute. As in discussions of whether the
> upload, or its copy, would "really be you", there is an element of
> definition or choice in looking into a complex contradictory person and
> saying what their "real" preferences are. If you aren't going to appeal
> to any criteria about why this is a good choice, but just declare it by
> definition, then there isn't much more to talk about.
The construction of volition is something I'm working on writing up, but
it may be a while before you see anything from me on the subject.
But if I had a genie built using your definition of "wanting", I would
never, ever make a wish to it.
The term "real preferences" may mean rather different things to Friendly
AI creators and economists, as Dan Fabulich points out.
>> I wouldn't *use* the term "want" for a corporation. ... Also the
>> above does not seem to be a good example of self-deception, just
>> simple deliberate lying.
>
> Corporations, and other organizations can have behavior that is very
> much like individual self-deception. So its too bad that you reject
> such examples, as they are much easier to analyze.
Er... it looks to me like they're easy to analyze because they're simpler
yet unrelated phenomena, i.e., deliberate deception rather than
self-deception. Why would anyone construe it otherwise? Why not pick an
uncontroversial example like only 2% of high school seniors believing they
are below average in leadership ability?
The form of your argument seems to be: "Look at this corporation engaging
in deliberate deception. We would say this corporation is lying about
what it really wants, right? The lie benefits the corporation, so your
cheater-detectors should go off. Now look at this human activity of
self-deception. This is similar, right? And it benefits the human,
right? So your cheater-detectors should go off and attribute to the human
the same kind of lying, cheating behavior that you attribute to corporate
PR departments." But this is not a valid analogy. In one case we have
human deliberation pulling the puppet strings, in the other, evolutionary
adaptation. Evolution doesn't want you to lie, evolution wants you to be
honest and self-deceived. In linguistic political organisms, there are
*separate and distinct* selection pressures on motivations and actions,
which may come into conflict. You are taking human behaviors that result
from these conflicting selection pressures, and invoking cheater-detectors
on apparently self-beneficial outcomes to attribute selfish
intentionality. To this I object because it is *the wrong explanation*.
Furthermore, I object to it as a temporary scaffolding in the explanation
because it is an emotionally destructive thing to say, just as telling
people that it is rational to defect in the (iterated) Prisoner's Dilemma
is emotionally destructive, and yet appealing on account of its apparent
tough-mindedness.
> Also closely related is the question of what a nation "wants." You see,
> it seems that in many ways the voting/politics person within us makes
> different choices than we do personally. Personally you might buy
> foreign products, or hire a foreign worker, but politically you might
> want to prohibit them. It is not just that people may be ignorant about
> social processes; our political selves seem to have different
> preferences from our non-political selves! Politically, we talk as if
> we are more high-minded. So which selves should political outcomes
> correspond to? Should people get the products they would want as
> ordinary people, or the products they say they want as political people?
Why should there be a hidden register in the brain containing an answer to
this question? As I understand what it means to help people, it is not
"helping" to fulfill temptations they are ashamed of and ignore principles
they are proud of. It may also not be "helping" people to extrapolate
their principles farther than they did when they were choosing those
principles. Defining volition is an FAI-complete problem.
-- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Fri Jul 11 2003 - 22:38:30 MDT