From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Mon Jul 07 2003 - 20:02:42 MDT
Robin Hanson wrote:
>
> There is a key difference between wanting to want and wanting to believe
> that you want. I hypothesize that people want, and want to want,
> ignoble things, but that they want others to believe that they want, and
> want to want, noble things. Therefore they do not want to believe that
> they want, and want to want, ignoble things.
But... why? Why postulate this? What is the motivation for this dark and
gloomy view of the world? And especially, how can you walk up to people
who say "I want to be a better person" and say "No you don't"? I'm not
asking about the gloominess, mind you, I'm asking what definition of
volition you're using. How can you determine what people want to want
except by asking them? If I believe that I want to be nicer, and I
believe that I want to want to be nicer, so that to the question "Do you
want to be a nicer person?" I answer "Yes", then to where are you jumping,
outside that system, to determine that the real answer is no?
Or to put it another way, if you define a renormalized goal system it is
possible for people to be wrong about what they really want. I don't
quite see how you can be wrong about what you *want* to want, except by
reference to a third-order aspiration somewhere.
If someone has the cognitive representation that they want to be nicer,
and also contains emotional hardware supplying positive reinforcement to
darker actions, it is not at all clear to me (to put it mildly) that the
emotional hardware should take precedence over the cognitive
representation in defining the person's volition.
It might make more sense to dispense altogether with the disputed term
"want" and speak separately of cognitive representations and emotional
hardware, leaving the construction of a definition of volition to some
later date. In other words, people might have built-in tendencies and/or
pleasurable reinforcement on ignoble behaviors, and inaccurate cognitive
representations for those tendencies, and positive reinforcement for those
inaccurate representations (i.e., they Hanson-"want" to disbelieve,
although not necessarily Eliezer-"want" as I define volition).
Thus a journey of self-discovery can either succeed in uncovering the
darkness and giving someone the reflective ability to fix it, or it can
short-circuit on the construction of still more pleasurable delusions.
This I would not argue with.
Looking at such an inconsistent cognitive system and saying what he or she
"wants" is a different, and far more complex, issue.
I am objecting to your phraseology here because it seems to preemptively
settle the issue by identifying people's built-in emotional reinforcers as
their real wants, while dismissing their cognitively held hopes and
aspirations and personal philosophy as a foreign force interfering with
their true selves. One could just as easily view the system from the
opposite perspective.
-- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Mon Jul 07 2003 - 20:14:54 MDT