Re: Why Does Self-Discovery Require a Journey?

From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Mon Jul 07 2003 - 20:02:42 MDT

  • Next message: Technotranscendence: "Not that anyone missed me..."

    Robin Hanson wrote:
    >
    > There is a key difference between wanting to want and wanting to believe
    > that you want. I hypothesize that people want, and want to want,
    > ignoble things, but that they want others to believe that they want, and
    > want to want, noble things. Therefore they do not want to believe that
    > they want, and want to want, ignoble things.

    But... why? Why postulate this? What is the motivation for this dark and
    gloomy view of the world? And especially, how can you walk up to people
    who say "I want to be a better person" and say "No you don't"? I'm not
    asking about the gloominess, mind you, I'm asking what definition of
    volition you're using. How can you determine what people want to want
    except by asking them? If I believe that I want to be nicer, and I
    believe that I want to want to be nicer, so that to the question "Do you
    want to be a nicer person?" I answer "Yes", then to where are you jumping,
    outside that system, to determine that the real answer is no?

    Or to put it another way, if you define a renormalized goal system it is
    possible for people to be wrong about what they really want. I don't
    quite see how you can be wrong about what you *want* to want, except by
    reference to a third-order aspiration somewhere.

    If someone has the cognitive representation that they want to be nicer,
    and also contains emotional hardware supplying positive reinforcement to
    darker actions, it is not at all clear to me (to put it mildly) that the
    emotional hardware should take precedence over the cognitive
    representation in defining the person's volition.

    It might make more sense to dispense altogether with the disputed term
    "want" and speak separately of cognitive representations and emotional
    hardware, leaving the construction of a definition of volition to some
    later date. In other words, people might have built-in tendencies and/or
    pleasurable reinforcement on ignoble behaviors, and inaccurate cognitive
    representations for those tendencies, and positive reinforcement for those
    inaccurate representations (i.e., they Hanson-"want" to disbelieve,
    although not necessarily Eliezer-"want" as I define volition).

    Thus a journey of self-discovery can either succeed in uncovering the
    darkness and giving someone the reflective ability to fix it, or it can
    short-circuit on the construction of still more pleasurable delusions.
    This I would not argue with.

    Looking at such an inconsistent cognitive system and saying what he or she
    "wants" is a different, and far more complex, issue.

    I am objecting to your phraseology here because it seems to preemptively
    settle the issue by identifying people's built-in emotional reinforcers as
    their real wants, while dismissing their cognitively held hopes and
    aspirations and personal philosophy as a foreign force interfering with
    their true selves. One could just as easily view the system from the
    opposite perspective.

    -- 
    Eliezer S. Yudkowsky                          http://singinst.org/
    Research Fellow, Singularity Institute for Artificial Intelligence
    


    This archive was generated by hypermail 2.1.5 : Mon Jul 07 2003 - 20:14:54 MDT