From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Mon Jul 07 2003 - 17:55:07 MDT
Robin Hanson wrote:
>
> It seems to me that an alternative hypothesis should be entertained:
> that we are greatly self-deceived about what we want. Maybe the things
> we actually want are typically not as noble as the things we want to
> believe that we want. Maybe we really want social approval more than we
> admit, for example. But believing ourselves to be noble may help us to
> convince others that we are noble, and so to convince them to associate
> with us. Also, we may want to believe that our career/spouse/etc. is
> just the sort of career or spouse we wanted, so that we can reassure our
> associates that we are not considering switching careers/spouses/etc.
>
> This alternative hypothesis suggests that most literary journeys of
> self-discovery are really aids to self-delusion, helping people tell
> themselves comforting stories about themselves. Real journeys of
> self-discovery would largely be dark affairs, wherein mounting evidence
> forced people to believe ignoble things about themselves that they would
> rather not tell others. And those who do struggle over decades to learn
> the truth about what people want, and who are willing to tell others,
> would face largely indifferent or hostile audiences. Young optimists
> would have evolved to ignore the claims of old cynics, since the young
> optimists of the past who did not were less attractive as mates and
> associates.
This definition of "want" seems to me to be one of the classically
postulated failure modes in genie stories - defining what people "really
want" as including wants they would not want to want. If someone does not
want to want something, it is not "help" to satisfy that want. I would
say that volition and its satisfaction should be defined by taking the
renormalized goal system into account, including people's aspirations to
be better people. It is not helpful if a genie defines your "wish" by
reference to your dark, repressed desires, and in fact, it is not helpful
if a genie defines your "wish" by inducing features of your apparent de
facto preferences which you would reject if you understood them.
What is the purpose of self-discovery if not self-alteration?
-- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Mon Jul 07 2003 - 18:05:46 MDT