Re: Dyson (Was: Paths to Uploading)

Eliezer S. Yudkowsky (sentience@pobox.com)
Sun, 10 Jan 1999 13:25:21 -0600

KPJ wrote:
>
> When you select a meme system you should take into account the ramifications
> of adopting the meme system. After you have done it, you will find it much
> harder to change from it.
>
> As a hypothetical example of why we select the rule of avoiding killing:
>
> "If we adopt the meme system that we should avoid killing then we need not
> put lots of resources on avoid being killed (cf. Bosnia). Therefore, we
> should avoid killing."

I see. So, in other words, you're walking a totally predetermined trajectory where your current meme system determines the meme system you'll adopt next. (As adjusted for your innate instincts, of course.) Your walk takes into account no sort of objective right or wrong whatsoever, just your current set of prejudices. If your meme system told you to adopt a meme system that told you to nuke Cincinatti, you'd basically have no defense. If you'd been born a worshipper of Great Moloch, you'd stay that way until the end of time. You don't claim that the truth has anything to do with what you believe, so why should I lend credence to your beliefs, why should I think your beliefs are covariant with the truth?

You could do anything, kill me or kiss your worst enemy, and deny that reasoning has anything to do with it. It appears as if your meme system makes you a sentience to avoid, maybe even to terminate with extreme prejudice. Thank you for the warning. 

-- 
        sentience@pobox.com         Eliezer S. Yudkowsky
         http://pobox.com/~sentience/AI_design.temp.html
          http://pobox.com/~sentience/sing_analysis.html
Disclaimer:  Unless otherwise specified, I'm not telling you
everything I think I know.