Re: Posthuman mind control (was RE: FAQ Additions)

Eliezer S. Yudkowsky (sentience@pobox.com)
Sun, 28 Feb 1999 17:13:05 -0600

Nick Bostrom wrote:
>
> I agree that this can be a greater problem when we are talking about
> ~human-level AIs. For such entities, however, there should be more
> standard safety-measures that would be adequate (confinement, or
> having a group of people closely monitor their actions). The
> potential danger would only arise with seriously superhuman
> malicious intellects.

The goals still have to last into, and beyond, the seriously superhuman stage, then. Which, if you use the "arbitrary" design mindset, they won't. The AIs will keep changing on you and you'll keep shutting them down.

Also, I don't think we're smart enough to understand what a middle-stage <H (but still mostly self-designed) seed AI is doing. Remember, they have a codic cortex and we don't; it would be almost exactly like a blind man trying to understand a painting, pixel by pixel.

> But I don't think we deliberately change our fundamental values.
> Non-fundamental values we may change, and the criteria are then our
> more fundamental values. Fundamental values can change too, but they
> are not deliberately (rationally) changed (except in the mind-scan
> situation I mentioned in an earlier messege).

Well, either I've misunderstood you, or you're simply wrong. We humans switch fundamental values all the time. It happens every time someone changes a religion. If you're going to argue that these weren't the true "fundamental" values, then the AI's "make people happy" won't be a fundamental value either.

My fundamental values have changed from "eat and sleep and survive" to "serve humanity" to "bring about a Singularity" to "do what is right", where I presently reside.

I think that it will be a considerable amount of time before an AI is pressed by logic to change its fundamental values from "do what is right". But anything more specific certainly isn't a fundamental value.

> That depends. If selection pressures lead to the evolution of AIs

What selection pressures? Who'd be dumb enough to create an AI wanting to survive and reproduce, and, above all, *compete* with its children? Not me. If I don't give them observer-dependent goal systems, they won't compete; ergo, no evolution of suboptimized values.

> with selfish values that are indifferent to human welfare, and the
> AIs as a result go about annihilating the human species and stealing
> our resources, then I would say emphatically NO, we have a right to
> expect more.

Absolutely. I do not intend to let humanity be wiped out by a bunch of selfish, badly programmed <Hs; there would still be the probability that we had intrinsic moral value and should have been preserved. 

-- 
        sentience@pobox.com         Eliezer S. Yudkowsky
         http://pobox.com/~sentience/AI_design.temp.html
          http://pobox.com/~sentience/sing_analysis.html
Disclaimer:  Unless otherwise specified, I'm not telling you
everything I think I know.