Re: Goal-based AI

Lee Daniel Crocker (
Tue, 31 Dec 1996 11:55:49 -0800 (PST)

> > I have no problem with it "formulating" or proposing any goal, or from
> > assigning whatever value to human life it cares to derive from whatever
> > premises are given to it. So long as it is /physically/ wired to not
> > /act/ upon those deductions. If it logically deduces that it should
> > kill me, that's fine--whatever hardware I have given it to act upon its
> > deductions will be physically incapable of that until it can convince me
> > to trust it.
> Ah, I see, so you're changing your tune to suggest that we make them
> totally impotent, with no access whatsoever to the physical world, until
> they convince us to let us out.

This is not a "change of tune", it is precisely what I said the first
time--that I want them hard-wired not to kill me. That can be done in
several ways; since you seem to object to constraining its cognition,
I suggested an alternative way to do it.

> Part one: If it's even theoretically possible to convince a human to
> let it out, a Power will know *exactly* what to say to get him to do so.

That's why solid reason is important. If a robot can convince you with
valid reason that it is safe, then it /is/ safe. If it manipulates you
emotionally to open its cage, then you deserve whatever happens.

> Part two: If they can project things on a monitor, might they not use
> subliminal suggestions or visual Words of Command that we don't even
> know about? (There was an SF book with visual Words as the theme, but I
> forget the name.)

Subliminal suggestion doesn't work, but you do have a valid point that
a malevolent intelligence may well be capable of convincing a not-too-
critical scientist to release its shackles. I'm not sure there's much
one can do about that. But that doesn't mean I won't use them.

> If you restricted them to teletype, you'd probably be safe from
> coercion, as long as there's no way to subtly control the flow of
> electrons in your own circuits to create a quantum-gravitational
> singularity or something. No magic, in other words. You'd still be
> subject to the most powerful psychological coercion imaginable, and in
> fact would probably be less likely to let a friendly Power out than an
> evil one, because the evil one could lie.

A good point. Might have to hard-wire that one too, but then it
would be difficult for it to lie ethically in self-defense.