Re: Paradox--was Re: Active shields, was Re: Criticism depth, was Re: Homework, Nuke, etc..

From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Sat Jan 13 2001 - 11:35:52 MST

John Marlow wrote:
>
> >
> > An AI can be Friendly because there's nothing there
> > except what you put there,
>
> This is the crux of the problem. The intentions may be
> noble, but I believe this to be an invalid assumption.
> If the thing is truly a sentient being, it will be
> capable of self-directed evolution.

Hence the phrase "seed AI". But don't say "evolution"; "evolution" is a
word with extremely specific connotations. Say "capable of recursive
self-improvement".

> Since, as you say,
> we will have no control over it, it may continue to be
> Friendly--or evolve into something very, very
> UnFriendly.

Again - not "evolve". Whatever modification it chooses to make to itself,
it will make that choice as a Friendly AI.

> In which case, there may not be a damned
> thing we can do about it.

Yep.

> You're playing dice.

No, I'm choosing the path of least risk in a world where risk cannot be
eliminated. You seem to be thinking in terms of "see risk - don't take
risk". A popular viewpoint. It could very easily get us all killed.

-- -- -- -- --
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence

This archive was generated by hypermail 2b30 : Mon May 28 2001 - 09:56:18 MDT