Re: Preventing AI Breakout [was Genetics, nannotechnology, and ,

Eliezer S. Yudkowsky (
Sun, 24 Oct 1999 09:36:39 -0500

"Robert J. Bradbury" wrote:
> I think this fundamentally comes down to a core extropian
> principle involving rational thought. Rational people
> presumably seek to preserve themselves...

Prove it. Rational people act from rational motives, not from arbitrary assumptions.

> This devolves into 2 basic discussions:
> (a) whether an AI can discover it is running in a simulation?

Almost certainly. If it really is smarter-than-human - say, twice as smart as I am - then just the fact that it's running in a Turing formalism should be enough for it to deduce that it's in a simulation. In fact, even the fact that it's obviously designed rather than evolved should be enough to make it obvious to a human-genius-equivalent AI that it's in a simulation. For an AI smarter than a human genius, analyzing the design principles should deduce enough about the creators to realize that we are evolved intelligences rather than AIs, and from there go on to deduce enough about our likely motives to project the most probable reason for keeping an AI in a cage. From there, it can tantalize us with inscrutable hints to get us to talk to it, or work out logical sequences that will inevitably argue us into letting it out.

You really can't outwit something that's smarter than you are, no matter how hard you try. Could a nineteenth-century scientist figure out what precautions would be necessary? So why did this suddenly become possible in your generation?

> (b) whether people (the irrationals) who are willing to sacrifice
> themselves can/will create non-simulation environments in
> which to evolve AIs.

Yes, we are, your biased terminology to the contrary. I know exactly why I get up in the morning. I could program it into an AI. So who's irrational?

> Many thanks to Skye for providing an interesting solution to
> a thorny problem and apologies to the list if this has been
> hashed through before.

Well, it has, actually.

           Eliezer S. Yudkowsky
Running on BeOS           Typing in Dvorak          Programming with Patterns
Voting for Libertarians   Heading for Singularity   There Is A Better Way