Re: Preventing AI Breakout [was Genetics, nannotechnology, and ,

Joseph Sterlynne (
Sun, 24 Oct 1999 20:26:07 -0600

> Hal Finney
>> Robert J. Bradbury

>I suspect that it is possible in principle to keep an AI within a
>simulation, but in practice it may be able to notice aspects of the
>environment which you have imposed ad hoc and which don't seem "natural".

It seems that everyone is confident that an AI of sufficient intelligence will be able to discover that it exists within a simulation. This is a little surprising. I won't say that we could absolutely guarantee its eternal ignorance but I don't see why an AI would not simply accept the laws of its environment as the laws of a nonconstructed universe.

>I think if you could actually set up the AI so that it actually evolved
>from a starting environment which did not have much more ad hoc features
>than our own, you might be able to fool it.

Some have been saying that a very intelligent AI would not be fooled because it would recognize the regularities or mathematical coherence of its universe. We have discerned certain regularities and codified them as laws of physics. Does regularity imply intelligent construction? An AI might simply conclude that it has figured out the fundamental laws of physics. That doesn't necessarily offer it metalevel knowledge.

>However that might take billions of years of simulated evolution. Any
>shortcuts you take might be noticeable by a super-smart AI.

You could always take shortcuts and then play Creationist God. None of us believes that dinosaur fossils were placed there by some deity; it is conceivable, though, that they were. If you're careful enough such a trick could work.

>Even if you succeeded, it's not clear what use an AI would be which lived
>in a world which is probably so dissimilar to our own.

That depends, perhaps, on how specialized the AI's work will be.

>In our own case, we do observe a great many coincidences which suggest
>that the universe is specially set up for life to be possible.

I have heard this sort of statement made a number of times on this list and elsewhere. It is often, if I am not mistaken, offered as the Anthropic Principle. I confess that do not discern any sense in this notion. Remarking upon the amazing "coincidences" which are simply causes of some existing thing is just circular. Life is just the result of the configuration of the universe.

>However few people conclude from this that we are running in an artificially
>created simulation. (The people who do, we call religious, and apparently
>we are supposed to bash them. Bash! Bash!)

Bash. But one could suspect that we are in an artificial environment and not be religious in the familiar sense.