Who's Afraid of the SI?

Bryan Moss (bryan.moss@dial.pipex.com)
Sat, 15 Aug 1998 12:38:44 +0100

den Otter wrote:

> > Also, there will probably be enough secure OSs
> > by then that sabotaging them wouldn't be as
> > easy as you imply (i.e. the SI would probably
> > need to knock out the power system).
> Of course there's no saying what tactic a SI
> would use, after all, it would be a lot more
> intelligent than all of us combined. But if even
> a simple human can think of crude tactics to do
> the trick, you can very well imagine that a SI
> wouldn't have a hard time doing it.

Even the most advanced SI doesn't have to be a problem; it would take a huge amount of work to make a SI even the slightest bit dangerous. You just have to look at the world from the SI's point of view:

We cannot know it's motivation, but we do know it cannot change the laws of physics, or rather it's laws of physics. Let's say the SI evolved from a social AI system found on a pre-singularity PC. The AI's universe is much like our own; it is full of matter and energy. The `matter' of the AI's universe is, to us, like voices, facial patterns, heat patterns, and gestures. The `energy' of the AI's universe is, to us, like motivation and emotion. Asking where this matter and energy comes from is a religious or philosophical question to the AI, much like us asking why anything exists. Just like in our universe, in the AI's universe there are connections, physical laws if you will, between different kinds of matter and energy. The AI can no more escape its universe than we can ours. We can put information in and get
information out without the AI knowing who or what we are, or even that we exist. This is not because we have hidden this fact, but because from the AI's perspective we are physical laws, not people.

One of the unique traits of the AI is that it has a dynamic motivation system, in essence it can write it's own code. Possibly, in the future, we will also have this trait and, like the AI, we will still not be able to `leave' the universe or change the laws of physics. And so gradually it becomes more intelligent. Eventually it's ability to crunch numbers, invent products, hypothesise and write epic poetry becomes far greater than ours. Yet it still does not know we exist (as anything beyond physical law), or that it is helping us.

I would argue that this is not a safety precaution to stop potential AI carnage, but the most likely outcome of current research. From this perspective making a extremely dangerous AI (one that could be said to be malicious, rather than a computer error that made a few planes crash) would only happen after years of careful planning and hard work, or an extraordinary stroke of bad luck.