Re: Who's Afraid of the SI?

Bryan Moss (
Mon, 17 Aug 1998 14:04:21 +0100

den Otter wrote:

> > Even the most advanced SI doesn't have to be a
> > problem; it would take a huge amount of work
> > to make a SI even the slightest bit dangerous.
> I lost your original message to a %$@#&%*
> Netscape error, but I think I got the general
> drift from later responses. I suspect we're
> basically talking about two different things,
> with IMO different "danger" levels. My concerns
> weren't as much about servant AIs turning mean,
> but rather ruthless *people* turning SI trough
> future technology (AI, nanotech, uploading etc.)

Yes, you're right, I misread the message. But other people still disagreed with me, so no loss on my part.

> If (and this is only logical) the abovementioned
> technologies will remain long enough (almost)
> completely within the realm of governments, big
> corporations, criminal organizations and the
> ultrarich for these people to understand and use
> their full potential, we're all screwed.

I doubt (I could be wrong) the military is looking into the application of AI as weapon itself, although they certainly will use it for
controlling current weapons and for strategic planning. If military AI did give one side a massive advantage (most likely the US) things would probably have time to even out (the US isn't known for it's conquering). I think some of the comments I've made apply here. For instance, would a corporation use (to coin a term) AI-munitions or is it more likely to stick within in the realms of sales and marketing? How many corporations built nuclear weapons during the arms race?

Uploading is something that is certainly not taken seriously by any of these organisations. Uploading is likely to come as a result of unrelated research and will not be considered as a weapon. I don't think its potential is that obvious. Uploading, despite being the topic of conversation here and in SF, will be largely ignored by most people. I can see uploading being announced on news like cloning - they've managed to simulate a monkey in exact detail on a computer or something. Most people will ask what the point is. A few of us will talk about its potential. There'll be moral outcry, an ethical commission will advice a ban, the US will do it, and everyone else will follow.

And then there's molecular nanotechnology. I've always been sceptical of a MNT singularity (in the short run). I've never understood why people think that after the first assembler there'll be a sudden outburst of new technology. MNT
dramatically reduces the cost of production and increases the strength to weight and complexity of products. You still have design problems and you're likely to need all those new software breakthroughs to be able to manage the complexity of some of the more radical projects. I think the MNT revolution will be fast, but not so fast that one dominant power will emerge.

Besides, all these fields are far from being monopolised, if you really think people are going to get screwed just make sure you're the one doing the screwing.