Re: Singularity: AI Morality

Nick Bostrom (bostrom@ndirect.co.uk)
Tue, 8 Dec 1998 23:01:33 +0000

John Clark wrote:

> I don't think security is possible by hardwiring certain actions into
> an AI and any attempt to do so would only increase the possibility of it
> becoming insane. Security is hard enough to maintain when you only
> have to worry about criminals of average or more likely below average
> brain power, you can't outsmart somebody vastly more intelligent than you
> are.

I think the trick is not to use coersive measures, but rather to wisely select the values we give to the superintelligences, so that they wouldn't *want* to hurt us. If nobody wants to commit crimes, you don't need any police.

Nick Bostrom
http://www.hedweb.com/nickb n.bostrom@lse.ac.uk Department of Philosophy, Logic and Scientific Method London School of Economics