Re: Singularity: AI Morality

Samael (
Wed, 9 Dec 1998 16:25:21 -0000

-----Original Message-----
From: Billy Brown <>
To: <> Date: 09 December 1998 16:18
Subject: RE: Singularity: AI Morality

>Nick Bostrom wrote:
>> I think the trick is not to use coersive measures, but rather to
>> wisely select the values we give to the superintelligences, so that
>> they wouldn't *want* to hurt us. If nobody wants to commit crimes,
>> you don't need any police.
>And others have posted similar thoughts.
>Guys, please, trust the programmers on programming questions, OK? The
>of things you are talking about sound reasonable, and might even be
>in a static system, but they are not even theoretically possible in the
>situation we are discussing. The problem is that we don't know how to
>a Transhuman AI - all we can do is make something that might evolve into
>on its own. If we try to put constraints on that evolution then the
>constraints also have to evolve, and they must do so in synch with the rest
>of the system.

The problem with programs is that they have to be designed to _do_ something.

Is your AI being designed to solve certain problems? Is it being designed to understand certain things? What goals are you setting it?

An AI will not want anything unless it has been given a goal (unless it accidentally gains a goal through sloppy programming of course).