Nick Bostrom wrote:
> What Billy calls "mind control system" could
> in my opinion better be called "motivation system". Every agent needs
> a motivation system, and just because its motivation system was
> deliberately designed by another being rather than happened
> accidentally (due to genetical and environmental factors) doesn't
> make it any less part of what that agent is.
A proposed standard for distinguishing between the two cases:
Motivation System - A combination of goals and knowledge that leads an entity to prefer some actions, and avoid others. The entity is free to examine the contents of its own moral system, and to weigh its merits against those of any other moral system it may encounter. If it decides that some aspect of its moral system needs to be modified, it can do so.
Mind Control System - A moral system that incorporates special software measures intended to ensure that it is never altered. The entity is either unable to think about its moral system, or unable to compare it to other systems, or perhaps simply unable to make changes to it.
Billy Brown, MCSE+I
bbrown@conemsco.com