RE: Posthuman mind control (was RE: FAQ Additions)

Billy Brown (
Thu, 25 Feb 1999 16:35:12 -0600

Nick Bostrom wrote:
> What Billy calls "mind control system" could
> in my opinion better be called "motivation system". Every agent needs
> a motivation system, and just because its motivation system was
> deliberately designed by another being rather than happened
> accidentally (due to genetical and environmental factors) doesn't
> make it any less part of what that agent is.

A proposed standard for distinguishing between the two cases:

Motivation System - A combination of goals and knowledge that leads an entity to prefer some actions, and avoid others. The entity is free to examine the contents of its own moral system, and to weigh its merits against those of any other moral system it may encounter. If it decides that some aspect of its moral system needs to be modified, it can do so.

Mind Control System - A moral system that incorporates special software measures intended to ensure that it is never altered. The entity is either unable to think about its moral system, or unable to compare it to other systems, or perhaps simply unable to make changes to it.

As I understand it, your contention is that an entity with a strong motivation system might be capable of changing it, but would never actually choose to do so. Is that substantially correct?

Billy Brown, MCSE+I