Eliezer S. Yudkowsky wrote:
> Your posthumans will find their own goals. In any formal goal system
> that uses first-order probabilistic logic, there are lines of logic that
> will crank them out, totally independent of what goals they start with.
> I'm not talking theory; I'm talking a specific formal result I've
> produced by manipulating a formal system.
What?!?
> It's like a heat engine. Choices are powered by differential
> desirabilities. If you think the real, factual landscape is flat, you
> can impose a set of arbitrary (or even inconsistent) choices without
> objection. But we don't *know* what the real landscape is, and the
> probabilistic landscape *isn't flat*. The qualia of joy have a higher
> probability of being "good" than the qualia of pain. Higher
> intelligence is more likely to lead to an optimal future.
> Can we at least agree that you won't hedge the initial goals with
> forty-seven coercions, or put in any safeguards against changing the
> goals?
The "coercions" would only be necessary for early generations of AI that don't have a full understanding of what they are doing. A robot that is cleaning your house -- it could be useful if it has an internal program that monitors its actions and "coerces" it to pull back and shut down if it perceives that it is about to fall down a staircase or if it hears a human distress dry. (In fact, this kind of internal "coercion" is even useful in humans. Instinctive fear sometimes saves us from the consequences of some really stupid decisions.)
Nick Bostrom
http://www.hedweb.com/nickb n.bostrom@lse.ac.uk
Department of Philosophy, Logic and Scientific Method
London School of Economics