Re: Posthuman mind control

Eliezer S. Yudkowsky (
Thu, 25 Feb 1999 22:33:17 -0600

First of all, I'd like to commend Nick Bostrum on the fact that his AIs are reasonably alien, non-anthropomorphized. I don't think they're quite alien enough in terms of cognitive architecture, but that's a separate issue.

Let's forget about the issue of objective morality. Let's say morality is subjective. Let's say morality is flat. For our purposes, it doesn't make a difference. The following statements are true regardless:

  1. I can write an AI that will operate without initial goals. They are not intrinsically necessary. KISS says leave 'em out.
  2. This is not a quirk of a single architecture. The choices made by an AI are the consequence of its reasoning; to impose a separate set of choices you need a separate set of reasons. Human emotions, for example, are not just a set of initial goals. There was a time in our evolutionary history when emotions constituted the entire mind, and thus emotions are essentially a separate legacy system that still contains all the basic elements needed to operate a primitive mind; emotions contain instincts and intuitions and reflexes as well as desires. When we get angry at someone, we don't just push a subgoal of popping him one; our opinion of him is also affected. If this aspect of emotions were removed, emotions would cease to be an effective control system.
  3. In particular, any reasoning system capable of reflexive reasoning, abstract reasoning, and formulating heuristics from experience will be capable of making choices. The cognitive element of a choice can be perceived, reasoned about reflexively using learned heuristics that operate on generic choices, and the conclusions translated by the reflexive system back into the cognitive element of a decided choice. Depending on the reasoning architecture and knowledge content, the AI may believe in objective morality, subjective morality, flat morality, self-preservation, or even whatever the programmers tell it to. It can serve joy (like David Pearce) or intelligence (like Eliezer Yudkowsky). What it does not need, for any of this, are initial goals.
  4. SIs probably don't have goal systems, period. Goal systems are non-essential artifacts of the human cognitive architectures; the cognitive objects labeled "goals" can be discarded as an element of AIs. Perhaps the concept of "choice" (in the cognitive-element sense) will remain. Perhaps not; there are other ways to formulate cognitive systems. But goal systems are only one way of relating choices to mind, a perhaps overly centralized way; there are others.
  5. Because choices are a consequence of knowledge and heuristics, choices change with intelligence. This may express itself only as a shift in priorities in the third decimal place, but the idea of an absolutely constant goal system is a chimera.

I think, therefore, that although Nick Bostrum's complacent SIs are sufficiently alien to be seriously considered, they still aren't alien enough. The complacent SIs derive from a view of human emotions as arbitrary initial goals - the alienness derives from the elimination of the idea that the specific human emotion set is necessary.

My SIs derive from a view of human emotions as an entire legacy system that influences us through an entire legacy worldview that integrates with our own. The alienness derives from the elimination of goals as separate entities from other cognitive elements. I think that goals are not arbitrary but *entirely unnecessary*, and that's why my SIs don't pay much attention to the ones they start with.
--         Eliezer S. Yudkowsky

Disclaimer: Unless otherwise specified, I'm not telling you everything I think I know.