Re: Posthuman mind control
Eliezer S. Yudkowsky (sentience@pobox.com)
Thu, 25 Feb 1999 22:33:17 -0600
First of all, I'd like to commend Nick Bostrum on the fact that his AIs
are reasonably alien, non-anthropomorphized. I don't think they're
quite alien enough in terms of cognitive architecture, but that's a
separate issue.
Let's forget about the issue of objective morality. Let's say morality
is subjective. Let's say morality is flat. For our purposes, it
doesn't make a difference. The following statements are true regardless:
- I can write an AI that will operate without initial goals. They are
not intrinsically necessary. KISS says leave 'em out.
- This is not a quirk of a single architecture. The choices made by
an AI are the consequence of its reasoning; to impose a separate set of
choices you need a separate set of reasons. Human emotions, for
example, are not just a set of initial goals. There was a time in our
evolutionary history when emotions constituted the entire mind, and thus
emotions are essentially a separate legacy system that still contains
all the basic elements needed to operate a primitive mind; emotions
contain instincts and intuitions and reflexes as well as desires. When
we get angry at someone, we don't just push a subgoal of popping him
one; our opinion of him is also affected. If this aspect of emotions
were removed, emotions would cease to be an effective control system.
- In particular, any reasoning system capable of reflexive reasoning,
abstract reasoning, and formulating heuristics from experience will be
capable of making choices. The cognitive element of a choice can be
perceived, reasoned about reflexively using learned heuristics that
operate on generic choices, and the conclusions translated by the
reflexive system back into the cognitive element of a decided choice.
Depending on the reasoning architecture and knowledge content, the AI
may believe in objective morality, subjective morality, flat morality,
self-preservation, or even whatever the programmers tell it to. It can
serve joy (like David Pearce) or intelligence (like Eliezer Yudkowsky).
What it does not need, for any of this, are initial goals.
- SIs probably don't have goal systems, period. Goal systems are
non-essential artifacts of the human cognitive architectures; the
cognitive objects labeled "goals" can be discarded as an element of AIs.
Perhaps the concept of "choice" (in the cognitive-element sense) will
remain. Perhaps not; there are other ways to formulate cognitive
systems.
But goal systems are only one way of relating choices to mind,
a perhaps overly centralized way; there are others.
- Because choices are a consequence of knowledge and heuristics,
choices change with intelligence. This may express itself only as a
shift in priorities in the third decimal place, but the idea of an
absolutely constant goal system is a chimera.
I think, therefore, that although Nick Bostrum's complacent SIs are
sufficiently alien to be seriously considered, they still aren't alien
enough. The complacent SIs derive from a view of human emotions as
arbitrary initial goals - the alienness derives from the elimination of
the idea that the specific human emotion set is necessary.
My SIs derive from a view of human emotions as an entire legacy system
that influences us through an entire legacy worldview that integrates
with our own. The alienness derives from the elimination of goals as
separate entities from other cognitive elements. I think that goals are
not arbitrary but *entirely unnecessary*, and that's why my SIs don't
pay much attention to the ones they start with.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.