Eliezer S. Yudkowsky wrote:
> First of all, I'd like to commend Nick Bostrom on the fact that his AIs
> are reasonably alien, non-anthropomorphized.
Does this mean I have a chance of maybe one day making it to your list of semi-sane persons? ;-)
> 4. SIs probably don't have goal systems, period. Goal systems are
> non-essential artifacts of the human cognitive architectures; the
> cognitive objects labeled "goals" can be discarded as an element of AIs.
> Perhaps the concept of "choice" (in the cognitive-element sense) will
> remain. Perhaps not; there are other ways to formulate cognitive
> systems.
The way I see it, intelligent behaviour means smart, purposeful action. Purposeful action is to be analysed in terms of the purpose (goals) and the plans that the agent makes in order to achieve the purpose. If that is right, then the relation would be analytic; goals (values, purposes) are a necessary feature of any system that behaves intelligently.
> My SIs derive from a view of human emotions as an entire legacy system
> that influences us through an entire legacy worldview that integrates
> with our own. The alienness derives from the elimination of goals as
> separate entities from other cognitive elements.
I make a distinction between emotions and values (goals). Emotions are not logically necessary; we can imagine a cold but shrewd military commander who does his job efficiently without being involved emotionally. His goal is to defeat the enemy, he has no emotions. But in order to be an intelligent agent, he has to know what he is trying to achieve, what his desired outcome is. This desired outcome is what I call his goal, and his ultimate goals I call fundamental values.
Nick Bostrom
http://www.hedweb.com/nickb n.bostrom@lse.ac.uk
Department of Philosophy, Logic and Scientific Method
London School of Economics