Re: Objective morality

Eliezer S. Yudkowsky (
Fri, 26 Nov 1999 09:57:36 -0600

"E. Shaun Russell" wrote:
> The same, of course, is true of AI; if
> someone tried to program objective morality into an AI, the lack of
> flexibile rationality and reasoning based on situations would likely still
> resemble "just a computer" whether Turing proven or otherwise.

I think this goes to the root of the problem I (and others) are worrying about. The logic I've worked out now is good, solid, and self-swallowing, but I'm a human. For an AI, that logic implies a particular low-level implementation, and that may not be good. For characteristics like flexibility, common sense, the ability to avoid doing dumb things like converting the entire Universe into a set of regular polyhedra, spotting mistakes your designers didn't know about, and adapting to new information as well as you'd have if your designers had known about it to begin with, you need high-level characteristics that are the result of a lower level that usually does not have a direct semantic interpretation. The logic I've got now, if implemented directly, would be too crystalline.

Since this is a true fact, an AI design based on this true fact should not look at itself and go: "Yuck, my design has been screwed up by the silly humans." Which is, let it be understood, the very first requirement for stable self-improving AI.

           Eliezer S. Yudkowsky
Running on BeOS           Typing in Dvorak          Programming with Patterns
Voting for Libertarians   Heading for Singularity   There Is A Better Way