Re: Singularity: Individual, Borg, Death?

Eliezer S. Yudkowsky (sentience@pobox.com)
Fri, 04 Dec 1998 16:07:04 -0600

Hal Finney wrote:
>
> There are obviously a multitude of possible moral systems. If we think
> of the moral system as an algorithm which, given a situation and a list
> of actions, produces a rank ordering of the actions, then there would
> probably be potentially an infinite number of moral systems, since there
> are an infinite number of computer programs.
>
> How do we single out the one or ones which represent "absolute morality"?
> What does this concept mean? I can't get a grip on it. Is there
> really such a thing?

Incidentally, after producing a rather speculative post along the lines of "Dunno, but I hope so", I should like to remind everyone why I hope so:

There's an infinite set of possible choices. If all choices are equally right, why do you get up in the morning? In short, "objective morality" is the *correct* choice.

If there's no distinction between a "correct" moral system and an "incorrect" moral system, no way of singling out the best, if they're all indistinguishable from each other - well, why do you make any choice at all? I acknowledge this possibility, but I also acknowledge that there may be a method of distinguishing between choices, and it's the second possibility that yields meaningful guidance.

Don't tell me that the "correct" choice depends what system you're in. The system *is* the set of correct-to-the-system choices; that's the *definition* of "system". Which *system* is "correct"? If all the systems are indistinguishable, why not choose the system where I jump off a building? Again, I act on the assumption that systems are distinguishable, because the alternative doesn't yield meaningful directions.

You have some people who say, "The choices are all the same; I'll do what seems best to me." At a higher level of self-awareness, you have: "I'll stick with the evolutionary system I was born in, they're all the same." At a higher level of self-awareness, you say: "Which new system you choose depends on how your current system evaluates that choice." Choices, systems, trajectories... but I want to jump out of the system and choose the real answer.

-- 
        sentience@pobox.com         Eliezer S. Yudkowsky
         http://pobox.com/~sentience/AI_design.temp.html
          http://pobox.com/~sentience/sing_analysis.html
Disclaimer:  Unless otherwise specified, I'm not telling you
everything I think I know.