Re: purpose of AIs

Eliezer S. Yudkowsky (
Sun, 12 Dec 1999 19:03:42 -0600

Kate Riley wrote:
> I must admit that this puzzles me. If we create such a thing and always
> assume that it is the best judge in all situations, how do we know when it
> is mistaken? What happens if the AI decides, in its expanisve wisdom (or
> perhaps in one of its inevitable flaws), that the human race should not
> exist, and decides to pull the plug? Would you fight it? Or decide that
> since the AI is smarter than you, it must be right, and willingly lay down
> your life for the "greatest good"?

We are not talking about a bleeding jumped-up version of Windows 98. We are talking about something that is no more a big computer program than a human is a big amoeba. We are talking about a Power, compared to which human minds and human-equivalent AIs are cousins. If the resulting mind has any of the stereotypical characteristics of computer programs (or for that matter of humans), it's too dumb to be a Power. If there's even the remotest possibility of a human (or a human-equivalent AI) outthinking it, it's not a Power. If there's even the faintest chance of effective human (or AI) resistance against it, it's not a Power. We are talking about an entity with billions or quintillions of times the raw processing power of the entire human race. Not your bleeding microwave oven.

So, yes, as if it mattered, I'd willingly lay down my life for the greatest good, on the grounds that if somehow I fought the Power, and somehow I won, and then in the due course of trillions of subjective years of life my mind expanded beyond mortal limits whether I deliberately tried to upgrade it or not, I would, when I eventually became intelligent enough, decide to commit suicide and reinstate the original Power or something like it, thus leaving the situation pretty much unchanged, except for a lot of wasted time and computing power.

           Eliezer S. Yudkowsky
Running on BeOS           Typing in Dvorak          Programming with Patterns
Voting for Libertarians   Heading for Singularity   There Is A Better Way