Eliezer writes:
> We are not talking about a bleeding jumped-up version of Windows 98. We
> are talking about something that is no more a big computer program than
> a human is a big amoeba. We are talking about a Power, compared to
> which human minds and human-equivalent AIs are cousins. If the
> resulting mind has any of the stereotypical characteristics of computer
> programs (or for that matter of humans), it's too dumb to be a Power.
How would you know whether a given AI is a Power? Maybe it is bluffing you. Maybe it got up to IQ 1200 and hit a bottleneck, and it decided, what the hell, might as well enjoy itself among all these stupid humans. It's not all THAT wise, and smart, and Powerful, but it can pull a pretty good bluff, the one-eyed man in the country of the blind.
So if some marvelous AI (or god, for that matter), says, "Behold, Elizer, for I am the one true AI, who sees all and knows all, and it is my will that you lay down your life for the Absolute Good of the universe," won't you have a teeny bit of skepticism? How could you know whether it is really as smart as it claims?
Hal