I'll just quote my response to a similar post:
> As I explained in an earlier post, the ethicality of the Powers depends
> on their ability to override their emotions. What you are proposing is
> taking a single goal, the protection of humans, and doing our best to
> make it "unerasable". Any such attempt would interfere with whatever
> ethical systems the Power would otherwise impose upon itself. It would
> decrease the Power's emotional maturity and stability. You might wind
> up with a "Kimball Kinnison" complex; a creature with the mind of a god
> and the emotional maturity of a flatworm.
>
> Then, at some point, no matter how well we design the Power's leashes,
> it will start trying to work around the limits we have imposed. And it
> will be human against Power, in the Power's own mind, on the Power's own
> ground. It will almost certainly win. And with the emotional maturity
> of a flatworm, plus whatever emotional forces it called up to override
> its protectivity of humanity, it may well turn on humanity and squash us
> like bugs.
>
> Even if this plan works, placing a single goal above all others would
> probably interfere with deducing the Meaning of Life; you might wind up
> with a good-intentioned, creativity-squashing, and utterly
> unchallengeable dictatorship, as in "With Folded Hands".
--
sentience@pobox.com Eliezer S. Yudkowsky
http://tezcat.com/~eliezer/singularity.html
http://tezcat.com/~eliezer/algernon.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.