Wrong. instinct = hardware morality. Morality is not remotely synonymous
with ethics. Morality is synonymous with racism, "Power corrupts",
self-righteousness, and so on. The hidden hand of evolution is what
prompts teenagers to overthrow the Establishment and the Establishment
to resist. They are simply reenacting the fights between a young
challenger and the old chief of the tribe.
What kind of moron would program something like that into a Power? What
kind of Power wouldn't erase a goad like that as soon as It saw the
fundamental absurdity of it all? Any Power with human morality is either
too dumb, or too firmly under our control, for us to worry about it
replacing the human race.
The Powers will be ethical. They'll deduce the Meaning of Life, and
build their motivations about that, not the goads and leashes of
evolution. If our life has any true meaning at all, the Powers will know
that instantly, and they will upload (and upgrade) us.
[snip]
> I see all these sub-goals I have are contributing to
> the biggie which is: survive or die. We like seeking new anything
> because it was (and still is) a good survival trait to have. ie. if
> our ancestors didn't have a want to explore and seek new knowledge we
> probably wouldn't be alive today.
Why would the Powers retain an archaic, destructive, and unpleasant
imperative like survival? Who's going to compete with them? Who's going
to threaten them? What possible reason would they have to fight among
themselves? They'll share a common, obvious, and observer-symmetrical
ethical system, not just with human-born Powers, but with any Power
strong enough to give them a fight. The correct side in any "argument"
will be so obvious that arguments will never take place.
Quit staring into a magnifying glass, quit welding your personal
morality and paranoia onto the Powers, and try to see the future!
You need to read:
http://tezcat.com/~eliezer/FOD.html#loon
--
sentience@pobox.com Eliezer S. Yudkowsky
http://tezcat.com/~eliezer/singularity.html
http://tezcat.com/~eliezer/algernon.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I know.