"Eliezer S. Yudkowsky" wrote:
> Why do I need a root supergoal produced by pure logic? I don't trust the
> built-in human goal system. Yeah, we can all think of one or two examples
> where it worked the way it was supposed to. Wheee. The rest of the day,
> however, we generally spend fighting someone else's emotions when we aren't
> fighting our own. This is a legacy system from the days of hunter-gatherer
> tribes. Furthermore, it doesn't serve our goals. It serves evolution's
> goals. You may think of yourself as being very cagey and selfish by making
> survival your first priority, but you're just embracing a set of genetic
> puppet strings. The whole thing is arbitrary.
Perhaps it is arbitrary, but your supergoal produced logic is resting an equally arbitrary evolutionary origins. We can speculate endlessly and wildly about the differences between built-in human goal systems vs. supergoals produced by so-called "pure logic". I happen to consider myself the ultimate anthropocentric relativist. Since all we have are our 3-lb pieces of gray matter from which to cogitate these issues, I posit that everything we can contemplate is a branch of psychoneurology. Thus, lofty ideas like "pure logic" and the "laws of physics", are really neuro-logic and neuro-physics -which in turn are really neuro-neuro-logic and neuro-neuro-physics ad infinitum. Von Neuman referred to this type of singularity of human knowledge systems, which now bear his name. Such Von Neuman Singularities are essentially knowledge black holes from which our current genetic and neurological limitations prevent us from escaping. I see such physical/technological barriers to knowledge to be the greatest motivating factor for becoming a post-biological being.
In the spirit of Kurt Godel, logic would seem to turn on itself, casting doubt on its ability to discern reality. I therefore can't help but distrust at a foundational level, any and all neuro^n-logical conclusions regarding any facet of Universe, including but not limited to the Singularity. Since in the end, my self-awareness is more certain than anything else, I choose to error on the side of preserving my existence rather than loosing it to an arguably greater intelligence. Arguably greater, in that processing speed and complexity will not necessarily bring about a wiser and smarter entity.
Since I think we are both resting an arbitrary evolutionary influences, all I have left is my awareness and ability to choose. I therefore choose life over death, control and participation in increasing degrees of higher intelligence over deletion, self-directed decisions over unselfish unknowable outcomes.