"Eliezer S. Yudkowsky" wrote:
> Why do I need a root supergoal produced by pure logic? I don't trust the
> built-in human goal system. Yeah, we can all think of one or two examples
> where it worked the way it was supposed to. Wheee. The rest of the day,
> however, we generally spend fighting someone else's emotions when we aren't
> fighting our own. This is a legacy system from the days of hunter-gatherer
> tribes. Furthermore, it doesn't serve our goals. It serves evolution's
> goals. You may think of yourself as being very cagey and selfish by making
> survival your first priority, but you're just embracing a set of genetic
> puppet strings. The whole thing is arbitrary.
Perhaps it is arbitrary, but your supergoal produced logic is resting an equally arbitrary evolutionary origins. We can speculate endlessly and wildly about the differences between built-in human goal systems vs. supergoals produced by so-called "pure logic". I happen to consider myself the ultimate anthropocentric relativist. Since all we have are our 3-lb pieces of gray matter from which to cogitate these issues, I posit that everything we can contemplate is a branch of psychoneurology. Thus, lofty ideas like "pure logic" and the "laws of physics", are really neuro-logic and neuro-physics -which in turn are really neuro-neuro-logic and neuro-neuro-physics ad infinitum. Von Neuman referred to this type of singularity of human knowledge systems, which now bear his name. Such Von Neuman Singularities are essentially knowledge black holes from which our current genetic and neurological limitations prevent us from escaping. I see such physical/technological barriers to knowledge to be the greatest motivating factor for becoming a post-biological being.
Paul Hughes