Perhaps I should have phrased RA1 differently, to distinguish between absolute knowledge and probabilistic knowledge. I stick by my guns, however, in that I have yet to observe rational debate between two parties absolutely convinced of what they're saying, or even with just one party absolutely convinced.
Nick Bostrom wrote:
>
> I think Eliezer's interesting argument is unsound, because I think
> one of it's premisses (RA1) is false.
>
> Eliezer wrote:
>
> > Rational Assumptions:
> > RA1. I don't know what the objective morality is and neither do you.
> > This distinguishes from past philosophies which have attempted to "prove"
> > their arguments using elaborate and spurious logic. One does not "prove" a
> > morality; one assigns probabilities.
> [snip]
> > LA1 and RA1, called "Externalism", are not part of the Singularity logic per
> > se; these are simply the assumptions required to rationally debate
> > morality.
>
> I don't see why a rational debate about morality would be impossible
> if you or I knew the "objective morality". People often
> rationally debate issues even when one party already knows where the
> truth lies.
But is that party _certain_? Probably not. People don't learn truth by being certain. Perhaps I should have phrased RA1 differently, to distinguish between absolute knowledge and probabilistic knowledge. I stick by my guns, however, in that I have yet to observe rational debate between two parties absolutely convinced of what they're saying, or even with just one party absolutely convinced. Even mathematicians, about the only people on the planet with real access to any kind of absolute truth, don't violate this rule. If someone is arguing with the mathematician, either it's another mathematician and the proof is so complex that both are uncertain, or it's some idiot who wants to prove he can trisect an angle. Most religious debate, or Plato and Aristotle's travesties of philosophy, illustrate very clearly what happens when one party tries to "prove" things using spurious logic rather than probabilistic reasoning.
And note that if one of us stumbled across a hydrogen-band SETI message containing the Meaning of Life, my logic goes out the window. So you're certainly correct in that if RA1 is wrong, it takes my argument with it.
> As for RA1, I think it could well be argued that it is false. We may
> not know in detail and with certainty what the moral facts are (so
> sure we'd want to assign probabilities), but it doesn't follow that
> we know nothing about then at all. In fact, probably all the people
> on this list know that it is wrong to torture innocent people for a
> small amount of fun. We could no doubt write down a long list of
> moral statements that we would all agree are true. Do you mean that
> we all suffer from a huge illusion, and that we are all totally
> mistaken in believing these moral propositions?
First of all, I don't know that it is wrong to torture innocent people for a small amount of fun. It is, with at least 90% probability, not right. I can't be at all certain that "2+2 = 4", but I can be almost totally certain that "2+2 does not uniquely equal 83.47". I don't become really convinced of something because evidence is presented for it, but only when I can't come up with any plausible alternative. Now, it may be that intensity of experience is the ultimate good, or a kind of thought that pain stimulates, or that pain is morally null but the joy the torturer experiences is not, or even that all things that happen are good and chosen by God. It does seem unlikely enough that pain is right, and likely enough that pain is wrong, and sufficiently in violation of social cooperation to boot, that I would cheerfully shoot such a torturer.
> Now, if we accept the position that maybe we already know quite a few
> simple moral truths, then your chain of reasoning is broken. No
> longer is it clear that the morally preferred action is to cause a
> singularity no matter what.
I'm not too sure of this. The hedonistic qualia chain, which states that the qualia of pleasure and pain are physically real positive-value and negative-value goals, is sufficiently in correspondence with the general postulates used by society, and hence with the game theory of social cooperation, that I would not grossly violate it simply to get to the Singularity. For me, the ends cannot justify the means. Any being short of superintelligence is still bound by game theory. There is absolutely no way in which I consider myself responsible for an ethical superintelligence's actions, however, even if I personally created the SI and successfully predicted the actions.
> For example, if it is morally preferred
> that the people who are currently alive get the chance to survive
> into the postsingularity world, then we would have to take this
> desideratum into account when deciding when and how hard to push for
> the singularity.
Not at all! If that is really and truly and objectively the moral thing to do, then we can rely on the Post-Singularity Entities to be bound by the same reasoning. If the reasoning is wrong, the PSEs won't be bound by it. If the PSEs aren't bound by morality, we have a REAL problem, but I don't see any way of finding this out short of trying it. Or did you mean that we should push faster and harder for the Singularity, given that 150,000 people die every day?
> In the hypothetical case where we could dramatically
> increase the chances that we and all other humans would survive, by
> paying the relatively small price of postponing the singularity one
> year, then I feel pretty sure that the morally right thing to do
> would be to wait one year.
For me, it would depend on how it affected the chance of Singularity. I don't morally care when the Singularity happens, as long as it's in the next thousand years or so. After all, it's been fifteen billion years already; the tradeoff between time and probability is all in favor of probability. From my understanding of causality, "urgency" is quite likely to be a human perversion. So why am I in such a tearing hurry? Because the longer the Singularity takes, the more likely that humanity will wipe itself out. I won't go so far as to say that I'm in a hurry for your sake, not the Singularity's - although I would personally prefer that my grandparents make it - but delay hurts everyone.
> In reality there could well be some kind of tradeoff like that.
There's a far better chance that delay makes things much, much worse. Delay appeals to the uncertain, but that doesn't mean it's not fatal.
> It's
> good if superintelligent posthumans are created, but it's also good
> if *we* get to become such beings. And that can in some cases impose
> moral obligations on us to make sure that we ourselves can survive
> the singularity.
Why not leave the moral obligations to the SIs, rather than trying (futilely and fatally) to impose your moral guesses on them?
> You might say that our human morality - our desire that *we* survive
> - is an arbitrary effect of our evolutionary history. Maybe so, but I
> don't see the relevance. If our morality is in that sense arbitrary,
> so what? You could say that the laws of physics are arbitrary, but
> that does not make them any less real. Don't forget that "moral" and
> "good" are words in a human language. It's not surprising, then, if
> their meaning is also in some way connected to human concepts and
> anthropocentric concerns.
I don't just think it's arbitrary, I fear that it is _wrong_.
Perform reality<->morality inversion, move back 500 years:
"You might say that our human reality - our opinion that the Earth is flat - is an arbitrary effect of our evolutionary history. Maybe so, but I don't see the relevance. If our reality is in that sense arbitrary, so what? You could say that human sacrifice is arbitrary, but that does not make it any less good. Don't forget that "real" and "true" are words in a human language. It's not surprising, then, if their meaning is also in some way connected to human concepts and anthropocentric concerns."
Same objections.
--
sentience@pobox.com Eliezer S. Yudkowsky
http://pobox.com/~sentience/AI_design.temp.html
http://pobox.com/~sentience/sing_analysis.html
Disclaimer: Unless otherwise specified, I'm not telling you
everything I think I know.