From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Tue Feb 11 2003 - 21:13:32 MST
Wei Dai wrote:
>
> I find it interesting that probabilities must work differently for these
> two types of people. Consider a thought experiment where you're shown a
> computer printout which is supposed to be the millionth bit in the binary
> expansion of pi. However the computer is somewhat faulty and in 1% of the
> branches of the multiverse it gives an incorrect answer. Now you're asked
> to guess the true value of the millionth bit of pi (let's call this X). If
> you guess correctly you'll be rewarded, otherwise punished, with about
> equal severity. Suppose you see the number 0 on your printout, and that
> you don't have any other information about what X is, and you can't
> compute it in your head. Obviously both the egoist-temporalist and the
> altruist-Platonist would choose to guess 0, but their reasoning process
> would be different.
>
> Before seeing the printout, both types believe that the probability of X
> being 0 is .5. After seeing the printout, the egoist-temporalist would
> apply Bayes's rule and think that the probability of X being 0 is .99.
> He reasons that guessing 0 lead to a .99 probability of reward and .01
> probability of punishment. His expected utility of choosing 0 is
> .99*U(reward) + .01*U(punishment).
>
> An altruist-Platonist would instead continue to believe that the
> probability of X being 0 is .5.
No.
> He reasons that if X is 0, then his
> current measure is .99m (where m is the measure of himself before seeing
> the printout), and if X is 1, then his current measure is .01m. So
> guessing 0 would lead to a .5 probability of .99m people being rewarded
> and .5 probability of .01m people being punished. His expected utility of
> choosing 0 is .5*U(.99m people rewarded) + .5*U(.01m people punished).
> Note that if he did apply Bayes's rule, then his expected utility would
> instead become .99*U(.99m people rewarded) + .01*U(.01m people punished)
> which would weight the reward too heavily. It doesn't matter in this case
> but would matter in other situations.
If he chooses to apply Bayes's rule, then his expected *global* utility is
p(1)*u(.99m rewarded) + p(1)*u(.01m punished), while his expected *local*
utility is p(.99)*(1 rewarded) + p(.01)*(1 punished). If he sees X=0
locally it doesn't change his estimate of the global truth that .99m
observers see the true value of X and .01m observers see a false value of
X. Roughly speaking, if a Bayesian altruist-Platonist sees X=0, his
expected global utility is:
p(.99)*u(.99m observers see 0 => .99m observers choose 0 => .99m observers
are rewarded && .01m observers see 1 => .01m observers choose 1 => .01m
observers are punished)
+
p(.01)*u(.99m observers see 1 => .99m observers choose 1 => .99m observers
are rewarded && .01m observers see 0 => .01m observers choose 0 => .01m
observers are punished)
=
p(1)*u(.99m observers are rewarded && .01m observers are punished)
=
p(1)*u(.99m observers rewarded) + p(1)*u(.01m observers are punished)
-- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Tue Feb 11 2003 - 21:16:04 MST