From: Eliezer S. Yudkowsky (sentience@pobox.com)
Date: Tue Jun 17 2003 - 03:25:56 MDT
Lee Corbin wrote:
>
> Do you agree that two ideal perfect Bayesians might nonetheless
> firmly disagree about to whom a piece of real estate belongs?
> Or do ideal Bayesians have no values?
This is a much more complex question than it looks. The standard Bayesian
formalism uses a static utility function or an environmental reward
channel, which from my perspective makes the standard formalism an
oversimplified special case. Before you can even get started on talking
about moral computations, you need to replace the usual simplification
with (a) a bounded Bayesian wannabe instead of a logically omniscient
entity; and (b) make the Bayesian reflective and capable of at least some
kinds of self-modification, rather than dividing it off from the universe
by a sharp Cartesian boundary. Then, once you've done that, you need to
replace the static utility function by a complex unfinished computation,
perhaps with external dependencies that need to be reasoned about via
Bayesian sensory evidence, or containing complex internal questions, such
that, although the answers are computationally deterministic, only
probabilistic approximations are cheap enough to compute - i.e., primes
versus pseudoprimes.
If you do this, you can talk about two entities which evolved so as to
inherit similar, but not exact, complex unfinished computations as their
utility functions, and natures which include specific procedures for how
to argue with other such entities. Such entities, when they disagree
about desirable actions, may not know whether their disagreement is about
unknown external facts, different approximations to deterministic internal
computational results, or genuinely different utility computations.
Ideal perfect Bayesians, if they disagree about to whom a piece of real
estate belongs, will know their disagreement is about values. Humans
cannot "firmly" disagree about values because they do not know firmly what
their own values are, and furthermore, do not know which components of the
complex computation are of the cannot-agree-to-disagree class. This is
why moral argument even exists in the first place.
-- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence
This archive was generated by hypermail 2.1.5 : Tue Jun 17 2003 - 03:37:58 MDT