Eliezer S. Yudkowsky wrote:
> > But I don't think we deliberately change our fundamental values.
> > Non-fundamental values we may change, and the criteria are then our
> > more fundamental values. Fundamental values can change too, but they
> > are not deliberately (rationally) changed (except in the mind-scan
> > situation I mentioned in an earlier messege).
>
> Well, either I've misunderstood you, or you're simply wrong. We humans
> switch fundamental values all the time. It happens every time someone
> changes a religion. If you're going to argue that these weren't the
> true "fundamental" values, then the AI's "make people happy" won't be a
> fundamental value either.
My answer to this is a superposition of three points: (1) I explicitly allowed that fundamental values could change, only (except for the mind-scan) the change wouldn't be rationally brought about. For example, at puberty peoples' values may change, but it's not because of a rational choice they made. (2) Just because somebody calls a certain value fundamental doesn't mean it actually is fundamental. Especially ideological values are, for humans, often better described as their official policies rather than their true driving force. Otherwise everybody would be perfectly willing to suffer torment and death for the sake of their professed "fundamental values"; but since they don't, those values are either not fundamental or are just part of their fundamental values. (3) With imperfectly rational beings (such as humans) their might be conflicts between what they think are their fundamental values. When they discover that that is the case, they have to redefine their fundamental values as the preferred weighted sum of the conflicting values (which thereby turned out not to be truely fundamental after all).
> My fundamental values have changed from "eat and sleep and survive" to
> "serve humanity" to "bring about a Singularity" to "do what is right",
> where I presently reside.
>
> I think that it will be a considerable amount of time before an AI is
> pressed by logic to change its fundamental values from "do what is
> right". But anything more specific certainly isn't a fundamental value.
"Do what is right" sounds almost like "Do what is the best thing to do", which is entirely vacuous.
> > That depends. If selection pressures lead to the evolution of AIs
>
> What selection pressures? Who'd be dumb enough to create an AI wanting
> to survive and reproduce, and, above all, *compete* with its children?
> > with selfish values that are indifferent to human welfare, and the
> > AIs as a result go about annihilating the human species and stealing
> > our resources, then I would say emphatically NO, we have a right to
> > expect more.
>
> Absolutely. I do not intend to let humanity be wiped out by a bunch of
> selfish, badly programmed <Hs
Nick Bostrom
http://www.hedweb.com/nickb n.bostrom@lse.ac.uk
Department of Philosophy, Logic and Scientific Method
London School of Economics