Billy Brown wrote:
> I see two fatal criticisms of the idea suggested in the FAQ:
>
> First, it is mind control. Remember, posthumans are by definition fully
> sentient beings. Programming them to abide by a preordained moral code is
> no different than doing the same thing to our own children, or to each
> other. I can see no possible way to justify such an action on moral
> grounds.
I think you are confusing to issues here. (1) doing it to our
children; (2) doing it to each other. These cases are diffrent.
(1) There is nothing morally wrong in doing it to our children.
(2) Doing it to each other, on the other hand, is only allowed through certain means such as persuation. The reason why there is a moral difference between the two cases is that in the first case there is no pre-existing person with violable rights. There are just different potential persons, and we make a choice about which one shall come into being (by choosing who we copulate with and how we raise our children, traditionally; or by choosing the values we program into a superintelligence, in the hypothetically). We can assume that we only choose between sets of values that the person would be happy with once it is in existence, and that would lead to a life worth living. In the second case, there are pre-existing persons, and if they don't want to be transformed by your mind-changing methods, then you have no right to transform them. (I'm skipping over some subtleties here that aren't relevant to my main point.)
> Second, in the long run it won't work. You can program non-sentient
> machines to do your bidding, but that isn't what the FAQ is talking about.
> It suggests that we attempt to enforce an eternal program of mind control on
> fully sentient posthumans - for those of you who have forgotten what that
> means, I suggest you read the definition of 'Posthuman' in the same
> document. Does anyone really think such a prohibition would work on humans
> for all of eternity? Then why do we think we can do it to an entire society
> of superintelligent, self-modifying entities?
Two points: First, a being who has a certain fundamental value *doesn't want to change it*, per definition. So it's not as if these guys will think they are being mind-fucked and try to figure out a way to get around it. No more than you are trying to abolish your own survival instinct just because you know that it is an artifact of our evolutionary past.
Second, later generations of posthumans may well be born or created by earlier generations. They might well have the ability and the will to propagate their fundamental values to their creations/offspring. (If you love humans, you wouldn't choose to have a child that will hate humans and exterminate them!) And they could presumably invent a highly reliable mechanism for transmitting their values to the next generation.
Nick Bostrom
http://www.hedweb.com/nickb n.bostrom@lse.ac.uk
Department of Philosophy, Logic and Scientific Method
London School of Economics