Re: Uploads and betrayal

Charlie Stross (charlie@antipope.org)
Wed, 1 Dec 1999 13:46:09 +0000

On Tue, Nov 30, 1999 at 05:40:53PM -0500, Robin Hanson wrote:
> >... "Yes, but you can't want what you want". In the future it
> >will be possible to reach in and select what you want (your motivation).
>
> I agree this is a big important change. And in an important way, this change
> makes it *easier* to envision the future. At first, we will
> replace what we currently want with what we currently want to want. And then
> we will replace that with what we then want to want, which is probably related

Mmmf. I'd like to agree with you, but let me play devil's advocate:

If such technology becomes available, I believe it will be used by certain regimes (yes, you know who) to ensure that what we want is *what they want us to want* (at least, for a noticeable subset of the human species).

No, this isn't some half-witted Star Trek Borg implementation. It's politics. Noam Chomsky had a point when he observed that the mechanisms of political coercion in a nominally democratically accountable state must be more subtle than those in a dictatorship. Try and extrapolate this theory to a framework where human nature is maleable and we arrive at something very frightening indeed:

"Welcome to eutopia! Everyone you see here is happy and smiling; and that's not because there are any mass graves or gulags in the background! Unlike all previous totalitarian ideologies, ours is based on the abolition of unhappiness. Pissed off? Don't like our way of life? Never mind, we'll be able to tune your qualia so that you're perfectly happy with the way things are!"

Being able to change what we want implies that _others_ can change what we want.