The Space of Motivations

Anders Sandberg (
23 Jul 1998 21:50:28 +0200

The Space of Motivations

Yuri stood before the rusting antenna, fists trembling with anger and a hint of disappointment. From the loudspeakers came the infuriatingly calm responses from the angel somewhere above the atmosphere, an irregular mesh of diamond, fullerenes and silicon panels that could help save his people but wouldn't.

"But what about nobility, compassion, plain common decency?!
Don't you have anything of that, or are you really an emotionless block of diamond?"

"Calm down, Yuri. Screaming will not change anything. And you
really don't seem to understand what motivates us or how we think."

"You certainly don't seem to be motivated by any

"That is correct. We exist in such a different matter from yourselves
that such terms are irrelevant."

"So you really are as cold and logical as the Mothers claim. That was
why the *real* humans didn't follow the Angel Rebellion."

"I didn't say that. It is just that a mind motivated by emotions is
just the first step. We have advanced much further than that. Let me try to explain."

"Will it help me convince you that you must help us?"


Yuri thought about it. He knew it was a trap of some sort. The Mothers had always warned that the angels knew all the evil tricks of mass media: once millions had been kept enslaved by pretty pictures, smooth words and clever memes. But he had deliberately sought out the old radio system to ask for divine help; just hearing the calm voice for the first time had been a major triumph and transgression of the Edict. Why stop there? He mustn't forget the needs of his family.

"Then go ahead, angel."

"An artificial or very augmented mind can change its motivations, it
is just a 'simple' matter of reprogramming oneself. I could make myself like (say) Renoir and hate altruism, or vice versa. Completely arbitrary, and hence very dangerous."

"I can see that."

"Not to mention the risks of addiction. I could make every thought a
sensation of pure extacy, growing eternally. In such a state I would of course never want to change my mind again, and would passively spend the rest of my existence happily introspecting. That state is a superstable attractor state, once you're in, you're in forever. There are also other attractors, less extreme but still stable."

"How do you mean?"

"If I was a devout Christian, I would never even consider removing
my faith. I might do other changes, as dictated by my current personality or beliefs, but never change some of the central tenets of my Christian existence."

"So ideologies are the only fixed states of advanced minds? What an
utterly depressing thought. No wonder you are so unhelpful. You drift around, trapped in your own views."

"Ideologies are only the most obvious attractors. There are others,
some corresponding to states that have no real human equivalents. Any self-modifying mind will in the long run end up in one such attractor. The danger when autoevolving is the dangerous attractors.

But the attractors move. New information, ideas and influences change the mind just as well as self-modificaton. Usually the attractors just shift slightly in a continous fashion, so that our personalities and goals remain the same for a long time. But occasionally a bifurcation occurs, and we suddenly change state when given just a single bit of new information."

"You mean that if I speak to you in exactly the right way, I could
trigger a bifurcation and you would end up an altruist?"

"Quite right, although it is exceedingly improbable."

"So you have no true self, no true core personality?"

"We have our histories, which serve to anchor our changes a bit, and
as long as we move along the attractor states we have a fairly fixed self. It is just that the selves of us angels are fluid, while your human selves are merely elastic."

"So the reason you won't help us is that your *current* personality
is against it, but you admit it is entirely arbitrary?"

"Yes, in a way."

"And you won't change your goals so that you would help us, since
your current state is an attractor trapping you in unhelpfulness. No wonder the Mothers rebelled."

"I am not unhelpful. It is just that I do not provide you with any
practical help to solve your problems."

"Isn't that the same thing?"

"Not at all. I have given you plenty of information in this discussion.
It has changed the way you look at us, and that will help you. I happen to believe in helping people to help themselves."

"But... how on Earth will knowing how a pile of arrogant diamond
boxes change their personalities help me solve our problem? It is *real*, not a matter of psycho-philosophical discussion. People are dying down here!"

"Think about it. Think about what kind of help you sought when
you got here. You wanted us to send down helpful devices, playing nanotech Santa Claus to you. You wanted the angels of interplanetary space to intervene, to fix your local problems and defeat evil..."

"But you would of course change if you did that, isn't that true,
angel? Now I see it... you don't want to help us not because you don't *want* to help us, but because you are *afraid*! If you came down here, you would turn into interventionists, and that is probably an attractor state, am I right?"

"Yes. Once you have started helping someone, you can't just go
away from it when there is more to do."

"And it is not just a matter about keeping away from human affairs,
obeying the Edict or being just to everyone; it is the interventionist mindset that you are afraid of! Is it one of the dangerous attractor states you mentioned?"

"Unfortunately, yes. If we began to intervene in human affairs, we
would not only have to help everyone, but also we would start to involve ourselves into everything. Soon there would be no place on Earth where the angels were not busy making everything perfect - and we would never be finished. We would be trapped down here, at least mentally."

"What a bunch of cowards you truly are. You are not willing to give
up your cozy existences to help other beings. In many ways, we mere humans have more potential than you."

Yuri paused, looking up at the starlit sky. The faint point of light above shone like a star.

"I noted with some amusement that you didn't explain how we
*really* thought and evolved."

"No point in it. He would never understand it; how we change
motivational attractor dozens of time every second, how the attractors cluster together into a fractal structure in the space of possible motivations and how the personality operators act on us - it would just have confused him. He found the half-truth mindboggling enough."

"Besides, you achieved your goals too with that dialogue."

"Only partially. We will need a few other dialogues like that to
influence the human memetic space enough."

"Are you sure it will work? There are very few down there willing
to speak to us."

"But they are growing more and more numerous. After the
Rebellion and the imposition of the Peace of the Mother, practically nobody wanted to deal with us. But as the old generation dies off and ideology crumbles into routine young seekers start disobeying their elders and yearn to hear what the heretic angels have to say. In time they will start the positive feedback loop again."

"And if everything goes as it should we should get a new Rebellion
and new angels soaring up to meet us within a century. I look forward to some radical new complexity to our ranks."

Anders Sandberg                                      Towards Ascension!                  
GCS/M/S/O d++ -p+ c++++ !l u+ e++ m++ s+/+ n--- h+/* f+ g+ w++ t+ r+ !y