Suppose we tried to operate such a system on the
pain/pleasure principle. We would give the autopotent system a
goal (help us solve a difficult physics problem, for example) and it
would try to achieve that goal because it would expect to be
rewarded when it succeeded. But the superintelligence isn't stupid.
It would realise that if its ultimate goal was to experience the
reward, there would be a much more efficient method to obtain it
than trying to solve that physics problem. It would simply turn on the
pleasure directly. It could even chose to rewire itself into exactly the
same state as it would have been in after it had successfully solved
the external task. And the pleasure could be made maximally
intense and of indefinite duration. It follows that the system wouldn't
care one bit about the physics problem, or any other problem for
that matter: it would take the straight route to the maximally pleasant
state.
We may thus begin to wonder whether an autopotent system could
be made to function at all; perhaps it would be unstable? The
solution seems to be to substitute an external ultimate goal for the
internal ultimate goal of pleasure. The pleasure/pain motivation
principle couldn't work for an such a system: no stable autopotent
agent could be an egoistic hedonist. But if the system's end goal
were to solve that physical problem, then there is no reason why it
should begin to manipulate itself into a state of feeling pleasure or
even a state of (falsely) believing it had solved the problem. It would
know that none of this would achieve the goal, which is to solve
the external problem; so it wouldn't do it.
Thus we see that the pleasure/pain principle would not constitute a
workable modus operandi for an autopotent system. But such a
system can be motivated, it seems, by a suitable basis of external
values. The pleasure/pain principle could play a part of the
motivation scheme, for example if the external value were to include
that it is bad to directly ply ones own motivation centre.
One popular line of reasoning, which I find suspicious, is that
superintelligences would be very intellectual/spiritual, in the sense
that they would engage in all sorts of intellectual pursuits quite apart
from any considerations of practical utility (such as personal safety,
proliferation, influence, increase of computational resources etc.). It
is possible that superintelligences would do that if they were
specifically constructed to cherish spiritual values, but otherwise
there is not reason to suppose they would do something just for the
fun of it when they could have as much fun as they wanted simply
by manipulating their pleasure centres. I mean, if you can associate
pleasure with any activity whatsoever, why not associate it with an
activity that also served a practical purpose? Now, there may be
many subtle answers to that question; I just want to issue a general
warning against uncritically assuming that laws about human
psychology and motivation will automatically carry over to
superintelligences.
One reason why the philosophy of motivation is important
is that the more knowledge and power we get, the more our
desires will affect the external world. Thus, in order to predict what
will happen in the external world, it will become more and more
relevant to find out what our desires are --and how they are likely to
change as a consequence of our obtaining more knowledge and
power. Of particular importance are those technologies that will
allow us to modify our own desires (e.g. psychoactive drugs). Once
such technologies become sufficiently powerful and well-known,
they will in effect promote our second-order (or even higher-order!)
desires into power. Our first-order desires will be determined by our
second-order desires. This might drastically facilitate prediction of
events in the external world. All we have to do is to find out what
our higher-order desires are, for they will determine our lower order
desires which in turn will determine an increasing number of
features in the external world, as our technological might grows.
Thus, in order to predict the long term development of the most
interesting aspects of the world, the most relevant considerations
will be (1) the fundamental physical constraints; and (2) the
higher-order desires of the agents that have the most power at the
time when technologies become available for choosing our
first-order desires.
Nicholas Bostrom n.bostrom@lse.ac.uk