From: Ken Clements (Ken@Innovation-On-Demand.com)
Date: Thu Jan 17 2002 - 23:48:04 MST
Wei Dai wrote:
>
> Human individuals and societies tend to change because
> individuals die and are replaced, or we adopt some idea that is initially
> attractive and then find out that it doesn't work, or we learn something
> new and surprising about the nature of reality. SIs presumably will not
> have these problems. It seems plausible that they will quickly settle into
> a stable state with long term goals that don't change over time.
>
> In other words, I think rational design and goal-oriented action will play
> a much bigger role than survival-of-the-fittest kind of evolution in
> governing how SIs change over time.
>
The first rule here is that we have no way to predict anything about the
behavior of SIs. If we did, there would be no 'S' in there.
We are mostly defined by our limitations. We have to have air and water and
food and warmth and gravity and pressure and all kinds of psychological
connections. Our cells break down with time, and our self repair systems do
not regenerate all kinds of things we would like to have made new. But our
biggest limitation, up until now, is that we could not go in and mess with our
own source code.
Although we do not know what SIs will be like, we can take a good guess that
they will be in control of their own designs. This may mean that SIs are
inherently unstable. It may turn out that the goal-oriented action of making
modifications to oneself (so as to blow off limitations) is so addictive that
once you can do it, you must do it. If so, the opportunity for runaway
feedback loops will be legion.
Again, guessing, one would expect that your typical SI would be interested in
knowing more about how the Universe works. If over a period of, perhaps, a
long time some major discoveries are made, it is almost sure to be possible
for the SI to use this knowledge to add new capabilities to itself. It is
hard to believe that the SI will decide to just "Stay happy the way I am" (SI
Zen Masters notwithstanding). Would you?
-Ken
This archive was generated by hypermail 2.1.5 : Fri Nov 01 2002 - 13:37:35 MST