Eliezer writes:
> If, as seems to be the default scenario, all supergoals are ultimately
> arbitrary, then the superintelligence should do what we ask it to, for lack of
> anything better to do.
Before it is a super-intelligence, it will be as smart as a genius human.
Before it is a genius, it will be as smart as an ordinary human.
Before it is a human, it will be as smart as a very dumb human being
(albeit perhaps with good coding skills, an idiot savant of programming).
And before that, it will be a smart coding aid.
In those early phases, you are going to have to direct it. It could
no more choose its own goals than Eurisko or Cyc.
Even as it approaches human level, it's not going to be able to
spontaneously decide what to do. This isn't something that will just
emerge. It has to be designed in. The creators must decide how the
program will allocate its resources, what will guide its decisions about
what to work on.
I don't see how you can transition from a system which had designed-in
structures which control how it spends its time, to one which lacks such,
without it spinning its wheels.
The very notion of a system which "chooses its own goals" seems
contradictory. Choice presupposes a ranking system, which implies a
pre-ordained goal structure. Choosing your own goals is like proving
your own axioms, or lifting yourself by your bootstraps. It's not going
to get off the ground.
You can build the computer to do what you tell it to, and you probably
would need to do this, in the early stages. It's just not clear what
you replace it with when you decide to turn it off.
Hal
This archive was generated by hypermail 2b29 : Mon Oct 02 2000 - 17:37:13 MDT