On Sat, 23 Oct 1999, Skye Howard wrote:
I believe that Skye may have hit upon something every
interesting that partially solves a major problem
involving self-modifying AI that has plagued the
list and worried me greatly:
> It might be interesting if you could create an
> artificial environment where you could test such
> things. For example, if you had an artificial human
> body existing on some kind of programmed
> could instill these devices into it and see if all of
> the simulated functions could continue... this would
> be a ways beyond modern technology, though, because
> an artificial computer model of a genome and the full
> animal generated therefrom, not to mention more
> processing power and memory space than a medium sized
> country, might be necessary:)
The fundamental problem I fear with self-evolving AI is the connection to the real world as we perceive it. However, if a self-evolving AI is operating in a simulation of the real world, then the problem becomes much more tractable. First, the changes take place more slowly so we have a greater ability to monitor them. Second, if the program shows signs of modifying itself into self-conscious sabotage of extra-environmental entities it can be suspended/deleted.
The questions then become:
(a) Can we guarantee that the AI never discovers it is running on,
and more importantly escape from, a simulation machine?
This goes back to the entire thread of whether we can detect *we*
are running on a simulation or whether our reality is an illusion.
(b) How do we guarantee that everybody understands and
adheres to the rules that self-evolving AIs are only
allowed to exist in simulated worlds? {This is not
dis-similar from the problem of how do we guarantee
that petty dictators don't release bioweapons that
in the process of killing us, come back to "bite" them.}
I think this fundamentally comes down to a core extropian principle involving rational thought. Rationale people presumably seek to preserve themselves and do not undertake operations that have a significant risk of their own death in the process of killing others (in contrast to the tradeoff of saving others lives at the risk of ones own life, which may be quite rational depending on how you value the lives involved).
This devolves into 2 basic discussions:
(a) whether an AI can discover it is running in a simulation?
(b) whether people (the irrationals) who are willing to sacrifice
themselves can/will create non-simulation environments in
which to evolve AIs.
Robert