"Billy Brown" <bbrown@conemsco.com> writes:
> Anders Sandberg wrote:
> > As you can tell, I don't quite buy this scenario. To me, it sounds
> > more like a Hollywood meme.
>
> Yes, it does. I didn't buy it myself at first - but the assumptions that
> lead to that conclusion are the same ones that make a Singularity possible.
Yes and no. The assumptions below certainly make a Singularity possible, but the reverse is not true. Vinge's original (vanilla?) Singularity was simply that the effective intelligence-enhancing ability of mankind as a whole increased, and thic could happen through far less dramatic means.
I must admit that I'm disturbed by the *faith* many seem to put into the Singularity. It is not a necessary part of transhumanist thinking, it is just one wild prediction among many.
> Taking the objections one at a time:
>
> > OK, this is the standard SI apotheosis scenario. But note that it is
> > based on a lot of unsaid assumptions: that it is just hardware
> > resources that distinguish a human level AI from an SI (i.e., the
> > software development is fairly trivial for the AI and can be done very
> > fast, and adding more processor power will make the AI *smarter*),
>
> Actually, I assume there is a significant software problem that must be
> solved as well. That's what makes it all so fast - the first group that
> figures out how to make an AI sentient enough to do computer programming
> will be running their experiment on very fast hardware.
Well, one thing a fellow researcher told me was that the world-class lab at a famous American university where he did his postdoctoral work had worse computers than we have in our student labs (and our institute has little money compared to the university). So excellence in programming doesn't give you the best hardware. But you are talking about hardware in general, so let's continue.
> > that this process has a time constant shorter than days (why just this
> > figure? why not milliseconds or centuries?),
>
> The time constant for self-enhancement is a function of intelligence.
> Smarter AIs will improve faster than dumb ones, and the time scale of human
> activity is much harder to change than that of a software entity. In
> addition, an AI can have a very fast subjective time rate if it is running
> on fast hardware. Thus, the first smart AI will be able to implement major
> changes in days, rather than months. I would expect the time scale to
> shrink rapidly after that.
I agree with some of the above assumptions (time constant depending on intelligemnce, smarter AI can likely improve faster, human have a hard time changing timescales). But just because AI *could* be very fast doesn't mean it *will* be very fast - that is the assumption you are trying to prove. Can you give any support for why an AI program would have a specific speed?
In addition, the human-level AI might be significantly slower than humans, so that even if it can build a better AI it will be faster to use humans. AI will only speed its own development if the number of collaborating AIs times their ability divided by their speed is smaller than the number of collaborating humans times their ability divided by human speed. Collaboration of course introduces logistic problems.
> > that there will be no
> > systems able to interfere with it - note that one of your original
> > assumptions was the existence of human-level AI;
>
> No, my assumption is that the first human-level AI will become an SI before
> the second one comes online.
Why is the human level so critical? Self-enhancement might be doable by less complex systems or perhaps require significantly more advanced systems, depending on form of intelligence and designability. And system security could definitely be efficiently provided by less-than-human level AI to an extent that hampers even a very bright mind (ever tried to silently sneak past a watchdog?).
> > that this SI is able to invent
> > anything it needs to (where does it get the skills?)
>
> I presume it will already have a large database on programming, AI, and
> common-sense information. It will probably also have a net connection - I
> would expect a program that has access to the WWW to learn faster than one
> that doesn't, after all. By the 2010 - 2030 time frame that will be enough
> to get you just about any information you might want.
Even *skills*? Wow. Somehow I'm not entirely convinced. And if you really can learn skills through the net in this scenario, then the *humans* will be well on their way to SI.
There is a difference between data, information, knowledge and ability.
> > and will have
> > easy access to somebody's automated lab equipment (how many labs have
> > their equipment online, accessible through the Net? why are you
> > assuming the AI is able to hack any system, especially given the
> > presence of other AI?).
>
> Again, by this time frame I would expect most labs to be automated, and
> their net connections will frequently be on the same network as their
> robotics control software. You don't need to be able to hack everyone, you
> just need for someone to be stupid.
>
> Besides, the AI could expand several thousand fold just by cracking
> unsecured systems and stealing their unused CPU time. That speeds up its
> self-enhancement by a similar factor, which takes us down to a few minutes
> for a major redesign. I expect a few hours of progress at that rate would
> result in an entity capable of inventing all sorts of novel attacks that our
> systems aren't designed to resist.
OK, you are simply assuming nobody notices what appears to be a super worm or virus on the net? A program that not just takes of processor power, but also hogs net resources to send dense data (its thoughts) everywhere. In a world that is obviously highly dependent on the net, where many important systems appear to be net-connected and the majority of people have grown up with the net and its pitfalls?
Personally I think this is another big Hollywood meme: the crackability of systems. Somehow it seems so simple in movies to crack mothership computers... Sure, security will always have flaws. But are they really so exploitable that somebody can take over a lot of systems easily without making any fuss? (remember things like anomaly monitoring software - I have seen a neural network classify what the users are doing and what kind of people they are; that program would immediately notice something amiss). Most likely you could run SATAN and its derivatives to get a few accounts, but in the process a lot of system operators would go on alert.
Notice that you seem to be assuming that as soon as the good AI that so far has spent its existence single-mindedly programming better versions of itself notices a lack of computing power, it quickly teaches itself to crack all sorts of systems (without anybody noticing) and then goes on with a career as engineer. Sure, it is smart, but *that* smart, flexible and able to find applicable skills without anybody noticing anything?
> > And finally, we have the assumption that the
> > SI will be able to outwit any human in all respects - which is based
> > on the idea that intelligence is completely general and the same kind
> > of mind that can design a better AI can fool a human into (say)
> > connecting an experimental computer to the net or disable other
> > security features.
>
> I don't think intelligence is entirely general - my own cognitive abilities
> are too lopsided to permit me that illusion. A merely transhuman AI, with
> an effective IQ of a few hundred, might not be any better at some tasks than
> your average h
> An SI is a different matter. With an effective IQ at least thousands of
> times beyond human average, it should be able to invent any human cognitive
> skill with relative ease. Even its weakest abilities would rapidly surpass
> anything in human experience.
But you are assuming what you want to prove, namely that the AI can grow into an unstoppable SI. But you have not managed to show that the human-level AI will reach SI level just by being in the AI-researchers' big computer.
-- ----------------------------------------------------------------------- Anders Sandberg Towards Ascension! asa@nada.kth.se http://www.nada.kth.se/~asa/ GCS/M/S/O d++ -p+ c++++ !l u+ e++ m++ s+/+ n--- h+/* f+ g+ w++ t+ r+ !y