Billy Brown writes:
> I don't think we were really getting anywhere with the previous line of
> responses, so I decided to try it again from the beginning. Here goes:
Yep, this has been also my impression. We do really seem to have unresolvably different ways of seeing matters. Not that diversity is a bad thing, but it sure turns tedious having to reiterate things/sugarcoat them in different verbiage.
> Regarding Current Software
> Current PC software is written for hardware that is actually in use, not
> hypothetical designs that might or might not ever be built. This is
> perfectly logical, and I don't think it makes sense to blame anyone for it.
To name a few notorious perpetrators: Big Blue, Intel, Microsoft have been reserved a special circle in Dante's Hell. (At least in my private universion of it).
> If a better architecture becomes available, we can expect ordinary market
> forces to lead them to support it in short order (look at Microsoft's
> efforts with regard to the only-marginally-superior Alpha chip, for
> example).
> The more sophisticated vendors (and like it or not, that included Microsoft)
The trouble with Microsoft is that we're reasoning with a marketplace flattened by more than a decade of its influence as a point of reference. To be fair we must evaluate multiple alternate branches of reality-as-it-could-have-been, which necessarily makes for extremely subjective judgements. Your mileage WILL vary.
> have been writing 100% object-oriented, multithreaded code for several years
> now. They use asynchronous communication anywhere there is a chance that it
I hear you. It is still difficult to belive.
> might be useful, and they take full advantage of what little multiprocessor
> hardware is actually available. There is also a trend currently underway
Well, essentially all we've got is SMP. I guess it makes use of shared-memory paradigm which is a dead-end. Shared-memory is at least as unphysical as caches, in fact more so.
> towards designing applications to run distributed across multiple machines
I know clustering is going to be big, and is eventually going to find
its way into desktops. It's still a back-assed way of doing things,
maybe smart RAM will have its say yet. If only Playstation 2 would be
already available, oh well. Marketplace will sure look different a
year downstream. Difficult to do any planning when things are so in
flux.
> on a network, and this seems likely to become the standard approach for
> high-performance software in the near future.
> Regarding Fine-Grained Parallelism
> Parallel processing is not a new idea. The supercomputer industry has been
Heck, LISP is the second oldest HLL known to man, and Alonzo Church invented lambda calculus in the 1940's. Eniac was a RISC machine. Unix is a 1970's OS. GUIs/mice/Ethernet are 1970's technologies. Von Neumann invented cellular automata in the 1950's.
What has age anything to do with how good an idea it is? If anything, a brand-new untried idea is something to be wary of.
> doing it for some time now, and they've done plenty of experimenting with
> different kinds of architectures. They have apparently decided that it
Nope, sorry, don't think so. Most of IT landscape is shaped by vogues, and right now parallelism is getting fashionable (what a damnable word) again. (While ALife is heading into oblivion, which is imo a damn shame).
> makes more sense to link 1,000 big, fast CPUs with large memory caches than
> 100,000 small, cheap CPUs with tiny independant memory blocks. That fits
Heck, caches hierarchies are unphysical. Fat CPUs are incommensurable with nonnegligeable fast on-die SRAM blocks _and_ high good die yield. Wafer-scale integration is impossible without good die yield and failure tolerance. Kbit buses are impossible if you don't have embedded DRAM technology due to packaging constraints. Embedded RAM technology is hardly one year old. VLIW is at the threshold of going mainstream, and VLIW only makes good sense with kBit broad buses/on-die memory. High code density is impossible without threaded code, and threaded code requires stack CPU hardware support. Common HLLs don't support threaded code/stack CPUs. No language supports fine-grain maspar systems. Blahblahblah. I could go on for a long time but (I hope) my point is nauseatingly clear: it's a big ball of yarn buried in yonder tarpit, and requires a whole lotta muscle to haul it out. You have to do it in one piece because everything is coherent/contiguous/synergistic. A bit of tarry string wrestled from the pit won't excite anybody. Please go for the whole hog.
IT is just another acronym for Inertia Technology. We're caught in a local minimum, but this doesn't mean there is no lower one. And a (number of) decision(s) has been made in the past which made us land in this particular minimum. It could have been a different one.
Sorry if this sounds like a just another technoshaman mantra, but that's just how things are.
> perfectly with what I know about parallel computing - the more nodes you
> have the higher your overhead tends to be, and tiny nodes can easily end up
> spending 100% of their resources on system overhead.
There are codes where Amdahl is going to bite ya. There are a lot where overhead is not a problem. My particular problem (from the domain of physical simulation on a 3d lattice) is the latter case.
> Now, if someone has found a new technique that changes the picture, great.
> But if this is something you've thought up yourself, I suggest you do some
> more research (or at least propose a more complete design). When one of the
Heck I did it years ago. Somebody even used the writeup in a CPU design class. As I don't have a fab and several 100 M$ to burn (and, incidentally, more important research to do) I can hardly be expected to assault the buttress alone, can I? You'll wind up in the moat, all dirty & bloody, and have to listen to stupid French jokes and be used for carcass target practice in the bargain.
> most competitive (and technically proficient) industries on the planet has
> already tried something and discarded it as unworkable, its going to take
> more than arm-waving to convince me that they are wrong.
Right, the whole area of supercomputing is going to vanish into a logics cloud overnight -- because as everybody knows they are all monoprocessors. Beowulf is just a passing fad -- pray no attention to exponential growth of 'wulfers. Photolitho-semiconductor CPUs will scale in to 10, 100, 1000 GHz regime trivially. Einstein was dead wrong, and you can signal faster than speed of light in vacuum. It makes actual sense implementing a Merced in buckytube logic. People who proved that reversible cellular automata in molecular logics are the most efficient way of doing computation were just dweebs. Right.
> Regarding the Applicability of Parallelism
> The processes on a normal computer span a vast continuum between the
> completely serial and the massively parallel, but most of them cluster near
> the serial end of the spectrum. Yes, you have a few hundred process in
Says who.
> memory on your computer at any given time, but only a few of them are
> actually doing anything. Once you've allocated two or three fast CPUs (or a
> dozen or so slow ones) to the OS and any running applications, there isn't
> much left to do on a typical desktop machine. Even things that in theory
I guess I don't have a typical desktop machine, then. I could really use an ASCI Red here, or better one of these kCPU QCD DSP jobs.
> should be parallel, like spell checking, don't actually get much benifit
> from multiple processors (after all, the user only responds to one dialog
> box at a time).
Spell checking? I never do spell checking. I do have the C. elegans genome sitting on my hard drive here, though, which I'd love to do some statistical analysis on. Guess what? Another embarrasingly parallel app.
> On servers there is more going on, and thus more opportunity for
> parallelism. However, the performance bottleneck is usuall in the network
You know what? We're going to move to xDSL pretty quick. And we're going to need a database-backed web site, both for the intranet and the outside. No never fork no more...
> or disk access, not CPU time. You can solve these problems by introducing
> more parallelism into the system, but ultimately it isn't cost-effective.
> For 99% of the applications out there, it makes more sense to buy 5
> standardized boxes for <$5,000 each than one $100,000 mega-server (and you
> get better performance, too).
Well, I guess I must be pretty special, because the $100,000 mega-server doesn't make at all sense when you want to do multi-million particles MD. Lots of cheap PC with full duplex FastEthernet, very much yes. And there is no bottleneck, since from a certain minimal system size/node onwards the things scales O(N), and I mean _strictly_ O(N).
> Of course, there are many processes that are highly amenable to being run in
> a parallel manner (video rendering, simulation of any kind, and lots of
> other things), but most of them are seldom actually done on PCs. The one
Well, I hate to keep repeating this, but it is not really that rare you seem to think it is.
> example that has become commonplace (video rendering) is usually handled by
> a specialized board with 1 - 8 fast DSP chips run by custom driver-level
> software (once again, the vendors have decided that a few fast, expensive
> chips are more economical than a lot of slow, cheap ones).
Cheap!=slow. A $30 DSP can outperform a $300 CPU because it don't have to put up with legacy bloat.
> Side Issues
> 1) Most parallel tasks require that a large fraction of the data in the
> system be shared among all of your CPUs. Thus, your system needs to provide
YMMV. Mine don't.
> for a lot of shared memory if it is going to be capable of tackling
Shared memory does not exist, at least not >2-4 ports. If you attempt to simulate that, you will have to pay dearly in logic and cache coherence issues, which starts to slow you down very quickly (point of diminishing returns is just round the corner). You can simulate shared memory with message-passing, though. If you really, really really need it.
> molecular CAD, atmospheric simulations, neural networks, etc. That brings
<laugher>. Molecular CAD, weather codes and neural codes are already are or patently formulable in an embarrasingly parallel way. Really. Look on the code shelves.
> up all those issues of caching, inter-node communication and general
> overhead you were trying to avoid.
Caching doesn't exist (pray pay no attention to the clever fata morgana). Internode-communication is readily addressable (=solved) by having a failure-tolerant routing protocol with switch fabric built into the CPU. Next Alpha is going to have multi-10 GByte/s inter-CPU signalling with 15 ns signalling latency. 3d lattice topology (6 links/CPU) is really sufficient for most codes I care about.
See SGI/Cray, DSP clusters and Myrinet Beowulfs for illustration.
> 2) You also can't get away from context switching. Any reasonably complex
> task is going to have to be broken down into procedures, and each processor
> will have to call a whole series of them in order to get any usefull work
> done. This isn't just an artifact of the way we currently write software,
> either. It is an inevitable result of the fact that any interesting
> computation requires a long series of distinct operations, each of which may
> require very different code and/or data from the others.
Strangely, my needs are very different.
> Billy Brown, MCSE+I
> ewbrownv@mindspring.com