organization of internet material

Eugene Leitl (
Sat, 27 Jun 1998 22:48:11 +0400 (MSD)

Michael Nielsen writes:
> I am presently attempting to improve my use of the internet; a
> problem of applied Intelligence Amplification. I have a few questions
> and an observation which I would greatly appreciate comments on.

An interesting project, I'm looking forward to the summary.

> What are your most effective sources of information on the internet?

Documents, and people. People spontaneously segregate in virtual
locations, and generate interesting informations streams both
spontaneously, and upon cues. Of course you have to pay for these
resources in the same coin. Documents can be found by searching, both
locally and globally in web spider indexes. Unfortunately, the growth
of both nodes and documents therein has long outstripped the
capabilities of even distributed search engines. If apache was to
marry webglimpse, and would send the rebuilt index actively, or even
link up with others to act as a global distributed search engine
the problem would go away. While we are at that, we can fix the
backlink problem properly, and take care of stale links (if a document
is referenced more than once, it is likely to float in some cache
somewhere, so with proper addressing scheme it could survive even if
the original site is down -- once all references have vanished, the
space can be reclaimed by garbage collection).

Apart from documents and people there are interactive, dynamically
generated documents, map servers, e.g.
structure generators, translators, etc. These are becoming
progressively more important.

> How did you find those sources?

By word of mouth, hotlinks and search engines.

> How do you make use of those sources?

Personally I scan the email, crosspost relevant items to several
persons/mailing lists. The big problem remains mail management. Even
the best mail agents are incapable of handling more than 6k messages,
which forces me to start afresh frequently, leaving the gzipped INBOX
body in some vault. Once I can set up my home machine properly, I will
make a giant (well, by now several 100 MBytes of gzipped mail are to
be processed) of it, and index it with webglimpse.
I think permanent connectivity is not a pipe dream anymore, even
outside of California. Of course having a webcam, and NMEA feed from
a GPS receiver is the logical next step. Relying on cellular is
expensive, and radio modems are not ubiquitously available, not very
cheap. Also, I don't understand why people do not turn fine-grain
providers, linking up with neighbour's housenets, offering mutual
router capabilities. One should think a (even diskless) i386/i486
8 MByte Linux box with several NICs is not so very expensive.

I read the documents on the web, in the last time frequently abusing
the local mirror. Let's hope similiar services
will spring up for other fields, I'm missing this particularly for
chemistry and biosciences. Researcher's pages are very useful, but
sometimes hard to find. Almost forgot: mailing list archives are
sometimes very useful. I even made a nightly mirror of CCL archive,
unfortunately my current machine is unavailable for incoming ftp/http
requests so currently I'm the only user.

> Do you use filters or robots as an aid to processing material obtained
> from the internet?

I distrust robots, though I've never used one. I think I'll wait for
learning neural nets which learn from looking over my shoulder. I
think people are the best filters, and I'd wish more people would
participate in the effort.

> It strikes me that a very well-thought out document "How to use the net",
> along the lines of Mortimer Adler's "How to read a book" would be

Apropos books: my next project is OCRing my entire library, book by
book, and digest my CD collection into mp3. Ideally, these should live
in a wearable computer. seems like a good
starting point for a conversion. Particularly built-in networking and
IrDA make it sound ideal for hot-synching with the home unit.

> extremely useful. _All_ guides to the net I know of focus on what
> resources are available, not the applied skills which are necessary to
> make the most efficient use of the net.
> Michael Nielsen

Sorry, this has become unfocused/ranty. No time to fix it up, gotta