META: Problems with collaborative filtering (was Re: Improving the list

Bartley R. Troyan (btroyan@littlekoala.com)
Tue, 02 Mar 1999 02:02:59 -0500

Anders Sandberg wrote:
> While the kill thread and topic stuff might not be necessary, I like
> the idea of Kookify/Glorify! A more soft incentive to produce good
> posts.

I don't like the idea of Kookify/Glorify as I understand it. Let's say you all decide I'm a Kook and virtually everyone heavily Kookifies me by clicking the Kookify link at the bottom of every message I send to the list. I don't care how you prevent multiple votes (it's not that hard), but it won't make a difference in the end... If I post a lot of rubbish for a while, I will attain an extremely high level of Kookiness after a short time.

Now, let's say perchance six months down the road the tradition develops that people tend to filter out Kooks (based on an X-Kook-Level or X-Kook-Ratio header or something included in all of the messages). They rarely, if ever, read messages from someone as certifiably Kooky as me. And now that there are, say 100 Million AOLers, there are also 10K subscribers to Extropians mailing list and lots of other kooks posting all sorts of junk. So assume that a filtered version of the list is created, that the AOL hordes (no offence EvMick) can subscribe to on the web site. Extropians-No-Kooks provides a list view in which the messages from Kooks at level x or greater are not even sent out with normal list traffic (say, due to bandwidth and other resource constraints. It's not cheap to send 300 pieces of email to 10K different email addrs every single day. But that's the level of traffic one must anticipate 6 months or a year down the road. It's a problem the entire net will continue to face as it continues to grow rapidly).

Anyway, therefore I will never be de-Kookified, even if I suddenly experience a Zen-like enlightenment and start posting like a networked super-brain better than a merged consciousnesses of the best posters (you know who you are). I've been rolled into a local minimum in Kook-space and now I can't get out. And everybody loses out on my godlike wisdom and conspiracy-theory-debunking insights.

My point is, maybe both the Glorifieds and the Kooks should be given an automatic, periodic, proportional adjustment in the opposite direction of their current tendency, just to make sure nobody gets stuck in any holes.

> I have been thinking of running some program through the list archives
> to do data mining, for example build sociograms (who responds to
> whom?), find threads where someone participates, who glorifies who and
> perhaps extract clusters of discussion. Maybe the way of improving
> quality of the list lies in making archives more powerful?

A lot of that stuff is interesting to me as well. I'll bet you'd find some of cliques that respond primarily to each other, and ignore newbie/clueless posts as well as posts from Already Certified Kooks. Then there are the debaters, and the people with way too much time on their hands that respond and argue everything. (I'm not saying that's bad, I'm just saying, well, I don't have that much time. I can only afford to catch up with this list, all msgs dumped into a separate folder, once or twice a day, and I am forced to use the delete key without reading most messages).

Anyway, to implement the kind of data mining and analysis Anders describes, in an efficient and easy-to-program manner, we would need a database, of course. Anyone want to donate cycles and space on a mySQL server on a really fast box with a gig or two of storage? Give me a login there, I'll give you everything Anders described above and much more. We'd also need a way to easily modify headers and footers of list messages on a per-message basis. It could start small and grow to be an engaging, real-life experiment in collaborative filtering. I wouldn't mind a share of that $10K if you all like it well enough :-) And if list members mostly just ignore those funny new extra footers on the bottom of all the list messages, so what. We'll take them out. No harm done.

-Bart
(aspiring Extropian Kook)

> --
> -----------------------------------------------------------------------
> Anders Sandberg Towards Ascension!
> asa@nada.kth.se http://www.nada.kth.se/~asa/
> GCS/M/S/O d++ -p+ c++++ !l u+ e++ m++ s+/+ n--- h+/* f+ g+ w++ t+ r+ !y