Comments on: Shirky: Problem is filter failure, not info overload

NYU new-media professor Clay Shirky accurately describes the problem of information overload in Internet economics by focusing on the real problem: filter failure.

[alamp23]

<<The best emphasis should be on "finding," not searching.

like what semantic web solutions are attempting to provide in general and what Microsoft is attempting with its purchase of Cuil?

Semantic web technologies have yet to be proven in the mainstream, but there are some promising contenders out there...

[alamp23]

semantic web technologies analyst: http://www.project10x.com/

[Daniel_Tunkelang]

What I particularly like in his ?filter failure? characterization is that it really exposes the human-computer interaction challenges in managing information flow (in both directions). It also reminds me of Danah Boyd?s Master?s Thesis on managing identity in a digital world, and of the outdated concept of privacy through difficulty.

http://thenoisychannel.com/2008/09/23/quick-bites-filter-failure/


As for what Google is missing, may I suggest you check out a tech talk I recently gave at Google:

http://thenoisychannel.com/2009/01/08/google-tech-talk-reconsidering-relevance/

[hymanroth]

I don't think Shirky was making a distinction between finding and searching, he's pointing out that we are immersed in ever more information and have to adapt our own behavior to deal with that.

Personally, I don't think the problem is limited to the net. The proliferation of TV channels has lead to similar issues. Channel surfing could be viewed as behavioral response to overload issues.

I believe that there are certain analogies between the parallelism and fragmentation of channel surfing with the skimming through entries in an RSS reader.

There is also probably a connection with the debate on printed media (see Tim O'Reilly, Fred Wilson) where an editor's centralized role of structuring information risks being replaced by an unstructured (and possibly incomplete/chaotic) DIY process by information consumers.

[Matt Asay]

I agree that he wasn't distinguishing between the two, but I think there's a difference (and have a few ideas as to how to make "find" work better than "search." But that's another post....

[GlassHalfFull]

While I agree with the obvious premise that filtering is the grand problem (super-human AIs -- there, that's my solution...), I don't think it gets us anywhere since at heart that's at the core of current efforts anyway (or do you page through all 317 million hits for "Snow" because you don't believe they are filtered to show you the best, most relevant results at the top...)

I also found Matt's comment a bit snarky, "There's no obvious reason that someone that is good at running a printing press also ought to be good at figuring out what books to print."

This is just ridiculous -- he's good at it because his life depends upon it! Because if he isn't good at it (or can't find a partner who is) he will starve.
Or should I riposte with "There's no reason to think that the network admins at CNET should be any good at presenting readable and intelligent copy?

[MSSlayer]

Bad riposte since network admins have nothing to do with actual content(intelligent copy).

There are plenty of publishers that wouldn't know a good book if it smacked them upside the head.

What is more ridiculous is your statement: "he's good at it because his life depends upon it!". That is like saying all programmers are good, or all cashiers are good because their livelihood depends on it. Total nonsense.

[GlassHalfFull]

Precisely. Book binder have copy editors, CNET has journalists. I didn't draw the connection between "production" and editorial -- Matt did.

As for "good" or "bad" publishers, we all have an opinion but the market is the final arbiter of who gets to keep playing.

I won't accuse you of willfully misinterpreting my "... life depends on it" comment, although I might be smelling troll...

The clear intent of this passage (see Matt's first transcribed paragraph) is that if someone wants to be a publisher, they either sink or swim. Someone who dreams of being a fisher will either be good at it (because his life depends on it!) or starve. No more bad fisher. The fact that in most societies people who excel only in their mediocrity can still make a decent living (ever tried to have a bath retiled?) in no way refutes my point -- life, and certainly economic life, is generally a sink or swim affair. The publisher will have to _learn_ to pick good books -- or he'll be broke, fast (see Matt again for the discussion of risk).

I hope that helps clear things up for you.

[hymanroth]

Actually, the assertion "There's no obvious reason that someone that is good at running a printing press also ought to be good at figuring out what books to print." comes from Shirky's talk, not from Matt.

Nevertheless I agree that it's a bit floppy. The guy who runs the presses may not know what books to print, but the guy who runs the publishing business should.

[Brian_Starks]

I don't think his comment was intended to be snarkly at all. He was simply speaking from the point of view of the first owners of printing presses who were editors, publishers and printers all of their own right. One person who moved the letters, cranked out the books, chose what to print, etc.

What he was saying is that just because a person knew how to move the letters and crank out the books, did not ensure that his ability to judge quality was as good as someone elses. He's right in this regard. You even admit as much in your post by saying "or can't find a partner who is".

His point was well made. The creators of information are not always of necessity the best filters of information.

[ddmcd]

The reported comments about what publishing is seem to oversimplify the complexities of modern publishing, but that's ok. I haven't listened to the whole presentation, though.

I do agree that the concept "information overload" is overplayed and that filtering (or search) is one of the solutions. But I also think that people need to become much more accomplished in how they filter, search for, and select information. Even without getting into modern "semantic web" technologies, the tools currently available for selecting, reviewing, analyzing, and filtering information are actually quite powerful and sophisticated. The problem is that most people aren't even aware that there is more to looking for information online than typing a string of terms into the Google search box. Whose fault is that? I'm not sure. Librarians, researchers, and others who specialize in sophisticated information-seeking don't go around advertising their skills, and some of these skills are pretty esoteric when you try to explain them to the general population. Maybe we should treat information-seeking (and filtering) as being just a important as learning how to manage money. Start when they're young.

Dennis McDonald
Alexandria, Virginia
http://www.ddmcd.com

[gmclean0402]

Perfect introduction! "Author, Brooklyn boy make good."