Shirky: Problem is filter failure, not info overload

I didn't attend the Web 2.0 Expo in New York last year, and so the exceptional keynote speech of Clay Shirky, a New York University new-media professor, writer, and consultant on the social and economic effects of Internet technologies.

The keynote, "It's not information overload. It's filter failure," is an insightful exploration of Internet economics and an intelligent response to Nick Carr's "Is Google Making Us Stupid?" argument.

If you haven't watched it, you must. It does more to explain the dearth of effective information filters that we wade through today. It has application to open source (180,000-plus projects on SourceForge, but which are useful?), but far broader implications.

You can watch it here:

But here is where he lays out the crux of the problem, which I've transcribed:

The other problem that Gutenberg introduced into intellectual life was the problem of risk. If you owned a printing press, you could make money, if people bought your books, but you could lose money if people (didn't) buy your books. And since you had to print the books in advance, you were taking on all the risk of whether or not those books would sell....This is the problem of publishing.

The economic solution was pretty simple: make the publisher responsible for filtering for quality. There's no obvious reason that someone (who) is good at running a printing press also ought to be good at figuring out (which) books to print.

But the economic logic of print in advance, then sell it--high up-front cost and then recoup when you reach the people--that economic logic has come to mean that the word "publisher" has come to mean two things: people who decide what to publish and people who do the publishing.

(All of the media revolutions since Gutenberg have the same characteristic:) It cost me a lot of money to get started. And so (publishers) had to filter for quality.

Here's what the Internet did: it introduced, for the first time, post-Gutenberg economics. The cost of producing anything by anyone has fallen through the floor. And so there's no economic logic that says that you have to filter for quality before you publish...The filter for quality is now way downstream of the site of production.

What we're dealing with now is not the problem of information overload, because we're always dealing (and always have been dealing) with information overload...Thinking about information overload isn't accurately describing the problem; thinking about filter failure is.

I think there's a billion-dollar business resident in Shirky's thoughts, business that Google is missing with its focus on "search." The best emphasis should be on "finding," not searching. The need is for filters of a more refined, catered kind.

[alamp23]

<<The best emphasis should be on "finding," not searching.

like what semantic web solutions are attempting to provide in general and what Microsoft is attempting with its purchase of Cuil?

Semantic web technologies have yet to be proven in the mainstream, but there are some promising contenders out there...

[alamp23]

semantic web technologies analyst: http://www.project10x.com/

[Daniel_Tunkelang]

What I particularly like in his ?filter failure? characterization is that it really exposes the human-computer interaction challenges in managing information flow (in both directions). It also reminds me of Danah Boyd?s Master?s Thesis on managing identity in a digital world, and of the outdated concept of privacy through difficulty.

http://thenoisychannel.com/2008/09/23/quick-bites-filter-failure/


As for what Google is missing, may I suggest you check out a tech talk I recently gave at Google:

http://thenoisychannel.com/2009/01/08/google-tech-talk-reconsidering-relevance/

[hymanroth]

I don't think Shirky was making a distinction between finding and searching, he's pointing out that we are immersed in ever more information and have to adapt our own behavior to deal with that.

Personally, I don't think the problem is limited to the net. The proliferation of TV channels has lead to similar issues. Channel surfing could be viewed as behavioral response to overload issues.

I believe that there are certain analogies between the parallelism and fragmentation of channel surfing with the skimming through entries in an RSS reader.

There is also probably a connection with the debate on printed media (see Tim O'Reilly, Fred Wilson) where an editor's centralized role of structuring information risks being replaced by an unstructured (and possibly incomplete/chaotic) DIY process by information consumers.

[Matt Asay]

I agree that he wasn't distinguishing between the two, but I think there's a difference (and have a few ideas as to how to make "find" work better than "search." But that's another post....

[GlassHalfFull]

While I agree with the obvious premise that filtering is the grand problem (super-human AIs -- there, that's my solution...), I don't think it gets us anywhere since at heart that's at the core of current efforts anyway (or do you page through all 317 million hits for "Snow" because you don't believe they are filtered to show you the best, most relevant results at the top...)

I also found Matt's comment a bit snarky, "There's no obvious reason that someone that is good at running a printing press also ought to be good at figuring out what books to print."

This is just ridiculous -- he's good at it because his life depends upon it! Because if he isn't good at it (or can't find a partner who is) he will starve.
Or should I riposte with "There's no reason to think that the network admins at CNET should be any good at presenting readable and intelligent copy?

[MSSlayer]

Bad riposte since network admins have nothing to do with actual content(intelligent copy).

There are plenty of publishers that wouldn't know a good book if it smacked them upside the head.

What is more ridiculous is your statement: "he's good at it because his life depends upon it!". That is like saying all programmers are good, or all cashiers are good because their livelihood depends on it. Total nonsense.

[GlassHalfFull]

Precisely. Book binder have copy editors, CNET has journalists. I didn't draw the connection between "production" and editorial -- Matt did.

As for "good" or "bad" publishers, we all have an opinion but the market is the final arbiter of who gets to keep playing.

I won't accuse you of willfully misinterpreting my "... life depends on it" comment, although I might be smelling troll...

The clear intent of this passage (see Matt's first transcribed paragraph) is that if someone wants to be a publisher, they either sink or swim. Someone who dreams of being a fisher will either be good at it (because his life depends on it!) or starve. No more bad fisher. The fact that in most societies people who excel only in their mediocrity can still make a decent living (ever tried to have a bath retiled?) in no way refutes my point -- life, and certainly economic life, is generally a sink or swim affair. The publisher will have to _learn_ to pick good books -- or he'll be broke, fast (see Matt again for the discussion of risk).

I hope that helps clear things up for you.

[hymanroth]

Actually, the assertion "There's no obvious reason that someone that is good at running a printing press also ought to be good at figuring out what books to print." comes from Shirky's talk, not from Matt.

Nevertheless I agree that it's a bit floppy. The guy who runs the presses may not know what books to print, but the guy who runs the publishing business should.

[Brian_Starks]

I don't think his comment was intended to be snarkly at all. He was simply speaking from the point of view of the first owners of printing presses who were editors, publishers and printers all of their own right. One person who moved the letters, cranked out the books, chose what to print, etc.

What he was saying is that just because a person knew how to move the letters and crank out the books, did not ensure that his ability to judge quality was as good as someone elses. He's right in this regard. You even admit as much in your post by saying "or can't find a partner who is".

His point was well made. The creators of information are not always of necessity the best filters of information.

[ddmcd]

The reported comments about what publishing is seem to oversimplify the complexities of modern publishing, but that's ok. I haven't listened to the whole presentation, though.

I do agree that the concept "information overload" is overplayed and that filtering (or search) is one of the solutions. But I also think that people need to become much more accomplished in how they filter, search for, and select information. Even without getting into modern "semantic web" technologies, the tools currently available for selecting, reviewing, analyzing, and filtering information are actually quite powerful and sophisticated. The problem is that most people aren't even aware that there is more to looking for information online than typing a string of terms into the Google search box. Whose fault is that? I'm not sure. Librarians, researchers, and others who specialize in sophisticated information-seeking don't go around advertising their skills, and some of these skills are pretty esoteric when you try to explain them to the general population. Maybe we should treat information-seeking (and filtering) as being just a important as learning how to manage money. Start when they're young.

Dennis McDonald
Alexandria, Virginia
http://www.ddmcd.com

[gmclean0402]

Perfect introduction! "Author, Brooklyn boy make good."