More Long Tail: Everyone Is Still Wrong

Every now and then another study comes out about the long tail and gets discussed in the usual places. More often than not, the end result is additional confusion, because the thing they are talking about (Chris Anderson’s book) defines the concept in many different ways, depending on what the author feels like talking about at the moment.

The latest is a working paper called Is Tom Cruise Threatened: Using Netflix Prize Data to Examine the Long Tail of Electronic Commerce, by Tom Tan and Serguei Netessine of the Wharton Business School at the University of Pennsylvania [PDF, summarized here]. It got slashdotted here, written up in The Register here, and Chris Anderson responds to it here.

Here is what the paper does. It  takes the sample of 100 million DVD ratings provided by Netflix for the recently-completed Netflix Prize, and breaks down the trends in ratings from 2000, when Netflix stocked relatively small numbers of titles and had relatively few users, to 2005 when Netflix had many more DVD titles and many more users. Then they ask whether demand for “hit movies” and “niche movies” increased or decreased over that time, as reflected in the number of ratings in Netflix’s sample data set. Not surprisingly, they notice that when measured absolutely (top 10, top 100) the demand for hits decreases and when measured in percentage terms (top 1%, top 10%) it increases.

The problem is that the Netflix Prize data set, while fascinating to explore, has nothing to say about the long tail by itself. Between 2000 and 2005 the DVD as a format exploded, with many old titles getting put on the new format, and Netflix exploded as the convenience of online movie rental took off. But comparing early Netflix to late Netflix doesn’t tell us anything at all about the evolution of consumer taste in the online world, or about the relative diversity of demand from online and ‘bricks and mortar’ stores, which is supposed to be what this is all about.

To be charitable, the nearest we can get is that it’s a comparison of a restricted set of choices (2000) and a broad set of choices (2005), but given that the size of the available set of titles increased by a factor of about 5 while the user base increased by a factor of 50, interpreting the results as “the effect on demand of this increase in variety [of titles]”, as Anderson does, is simply seeing what you’d like to see.

If we are going to take Anderson seriously then we should adopt his standard definition when the long tail gets challenged:

This is a good moment to remind everyone of the normal definition of "head" and "tail" in entertainment markets such as music. "Head" is the selection available in the largest bricks-and-mortar retailer in the market (that would be Wal-Mart in this case). "Tail" is everything else, most of which is only available online, where there is unlimited shelf space. [link]

It’s a definition that is skewed to guarantee success for his model, and which is completely uninteresting (as I have posted about ad nauseam) but hey, it’s his definition. And the Netflix data has nothing to say about it.

So when Chris Anderson posts his favourite graph from the data and claims it’s a vindication of the long tail (“Netflix data shows shifting demand down the Long Tail”), it can only be because it looks like the schematic, unlabelled, number-free graphs in his book. It’s cherry picking the data for the most simplistic of reasons  because the two lines he’s comparing have no relation to what he talks about elsewhere, but hey, who cares?

Comments

[RAD]

I think the movie queue system used by Netflix makes it difficult to mine Long Tail effects as well. Nick Carr's post hints at this but from the perspective that Netflix is trying to milk profits from its customers (or perhaps minimize its own costs).

An alternative view is that Netflix has to work hard to keep customers after their first 6-12 months. The key measure is probably customer queue length.

The online queue system used by Netflix is very different from Blockbuster type rentals or Amazon DVD sales. Not only can Netflix influence rental patterns by adjusting the recommendation system as Carr suggests, but it can also manipulate the relatively opaque matching system that picks the DVDs to send out to its customers. Actually "influence" and "manipulate" overstate what is just a hard problem to optimize.

I still very much like The Long Tail theory. I don't believe The Long Tail out sells The Head but I do believe the trend towards diminished heads and fatter tails is real. To measure it you would have to gather numbers on the demand side (i.e. customers) rather than the supply side. Many people will use a combination of retail sales (BestBuy, Wal-Mart), mail based queue systems (Netflix, Zip.ca), and new head focused kiosks (Redbox).

[tomslee]

One of the things that the discussion at Nick Carr's place and your comment makes clear is that we have to dump the idea of "unfiltered demand" as a pure expression of our inner preferences. All demand is filtered because, at least, we are social creatures and because we are dealing with commercial entities.

It's a bit like the idea of "quality" of movies, which gets jettisoned pretty quick by people designing recommender systems. Even if there is something there, it's so tenuous and culturally shaped that it's best to just ignore it in discussions where it can be ignored.

So the idea that the Internet removes bottlenecks, allowing our innate preferences to come to the fore, is not one that can be sustained. There are always influences, always bottlenecks. And with Netflix in particular, I think the range of pressures you mention illustrate that.

[Dipper]

When any complex subject is discussed, there are always two extremes. At one extreme is the person who says "it's really all about factor X, everything else is just noise", and at the other is the person who says "factor X only explains part of the phenomenon, and may be an effect not a cause, and there's lots of other factors that are inter-related, and its almost impossible to conduct experiments that isolate these factors."

As a physicist I'm naturally drawn to the first approach, and the first approach makes more noise and sells more books. But for anything more complex than an electron in a vacuum, the second approach seems to be truer. So I agree.

[tomslee]

In general I agree there is a role for both approaches.

In this particular case, the trouble is that the long tail is not just a coarse theory, it's a wrong theory, or not even a theory at all. I wouldn't mind "it's all about factor X", but "factor X" keeps changing.

[Phil]

I'm teaching an undergraduate stats course at the moment, so simple representations of complex numerical data are on my mind. In that context, what bugs me about the Long Tail is what a junky graphic it is. Give me the numbers 1, 7, 8, 1, 7, 4, 2, 2, 1 and there's a number of things I could do with them, but it would never occur to me to represent them as a simple bar chart with the X axis sorted high to low; it's incredibly naive ("so the eight goes first because it's biggest, then the seven... then another seven...") as well as being very uninformative (why's that bar there? yes, I can see it's bigger, but why's it there?). H'mph, I say.

[tomslee]

But it's so smooth, with none of those bumpy bits that make other graphs so ugly.