Whimsley

Online Monoculture and the End of the Niche

Online merchants such as Amazon, iTunes and Netflix may stock more items than your local book, CD, or video store, but they are no friend to "niche culture". Internet sharing mechanisms such as YouTube and Google PageRank, which distil the clicks of millions of people into recommendations, may also be promoting an online monoculture. Even word of mouth recommendations such as blogging links may exert a homogenizing pressure and lead to an online culture that is less democratic and less equitable, than offline culture.

Whenever I make these claims someone says "Well I use Netflix and it's shown me all kinds of films I didn't know about before. It's broadened my experience, so that's an increase in diversity." And someone else points to the latest viral home video on YouTube as evidence of niche success.

So this post explains why your gut feel is wrong.

The argument comes from a paper by Daniel M. Fleder and Kartik Hosanagar called Blockbuster Culture's Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity. They simulate a number of different kinds of recommender system and look at how these systems affect the diversity of a set of choices. Towards the end of the paper they observe that some of their recommender systems increase the experience of diversity for every individual in the sample and yet decrease the overall diversity of the culture. So I wrote a program that does basically what they do in their paper and tweaked it to highlight this result.

The result is what's important here, rather than the particular algorithm used to generate this instance of it. But I know some people will want to know how the results are generated, so I'll give a short sketch. If you want more than this, Fleder and Hosanagar provide details, my tweaks to their model are available as source code (python) if you want, and if you post in the comments we could get into a discussion. But it's not important, trust me.

Each simulation starts with 48 customers and 48 products. Each product is described by two attributes, with values generated according to a normal distribution. So the products are distributed on a two-dimensional grid, with a value of about -3 to +3 along each axis. Each customer is assigned a taste for each attribute, so they also are scattered about in the same space. The idea is that a customer will prefer, other things being equal, a product that is close to it in these attributes. Here are two distributions of customers (blue) and products (red). You can see that most customers share a mainstream taste around the middle of the graph, but there are a few who have odd tastes off to the edges. Likewise, most products have attributes that are mainstream, but there are a few "niche" products closer to the edge.

In this particular simulation, a customer can choose the same item over and over again, so it simulates something like streaming radio more than a bookstore. Each simulation starts off with a priming phase, in which each customer makes 75 choices according to a function which favours nearby products, but with some randomness so that they may on occasion choose one further away. After 75 choices we turn on a recommender function. Whenever a customer goes to make a choice, the recommender system identifies a product and recommends it to the customer. The recommendation increases the chance that the customer will choose the recommended product. Fleder and Hosanagar look at a few recommender functions. The one I use works like this:

• The set of 48 customers is divided into equal-sized communities, with members chosen at random so they may not be close in taste.
• The recommender function chooses an item by looking at what customers in the same community have chosen. It recommends the one most popular among others in the community.

I'm just going to show you two simulations. Run 1 above - which I will call Internet World - treats the entire set of 48 customers as a single community. The other (run 28 above), which I will call Offline World, breaks it into 24 communities of two people each. In Offline World I will get recommendations from the people around me and you will get recommendations from the people around you, but these recommendations are separate and isolated. In Internet World we each get recommendations from all 48 customers.

Here are the results for the two simulation runs I'm going to focus on. The results of these simulations are far from the only possible outcome, but they show why the gut feeling may fail, and I've chosen them for that purpose.

In Internet World each customer experiences an average of 3.5 products over the course of 75 choices with an active recommender system, while in Offline World each customer experiences only 2.4 different products. So the wider set of people providing recommendations in Internet World has led to an increase in individual diversity. This is like saying that "Netflix shows me pictures I would never had heard about from my friends alone", or "Amazon recommended a book I had never heard of, and I liked it".

On the other hand, the overall diversity of the culture can be measured by the Gini coefficient of the products. A Gini coefficient of zero is complete equality (each product is chosen an equal number of times) and a Gini coefficient of 1 is complete inequality (only one product is ever chosen by anyone). And Internet World has a Gini of 0.79 while Offline World has a Gini of only 0.52. Internet World is less diverse than Offline World.

How can these seemingly contradictory results happen? Let's take a look.

In the following graph, each dot is a customer, arranged in their two-attribute preference space (just like in the graphs above). But this time the area of each dot is proportional to the number of unique products they experience. So in Run 1 (Internet World) you can see that the dots are, on average, bigger than the dots in Run 28 (Offline World). This shows the greater individual experience of diversity in Internet World; for example, there is a customer with attributes of (1.1, -0.8) who samples no less than 38 different products, and only seven of the 48 customers stay with a single product throughout the whole simulation. Meanwhile in Offline World  the most eclectic customer samples only nine and there are no fewer than 19 customers who sample just one product. The experience of individual customers in Internet World is of broader horizons and more selection, as recommendations pour in from far and wide, rather than from the limited experiences of their small community in Offline World. This picture has become the standard narrative of choice in the Internet World - our cultural experiences, liberated from the parochial tastes and limited awareness of those who happen to live close to us, are broadened by exposure to the wisdom of crowds, and the result is variety, diversity, and democratization. It is the age of the niche.

But wait!

Here is a graph of the products in each simulation. This time, the area of each dot shows its popularity: how often a customer chooses it.

You can see that on the left, in Internet World, a few products were chosen a lot, especially the one centred on about (-0.2, -0.2). In Offline World there are many more medium-sized dots, showing that the consumption of products is more equal. In Internet World one product has "gone viral" and gets chosen over 1500 times out of the total of 3600, while 26 products languish in the obscurity of being sampled fewer than ten times. In Offline World no single product is chosen more than 10% of the time, and only 14 products are sampled fewer than ten times. In short, niche products do better in Offline World than in Internet World.

While each customer on average experiences more unique products in Internet World, the recommender system generates a correlation among the customers. To use a geographical analogy, in Internet World the customers see further, but they are all looking out from the same tall hilltop. In Offline World individual customers are standing on different, lower, hilltops. They may not see as far individually, but more of the ground is visible to someone. In Internet World, a lot of the ground cannot be seen by anyone because they are all standing on the same big hilltop.

The end result is the Gini values mentioned before. Here are Lorentz curves for Internet World (blue) and Offline World (green), in which the products are lined up in order of increasing popularity along the x axis, and the cumulative choices for those products is plotted up the Y axis. 

So there it is. Individual diversity and cultural homogeneity coexisting in what we might call monopoly populism.

But don't think this is just about automated recommender systems, like the ones that Amazon and Netflix use. The recommender "system" could be anything that tends to build on its own popularity, including word of mouth. A couple of weeks ago someone pointed me to this video of Madin, a six-year-old soccer prodigy from Algeria, and the next day my son, who moves in very different online circles to me, was watching the same one. I know who Jim Cramer is even though we don't get CNBC in Canada because everyone is talking about him and helping his disembodied head to shoot down Jon Stewart. More people watched Tina Fey being Sarah Palin online than on Saturday Night Live, and Fey is now famous in countries where no one watches the TV show. Clay Shirky writes an essay and I get five different links to it in my Google Reader feed in one morning. Our online experiences are heavily correlated, and we end up with monopoly populism.

A "niche", remember, is a protected and hidden recess or cranny, not just another row in a big database. Ecological niches need protection from the surrounding harsh environment if they are to thrive. Simply putting lots of music into a single online iTunes store is no recipe for a broad, niche-friendly culture.

Tumbleweed blogs (firstname challenge again)

So some bloke called Graham emails me about my competition and he says, like, why don't you do it yourself then? And here's a list of names: http://bel-epa.com/area51/library/names.txt.

So I says, OK then. I will. And I write myself a little python like this:

import httplib
import re
import sys
conn = httplib.HTTPConnection("bel-epa.com")
conn.request("GET", "/area51/library/names.txt")

namelist = conn.getresponse().read().split()
conn.close()

activenames = {}
inactivenames = {}

print len(namelist)

for name in namelist:
 try:
 host = name + ".blogspot.com"
 p = re.compile('200[0-9]<')
 conn = httplib.HTTPConnection(host)
 conn.request("GET", "")
 response = conn.getresponse()
 body = response.read()
 mo = p.search(body) 
 if mo:
 activenames[name] = mo.group().rstrip('<')
 print name, mo.group().rstrip('<')
 else:
 inactivenames[name] = "null" 
 print name, "null"
 except:
 print "Failure on " + name 
conn.close()

f = open("blogdeath.csv", "w")
f.write("name,year\n")
for name, year in activenames.iteritems():
 f.write(name + "," + year + "\n")
for name, year in inactivenames.iteritems():
  f.write(name + ",1900\n")

And then I plot the results and they look like this.

Which means that of the 7489 potential firstname.blogspot.com URLs:

• 2692 don't seem to have anything going on ever or have weird styling so I can't catch the date headers blogger templates usually use.  A quick scan suggests that most are stillborn. See http://ned.blogspot.com/ or http://karim.blogspot.com/ for a couple of examples.
• 361 were born and died in 2000.
• 2001 and 2002 were the peak years for firstname blogs, with over 900 dying off each year. Not surprising that this is an early peak: the firstname places would be picked up quickly. Perhaps more surprising is that these blogs have been sitting there for 7 or 8 years, unattended, and blogger hasn't done anything with them.
• The number of dying blogs falls off - meaning that the number of active ones does as well probably. Until now, only 96 blogs have been updated in the first three weeks of 2009.
  

96 out of 7489. That's about one in 75 that are active. Pretty small numbers.

How does this silly survey compare with a bigger picture? The  Technorati annual review of blogging for 2008 says that about 7.5 million blogs were updated in the 120 days before their report of 133 million that they have ever indexed: about one in 20 or so. Given the longer timeframe, not too bad a match. So 19 out of 20 blogs that have ever been started are now moribund.

And here, for those of you who are interested, the 96 firstname blogs still going strong are listed at the end of this post.

My technorati ranking is about 33 at the moment but I have been as high as 90. That puts me, they say, in the top 1% of blogs. I get about 50 readers a day, many of whom are googlers who probably land here and don't find what they are looking for. And, on generous average, about one comment per post, so 99 out of a hundred bloggers get few readers and no comments. If these people are a typical sample, maybe one of them gets a comment per post. Maybe you should choose one and say hello.

elvina
adger
minta
cindi
worden
doretta
grover
carmen
charlton
hildagarde
melesa
lillian
miranda
delcina
barbette
urbanus
tina
jess
roland
bonny
madalyn
vivi
corabella
nikoletta
faunie
cherey
henka
terrill
olive
valli
kamila
cherice
lorette
daune
orville
sella
johnathon
aleecia
madelina
dottie
dario
roselyn
merrie
amalea
ephraim
trey
werner
antonino
levon
lavina
saxe
douglas
meira
danila
winston
lavinie
teresita
clovis
chas
letti
anders
edythe
aileen
luanna
huntlee
bancroft
sidonnie
fancie
marcelline
joella
pauline
engracia
oran
kirk
rhonda
leonardo
claudia
evette
kris
bird
zia
wilmette
merrel
fitz
cain
lindy
hope
dwain
mareah
sara
berkley
pinchas
josselyn
robinett
oprah
heidi

All the Web's a Stage

Timothy Lee has an excellent post at Freedom To Tinker. Here is some of it:

[T]alking about "free riding" as a problem the Wikipedia community needs to solve doesn't make any sense. The overwhelming majority of Wikipedia users "free ride," and far from being a drag on Wikipedia's growth, this large audience acts as a powerful motivator for continued contribution to the site. People like to contribute to an encyclopedia with a large readership; indeed, the enormous number of "free-riders"—a.k.a. users—is one of the most appealing things about being a Wikipedia editor.

This is more than a semantic point. Unfortunately, the "free riding" frame is one of the most common ways people discuss the economics of online content creation, and I think it has been an obstacle to clear thinking.

The idea of "free riding" is based on a couple of key 20th-century assumptions that just don't apply to the online world. The first assumption is that the production of content is a net cost that must either be borne by the producer or compensated by consumers. This is obviously true for some categories of content—no one has yet figured out how to peer-produce Hollywood-quality motion pictures, for example—but it's far from universal.
Moreover, the real world abounds in counterexamples. No one loses sleep over the fact that people "free ride" off of watching company softball games, community orchestras, or amateur poetry readings. To the contrary, it's understood that the vast majority of musicians, poets, and athletes find these activities intrinsically enjoyable, and they're grateful to have an audience "free ride" off of their effort.

The same principle applies to Wikipedia. Participating in Wikipedia is a net positive experience for both readers and editors. We don't need to "solve" the free rider problem because there are more than enough people out there for whom the act of contributing is its own reward.

You can see how we got into this problem. Britannica is an encyclopedia and it is an economic enterprise. Wikipedia ends in pedia so it must be an economic enterprise too. But it is not, and we should look elsewhere for our analogies. Once we stop thinking of Wikipedia in economic terms the supposed paradox disappears. Small wonder that, as Lee says, economists are the ones who have the hardest time understanding it.

Amateurs have performed on other world-stages before now. Sports provides prominent examples: rugby union,Wimbledon and the Olympics were all amateur-only for many years. What happened, of course, is that although these stages were not economic for their participants, the hosts ended up with a lot of money once the world of television and advertising intruded. And once the hosts got money the performers started acting like economic agents too and demanded their share. Giving an amateur performance on a non-profit stage is not a paradox, but giving an amateur performance on a profitable stage is being a sucker.

Wikipedia, for historical reasons, has so far kept advertising and profit out of the picture. But other "platforms of mass collaboration" have not - Amazon, YouTube, IMDB, Bebo have all shown that the owners of the stage can make a lot of money. And once they do, it's only so long before the actors start to demand a share, and the whole dynamic changes. 

The indiscriminate gushing over the age of mass collaboration has obscured these differences for now, but they won't for long. And then, I would guess, amateur production may go the way of the amateur olympics. But maybe not - let's hope that non-profit stages stay that way so that the economists' misunderstanding of Wikipedia can continue.

YouTube Becomes Big Business

It's one of those hoary old sayings - it's been around for maybe two whole years now - that while the Geezer Generation of passive consumers watched network TV, the Net Generation of cool participators go on YouTube and do their creative teenage thing. But it's no longer either/or. A few recent milestones highlight how YouTube is changing.

First, Avril Lavigne pipped a homegrown video to be first to 100 million viewers and now "six of the 10 most-watched videos of all time are straight music videos."

Second, CBS has reached an agreement with Google to show full-length TV shows on YouTube.

Third, Tina Fey's Sarah Palin sketches for Saturday Night Live have been watched more times on the Internet than on TV. Says the Associated Press:

There were 10.2 million people watching the season-opening "Saturday Night Live" when Fey first appeared as Palin, with Amy Poehler portraying Hillary Clinton, according to Nielsen Media Research. These days, that's a good-sized audience for prime-time, let alone late-night, TV.
Another 1.2 million people captured the episode on their DVRs and watched within the week. Through the middle of last week, NBC estimated that it had streamed the skit online more than 13 million times. Those are just the numbers NBC can keep track of; the skit was undoubtedly captured and posted or e-mailed many more times.
NBC perfected "widget" technology only a few months ago, allowing video of its material to be captured across the Internet while retaining a tie to the network's Web site. It has aggressively marketed the Fey skits to political and comedy blogs...
...There's also the chance for even more revenue. Only in the past few weeks has NBC Universal perfected the technology to place a movie studio advertisement at the end of the clip it distributes online. Pre-clip advertising would add even more value.

It's not actually YouTube - as the article says, NBC now posts its own videos and removes them from the YouTube site when viewers post there - but these developments show that Internet viewing can be complementary to, not competitive with, mainstreamTV.  And "going viral" is no longer reserved for amateur guitar players. In fact, in an example of the centripetal web, the Internet now lets US Network productions go where they could not go before - you can now watch Fey/Palin in the UK on the Guardian and The BBC web sites. 

Whether TV networks will end up hosting their own material a la NBC, or whether there will be more CBS-style Google/network deals that see the networks outsourcing the hosting to YouTube in return for a slice of the advertising money, who knows? But the trend is clear; the confrontation between Internet and TV is coming to an end. The old enemies were perhaps never really at each others throats, and now they will cohabit happily. And while YouTube will continue to host amateur videos (why not?) it will make money from the music videos and the TV networks as it moves to higher-quality images and longer shows. Google's expanded YouTube advertising initiatives will help these deals along.

The Carr-Benkler wager is looking more and more like a win for Carr.

Money Ruins Everything, but we have to talk about it anyway

In Money Ruins Everything (blog post, complete article), John Quiggin and Dan Hunter look out at the new forms of creative expression introduced by the Internet (blogs, wikis, citizen journalism, and to some extent open-source software) and conclude that today's most important innovations are driven by the collaboration of amateurs with non-economic motives. They give this development progressive political overtones by labelling it "the 'amateur collaborative content' movement" [p216] and explicitly identify it as a non-commercial alternative to the market. I don't know Dan Hunter's other writings but Quiggin, at least, is a social democrat whose views I usually agree with and whose writings I read often and respect, so I don't disagree lightly with him. But the picture they paint is a distorted one so I have to.

[Note to regular readers: you may want to move right along - much of what's here is stuff I've written earlier in other contexts. It's just more of the same, but it needs to be said.]

There are three major things wrong with the paper.

1. A portion of the activity they describe as non-commercial is in fact commercial.
2. They exaggerate the rise of the amateur in the Internet age.
3. They neglect the other side of the coin, which is the spread of commercialism into areas that were previously non-commercial.

I thought I'd talk about all three, but it's taking too long to write and I need to go and cook some dinner as part of my contribution to our household's creative production so I'll just address the first. If I were to say something about the other two it would be

(2) in assessing the rise of the new amateurism they neglect instances of amateur production that existed before the Internet and may be driven out by the rise of the Internet. And

(3) Facebook represents the commercialization of conversation, not the amateurization of collaborative content production.

The distinguishing technological feature of the collaborative web ("web 2.0") is the shift from peer-to-peer networks to a "platform" architecture that is built on top of the lower-level protocols of the Internet.

Wikipedia is a platform: it defines the ways in which you can interact with it, stores the changes you make, and provides security and authentication mechanisms among other things. Facebook is a platform. Amazon is a platform. YouTube is a platform. Blogging takes place on platforms. So the authors are wrong when they say that "As a result of various forces—notably the ascension of the general purpose computer, peer-to-peer technologies, and the internet—all manner of established verities in the content industry are falling." [p215] None of the platforms mentioned above operate in a peer-to-peer manner.

The distinction matters because when content is built on a platform, it is in some important senses owned by the platform-owner or aggregator. Private ownership is present, even if the content (videos, book reviews etc) is explicitly shared by its amateur producer. The suggestion that "Users can modify open source software as they see fit, and can choose whether to make their modifications publicly available, but cannot charge for the use of software derived from an open source program." [218] is incorrect for open source software itself (it applies only to GPL'd software and not software produced under other open source licenses such as the BSD license, and even then copyright owners - such as MySQL AB, now a part of Sun Microsystems  -  can and do charge for some versions of the software). More importantly for this paper, it is also misleading when it comes to collaborative content. Chad and Steve cannot sell an individual video produced by an amateur, but they can sell the entire collection of videos lock stock and barrel to Google for about a third of a billion dollars each; Michael Birch can sell Bebo and all its content (including Billy Bragg's songs) to AOL and pocket $600 million. Now that's commercial. Or you can get a seat at Davos, of course, which now seems to be the Cannes red carpet equivalent for our youthful webby leaders.

Amateur production on a non-commercial platform such as Wikipedia is non-commercial. And it's important. No argument there. But amateur production on a commercial platform is commercial activity. With one party in each transaction motivated my money, it's the sound of one hand shaking. Back in 2003 Tim Bray's dinner companion Robb Beal introduced the phrase "digital sharecropping" to distinguished building software "for any platform that is owned and operated by a company" from building sofware for the web. But now that dichotomy has faded: the web includes many platforms owned and operated by companies and sharecropping has moved online along with it. Nicholas Carr has been particularly pointed about the movement of digital sharecropping onto the web, and Seth Finkelstein has pointed to several examples including citizen journalism. It's a phrase that should be front and centre of everyone's mind when they see the phrase "networked production".

The distinction can often be seen in who is sharing what. On a non-commercial platform, the amateurs share and the platform owners share as well. At least, my understanding is that on Wikipedia not only the content but also the software and large amounts of data mining derived from it about users and so on is shared publicly. In contrast, on a commercial platform only the amateur material is shared. The contributions of the commercial part of the transaction (Amazon's sales data for example; data on user habits; the software that runs the platform itself) are not shared -- in fact this half of the story is hidden with as much zealousness as the source code of any closed-source company. One of the reasons that "The Long Tail" was such a flop of a book is that the data Chris Anderson relies on is unavailable for anyone else to inspect, and much of it was given to him in filtered form by the content owners.

A second failure is most obviously present in the following paragraph (page 245).

Copyright thus provides incentives to the intermediaries of the content industries—publishers, agents, movie studios, retail stores, etc.— where the processes of moving content from creator to user are expensive or capital-intensive. These “content processes” include the creation of the content, the selection of the content for commercial publication, its production and dissemination, its marketing, and its eventual use. Until recently each of these processes has been too expensive, too difficult,  or too specialized for amateurs to undertake. So until now, highly capitalized intermediaries were necessary to ensure that content moved into society.

Amazon, Google and so on are obviously highly capitalized. More so, in fact, than many small distributors of pre-web content. The web is seeing the movement of content into society consolidated into a handful of "highly capitalized intermediaries": very large and wealthy commerically-driven web site owners, including such old-school outfits as News Corporation, the owner of MySpace. It does no good to ignore this trend.

The paper also elides the distinction between amateur and professional motivations, with the temptation to caricature employed work as factory-line production of alienated labour, while amateur work is driven by love. To take a relevant example, what is the overlap between the motivations of John Quiggin (blogger) and John Quiggin (employee)? Is his work for the University of Queensland driven by the his "first rule" which is "never to give more than you get?" [231] And if not, why does he think that software engineers for commercial software companies writing closed-source programs are driven by different motivations? There is a possible argument (nicely made by Geof in a comment on a previous post) about alienation and the relationship between the developer and the code, but this does not cover production in sharecropping platforms. I'm more inclined to go with another comment by Dipper, that participation in many cases can be covered by standard utilitarian calculus.

One argument made by the authors is that the incorporation of money into production would drive out amateur efforts in a blood-donation kind of way. I'm happy to help push someone's car out from a snow drift for free, but I wouldn't do it for a dollar. But there are two groups of people with an incentive to keep money out of the equation: one is the promoters of real community-driven, shared production (the authors' camp, and one I'm quite happy with) and the other is the turbocapitalist platform owners such as Amazon (would you review a book for a nickel? would you trust the review of someone who did?) It is this shared interest in supressing the role of money for very different reasons that makes me most queasy in contemplating the future of the Internet. We need to be clear in distinguishing public goods from privately owned plantations. Unfortunately, in this paper, John Quiggin and Dan Hunter fail to make the distinction and the result is a distorted picture of web-based creative work. Money may ruin everything, but turning a blind eye to money doesn't get around that problem.

I feel like I'm being harsh on the authors here - as I say, I agree with their political sympathies, and the one I have read (Quiggin) is obviously smart and well informed on many issues. But serious social scientists need to do serious work on the nature of production on the Internet and not just adopt the turbocapitalist line. Me, all I can do is raise a few questions. But then, when it comes to social science, I'm just an amateur.

Don't be Pompous

Let's get this straight - I don't have anything  against Google any more than I do any other company. But there are times when it just goes out of its way to be a pretentious prat. Today is such a day.

Here is Google's unbelievable official response to the Microsoft attempt to buy Yahoo!  How about this for a paragraph:

Could Microsoft now attempt to exert the same sort of inappropriate and illegal influence over the Internet that it did with the PC? While the Internet rewards competitive innovation, Microsoft has frequently sought to establish proprietary monopolies -- and then leverage its dominance into new, adjacent markets.

David Drummond (Senior Vice President, Corporate Development and Chief Legal Officer) talks about two principles of the Internet: openness and innovation.

We take Internet openness, choice and innovation seriously. They are the core of our culture.

The beauty of this stance is that you can play the openness and innovation off against each other. Google's important software is just as proprietary, closed source, and hidden as that of Microsoft - in fact more so because M$ has shared source agreements with many companies while Google's core technologies are not shared with anyone. Google does not disclose information about things like water consumption at its server stations because it's a "competitive matter"; Google buys properties under other names because it can get a better deal. In cases like these where Google wants to be secretive they pull the innovation card and talk about innovation and the need to  prosper in a competitive market.

When Google wants to promote technologies that are complementary to its own, it makes them open source (for example, in its sponsorship of Firefox browser development) and talks in idealistic terms about "the community". The overlap of interests between Google and its "communities" is partial at best, however, and such talk is cheeky, to say the least, coming from some of the world's richest people. And if making its advertising-driven wealth isn't leveraging its dominance in search into new, adjacent markets then I don't know what is.

In the end, Google and Microsoft are both profit-maximizing companies. No matter how much they couch their goals in suitably vague idealistic terms ("don't be evil" and the "freedom to innovate" respectively) they respond to the incentives faced by all such institutions. I just wish they'd be up front about it and not put out the kind of drivel that Google did today.

The Netflix Prize: 300 Days Later

Today the Netflix Prize Competition has been running for 300 days.

Online DVD rental outfit Netflix caused a real buzz last October when it announced the competition. If anyone can come up with a recommender system for predicting customer DVD preferences that beats its own algorithm (Cinematch) by a certain amount, Netflix will hand over $1million. The prize got a lot of attention because it exemplifies the idea of crowdsourcing. Not only does Netflix rely on crowdsourcing of DVD ratings (user ratings of DVD titles) but the competition itself is an attempt to use crowdsourcing to develop the algorithms to make the most of those ratings. Instead of doing the work itself, or hiring specialists, Netflix lets whoever anyone enter their competition and pays the winner. The competition is still in progress: Netflix says it will run until at least 2011. So now the initial buzz has died down, what can we learn from the Netflix Prize?

First, the competition details (see here (PDF) for a short paper by two Netflix employees). Netflix made public a database of customer DVD ratings (tweaked to ensure privacy) that included over 100 million individual ratings of 17,770 titles by 480,189 people. If you sign up for the prize, you can download these ratings. Each rating involves one customer giving an integer number from one star (very bad) to five stars (very good) for a given title. For example, customer 296452 gave title 234 ("Animation Legend: Winsor McCay") a rating of 1 (very bad).

The idea is that competition entrants try to develop an algorithm by using the training set (which is the 100 million plus set of ratings), try it out on a set of probe set of test data that they also give you, and once they think they have a good algorithm, create a set of predictions for a qualifying set of users and titles, and upload it to Netflix. Netflix test these predictions against the actual rankings (which they keep private) for that qualifying set. They post the leading algorithms on a leaderboard.

The quality of any algorithm is determined by its root mean squared error (RMSE). To calculate the RMSE you take the difference between the rating the algorithm predicts and the actual rating, and square it so it's guaranteed to be a positive number. Then you take the average of all these over the set of data to get the mean squared error. Finally, taking the square root gives the RMSE, which is the roughly the size of a typical error.

A perfect algorithm would predict exactly what rating every user would give to every title and would have an RMSE of zero. A random set of predictions has an RMSE of 1.95. But the actual range of action is much narrower than this 1.95 range. A simple algorithm that uses the average rating for each title as the prediction - "let's see, the average rating for the 104,000 customers who rated Mean Girls was 3.514, so I predict you will give it a rating of 3.514" - gets an RMSE of 1.0540. Netflx's Cinematch algorithm has an RMSE of 0.9525. Netflix set the prize target at a 10% improvement over that, which is an RMSE 0.8563. So the range that recommendation systems can realistically cover - from naively simple to cutting-edge research - seems to be the narrow band between the middle three lines in the following diagram.

In the days and weeks after the prize was announced, progress was rapid. The Cinematch score was matched within a week. Within a month the leaders were half way to the winning prize with a 5% improvement. But getting further improvement progress has proved more and more difficult. It took another month to get to a 6% improvement, about 5 more months to get to 7%, and the current (July 29 2007) leader is at 7.8% improvement and has been unchanged for a month. Here is a graph of the progress, showing the three lines above and the prize leader progress:

At this stage it is not clear if the prize is winnable: the existing algorithms use a lot of linear algebra and some pretty fancy machine learning ideas (see a description by a leading participant here and some sample code for a similar approach here), the leading groups include university research labs from around the world, and many of the more obvious approaches have been explored. Certainly media and blog interest - huge in the early days - has dropped off in recent months. This New York Times article is one of the few from the last month or two.

Let's not get into the computer science of recommender systems - there's a good review from 2004 called Evaluating Collaborative Filtering Recommender Systems here if you want to know more. Instead, let's step back a bit and ask what this prize tells us so far, and look at a couple of things we can learn by poking around the massive data set that Netflix provided.

One question is: how good is an algorithm with an RMSE of 1, and is an algorithm with an RMSE of 0.8563 much better for the average customer? Actually, I guess that's two questions. Anyway, if the errors followed a normal distribution (which they don't, but we're talking back-of-envelope here) then if a customer actually rated a title as 2 (poor), an algorithm with an RMSE of 1.0 would predict somewhere between 1 and 3 about 70% of the time. Not bad, but not startling. If the algorithm gave ten recommended movies, then it would get on average seven out of ten within one unit of the customer's actual rating. Meanwhile, the RMSE=0.8563 algorithm would get 7.6 out of ten. While this is an improvement, and while it may be a remarkable technical accomplishment, it does not seem to be exactly a revolutionary leap compared to the really simple algorithms as far as customers go.

[Update, December 25, 2007: Yehuda Koren of leading team KorBell approaches the recommendation problem a different way, looking at ordering of recommendations rather than at matching them. His way is more appropriate, and gives much more encouraging results. See here.]

As soon as you start looking at the data set it becomes obvious why it is so difficult to get good results. Databases don't have the linear algebra and other mathematical tools for taking a run at the prize but they are convenient for exploring data sets, so I loaded the data into a SQL Anywhere  database (The developer edition is a free download, and I'll provide a perl script to load the data if you really want it) and started poking around. Here are a few of the more obvious oddities (all these observations have been posted elsewhere - see the Netflix prize forum for more):

• Customer 2170930 has rated 1963 titles and given each and every one a rating of one (very bad). You would think they would have cancelled their subscription by now.
• Five customers have rated over 10,000 of the 17,770 titles selected - and presumably they also have rated some of the others among the 60,000 or so titles Netflix had available when they released the ratings. Are these real people?
• Customer 305344 had rated 17654 titles. Even though Netflix make it easy to rate titles that you have not rented from them (so they can get a handle on your preferences) can this be real?
• Customer 1664010 rated 5446 titles in a single day (October 12, 2005).
• Customer 2270619 has rated 1975 titles. 1931 were given a 5, 31 were given a 4, 10 given a 3, 2 given a 2 (Grumpy Old Men and Sex In Chains) and a single title was given a 1. That title? Gandhi, which has an average rating of over 4 and which less than 2% of those who watch it give a 1.
• The most often rated movie? Miss Congeniality with ratings by over 232,000 of the 480,000 customers. And which title is most similar to it in terms of ratings (using a slightly weighted Pearson formula)? Bloodfist 5: Human Target.
• Most highly rated - Lord of the Rings: Return of the King (Extended Edition), with 4.7.

Some of the more bizarre facts above may be artifacts of whatever tweaking process Netflix put the data set through (although they claim not to have materially affected the statistics). While odd, bizarre users are not always difficult to deal with: if you have rated each of the last 1963 titles you've watched as 1 it is pretty easy to predict what you will rate the next title. But others are more tricky.

One reason for these oddities is one of the things that Evaluating Collaborative Filtering Recommender Systems identifies. They note that (on other, smaller data sets) even the best algorithms don't seem to get beyond an RMSE of 0.73 on a five-point scale, and speculate that the cause may be "natural variability". We users provide inconsistent ratings - sometimes we'd rate a movie a 3 and sometimes a 4, with no consistency. It may depend on our mood when we watched the movie - we may give a romantic movie a higher rating if we watched it on a first date than if we watched it a week later after being left broken hearted, or a demanding movie a low rating because we were tired and out of sorts when we watched it - or it may depend on our mood when we actually provide the rating.

There are other, more obvious reasons which, for reasons I don't understand, don't seem to get discussed much. Netflix itself and most competitors talk about the data in terms of "movies" and "users". But the "movies" in the list are not all movies: a lot are TV series or music video collections. The variability among the episodes of a series (Do you think Lost Season 1 deserves a 3 or a 4?) must make single-number ranking even more variable and these composite DVDs figure prominently among those titles that have the biggest variance in ratings.

Then there's the fact that a customer might not really be a single person. It might be a household with several viewers in it. So perhaps one person likes Terminator, one likes Bridget Jones, and one likes Spongebob Squarepants. Once we realize that the "user" might be a collection of people there is no strangeness between giving high ratings to each of these, but you can see how, depending on which household member entered the rating, the values may be quite different (perhaps this is why titles like 'N Sync: Making of the Tour, Pokemon Vol 9, and Boston Red Sox 2004 World Series Collectors' Edition have high variance - the person rating may not always have been the person who wanted to watch it). If the data set contains these inevitable variations (in addition to the plain kookiness on show in the Netflix set) then it may be that even the clevverest algorithm can make little progress in untangling all the intrinsic vagueness of the data.

So what I get from the Netflix prize is that there are probably significant limits to recommender systems. Even the smartest don't do a whole lot better than the simple approaches, and a lot of work is required to eke out even a little more actual information from the morass of data. It seems surprisingly difficult to get reliable, factual information on this important question of how useful they can be. Part of the reason is that they are new - Amazon has only been in business for about ten years after all - and part of the reason is that the behaviour of these systems is often a closely guarded secret despite the aura of openness that web companies cultivate.

This matters because there is a surprising amount riding on the effectiveness of recommender systems. Silicon Valley's new-economy enthusiasts see them as the key to developing a new level of cultural democracy: they see recommender systems as a trebuchet hurling rocks at the castles of the old elite of mainstream media, big publishers with big marketing departments, big-chain book stores and Hollywood sequels. Recommender systems are claimed to embody the "wisdom of crowds". The idea is that everyone just publishes stuff (blogs, wikipedia entries and so on) and amateur readers or viewers decide what has merit by their actions (rating stories, buying and rating books and DVDs and so on). The work of critics is "crowdsourced" to customers, but it is the recommender system that distills these ratings to yield the aforementioned wisdom.

If faith in recommender systems is misplaced, then the new boss may look much like the old boss only with more computer hardware. There is a danger that recommender systems may simply magnify the popularity of whatever is currently hot - that they may just amplify the voice of marketing machines rather than reveal previously-hidden gems. Even worse, their presence may drive out other sources of cultural diversity (small bookstores, independent music labels, libraries) concentrating the rewards of cultural production in fewer hands than ever and leading us to a more homogeneous, winner-take-all culture.

I'm no futurist, but I see little evidence from the first 300 days of the Netflix Prize that recommender systems are the magic ingredient that will reveal the wisdom of crowds.

Reputations

Regarding reputation-building on the Internet, Clive Thompson writes approvingly that

network algorithms do not favor the cagey or secretive. They favor the prolific, the outgoing, the shameless.

Said another way, network algorithms do not favour the quiet or the reflective. They favour the loud-mouthed, the self-promoting, the flashy.

O brave new world that has such algorithms in it.

Believe the Opposite: Radical Opacity

I'm afraid Clive Thompson has jumped the shark. From being a witty journalist at the interesting This Magazine he now fits right in at the boring Wired Magazine. On the way he seems to have lost his sense of irony (maybe they don't let you bring irony into Silicon Valley?) and his cynicism. As a result, he has also lost the plot. Come back Mr. Thompson!

His March 2007 article in Wired Magazine called The See-Through CEO coined the phrase Radical Transparency. Like other Silicon Valley catch phrases, it has that air of youthful rebellion, it is self-consciously ignorant of history (who needs history when all the interesting things are happening right now), and - most important of all - it imparts a feel-good sense of anti-corporate attitude to your next venture funding proposal or business plan. Because like other Silicon Valley catch phrases, Radical Transparency has about as much to do with rebellion as riding a mountain bike.

Here are some snatches from the article, and some recent events in the real world, mainly as reported by The Register - which has thankfully managed to keep its senses of both irony and cynicism - and mainly about Web 2.0 poster-offspring Google and its growing Google-hoard of companies.

"You can't hide anything anymore," Don Tapscott says. Coauthor of The Naked Corporation, a book about corporate transparency, and Wikinomics, Tapscott is explaining a core truth of the see-through age: If you engage in corporate flimflam, people will find out.

Meanwhile, Google plays cat and mouse with regulators. Leif Aanensen, deputy director general of the Norwegian Office of the Data Inspectorate, has been investigating Google's data retention policy:

   "We are not satisfied," he said. "We didn't get the proper answers."

   "Our main issue was their data retention policy and the use of the data they   stored. We asked them what they were doing with the personal data - are you   creating profiles - they didn't answer," he said.

Thompson writes: "You can't go halfway naked. It's all or nothing. Executives who promise they'll be open have to stay open."

Meanwhile, Google - who make repeated references to their own "radical transparency" - are closed-mouth about the introduction of new programs.
Paying select few video producers for example:

YouTube says anyone who wants to get paid can let it know by registering an interest, but provided no timescale for when it will cough up, or what the carve-up will be.

Or will there be   advertising   on the iGoogle front pages?

   The company has not made any noises about placing personalised ads on the new   iGoogle personalised homepage, but industry observers are fairly confident it   is only a matter of time.

When it comes to openness, Thompson writes "there's no use trying to resist. You're already naked." How Naked? Hard to tell, because it is not easy to find out what   information   Google keeps about you.

   "Upon arriving at the Google homepage, a Google user is not informed of   Google's data collection practices until he or she clicks through four links,"   says the section of the complaint which details Google's alleged deceptive   trade practices. "Most users will not reach this page. In truth and in fact,   Google collects user search terms in connection with his or her IP address   without adequate notice to the user. Therefore, Google's representations   concerning its data retention practices were, and are, deceptive practices.

   "As a result of Google's failure to detail its data retention policies until   four levels down within its website, its users are unaware that their   activities are being monitored," says the complaint in the section alleging   unfair trade practices.

Thompson writes:

Secrecy is dying. It's probably already dead.

Meanwhile, here's Google being radically opaque:

ord broke this month that Google has purchased 800 acres of land in Pryor, Oklahoma. The company has yet to confirm plans for the site, but I'm betting  on a new data center rather than an amusement park (in all fairness, you can   never tell with this bunch – Ed).

Oklahoma proves a handy spot to have a data center since the state's Governor signed a new law that affords the largest corporate energy users the right to keep their power consumption figures a secret.

Governor Brad Henry signed the energy law (House Bill 1038) just a couple of   days after news of Google's land purchase reached the local newspapers.  Coincidence? Sure.

The lawmakers behind the bill denied having chats with Google around any legislation. People familiar with the matter, however, did note that the law proves convenient for an entity such as Google that likes to keep as much information secret as possible.

If you're a demanding type who needs evidence of Google's secret ways, have a  listen to head of strategic development Rhett Weiss. He presided over a party celebrating yet another Google data center in South Carolina. When asked about  Google's water and power usage, Weiss confessed:   "We're in a highly competitive industry and, frankly, one or two little pieces   of information like that in the hands of our competitors can do us   considerable damage. So we can't discuss it."

What else does Google not tell us? Here's Nicholas Carr:

“We never,” says a Google representative, “comment on who we’re talking to, who we’ve considered, who we’ve rejected. We feel that when we come to an agreement, that’s the time to make an announcement.”

So please, Mr. Thompson - exercise some scepticism. Even a little would go a long way.

The liberation mythology of the internet

Nicholas Carr of Rough Type has been reading David Weinberger's Everything is Miscellaneous, and is disappointed. But in his disappointment he coins a phrase I really like: "the liberation mythology of the Internet".

I only reached the bottom of page nine, at which point I crashed into this passage about music:

For decades we've been buying albums. We thought it was for artistic reasons, but it was really because the economics of the physical world required it: Bundling songs into long-playing albums lowered the production, marketing, and distribution costs because there were fewer records to make, ship, shelve, categorize, alphabetize, and inventory. As soon as music went digital, we learned that the natural unit of music is the track. Thus was iTunes born, a miscellaneous pile of 3.5 million songs from a thousand record labels. Anyone can offer music there without first having to get the permission of a record executive.

"... the natural unit of music is the track"? Well, roll over, Beethoven, and tell Tchaikovsky the news.

There's a lot going on in that brief passage, and almost all of it is wrong. Weinberger does do a good job, though, of condensing into a few sentences what might be called the liberation mythology of the internet. This mythology is founded on a sweeping historical revisionism that conjures up an imaginary predigital world - a world of profound physical and economic constraints - from which the web is now liberating us. We were enslaved, and now we are saved. In a bizarrely fanciful twist, made explicit in Weinberger's words, the digital world is presented as a "natural" counterpoint to the supposed artificiality of the physical world.

There's much more at Rough Type, as Carr demolishes Weinberger's claim.