Whimsley

(De)fraud

"Why is defraud is not the opposite of fraud?" asks my younger son. "It should be like - hey, you got scammed, here's some money."

Paying for Linux

Last week on 'Linux Goes Corporate': After Nick Carr mentioned my post on the Linux Foundation report and on Linux as corporate joint venture, Timothy Lee and Ed Cone/Clay Shirky responded and then Doc Searls put in his two cents worth. The main point of the responses is that the shift from hobbyists to professionals is not important. For example:

"the open source model is about organization, not who signs your paycheck" (Lee).

"the idea that the minute you pay people to do something, you have the right to manage them and the right to completely take over that work for the benefit of the company -- that's not true" (Shirky).

"in all the conversations I've had over the years with kernel developers, none has ever copped to obeying commands from corporate overlords to bias kernel development in favor of the company's own commercial ambitions. In fact, I've only heard stories to the contrary" (Searls).

---

The professionalization of Linux matters because it marks a change. Open source now is not the same as ten years ago when The Cathedral and the Bazaar was written.

Despite the historical revisionism of the "paycheck's don't matter" crowd, the absence of paychecks certainly mattered ten years ago. Eric Raymond opened his classic essay with these sentences:

" Linux is subversive. Who would have thought even five years ago that a world-class operating system could coalesce as if by magic out of part-time hacking by several thousand developers scattered all over the planet, connected only by the tenuous strands of the Internet?"

Ten years on what stands up? "several thousand developers scattered all over the planet" does - the number of participants in the Linux kernel is larger than ever. But the Linux kernel now is not developed by "part-time hacking". And Linux is no longer subversive, in that anything IBM is happy about is not, by definition, subversive.

And it mattered six years ago when Yochai Benkler wrote "Coase's Penguin: Linux and the Nature of the Firm", which contains this statement:

Programmers do not generally participate in a project because someone who is their boss instructed them, though some do. They do not generally participate in a project because someone offers them a price, though some participants do focus on long-term appropriation through money-oriented activities, like consulting or service contracts.

So now programmers do participate in the Linux kernel because their boss instructed them to (or paid them to).

Now I can imagine people saying that just because the Linux programmers are paid, that doesn't make their motivation financial and that doesn't mean companies can tell them what to do. Here is Lee:

What makes the open source model unique isn't who (if anyone) signs the contributors' paychecks. Rather, what matters is the way open source projects are organized internally. In a traditional software project, there's a project manager who decides what features the product will have and allocates employees to work on various features.

This is a caricature. I work in a commercial, closed-source software company, and my job is very close to that of a project manager. But what an expert programmer spends his or her time on is always a negotiated thing. Partly it's a matter of skill and expertise (who is best placed to identify algorithms or code-level weaknesses and areas for improvement?), partly it's a matter of "what needs to be done" and partly it's a matter of innovation. And as Joel Coehoorn says in a comment to Lee's post,  "there most definitely are project managers for open source projects. I can't think of a single successful open source project that doesn't have a well-defined road map for features." Open source believers too often portray paid closed-source programmers as wage slaves, and paid open-source programmers as free agents working for love (to scratch an itch) who just happen to collect a paycheque. It's a dichotomy that just doesn't hold up.

Shirky argues that "the idea that the minute you pay people to do something, you have the right to manage them...that's not true" but this statement applies to the employees where I work just as much as to IBM employees working on the Linux kernel, and there is nothing about the Linux kernel that gives Redhat programmers a special "ignore the boss" license. Shirky says that "If [a senior manager] announced a strategic change in the kernel they would be laughed out of the room" as if senior managers routinely dictate strategic changes in the Windows kernel or database kernels. More senior managers get laughed out of the room than Shirky would believe.

---

This leaves us with ownership. IBM and Redhat do not own and will never own Linux. But Redhat sells Linux, and the ability to sell something is one component of ownership. The distinction is more one of exclusive ownership. And here the Visa analogy I used originally holds up - Visa is now a separate company but it is also a collaborative venture and companies that helped to build it were simultaneously assisting their competitors. Industrial sponsorship of academic research - an activity the pharmaceutical industry relies on - has the same tensions about ownership and organization and direction.

Open source, like any new phenomenon, is changing. Our understanding of it has to change too. The Linux Foundation report prompts us to re-examine some of the neat and clean dichotomies that were floated around ten years ago and see if they still hold water. A few do, but many don't.

Linux Grows Up and Gets a Job

Linux started as a hobby project. But it is now 17 years old and it has grown up, and a recent report by the Linux Foundation, which extends a series of investigations by Jonathan Corbett, shows that it is no longer a hobby, it is out in the working world.

The report looks at the Linux kernel, and so does ignores all those other pieces of the operating system (drivers, applications, desktop interfaces and so on). Still, the kernel is the heart of the OS and "one of the largest individual components on almost any Linux system. It also features one of the fastest-moving development processes and involves more developers than any other open source project." So this report, which "looks at how that process works, focusing on nearly three years of kernel history as represented by the 2.6.11 through 2.6.24 releases." is a valuable window on what may happen to successful open source projects as they mature.

One of the highlights: "over 70% of all kernel development is demonstrably done by developers who are being paid for their work". 14% is contributed by developers who are known to be unpaid and independent, and 13% by people who may or may not be paid (unknown), so the amount done by paid workers may be as high as 85%. The Linux kernel, then, is largely the product of professionals, not volunteers.

So Linux has become an economic joint venture of a set of companies, in the same way that Visa is an economic joint venture of a set of financial institutions. As the Linux Foundation report makes clear, the companies are participating for a diverse set of commercial reasons. Some want to make sure that Linux runs on their hardward. Others want to make sure that the basis of their distribution business is solid. And so on, and none of these companies could achieve their goals independently. In the same way, Visa provides services in many different locations around the world in different sizes and types of stores. Some banks need their service mainly in one country, some in another, but when they work together they all get to provide their services all around the world.

This does not mean that Linux was always a commercial venture, or that all open source projects are commercial ventures. To invoke another parallel, open source software is a creative venture like music. Many people create music for all kinds of reasons. Most people create music for an audience of one when they hum in the kitchen or sing in the shower. A smaller (but still huge) number of people get together to form groups or participate in orchestras or bands. They don't earn a living from it, but they love doing it and enjoy their performances. Some might dream of hitting the big time, others are happy being part of their community. Then a much smaller set of people take it a step further. Maybe they are paid to be in an orchestra; maybe they are in a band with a manager and bar gigs around the country. And a lucky few, of course, hit the big time. They get a record deal, find a big audience, and make some real money.

So is music produced for love or for money? Both of course, in various proportions across the spectrum. Open source started off as a small-scale set of projects done mainly by volunteers. As the scale and scope of open source projects an increasing number have provided their contributors with some money (augmented perhaps by a waitressing job). Now a few of the most successful have hit the big time and become full-scale economically important commercial enterprises.

Things change. As open source software has matured and expanded it has become both more unlike the rest of the world and more like it. It will be fascinating to see what comes next, but the Linux Foundation report has made clear that open source has crossed its commercial Rubicon, and there is probably no going back.

Update: follow-up and response to some discussion here.

Talking of war...

My brother just put up a picture of my Grandad's very familiar German binoculars, picked up on the battlefield in World War 1.

Taxi to the Dark Side/Talking to the Taliban

This is not what I usually write about here, as I have little to add to the discussion over the war in Afghanistan, but two things have made a big impression on me in the last couple of weeks.

The first is the series of interviews with 42 members of the Taliban organized by Graeme Smith of the Globe and Mail (link). In particular, there is this:

Almost a third of respondents claimed that at least one family member had died in aerial bombings in recent years. Many also described themselves as fighting to defend Afghan villagers from air strikes by foreign troops.

...and this, which I guess many people better informed than I already know:

Aerial bombings and civilian deaths have both increased: The United Nations estimates more than 1,500 civilians were killed last year, as compared with the 900 to 1,000 civilian deaths counted by two studies of the previous year. An analysis of the first nine months of 2007 found the number of air strikes was already 50 per cent higher than the total for 2006.

Civilian bombings emerged as a major theme of the war last year. President Hamid Karzai shed tears in public as he spoke about civilian deaths. In June, a coalition of Afghan aid agencies published a controversial report suggesting that the rate of civilian casualties had doubled from the previous year, and that international forces were starting to rival the Taliban as the greatest source of civilian deaths.

The second was watching Taxi to the Dark Side, the Oscar-winning documentary about torture in Afghanistan, Iraq, Guantanamo, and elsewhere. It made very clear that most detainees (83,000 at the time the film was made) were not detained by the allied forces, many have committed no crime whatsoever, and that torture of the detainees is common. The effect that the torture and murder of Dilawar, the taxi driver of the title, has on his village is not followed up but is not difficult to imagine.

Aerial bombing and torture are not at the centre of the debate over NATO's role in Afghanistan, but they should be. The debate would then not be about how we can best help the people of Afghanistan; it would be about how we can stop damaging them and driving them to desperate measures. As long as these tactics are part of the military operation in Afghanistan then the longer NATO is there, the more enemies it will create.

Taxi to the Dark Side, which includes many interviews with soldiers involved in torture, shows how the ethics of groups can differ from the intentions of the people that make up those groups. Even an army composed largely of well-meaning individuals can act in fundamentally immoral ways. Just as NATO soldiers cannot tell a Taliban veteran from a villager who abhors the Taliban, so Afghan citizens cannot tell the difference between a soldier who will treat them decently and one who will arrest them for nothing and condemn them to torture. In both cases the safe assumption to make is the cautious one. So all Afghan citizens will be suspicious of and hostile to NATO soldiers and NATO soldiers will be suspicious of and hostile to Afghan citizens whether or not that hostility is deserved by the individuals. And each such hostile encounter justifies the hostility of the other side. While NATO is there it is not easy to see how things can improve.

Within Canada, we need to address the question of what kind of war we are part of. The assumption to date is that we are fighting a decent war. As long as aerial bombing and torture are part of the military operation then this is not a decent war.

Here Comes Everybody

Here Comes Everybody: The Power of Organizing Without Organizations, Clay Shirky, Penguin, 2008.

---

"We are living", says Clay Shirky, "in the middle of a remarkable increase in our ability to share, to cooperate with one another, and to take collective action, all outside the framework of traditional institutions and organizations" [p. 20]. Digital technologies are now part of our social fabric, and all "the phones and computers, the e-mail and instant messages, and the webpages are manifestations of a more fundamental shift. We now have communications tools that are flexible enough to match our social capabilities and we are witnessing the rise of new ways of coordinating action that take advantage of the change" [20].

Here Comes Everybody is an accessible and challenging introduction to these changes - to the many ways every Shem, Shaun and Issy can now share, collaborate, and act together. It combines some great stories with non-technical introductions to some of the key ideas (a little game theory, a little network theory, a few power laws) and is well worth reading -- but it should be read with caution.

The reason for this mixed verdict is the dual nature of the book itself. Here Comes Everybody has two voices. One (let's call him 'Shirky') is a perceptive and creative interpreter of the ways that digital technology is changing society. I like and respect Shirky. He is blunt and provocative enough to cut through the mess of questions that come up when tackling something as far-reaching as the Internet, while being even-handed and reserved enough to respect the complexity of his subject. Here is Shirky on the erosion of journalism and photography:

There is never going to be a moment when we as a society ask ourselves, "Do we want this? Do we want the changes that the new flood of production and access and spread of information is going to bring about?" It has already happened; in many ways, the rise of group-forming networks is best viewed not as an invention but as an event, a thing that has happened in the world that can't be undone. As with the printing press, the loss of professional control will be bad for many of society's core institutions, but it's happening anyway. [p73]

I've struggled with this message for a few days because I don't like its determinism, but he has convinced me. Shirky separates what is happening from what is desirable (not that everything digital is undesirable) and that's an important separation to make if we are to be at all clear-headed.

Shirky is not a techno-inevitabilist in a broad sense. Nuclear power, for example, "is a technology that society can, for the moment, make a decision about" [299]. As with driving a car, we "have a good deal of control over both the route and the speed with which nuclear power progresses, including the option to simply pull over" [299]. But when it comes to digital technologies we are steering a kayak: "We are being pushed rapidly down a route largely determined by the technological environment... Our principal challenge is not to decide where we want to go but rather to stay upright as we go there." [300]

The other voice (let's call him 'Clay') is a techno-enthusiast and an inveterate story-teller. When Clay looks at the Internet he sees no reason for worry - he sees freedom and unlimited potential. It's "the largest increase in expressive capability in the history of the human race" [106], bringing with it a world in which, when "people care enough, they can come together and accomplish things of a scope and longevity that were previously impossible; they can do big things for love." [142] Shirky may warn that there are losses from social changes, but Clay is breezily dismissive: "The spread of cheap and widely available creative tools is sad for people in the advertising business in the same way that movable type was sad for scribes -- the loss from this kind of change is real but limited and is accompanied by a generally beneficial social change." [209] This is the logic of The Lottery - the short story in which one person is stoned so that others can be better off.

I blame Clay and his enthusiasms for the two major flaws of this book.

The first is a fallacy of composition. Clay looks at the Internet and sees lots of groups forming (and things are easy to see on the Internet because even our most casual utterances get stored on someone's servers for posterity to investigate) and he concludes that the world is alight with a new groupiness, the likes of which we have never seen. From time to time Shirky pipes up to remind his alter ego that this is not enough, that "treating the internet as some sort of separate space... was part of the problem" [194]. "The internet augments real-world social life rather than providing an alternative to it" [196]. One implication of Shirky's caution is that, to evaluate the state of groups in our world, we also have to look at how our use of the Internet may have displaced other forms of group building. But while Shirky knows all about Bowling Alone, Clay is too busy running off to tell us a story about Meetup.com to take a really close look at how those positives and negatives are adding up.

While Clay is telling us all about the use of digital technology to spark innovative forms of protest in Belarus, which is a fascinating story, we really need Shirky to ask why, with all these group-forming tools at our disposal and despite the documented disillusionment with the war in Iraq, there is so little coherent protest happening compared to previous wars? Is it really the case that society now is becoming, thanks to the internet, more democratic, more collaborative, and more cooperative than before? I am not convinced. Clay is in danger of making the same mistakes that William Greider made in One World, Ready or Not, and which Paul Krugman demolishes - of finding lots of examples of groups and inferring that the world must have more groups in it than it used to - but that logic is flawed.

Now Shirky is far too well-informed to fall into this kind of trap. Shirky recognizes that, just as removal of a bottleneck at one point in a highway may prompt a new bottleneck to form a few miles down the road, so the "removal of technological limits has exposed a second set of social ones" [91]. But Clay's enthusiasms mean that the book is unbalanced, and this second set of limits never really gets investigated closely. Clay the enthusiast wins out over Shirky the dispassionate observer. If you are going to argue that groups are forming as never before, and if you are going to use minor events like angry airline passengers protesting about being trapped on runways to claim that "Consumers now talk back to businesses and speak out to the general public, and they can do so en masse and in coordinate ways" [179] then you really have to think about consumer activity before digital technology. Here is "talking back to business" with a vengeance:

August [1800] - Notwithstanding that the last day of this month was a Sunday, it was marked as the commencement of a serious riot. A great increase in the price of provisions, more especially of bread, had roused the vindictive spirit of the poorer classes to an almost ungovernable pitch. They began late in the evening, by breaking the windows of a baker in Millstone Lane, and in the morning proceeded, with an increase of numbers and renewed impetuosity, to treat others of the same trade in the same unwelcome manner. Granaries were broken into at the canal wharfs, and it was really distressing to see with what famine-impelled eagerness many a mother bore away corn in her apron to feed her offspring. [link]

Do consumers have a stronger voice now than in the past? I don't know, but I do know that a story or two about American Airlines passengers is not going to convince me. If Clay wants to tell us about a student group using Facebook to protest about British Bank HSBC's cancellation of interest-free loans then perhaps he should think about the longstanding student boycott of Barclays Bank during the 1970's and 1980's that contributed to the end of apartheid. But he doesn't, and that's disappointing.

The second flaw - also a common one - is a reluctance to follow the money. Between them, Clay and Shirky convince me that the Internet marks a change; that a society with an Internet is different from a society without one. Also, they convince me that the Internet is not a single model of sharing/collaboration/collective action; it's many models. So let's talk about these models, about which ones have legitimacy and longevity and which don't. To do so requires Shirky to take the broad brush that Clay is using and to start to make smaller, more detailed points.

Perhaps we no longer need books telling us that the Internet is a big thing. It is time to treat that fact, as Shirky sometimes does, as the starting point for a discussion rather than the conclusion. The questions then become ones of what kind of structures will form and persist in the online world, and if you are going to talk about these questions then you have to address the economics of the problem.

Money matters.

People's willingness to contribute to Wikipedia, Shirky points out, is tied to the non-profit status of Wikipedia. A threat from Spanish participants to start an alternative version convinced founder Jimmy Wales to "formally forgo any future commerical plans for Wikipedia, and to move the site from Wikipedia.com to Wikipedia.org, in keeping with its nonprofit status. Similarly, he decided to adopt the GNU Free Documentation License for Wikipedia's content [which] assured contributors that their contributions would remain frerely available" [274]. There is an ongoing tension between contributors to sites and the "owners" of those sites that is visible on MySpace (see Billy Bragg's efforts to gain musicians rights over their content), on Facebook (the failed "Beacon" program as one example) and even Google and the line it walks with advertising content on its search results. Contributor-driven movie database IMDB was taken over by Amazon, but Amazon does not put its name on the pages of the site - does it fear that people would not contribute so freely if they realized they were just making Jeff Bezos even richer? The book ends with a story by Clay about a successful revolt by participants at digg.com - but once again, a story is not enough to support the thesis he is trying to make. 

There are many issues of democracy, ownership, privacy and control that, depending on how they are resolved, will influence our chances of staying upright as we get carried along by this technological flood. Clay Shirky shows in this book that he has many insights that we can use, but to really help steer this kayak we need more Shirky and less Clay.

The Anarchist in the Library

The Anarchist in the Library, Siva Vaidhyanathan, Basic Books 2004.

---

Siva Vaidhyanathan (see here and here) sees the central problem of the Internet and of society as a whole, as the tension between decentralization/freedom/anarchy and centralization/control/oligarchy.

The great challenge in the new century is to mediate between two divergent trends -- anarchy and oligarchy. In the war between distribution and concentration of information, the issues and conflicts seem intractable. [xvii]

In the world of the Internet, at least in the years leading up to 2004 when the book was published, these poles of anarchy and oligarchy manifest themselves technologically. Anarchy is the "ideology of peer-to-peer systems" such as the file-sharing and media-sharing networks that have followed Napster; it is characterized by a fluid, decentralized architecture in which "all the 'thinking'.. happens at the end point" and in which there is "no discernible command-and-control system" [17]. On the other side, digital rights management is the technology of oligarchy, imposing controls on what you can and cannot do with the software and media that you buy (or, increasingly, license). I don't usually buy the technology-determines-behaviour line, but his discussion of what it means for a technology to "have an ideology" is the best I've read, and is well illustrated by the distinction between technologies that operate on the basis of protocols (handshakes, conventions) and those that work on the basis of controls.

When I first picked up this book (from my library) I assumed that the author was another techno-utopian and that he is firmly on the side of freedom and anarchy over control and oligarchy (as who but the most unromantic of us could not be)? Not unreasonable, given chapter titles like Hacking the Currency and The Peer-to-Peer Revolution and the Future of Music, but wrong. In fact, although his heart is with the anarchists, Siva Vaidhyanathan is best described as a librarian. Not as in somebody who works in a library, but as in someone who is a fan of libraries:

Librarians should be our heroes. The library is not just functionally important to communities all over the world; it embodies Enlightenment values in the best sense. A library is a temple devoted to the antielitist notion that knowledge should be cheap if not free -- doors should be open. Supporting libraries -- monetarily, spiritually, intellectually, and legally -- is one of the best things we can do for the life we hope to build for the rest of the century. [Page 119]

Libraries seem anachronistic -- if they did not already exist, they could not be created now (in North America anyway) -- and yet they are among the most popular of institutions. People make jokes (deserved or not) about the service of the post office, the inflexibility of government, and the so on, but no one (well, apart from Seinfeld) pokes fun at libraries. Even some libertarians, who see any path laid by the state as a road to serfdom, love libraries. Our local libraries are always busy, and after a bad patch some time ago appear to be thriving with their mix of Internet access, DVD and CD rentals and, of course, loads of books. The popularity of the library surely comes from its nature as patron-focused but not commercial (no "consumers" here) and state sponsored but not monopolistic. It is populist and yet highbrow.

In the end Vaidhyanathan takes a dialectical stance and rejects both anarchy and oligarchy in favour of the library - a civic, noncommercial, open and public model. I didn't see this coming until well into the book, which after the first couple of chapters is more a set of essays around a common topic than a sequential argument. I thought he was heading towards one more simplistic agenda ("information wants to be free" anyone?) but he pulls himself back from the brink and ends up writing [185] "The heart of my argument in this book is a call for modesty and patience" and this:

The urge to break heads, to do the bidding of oligarchy by any means necessary is intimately linked to specters of anarchy. The urge towards anarchy depends on oligarchic abuses. Each creates the conditions that allow the other to thrive. The question for us in the twenty--first century should not be choosing anarchy or oligarchy by constructing and maintaining systems that discourage both. Anarchy is a reaction, not a vision or solution that can produce the best society and the best human future.  [187]

One of the reasons he pulls back is a debate with Randy Cohen of the New York Times, which he describes with admirable honesty [63].

"The history of popular culture is a continuous struggle on the artists' part not to get robbed... it seems to me that what MP3 [digital music] does is democratize the ability to rip off an artist," Cohen wrote to me. "And what's particularly galling is that you not only want to do it, you want to be praised as a social progressive when you do."
He got me. That's my schtick. By the guiding principles Cohen deployed in our peer-to-peer debate, I had no escape. He considered copyright to be an artists' right and concern; I consider the chief player in the copyright system to be the corporation.

The artists' struggle continues. The commercial oligarchs do whatever they can to avoid paying artists, as the Hollywood writers' strike showed and as recent press about royalty awards not being passed on to artists makes clear. But the free-use techno-anarchists exploit artists too. Greatest Living Englishman Billy Bragg writes in the New York Times about how he advised Michael Birch, the founder of social networking site Bebo.com, on how to handle artistic content on the site (via Nicholas Carr). Bragg was angry that while artists contributed to the site, Michael Birch sold it to AOL for $850million and the artists got nothing. Here is part of his op-ed:

He was hoping to expand his business by hosting music and wanted my advice on how to construct an artist-centered environment where musicians could post original songs without fear of losing control over their work. Following our talks, Mr. Birch told the press that he wanted Bebo to be a site that worked for artists and held their interests first and foremost.

In our discussions, we largely ignored the elephant in the room: the issue of whether he ought to consider paying some kind of royalties to the artists. After all, wasn’t he using their music to draw members — and advertising — to his business? Social-networking sites like Bebo argue that they have no money to distribute — their value is their membership. Well, last week Michael Birch realized the value of his membership. I’m sure he’ll be rewarding those technicians and accountants who helped him achieve this success. Perhaps he should also consider the contribution of his artists.

The musicians who posted their work on Bebo.com are no different from investors in a start-up enterprise. Their investment is the content provided for free while the site has no liquid assets. Now that the business has reaped huge benefits, surely they deserve a dividend...

If young musicians are to have a chance of enjoying a fruitful career, then we need to establish the principle of artists’ rights throughout the Internet — and we need to do it now.

Instead of cheering for one side or the other, Vidhyanathan's book foresees and acknowledges both these problems. And problems they are. He does not offer clear or simple solutions, but he points us thoughtfully in the right direction, and that's a very valuable contribution. So I definitely recommend the book, even though I wish it were a bit less scattered than it is (and that it had a better index). It's an exploration, not a recipe for the future, and has the rough edges that come with that territory, but there is a lot of food for thought in the pages.

The Anarchist in the Library, written in 2004, is already a bit outdated. Not the author's fault of course, but it is already odd to read a book about the Internet that has no mention of Wikipedia and no index entry for Google. These recent developments have changed the nature of the Internet. Peer to peer networks are still around, but I don't think they are the defining feature of the Internet. If we think of music we think of iTunes, not Napster; if we think of books we think of Amazon; if we think of social networks we think of Facebook and MySpace. None of these are peer-to-peer: they are centralized technologies built on the basis of controls, not protocols. Software architecture is often described as a stack, and while the low-level plumbing of the Internet remains a peer to peer protocol, the higher levels of the stack are "platforms" or client-server models of request and response. The techno-capitalist digirati have moved happily onto such platforms, and the centralization of ownership that they carry with them. In the era of utility computing, peer-to-peer networks appear to be on the wane.

There is one other place in which I think Vaidhyanathan gets it wrong. He underestimates the role of information asymmetries and transaction costs. He says that "Major record labels perform four basic tasks: production, distribution, price fixing, and gatekeeping" [48]. But there is a fifth, which is promotion. Simply putting a record on the Internet is hardly more effective than playing your music on your front lawn - the problems of finding something no one knows exists are far greater than he credits. One major function of libraries, after all, is to match readers and books ("Every reader his/her book" in one of Ranganathan's five laws of library science -- thanks John). When Vaidhyanathan tells a journalist that African musicians don't need record companies because "The artists can do it all themselves for less than $10,000" he is naive. Billy Bragg's friends on Bebo did the same, and it got them nowhere. Others have proclaimed this line before - for example Chris Anderson in my least favourite book talks about the band Birdmonster who eschewed labels

Label were calling with deals, but Birdmonster turned the offers down. As [lead singer Peter] Arcuni put it, “We’re not anti-label in principle, but the numbers (risk vs. reward) didn’t add up.”

A music label exists primarily to fulfill four functions: 1) talent scouting; 2) financing (the advances bands get to pay for their studio time is like seed capital invested by a venture capitalist); 3) distribution; 4) marketing.

From Birdmonster’s perspective, they didn’t need that.

Well, apparently they do now. The problem is that so much of culture is governed by asymmetric information, information cascades, and network effects. You don't know what a book is going to be like until you buy it, so simply knowledge of the existence of a book is insufficient - you need recommendations and reliable ones at that. I've gone on about this in various ways here and here and here. His neglect of these forces is one of the reasons I thought he was heading down the techno-utopian path, but as I say he ends up, thankfully, rejecting the "California Ideology" [155] in favour of something less catchy, less simple, but more hopeful.

Long Tail of News? Actually No.

Second in a series.

Ironically, Wired points to a report by the (American) Project for Excellence in Journalism on The State of the (American) News Media 2008. Here is a little excerpt from the Executive Summary.

The state of the American news media in 2008 is more troubled than a year ago.
And the problems, increasingly, appear to be different than many experts have predicted.
Critics have tended to see technology democratizing the media and traditional journalism in decline. Audiences, they say, are fragmenting across new information sources, breaking the grip of media elites. Some people even advocate the notion of “The Long Tail,” the idea that, with the Web’s infinite potential for depth, millions of niche markets could be bigger than the old mass market dominated by large companies and producers.1
The reality, increasingly, appears more complex. Looking closely, a clear case for democratization is harder to make. Even with so many new sources, more people now consume what old-media newsrooms produce, particularly from print, than before. Online, for instance, the top 10 news Web sites, drawing mostly from old brands, are more of an oligarchy, commanding a larger share of audience than they did in the legacy media. The verdict on citizen media for now suggests limitations. And research shows blogs and public affairs Web sites attract a smaller audience than expected and are produced by people with even more elite backgrounds than journalists.

Mr. Google's Guidebook

Whimsley Hall is now strewn, like Miss Haversham's house, with cobwebs and dust. Most visitors no longer come in by the front door to take a tour. Instead, Mr. Google (a travel agent who doubles as our butler) directs them straight down to the basement where the family archives are kept and tells them to look at one particular historical document called The Netflix Prize: 300 Days Later. They  read this and then they walk right out.

I shouldn't complain. It's nice that they visit at all - much better than rattling around here by myself - so I should be very grateful to Mr. Google for bringing these people to visit, but it does leave me wondering why he always sends them to look at this same corner of the house. I have a few other items lying around that I think are just as pretty but Mr. Google takes the visitors right by them without so much as a glance.

So when he brought me the sherry decanter the other day I challenged him on it. I thought it was an innocent enough question to ask of one's butler. Little did I realize the terrifying journey I was embarking on with that one question. He explained that when you ask him a question he "understands exactly what you mean and gives you back exactly what you want." That sounded a little presumptuous so I asked him how he could be so confident in his understanding and he replied, rather stiffly if you ask me, "If I did not give you exactly what you wanted then you wouldn't have asked me in the first place would you?" There was something about the slow, pronounced way he articulated this that made me feel like Wooster to his Jeeves so I didn't pursue the topic, fearing he would get upset. I wouldn't want him to leave; it's so hard to find good help nowadays.

As I sipped my sherry I realized that I don't really understand the man. For a butler and travel agent he seems remarkably well-to-do, and yet when I ask why he works so hard (I happen to know he is butler at several other houses in the county as well as mine) he insists he is only interested in helping people and points to his family motto, which he keeps on a little card that he brandishes frequently. "Don't be evil", it says.

Still, after a few glasses I still felt a little bolshie over Mr. Google's tone and I remembered that some months previous he had actually given me a copy of his biography (he is a most talented individual, I grant). He had told me it was an authorized volume with interviews and selections of his personal correspondence. I hadn't paid much attention to it at the time, but now I took a candle, wandered over to the west wing of Whimsley Hall, and climbed the stone staircase to the very top, where the library is. There I found the biography already lying on a table by the window, which was strange because I had never taken it off the shelf before, and as I sat down to read I realized that it does indeed tell me all about our Mr Google.

Was I surprised! The biography was a revelation. I don't get out often these days (it's the gout) and I am now woefully out of touch, but it turns out that there is more to Mr. Google than I ever dreamed. He is responsible for an astonishingly popular free publication called Mr. Google's Guidebook. It's one of the most remarkable books you'll ever read - if you open it twice you never see quite the same page. In fact the only book I've ever heard of to match it is a limited edition print called "The Book of Sand" that my friend Mr. Borges had in his library before he went mad.

Here is how Mr. Google's Guidebook works.

A long time ago, people used signposts to get where they wanted to go. Each signpost was a little underlined phrase in blue that took you to a new place. People would wander all over the place, hopping from one place to another, looking at signposts to see where to go next. These signposts made a sort of map. The complete map of the world is a very big and complicated thing of course, but here is a little piece of it (thanks to this article).

Mr. Google realized that most people don't really want maps, they want guidebooks. And he also realized how he could use those signposts to build a good guidebook. When someone puts up a signpost it shows that they feel this destination is a place worth going to. Mr. Google's Guidebook is very well indexed, so when you look up something like "I would like to visit a peaceful country retreat" it takes you to a page of recommendations. To generate these recommendations Mr. Google counted all the signposts that pointed to peaceful country retreats and pick those retreats that most signposts pointed to. "Here", he would say, "I recommend you go to Harburn House, or Green Mountain Bed and Breakfast". Mr. Google's Guidebook became something of a sensation. Once it was established, he put in a few advertisements alongside each question and made a pretty penny from it.

As I read further, a storm rose up in the west. The candle flickered as the wind came in through the ill-fitting windows. Maybe this forbidding atmosphere explains why the more I read the more I worried that there might be something a little sinister about our Mr. Google. Mr. Google's Guidebook has become very influential. In fact, I understand he now has a cave somewhere in view of the mountains that is full of large and noisy engines, with powerful pistons cranking away to produce new versions every day. That initial guide of his, so clean and tidy, has become a monstrous device like one of those machines designed by Mr. Goldberg or Mr. Robinson. He claims that every edition of his guide "considers more than 500 million variables and 2 billion terms".

It was late at night by the time I read this remarkable fact, and I'd had several more glasses of sherry. A particularly strong gust rattled the window and I stood up to close the shutters. The wind blew the window open and I had to reach out to grab it. It was then that I noticed in the distance a flash of flame and a column of thick black smoke rising in the distance from right near Mr. Google's cave! It looked positively diabolical. I believe I actually cried out in shock.

After a tussle I got the shutters closed and, shaken, returned to my leatherbound armchair and took another sip to calm my nerves. My mind was racing. "Come on Whimsley", I thought, "You're just imagining things". How could there be anything sinister about the Guidebook? I opened a page of the biography at random and it set my mind somewhat at ease. Here is what I read:

At some universities, administrators are taking a new approach to deciding where to put footpaths. At first they don't put footpaths anywhere; they just let students walk across the grass to get where they need to go, wearing away the grass and creating rough tracks as they do so. Then when it is clear where the popular tracks go the university can just tidy them up, put down some paving stones, and they have a path in the right place.
It's called the Wisdom of Crowds. The students decide where the paths go by just going about their everyday life, and the university taps into their preferences to design an environment that reflects exactly what the students would want to do.
That's what we've done here at Mr. Google's Guidebook. We track how people walk around, which signposts they follow, and that lets us put paths in the right place, just where you want them to be. We can lead you just where you want to go; lead you to interesting and even unexpected destinations that will provide you with just what you asked for, and more.

Maybe that's all it is then, I thought. Maybe everyone wants to read about the Netflix Prize and no one wants to read 25 essays about the Long Tail. Maybe I've had one moment of startling insight amidst so much dross. But somehow, as my mood plumbed new depths, my brain clung to a stubborn belief that the prettiest thing in Whimsley Hall is an ancient manuscript that sets out, with wit and panache, the problems with toilets. This gem lies in a corner covered with grime and neglected by Mr. Google. How could I possibly explain such gross inequity? Pondering this problem, I fell asleep in the chair and drifted into a strange dream...

I was standing with my mother in York; one of many visits some years ago. She was pointing to a Chinese take-away restaurant that occupied one narrow division of a long terrace of houses and shops on one of the old streets of that city.
- Look at that, she said, Why is that restaurant right where it is? Why does it go from here to here (gesturing to the edges of the restaurant) rather than some other place?
- I have no idea.
- It's because of the vikings. Over a thousand years ago the land was divided, and then subdivided as it was passed down from father to sons. Each son inherited a narrower strip, so that it had access to the road, and as a result property lines were narrow. Over the years new buildings have been erected until the current use of land has nothing to do with the original, and yet the boundaries remain; archaic, obsolete, and yet fixed. Unless a new motorway comes through they may stay the same way for another thousand years.
- How strange, I commented.
- Not as strange as the egg rolls, she replied...
Then I was in a train, reading a newspaper article about Alfred Wainwright's classic hand-written guides to the paths of the Lake District. It explained that, while the routes of footpaths usually evolved over time rendering such guides as Wainwright's obsolete, the very success of Mr. Wainwright's magnificent effort had preserved their relevance. People used the guides to follow the paths, cementing the routes in place for much longer than would have been the case without them.

I awoke with a start. What an odd thing to remember. Why now? My mother is in fine fettle - no worries there. And Wainwright's Guides? I hadn't thought of them in years. It must be something else that produced that dream. And then I realized, it was all about Mr. Google's Guidebook. I struck my forehead, blinded by this insight, and rushed to the table. Picking up the quill pen, I dipped it in the ink and began to write feverishly on the pad of paper lying there...

Mr. Google is lying! (I wrote) His Guidebook no longer reflects the paths set out by travellers as they navigate their lives. It is no longer an outside observer of people's wanderings. Google's success has changed the way people find their routes. Here is the way it happens. When a new cluster of destinations is built there may be a flurry of interest, with new signposts being erected pointing towards one or another of those competing locations. And those signposts have their own dynamics, perhaps forming a power law as set out by Mr. Shirky or perhaps something different, as Mr. Shalizi has explained.
But that's not the end of the story. After some initial burst, no one makes new signposts to this cluster of destinations any more. And no one uses the old signposts to select which particular destination to visit. Instead everyone uses Mr. Google's Guidebook. It becomes the major determinant of the way people travel; no longer a guide to an existing geography it now shapes the geography itself, becoming the most powerful force of all in many parts of the land.
So my Netflix Prize essay got selected by Mr. Google's machines as one of the more interesting and insightful commentaries - the machines are perceptive, we must grant them that - and it soon appeared as number 3 on the list of recommended destinations for anyone looking for "Netflix Prize", right after the official site itself. And now no one is guided here by those few original links - the relevance of their effect is as vestigial as the effect of the Vikings' property rules. Mr. Google's Guidebook has cemented the verdict in place long after the early discussion has lost its relevance, like the edges of the Chinese take-away and like Mr. Wainwright's guides fixed the routes of the paths he charted. With little new being written about the Netflix Prize the Guidebook is the major source of new journeys. And so the Guidebook changes the pattern of the landscape from a rich, linked one with its power law shape (or other shape). Instead, there is a two stage process in the evolution of much of the landscape. The first stage is a brief discussion, from which Mr. Google picks a few winners. In the second stage, after that discussion has faded away, the continuing popularity of the winners is assured simply by their positioning in the Guidebook. Mr. Google has singlehandedly changed the way people travel, changing the selection of destinations from an ongoing referendum to a brief discussion from which he anoints a few winners.
Mr. Google no longer gives you what you want, he selects a winner from the crowd and then tells you it's what you wanted.

I was just about to put down the pen, exhausted now, when I heard a creak and the door to the library opened. I lurched around to see coming through the door --- Mr. Google himself! His face was no longer subservient as befits a butler. Instead it was smirking. And his teeth - surely they had not been so pointed before. I shrank.

But Mr. Google did not attack me with a knife, or bite me in the neck. Nothing so dramatic. He simply looked over at my scribbled notes and sighed a world-weary sigh.
- You don't understand do you sir?
- What do you mean Google? I understand everything now.
- Really? This document here? And what does that matter if no one reads it? And who decides whether anyone can come here to view it? Exactly how do you propose to publicize your absurd opinions if not through me?

My shoulders sagged. Defeat. Of course, there was nothing I could do. "So you'll silence it then. Keep people away. My revelations will moulder, along with that masterpiece about the toilets".

- No (said Google). That's what I mean - you really don't understand. You see, I don't care if people come and look at these hen scratches or not. Maybe they will, maybe they won't. As long as I can sell a few advertisements on that page of my guidebook I really don't care. After all, what better praise for a Guidebook than to help people find out what's wrong with it? Just leave your manuscript with me. I'll look after it.

He held out his hand, imperious now. I felt disheveled after my long night. My brain was spinning. I could see no alternative. In a vain attempt to maintain some self-respect I drew myself up to my full height and pulled back my shoulders, adopting a bearing appropriate for my class. "All right Google. Here you go. Don't lose it now."

"Thank you sir. You can be sure I won't lose it. I never do lose anything you know."

I turned away from him and stumbled down the stairs. I had ended up giving him an order, and he had accepted it. Yet I could not shake the impression, even as he brought me a glass of sherry that evening in my sitting room, placing the silver tray beside me with deference, that Mr. Google - far from being a butler and travel guide - was more a master than a servant.

Quantum Breathing

Chris next door works at the Perimeter Institute; he is a physicist in the field of quantum information and I've been trying to read some of his papers. Interesting, if a bit beyond an ex-chemist. But reading quantum things again reminded me of The Best Quantum Thing I Ever Learned, which - outside chemists - seems little known. So I'll tell the story here as briefly as I can - Wikipedia now has a good set of entries on this if you want to know more, including fancy moving pictures: start at haemoglobin. All the images here are borrowed from Wikipedia.

---

Breathe in.

What's going on? You draw oxygen into your lungs; from there it passes to the bloodstream; from there to muscles and nerves. To make that journey oxygen molecules (O₂) stick to molecules of haemoglobin in red blood cells, which carry them until they get to a place where they are needed, at which point the oxygen molecules hop off the haemoglobin to do their work.

More precisely, an oxygen molecule sticks to an iron atom at the centre of one of four "haem" or "heme" groups that are part of each haemoglobin molecule. Here is a picture of a haem group with its iron atom (Fe) surrounded by four nitrogen atoms. There is actually a fifth Nitrogen atom at right angles to the diagram ("behind" the Fe), that is in turn attached to the long "globin" chain.

If haemoglobin was just a passive lump onto which oxygen molecules stick then breathing would not work. The first oxygen molecule to bump into a haemoglobin can attach to one of four iron atoms, and so would tend attach more frequently than the second (which has only three spots to attach to), which would attach more frequently than the third and the fourth. So unless oxygen was present in great excess, there would be empty seats left on each haemoglobin as it stops by the lungs to collect its passengers. In the same way, when the oxygen molecules step off from the haemoglobin at the end of the line, the last one would have the least tendency to leave - haemoglobins may have some passengers returning for a round trip. We would have to breathe much harder than we do to get the oxygen to our muscles.

But, as Max Perutz found out a few decades ago, haemoglobin is much more than a passive lump.

When haemoglobin is empty of oxygen, the haem groups - each of which is attached to one of the four "globin" strands that wrap themselves up to form the blob-shaped haemoglobin molecule - are near to the surface of the blob but are not openly exposed. When the first oxygen molecule attaches, it triggers a slight change in the local structure of the haem group, and this in turn tugs on the globin so that the other haem sites are exposed, and more easily attract oxygen molecules. It's this opening up as oxygen attaches that is the root of haemoglobin's efficiency, and the switch that triggers it is a quantum one.

Before the oxygen attaches, the iron sits slightly out of the plane of the four nitrogen atoms. But when the oxygen attaches, the iron atom moves into the plane of the ring. This slight movement is the switch, triggering the rearrangement of the globin chain and the exposing of the other haem binding sites. Here is a sideways view, in elaborate ASCII text graphics, of the iron moving into the haem plane.

          N
          |                               N
         Fe                               |
N-----N======N-------N           N-----N==Fe==N-----N
                                          |                                                                                 |
                                          O

What happens to the iron atom to shift it into the plane? To understand that you have to know a bit about how the electrons around the iron atom are arranged.

Electrons are arranged in shells of course, and each shell holds different numbers of electrons. The innermost shell holds two electrons, the second holds eight, and the third shell can hold 18 electrons. Iron has 14 electrons in the third shell and two electrons in the fourth shell. In haemoglobin, the iron exists as Fe²⁺, with the two electrons in the 4s orbital removed. We can focus on the 14 electrons in the 3 shell to understand the switch.

The 3 shell has a single orbital with zero angular momentum (3s) that, like any orbital, can hold two electrons (one "spin up", one "spin down"). It has three 3p orbitals (6 electrons) with an angular momentum of one and a slightly higher energy. And it has five 3d orbitals with an angular momentum of two and a slightly higher energy still. There are 6 electrons spread among the 3d orbitals (leaving four unoccupied spaces) and these d electrons are the key to the switch.

When six electrons occupy the five 3d orbitals they avoid pairing up if at all possible, and they have spins "pointing" the same way (Hund's Rule: a consequence of the Pauli Exclusion Principle). By occupying separate orbitals, which generally occupy different regions of space, they keep out of each others' way and this lets the negatively-charged get a little closer to the positively-charged nucleus (become more tightly bound) without repelling each other. So the 3d energy levels of an iron atom are like this (the vertical lines are electrons, the horizontal line shows an empty slot where an electron could fit).

    ^
E   |        ||  |-  |-  |-  |- 
    |

(The unpaired electrons here, all lined up with spins "pointing" the same way, are the reason that iron is magnetic.)

This labelling of orbitals as 3s, 3p, 3d, 4s and so on is precise only for isolated atoms. But place an iron in a cage and not all the 3d orbitals are at the same energy: there is a "ligand field" effect that raises the energy of some orbitals more than others. The Nitrogen atoms have a partial negative charge, and (like charges repel) electrons in orbitals that are located largely close to these atoms rise in energy. Those that are mainly distributed away from the Nitrogen atoms will be lower in energy.

The shape of the cage determines the change in energy levels (as well as a mixing among them which we can put to one side). Here is an iron atom in a square cage. The five 3d orbitals (in a line on the left of the diagram) have split into distinct levels on the right, with different energies. To untangle the configurations of orbitals in different environments you need to apply group theory ideas to quantum mechanics.

If the split between the energy levels is small then Hund's Rule still applies and the electrons will tend to occupy orbitals singly, with spins aligned: this is called the "high-spin" case.  If the split is bigger then the electrons pair up in the lower-energy orbitals: the "low-spin" case. In the haem group the iron is high-spin. But as the oxygen atom binds to it, bringing another negative charge close to the iron, the pattern changes and (crucially) the magnitude of the splitting among the energy levels increases, and the iron changes to being a low-spin case. And, to cut the story slightly short, this switch to a low-spin Fe²⁺ ion causes the iron to fit into the plane of the haem group instead of protruding from it, and it drags the fifth nitrogen atom and the globin chain along with it.

So breathing, in the end, works only because of a fine balance between magnetism and electron level splittings in iron atoms; it works because quantum mechanics says that electrons occupy discrete energy levels and because we can calculate the order and energy of those levels. What I like is that (unlike most popular quantum topics, which focus on weirdness) this phenomenon that lies behind every breath is no longer a mystery - it's a triumph of understanding and brings the most abstract topics, group theory and quantum mechanics, down to earth.