Whimsley

An Uncertain World II: Adapt, by Tim Harford

Another long post: PDF here (but lacking some last minute changes) for anyone who prefers it. 

There is a contradiction at the heart of Adapt, the new book by the Financial Times' Undercover Economist, Tim Harford, and once the contradiction is unpicked, the rest of the book unfortunately unravels. Apparently Nature, the Sunday Times, and the Financial Times loved the book, and his review page has nice comments from lots of smart people, but here goes anyway.

Table of Contents

• The thesis
• The contradiction
• Consequences
  • Capecchi and Tharp
  • Whole Foods, Timpson, Chile, and Wal-Mart
• Centralized experimentation
• The Financial Crisis
• Foxes and Hedgehogs
• If not trial-and-error, then what?

The thesis

Adapt argues that, to deal with the complexity and unpredictability of the modern world, we should take our inspiration from the market, and apply its methods to other parts of our world. It opens with a quotation from Hayek, and Harford is inspired by the market's evolutionary, decentralized, trial-and-error nature. Individual firms may fail in large numbers, but that is not a problem for the whole: "The difference between market-based economies and centrally-planned disasters, such as Mao Zedong's Great Leap Forward, is not that markets avoid failure. It's that large-scale failures do not seem to have the same dire consequences for the market as they do for planned economies." (11)

The year 2011 is an odd time to make such an argument, even though he does follow the above sentence with "The most obvious exception to this claim is also the most interesting: the financial crisis that began in 2007. We'll find out why it was such a catastrophic anomaly in chapter six". (11)

Harford continues: "trial and error is a tremendously powerful process for solving problems in a complex world, while expert leadership is not. Markets harness this process of trial and error, but that does not mean that we should leave everything to the market. It does mean - in the face of seemingly intractable problems, such as civil war, climate change and financial instability - that we must find a way to use the secret of trial and error beyond the familiar context of the market." (20)

And Harford prescribes his remedy liberally: "The adaptive, experimental approach [trial and error] can work almost anywhere" (35) he says, and he applies it to climate change (chapter five), to business strategy (chapter seven), to individuals (chapter eight), and to financial collapses (chapter six).

The contradiction

Theorem: In systems with multiple levels, trial-and-error cannot be an optimal strategy at each level.

Proof (by contradiction, and a bit like the Unexpected Hanging Paradox):

Consider an economy consisting of a government, firms, and employees. There are two sets of decision-maker in this economy: the government must decide on a strategy for the economy as a whole and firms must decide how they will instruct their employees to work.

To keep things really simple, limit the available strategy space for each decision-maker to only two options: 

• trial-and-error delegates decisions to the level below (government to firms, firms to employees)
• eggs-in-one-basket mandates to the lower level what they must do. 

If trial-and-error is the optimal strategy at the level of the economy, the government will adopt it, allowing each firm to identify and pursue a strategy of its own. Some firms will choose trial-and-error, while others adopt eggs-in-one-basket. Which firms will succeed? There are two cases.

Case A: some trial-and-error firms succeed, and some eggs-in-one-basket firms succeed. In this case, trial-and-error is not the optimal choice at the level of the firm, it is simply one strategy among others (the others in this case being eggs-in-one-basket) that may be worth pursuing.

Case B: only trial-and-error firms succeed. In this case, trial-and-error is the optimal strategy for firms. But (here's the catch) if trial-and-error is the best strategy at the level of the firm, then the optimal strategy for the government is eggs-in-one-basket, mandating that all firms must operate in a specific (trial-and-error) fashion.

In short, if trial-and-error is the best strategy at the level of government, then we cannot say it is the best strategy at the level of the individual firm. And if it is the best strategy at the level of the firm, it cannot be the best strategy at the level of the economy.

Consequences

The contradiction appears in stories throughout the book. Here are two examples.

Capecchi and Tharp

Adapt tells us the stories of Mario Capecchi (pp 97–100) and of Twyla Tharp (pp 247–256).

Tharp is a brilliant and determined choreographer whose ambitious production Movin' Out was panned when it premiered in Chicago. According to Adapt, Tharp is a believer in the process of experimentation, filming hours of improvised dance in the search for just a few interesting moves. She treated the Chicago experience as a failed experiment and "made peace with her losses and immediately set about the hard work of winning back both the critics and the audiences." (254) She reworked the show, and the result was a piece of groundbreaking dance that won her rave reviews when it moved to Broadway. An inspiring story.

Capecchi is a brilliant and determined biologist whose ambitious proposal to "make a specific, targeted change to a gene in a mouse's DNA" (99) was panned when he submitted it to the NIH funding agency. According to Adapt, Capecchi is a "stubborn genius" who survived a remarkable childhood as a street urchin. He refused to treat the NIH proposal as a failed experiment, refused to make peace with his losses, and instead treated the NIH experience as an obstacle to be worked around. He got grants for two other less-ambitious projects and used that money to carry out his panned mouse-gene project anyway. The result was a piece of groundbreaking research that won him the 2007 Nobel Prize in medicine. An inspiring story.

Harford uses the story of Twyla Tharp to promote the virtues of trial-and-error at the level of individual, and the story of Mario Capecchi to promote the virtues of trial-and-error at the level of the organization. But trial-and-error at the organization level means accommodating people like Capecchi, who is clearly not a trial-and-error individual and who exhibits precisely those traits (rejecting critics views, refusal to change) that Harford warns us against in the Tharp story.

Whole Foods, Timpson, Chile, and Wal-Mart

Adapt tells us the story of American high-end grocery chain Whole Foods and UK bric-a-brac retailer Timpson (pp 224–230). These two companies promote decentralization and experimentation, giving teams (in the case of Whole Foods) or stores (in the case of Timpson) independence from the company so that they can find their own successful strategies. Both exemplify Hayek's 'now familiar words "knowledge of the particular circumstance of time and place"' (227). Harford uses their stories to show that "the world is increasingly rewarding those who can quickly adapt to local circumstances" and to promote strategies such as delegation of power to the front lines of the organization. Peer monitoring "offers a subtlety and sensitivity that monitoring from corporate HQ simply cannot match" (229).

He contrasts the decentralized efforts of Whole Foods and Timpson with the failure of centralized planning, exemplified by "one of the most surreal examples of the planner's dream" (69), Salvador Allende's Project CyberSyn which looked to use a supercomputer to collect centralized reports of economic activity throughout Chile in the early 1970's and to tune the economy. The project was "not a success", and shows us "the way in which our critical faculties switch off when faced with the latest technology." (70) The dream of "information delivered in detail, real-time, to a command centre from which computer-aided decisions could be sent back to the front line" persisted in the form of Donald Rumsfeld and his failed conduct of the Iraq invasion (chapter 2). The lesson is that "such [centralized] systems always deliver less than they promise, because they remain incapable of capturing the tacit knowledge that really matters." (71)

Wal-Mart is mentioned only briefly in Adapt, on p 226. "Of course, this kind of business model [Whole Foods] is not the only way to succeed in the supermarket trade. Far more centralised supermarkets such as Wal-Mart in the US and Tesco in the UK are clearly very profitable".

Wal-Mart owes its success to massive centralization of a kind that makes Project CyberSyn look unambitious. Famously, the Wall Street Journal reported in 2007 that "Wal-Mart's centralized thermostat system in Bentonville, Arkansas, its corporate headquarters, actually uses a monitoring team to control the temperature for every store from this centralized location." Centralization and scale permit remorseless cost-cutting precisely by minimizing local experimentation even in such minute decisions as store temperature.

Again, the contradiction plays out. Harford wants to argue that at the level of the economy as a whole and at the level of the individual firm, trial-and-error triumphs over misguided centralization. Yet trial-and-error at the level of the economy permits individual firms like Wal-Mart to pursue centralized, eggs-in-one-basket strategies, some of which turn out to be better than experimentation. You can't eat your cake and have it too.

Centralized experimentation

Harford does note that Wal-Mart and Tesco "still experiment but have managed to centralise and automate that experimentation" (226), but gives no further details. This sentence shows another failure of the book: a blurring of the line between experimentation (trial-and-error) and decentralization. Throughout most of the book he uses experimentation as a synonym for decentralization (tacit knowledge and all that) and is in favour of both, but sometimes - as here - he separates the two to make his argument fit.

The most dramatic case where he separates experimentation from decentralization is in the chapter on development aid (chapter 4). Part of the issue in pursuing a trial-and-error strategy is to identify successes – to design a "feedback loop" that permits successful ideas to evolve further – and chapter 4 looks at the use of randomized trials in development projects to provide that feedback.

Harford argues throughout chapter 4 that development aid projects can easily go wrong despite, or because of, the best intentions of those involved. To identify successful strategies he argues for rigorous experimentation based on clinical trial methodologies, and randomized trials. Why, I wondered throughout this chapter, does the use of randomized trials come up particularly in the area of development aid? I believe (and here I may be atributing ideas to Harford that are not his) that it comes from the usual economist's idea that in development aid we must avoid woolly, sentimental thinking and adopt a hard-headed approach if we are really to Do the Right Thing.

The problem is, randomized trials are often not incentive compatible for the participants. In one example, Harford describes an experiment in Kenya to deliver a new set of textbooks to schools. The charity funding the programme "chose twenty-five schools at random" and distributed the books. The experimenters found, surprisingly, "little evidence that textbooks were helpful". Chalk one up for randomized trials.

(I'm going a bit out on a limb for the next few paragraphs: if others with more knowledge of the subject can correct me, go for it).

What are the incentives at work at the level of the individual school? Given a choice between a set of new textbooks and no set of new textbooks, most schools would choose the books because the expectation was that they would improve the education experience. To succeed, randomized trials demand either a setup where there is no expectation of likely outcomes (as in another example he gives, an 18th century sea doctor treating scurvy with "oranges and lemons" or "cider, acid, or brine"), or where the subjects of the experiment are deprived of the right to choose. If there is an expectation, going in to the experiment, that one option or the other is more likely to be beneficial, then it is in the interests of the experimental subjects to go with that option rather than participate in a randomized trial.

It is not surprising that Harford can find stories of randomized trials in the case of medicine and of development aid. Both scenarios rely on a powerless, voiceless set of experimental subjects. In scenarios where the subjects have a voice, randomized trials are rare despite their system-wide benefits because the incentives don't line up. Are there cases of trials in North American or British school systems where a random selection of schools get access to a new, potentially beneficial teaching aid? If there are, Harford has apparently not found them.

So here again the contradiction is at work. Trial-and-error at the level of the development project demands centralized control. The school textbook project demands that trial-and-error not be an option at the level of the individual school.

Such contradictions occur throughout the book. I don't know much about development aid or randomized testing, but I do know a bit about technology. Harford gives Google a positive write-up for its famous "20% time" program for employees (pp 231–234), showing that it succeeds by casting aside those projects that fail. But in demonstrating why failures need to be rigorously abandoned (a "tight feedback loop") he writes "According to the TechRepublic website, two of the five worst technology products of 2009 came from Google - and they were major Google products at that, Google Wave and the Android 1.0 operating system for mobile phones. Yet most internet users know and rely on Google's search, Google Maps and Image search, while many others swear by Gmail, Google Reader, and Blogger." (234). By his own logic, Google should have abandoned the duds, and it did throw Wave overboard, but of course in the case of Android it persisted and now Android is the most widely-used smartphone operating system in the world. And Google Maps and Blogger, at least, are not products of the innovation program but were developed outside Google and then purchased.

Adapt also uses business professor Clayton Christensen's Innovator's Dilemma to bolster its case, in which Christensen shows how surprisingly primitive but cheap and "good-enough" technologies tend to displace advanced, high-end technologies in a process called "disruptive innovation". For the record, I found that to be an excellent book. But I first heard Christensen talk on the subject at a BlackBerry conference where he explained that the threat to BlackBerry was that its end-to-end design was vulnerable to a horizontal model that would not give the same highly-tuned experience, but which would do a good-enough job at lower prices. It sounded convincing, but within a year (I think) the real competitor turned out to be Apple's iPhone - an even more high-end, even more end-to-end design, and quite the opposite of the prediction. Apple, of course, is a hugely centralized company that puts all its eggs in one basket when it releases new technologies.

The Financial Crisis

By this time, you can see that I think Adapt suffers from the very cognitive dissonance that its author warns against in his final chapter. In the face of contrary evidence, the author finds ways to accommodate the facts within his framework by stretching the argument in ways that are ultimately unconvincing. It's exactly this kind of flaw that Tetlock identified among his worst-performing experts (the hedgehogs). Nowhere is it more obvious than in Adapt's chapter on the financial crisis.

To identify successful strategies, Harford argues that "we should not try to design a better world. We should make better feedback loops" (140) so that failures can be identified and successes capitalized on. Harford just asserts that "a market provides a short, strong feedback loop" (141), because "If one cafe is ordering a better combination of service, range of food, prices, decor, coffee blend, and so on, then more customers will congregate there than at the cafe next door", but everyday small-scale examples like this have little to do with markets for credit default swaps or with any other large-scale operation.

The chapter on the financial crisis misses the boat completely.

One part of the chapter deals with arranging incentives to encourage whistleblowers. There is only one problem with this, which is that a lack of whistleblowers was not the problem with the mortgage market. There were people who saw the market coming, said so, and put money behind their words, and their stories have been chronicled in books like Michael Lewis's The Big Short. But such people were written off as Chicken Littles. The problem was that the "short, strong feedback loop" of the market was neither short nor strong: it sent fundamentally misleading messages, because that's what a bubble is.

The second idea Adapt has about financial crises is to provide greater visibility for regulators into problems and stresses in a system, and it is illustrated by a story about the final weekend in September 2008, as Lehmann Brothers collapsed. But again, that weekend was simply the final bursting of the pimple and, while replaying that weekend may have led to different outcomes, the time was long past when the crisis could have been averted.

The final idea is borrowed from engineering systems, and has been floated again by others recently (Justin Fox here), and is to decouple various parts of the financial system so that failures in one area cannot spill over to failures in other areas. This may or may not work - I'm no expert - but I can say this: the proposal has nothing to do with the thesis of the book. The coupling between different parts of the financial system came about precisely because of an unwarranted belief in the virtues of the innovative, market-driven processes that Harford is promoting, and were governed by exactly the kind of feedback that he relies on (price, profits) elsewhere.

Foxes and Hedgehogs

Like Future Babble (see previous post), Harford includes the work of Philip Tetlock early in the book, to show us not to trust experts. And that's fine. But he fails to see the depth of the paradox of expert failure: it applies to those who would replace experts as much as it applies to experts themselves.

Tetlock divided his experts into foxes (good at many things) and hedgehogs (good at one thing) and argued that hedgehogs are over-confident because they "reduce the problem to some core theoretical scheme'… and they used that theme over and over, like a template, to stamp out predictions". And that's exactly what Harford does here. He sees evolution as a fox-like strategy (trying many things and selecting a few) but doesn't notice that at the level of individual species, evolution gives us both foxes and hedgehogs, and both do perfectly fine.

Once the contradiction at the heart of the book is clear, it is not surprising that the book itself cherry picks examples where trial-and-error has succeeded, or where eggs-in-one-basket has failed. But such stories, while entertaining, make a notoriously shaky foundation for any kind of general structure, and so it proves here.

If not trial-and-error, then what?

So in the end, Harford fails in his attempt to sell trial-and-error as a panacea. It's easy to knock, I can hear you say, but do you have anything better to offer? Well probably not, but let me at least sketch some preliminary thoughts very briefly.

Both Gardner and (especially) Harford place great emphasis on the usefulness of particular knowledge, and the need to recognize our limitations when it comes to seeing the future and to planning. "Allow room to experiment, to revise, and to adapt" is not bad advice so far as it goes. But Harford pushes this argument too far, and so tumbles into contradiction. He seeks to use this lesson as a general, all-purpose lesson, which means that he is again failing to acknowledge our limitations when it comes to seeing the future and to planning.

We need to accept that there is no algorithm for success. In fact, any such recipe would be self-defeating. The process of achieving success is irreducibly specific, irreducibly individual, and irreducibly paradoxical. It is not the realm of science, logic and analysis - it is the realm of art, precisely because art is comfortable with paradox and self-contradiction in a way that science and logic is not.

Or: "If I knew the jazz of the future, I'd play it" as someone said.

 

The Myth of Digital Democracy, by Matthew Hindman - reviewing the reviews

The Myth of Digital Democracy, Matthew Hindman, Princeton University Press  2009

---

The last sentence of Matthew Hindman's The Myth of Digital Democracy is "It may be easier to speak in cyberspace, but it remains difficult to be heard". The book is about collecting and analyzing the following large data sets on the way to this conclusion:

• The links among 3 million American political web pages together with data showing how Google leads its users to political sites. Hindman concludes that "link structure is an effective proxy for audience share" and that "communities of Web sites on different political topics are each dominated by a small set of highly successful sites". The scale of online concentration is so profound, he argues, that claims the Internet "democratizes" politics are misleading. For example, when it comes to blogs, "the top blogs are now the most widely read sources of political commentary in the United States", but these widely-read bloggers are very few in number (a few dozen) and they are "overwhelmingly.. well-educated white male professionals". The kind of voices that get heard in political discussion are the same kind that were heard through offline media, only perhaps more so. "The vigorous online debate that blogs provide may be, on balance, a good thing for US democracy. But as many continue to celebrate the democratic nature of blogs, it is important to acknowledge that many voices are left out."
• Data from Hitwise of search-engine-directed traffic show that online politics is a tiny sliver of Internet traffic, and that "Scholars, public officials, and journalists have paid a great deal of attention to online politics. Citizens themselves, though, have directed their attention elsewhere." Not too surprising perhaps.
• Data from Hitwise and other sources, of patterns of concentration in [American] online and traditional news media. He concludes that online media is much more concentrated (a few outlets get a larger share of the traffic) than many offline industries, particularly radio. The biggest story is what he calls "the missing middle":
  
  From the beginning, the Internet has been portrayed as a media Robin Hood - robbing audience from the big print and broadcast outlets and giving it to the little guys. But the data in this chapter suggest that audiences are moving in both directions. On the one hand, the news market in cyberspace seems even more concentrated on the top ten or twenty outlets than print media is. On the other, the tiniest outlets have indeed earned a substantial portion of the total eyeballs... It is the middle-class outlets that have seen relative decline in the online world. Moreover, it is overwhelmingly smaller, local media organizations that have lost out to national sources. [p100]

It is a refreshing change to read a book about the cultural and political impact of the Internet that actually looks closely at Internet traffic (what people read) rather than at the number of sites (what people write), and it's this perspective that leads Hindman to his myth-busting conclusions. The main flaw of the book is that it falls between two stools: it's clearly an academic work that started as a set of papers or a thesis, but it is looking for a wider, popular audience. To reach that audience, Hindman should have got rid of many technical details and written a book with more narrative, but if you don't mind reading technical studies, this is a good one, and I recommend it.

The Myth of Digital Democracy has been out for a year or so now, so after I finished it, I looked at some reviews, and got a surprise. The books detractors argue that no one claimed the Internet democratizes politics, and if they ever did then they don't any more, and if they still do then they mean something different.

So here are links to some critiques of Hindman's book, and some words in defence of The Myth of Digital Democracy:

• Charlie Beckett of the LSE's think tank POLIS writes that the idea "that the Internet is innately democratic and that it will have revolutionary political consequences" is "a straw man". He goes on: "I always struggle in lectures or talks when I have to find quotes from these digital utopians. I can always cite lots of people (like me) who argue that the Internet has given us great tools and that it offers huge potential for civic engagement and public self-expression. I can even find examples, from Mysociety to Iran and Twitter that show concrete cases. But I don’t know many serious people talking about a revolution."
  
  This is a little cheeky from someone whose own book is called SuperMedia, and which is publicised as a "manifesto" for a "radical new relationship between the media and the public", and which apparently "explores the potential for an entirely new type of journalism... and makes the case that journalism could be the catalyst for change needed to solve many of the world's problems." All sounds pretty revolutionary to me. Beckett is getting hung up on the title, which was probably the publisher's idea anyway, and needs to read the book more closely.
  
  As for actually finding quotes about the Internet's innately democratic and revolutionary (or at least radically disruptive) nature, well there's always Google's description of its own PageRank search algorithm as "drawing on the uniquely democratic nature of the Internet" and its recent claims that in Iran, citizen video reporting via YouTube "appears to have become an essential part of their struggle". Or there's Lawrence Lessig in Remixed: "The neutral platform of the Internet democratized technical and commercial innovation. Power was thus radically shifted." Or Yochai Benkler at the very beginning of his influential The Wealth of Networks: "Enabled by technological change, we are beginning to see a series of economic, social, and cultural adaptations that make possible a radical transformation of how we make the information environment we occupy as autonomous individuals, citizens, and members of cultural and social groups. It seems passe today to speak of 'The Internet revolution.' In some academic circles, it is positively naive. But it should not be." And that's ignoring all the talk of Twitter Revolutions in Moldova and Iran, and Egypt's Facebook Revolution.
  
  Discussion of the Internet's impact on all aspects of our society is routinely cast in a revolutionary light, with a strong dose of "power to the people", and if such language is usually kept out of academic journals, that's no reason to ignore it.
  
• For Mark Bahnisch of Inside Story, Hindman "narrows down the possible targets for his myth-busting to the claim that the internet will bring about 'democratisation.' But it's unclear who's actually making this claim, and Hindman ignores most of the more nuanced and specialised scholarship on the topic. If what he has to say when he gets down and dirty with the blogs later on is any indication, the real target is a nuch of journos and op-edders writing in the New York Times circa 2004." 
  
  Bahnisch is saying Hindman should ignore "journos and op-edders" with audiences of millions and focus instead on "nuanced and specialised" scholarship that general readers like me will never read. Rubbish. The real debate about the role of the Internet is a public one, and Hindman has done exactly the right thing to tackle this perception that's out there among the unwashed masses like myself. If the nuanced and specialized among us have good things to say, they should get out there in public and say them.
  
• When it comes to the concept of democratization, Bahnisch and others people bring up "Michel Bauwens' concept of equipotentiality" as an alternative meaning interpretation of "democratization". Equipotentiality is a fancy word for speculations about how peer-to-peer networks function: no one said that democratization means a wider set of voices are heard, simply that anyone can potentially reach an audience of millions, and if only a few do, well good for them.
  
  The whole point Hindman's approach makes is that concepts, whether they coin new words or not, need to be grounded in reality. Is the loose world of blogs equipotential? Who knows, given the vagueness of the concept? But that potential is certainly not equirealized and that's a point worth making.
  
• Allison Hayward writes that "to observe that 'digital democracy isn't always and everywhere the rule is not the same thing as saying that the Internet hasn't 'democratized' politics... For the Internet to be 'democratizing' we shouldn't require that it be revolutionary, only that at the margins it provide a broader population with more opportunities to contribute, volunteer, engage, and advocate, and make changes that are sustainable over time."
  
  Sigh. Hindman has demonstrated that many of these functions are not accessible to a broader population in any meaningful way. The book challenges those who believe the Internet is "democratizing" in whatever way to show there is some substance to these claims, and the rebuttals are wishful thinking based in, well, nothing much really.
  
• Matt Bai argues that Hindman's data is old, and "the political impact of the Internet is spreading so quickly that it's almost impossible to capture and quantify": he disparages Hindmans's graphs and equations and prefers anecdotes about internet activists from modest backgrounds. 
  
  By the time I'd read this review I was feeling pretty sorry for Matthew Hindman, because it is clear that he just can't win. Bai has a belief in the power of the Internet that borders on the mystical ("spreading so quickly that it's almost impossible to capture and quantify"), so what would it take to persuade him otherwise? Nothing short of careful, detailed data I'm sure. But when someone does the work, well it takes time, and so it is dismissed as "old" and full of obscure graphs and equations. There is little point entering such an argument.
  
• Along these lines Henry Farrell, in the Times Higher Education Supplement, suggests that Hindman should just not engage with these claims, that he spends too much time "refuting bad ideas" and tackling "stupid claims for the democratic benefits of technological pixie dust" and goes on to say that "To really understand how the internet affects democratic politics, we need to forget about the internet evangelists. Not only were they badly wrong, but their notions of democracy were sloppy and unhelpful."
  
  I sympathize with Henry Farrell: it would be nice if we could forget about bad ideas and move ahead on more constructive paths, and I hope people do. But sloppy or not, those notions of democracy are out there, and are still influential, and Hindman has made a valuable contribution towards refuting them. 

The Pirate’s Dilemma by Matt Mason: A Review

[Shinier and better version here.]

In The Pirate's Dilemma Matt Mason claims to speak from the perspective of rebellious and subversive youth culture while he promotes the worst kind of corporate astroturfing which is too bad because his message that many new ideas emerge from outside the mainstream and from outside the market is important and his message that pirates who skirt the edges of the law to bring culture to new audiences have done much to improve our society also matters, but when an author wants us to believe he is anti-establishment while he praises the vitamin water company Glacéau for "keeping it real" in its advertising campaigns with 50 Cent while telling us that it was sold to Coca Cola for $4.1 billion in 2007, praises Procter & Gamble for its viral video campaign, is entranced by the way that Nike's Air Force One sneaker owes its success to the "remix" well where do you start? Mason loves the idea that youth culture, existing in spaces outside the mainstream and outside the commercial world, has had a huge impact on our modern world and I am with him 100% but the lives of many idealistic, left-wing youth become enmeshed in compromise as we get older and we stoke the fires of capitalism during the day while trying to throw a little water on those same fires in the evening and I understand how this happens because like many middle-aged people I wrestle with the contradictions and compromises involved, and I admire those few who have stuck to their principles, often at a real cost to their careers and personal lives, which is why the few people who really piss me off, whom I actively scorn and who get my blood boiling, are those like Matt Mason who don the mantle of rebellion and anti-corporate politics while consulting for Disney, Pepsi, and P&G and who claim that selling YouTube to Google for $1.65 billion is a form of rebellion and who babble about the benefits of sharing because "it's not all about the money any more" while giving presentations to the people who brought you McDonald's "I'm Loving It" campaign and who place themselves on the romantic side of the battle between graffiti and advertising in "a turf war that has raged for centuries between the establishment and a secretive, loose-knit network that doesn't like the top-down, one-way flow of information in public spaces" [103] only to step slickly into approvingly quoting advertising agency Droga5 on creating "a dialogue between advertising and graffiti" which really means using graffiti for commercial ends and making a buck and if that's not selling out to the man then what the fuck is really? because the punk spirit Mason loves so much has nothing to do with business models or change agents or entrepreneurial spirit or building a brand or even combining altruism with self-interest because the spirit he writes of was defiantly and nihilistically anti-corporate and Matt Mason lives in a corporate world however much he'd like to think otherwise and when he claims that pirates are those who are "pushing back against authority, decentralizing monopolies, and promoting the rule of the people: the very nature of democracy itself" well I see what he means but then when he goes on to claim that the anti-authoritarian ideals of youth culture are becoming ... a new more extreme, invigorated, and equitable strain of the free market--the decentralized future of capitalism [171] then I just want to shake him by the neck and shout at him that you're obviously not stupid Matt Mason so why don't you do what you know you should do and follow the fucking money before making pronouncements about sharing and decentralization when it's still the case that money is not shared and money is not decentralized because sharing is one thing but if I share and you get the money then I'm not being altruistic I'm just being a sucker and you're not promoting community you're exploiting the good intentions of those who are spending their time and talent on your venture and if you want to impress me with the subversive role of DVD bootlegging don't quote billionaire Mark Cuban and Disney co-chair Anne Sweeney and billionaire Steve Jobs at me because if they have found a way to co-exist with piracy it doesn't mean that these companies stand for a more democratic and equitable form of capitalism it just means they've found ways of using or co-existing with piracy in a way that promotes their own interests over those of their rivals and I end up not taking him seriously at all which is unfortunate because he has many entertaining stories of hip-hop, pirate, and punk culture although I end up not knowing whether to trust them because where the book overlaps with things that I know anything about he is often ludicrously wrong like when he repeatedly refers to Linux as a company [148, 150, 238] or when he quotes Courtney Love saying that record companies "figured out that it's a lot more profitable to control the distribution system than it is to nurture artists... They own the plantation" while completely failing to notice that big chunks of the Web 2.0 world he loves work on exactly the same model, that owning the platform gives control of the distribution system and that's where the money is or when he gives us a canned history of Wikipedia [149-150] which is derived from one interview with Jimmy Wales so it's no surprise that it gets several key facts wrong or when he identifies Steve Jobs with openness and sharing [145] and claims that the notoriously secretive and proprietary Apple won the music wars because it "truly understood sharing" [158] when the fact is that the only thing Apple really wants to share is the music they don't own and not the technology that they do own you can ask Palm about that who can't sync their own phones with iTunes or you can ask the developers who have left the iPhone App Store over Apple's arbitrary and opaque approval process, so Matt Mason in the end sits for me with people from an earlier generation like Kevin Kelly who claims to be a maverick while working for Conde Nast or Chris Anderson who claims to be on the side of small and scrappy businesses against big companies while promoting Amazon or Stewart Brand or John Perry Barlow who strive to combine activities like consulting for senior management at large corporations with statements like "I'm an anti-company man" if you can believe it I mean do you have any self-awareness at all? I want to ask them I think it's the ego that gets to me and the fact that they have been successful in leading people and particularly young idealistic people with good intentions into activities that they think are progressive and politically anti-establishment but which end up just feeding money into the pockets of Silicon Valley venture capitalists and the lucky guys who get to sell their startups to Google for a nice billion or so as if that's a triumph of the little guy give me a break.

Review: Market Rebels, by Hayagreeva Rao

Market Rebels: How Activists Make or Break Radical Innovations, Hayagreeva Rao, Princeton University Press 2009

---

For decades, economists have extended their intellectual reach beyond mere money in an attempt to encompass all the social sciences in their analytical framework. But now the boot is on the other foot and it looks like even core economic observations may be better explained by other social sciences. Robert Solow apparently said that attempts to explain differences in economic growth across countries typically end in "a blaze of amateur sociology". The focus on psychology in explanations of the banking crash shows that growth is not the only area of economics where the discipline runs out of steam before reaching its destination. The rise of behavioural economics, surely a last-gasp attempt by economists to match their models to the real world without changing departments, suggests that the condition goes deep.

Despite its title, Hayagreeva Rao's Market Rebels (Open Library link, publisher's page) challenges the economic analysis of innovations. At 180 pages and full of case studies it's easy to read quickly, but I was so taken by it that I went through it a second time and found much that I had missed. Rao does not hammer the reader over the head with the implications of his case studies, but for me as a non-sociologist and non-economist the implications are huge and I'll be thinking about the book for a long time.

The case studies are diverse, but are centered around a single claim: the "joined hands of activists" play an important part in the creation, diffusion, and blocking of innovations. Collective action matters. Rao describes how hobbyists were key to the cultural acceptance of the car and the development of the personal computer; how microbrewers brought diversity back to beer; how nouvelle cuisine grew from the rebellious student movements of Paris 1968; how shareholder activism has pushed large companies to change behaviours; how community activists attempted to stall the spread of chain stores and then of big-box stores; how the green movement blocked the development of biotechnology in Europe. These studies, many based on his own research, help to bring activist groups and their campaigns out from the wings and into the spotlight as we think about innovation and social change, and by doing so Rao is performing a valuable service.

The book is not strong on systematic analysis. The closest he gets to describing what determines whether movements succeed or fail is that successful movements must create "hot causes" and "cool mobilizations". Both concepts are tied in to the concept of "identity", which for Rao is the underlying motive that causes people to join with or against social movements. A "hot cause" is the spark that successful activists use to light a fire. It's a lightning-rod incident or issue that arouses strong emotions such as pride or anger. Examples include the frustrated demonization of "big beer" by real ale enthusiasts; the outrageous bonus paid to Home Depot CEO Robert Nardelli which crystallized the shareholder rights movement. But "hot causes" by themselves are not enough for a prolonged campaign. "Cool mobilizations" are actions that keep a movement going forward by "engaging audiences in new behaviors and new experiences that are improvisational and insurgent". Examples include setting up a microbrewery or formulating shareholder resolutions. The concepts are useful, but it's a shame Rao doesn't have a stronger turn of phrase. Naming concepts can be key to owning them, and "hot cause" and "cool mobilization" are too literal and clumsy to take on the weight they need. But this is a detail (and I don't have better ideas).

For someone who has spent most of their non-fiction reading time reading economics and economics-inspired books in recent years, Rao's is a welcome and refreshing change. Economic analysis too-often reduces the political left-right split to the false dichotomy of market vs state, but this reduction maps badly on to the real experience of political activism. Those who protest Monsanto's private-sector use of genetic engineering are often the same as those who protest state-driven wars. Many of those who oppose new Wal-Mart stores also oppose the extension of surveillance powers by the state. Where do such activists see themselves in a market vs state debate? For many, they don't: market vs state is not what it's about. So it's not surprising that economists have a blind spot when it comes to social movements, and that the discipline systematically minimizes their impact. By putting social movements at the centre of his stories, Rao shows that they can and do have an influence, and that they deserve a place in any serious look at institutions that shape social change.

Although he says almost nothing about the Internet and digital collaboration, Market Rebels' focus on innovations makes the book obviously relevant. Rao's analysis is a welcome alternative to the usual focus of widely-read writers like Yochai Benkler and Clay Shirky. These writers take the economics point of view and focus on issues such information as a public good, lowering transaction costs for online exchanges, and the vanishingly small marginal cost of reproduction of digital information. Rao's unspoken counterargument, which convinces me, is that group formation is not a problem of information, it's a problem of identity. If he is right then although we can expect to see many examples of successful groups in the online world, we won't see not a huge flowering of groupiness compared to the information-starved analogue world.

What's more, if Rao is right and initiatives such as Wikipedia, blogging and the Open Source movement really are social movements, then they may have a limited lifespan. Digital activist identity is a rebellious and anti-establishement stance, but such a stance can only be maintained while the movement is oppositional. Open Source, with a longer history behind it than Wikipedia or blogging, has changed its identity to being more professionalized, more establishment. Its participants are less likely to be the radicals of yesteryear. Once Wikipedia is established as a success for a few years, rebellion becomes irrelevant as a motive and one may wonder whether the activists will find the "cool mobilizations" to maintain involvement and participation.

If there is an economics tie-in with Rao's analysis, it's with the analysis of identity pioneered by Rachel Kranton and Robert Akerlof (and which I sketched in the final chapter of No One Makes You Shop at Wal-Mart). Rao does little to pick apart the concept of identity and it looks to me like the K&A analysis would have been helpful to him. For Kranton and Akerlof, identity is a set of social categories (car enthusiast, green activist), a set of prescriptions that go along with those categories, and a set of costs and benefits associated with following or not following these prescriptions. We each choose an identity from the range that society provides ("environmentalist", "conservative", etc). Forcing this choice is the object behind Rao's "hot cause": the lightning-rod issue that polarizes participants into supporters and opponents ("which side are you on, boys, which side are you on?", as the greatest living American sings). Once you have chosen an identity, you must affirm it by following the prescriptions associated with that identity (shopping at independent stores, eating nouvelle cuisine, etc) or you pay the price of dissonance if you take actions that go against those prescriptions (shopping at Wal-Mart, eating classical cuisine). Creating a strong set of such prescriptions is the essence of Rao's "cool mobilizations", which serve to maintain a sense of solidarity and identity among movement members.

One of the more common criticisms of No One Makes You Shop at Wal-Mart was that, although I argued in favour of collective action as a corrective force to free markets, I had little to say about what forms that action should take. It's a fair knock, and I'm happy that I can now point such readers to Rao's book. Not only does he take on several of the issues that I cover (Wal-Mart and big-box stores, biotechnology, real ale) but he takes them much further than I could ever had done, and in a wonderfully specific and constructive way that provides concrete guidance to activists. I take my hat off to him.

Review: The Future of the Internet and How To Stop It by Jonathan Zittrain

The New York Times recently asked: Do We Need a New Internet?:

At Stanford, where the software protocols for original Internet were designed, researchers are creating a system to make it possible to slide a more advanced network quietly underneath today’s Internet. By the end of the summer it will be running on eight campus networks around the country.

The idea is to build a new Internet with improved security and the capabilities to support a new generation of not-yet-invented Internet applications, as well as to do some things the current Internet does poorly — such as supporting mobile users.

The Stanford Clean Slate project won’t by itself solve all the main security issues of the Internet, but it will equip software and hardware designers with a toolkit to make security features a more integral part of the network and ultimately give law enforcement officials more effective ways of tracking criminals through cyberspace.

Ed Felten of Princeton University responds with an orthodox hacker-purist line:

[The first misconception] is the notion that today's security problems are caused by weaknesses in the network itself. In fact, the vast majority of our problems occur on, and are caused by weaknesses in, the endpoint devices: computers, mobile phones, and other widgets that connect to the Net. The problem is not that the Net is broken or malfunctioning, it's that the endpoint devices are misbehaving -- so the best solution is to secure the endpoint devices...

It's an appeal to ye-olde Internet mythologie, complete with deferential references to "the founders" and their foresight, as if the Internet were some real-world Seldonian Foundation.

Neither position is good enough, and Jonathan Zittrain's wise book The Future of the Internet And How to Stop It [home page, Open Library entry] does a great job of explaining why and of providing some better ways of thinking about the problem. Yes, designing security into the Internet will inevitably cripple the very flexibility and permissiveness that has made the Internet a continuing source of unpredictable and surprising innovations. But sentimental idealization of the Original Internet and its "end-to-end" design won't do either [p165].

[U]sers are not well-positioned to painstakingly maintain their machines against attack, leading them to prefer locked down PCs [as Felten appears to advocate, ed], which carry far worse, if different, problems. Those who favor end-to-end principles ... should realize that intentional inaction at the network level may be self-defeating, because consumers may demand locked-down endpoint environments that promise security and stability with minimum user upkeep. This is a problem for the power user and consumer alike.

...When endpoints are locked down, and producers are unable to deliver innovative products directly to users, openness in the middle of the network becomes meaningless. Open highways do not mean freedom when they are so dangerous that one never ventures from the house.

The passage shows the strengths and weaknesses of the book. Zittrain takes an unusually and refreshingly pragmatic, realistic view of the Internet, rejecting old approaches as and when needed. On the other hand, you can see from the first two sentences that his prose can be repetitious and dry. You have to work your way through some dense thickets to get to through this forest - but if you are ready to make the effort, it's a worthwhile journey: one of the best Internet books I've read.

Zittrain's central concern is a dialectical contradiction at the heart the Internet:

1. The Internet's success comes from its remarkable ability to repeatedly generate new and unexpected uses.
2. This "generativity" (yuck, what a clumsy word), comes in turn from the deliberate dumbness of the Internet protocols themselves; they were deliberately designed to permit any kind of traffic, for any purpose, to pass between end points of the network.
3. But the more the Internet becomes "prime time", the more it attracts spammers, virus writers and information thieves to prey on the online population, and the Internet's dumbness is free for them to use as well.
4. The more these highway predators threaten our online experience, the more we seek to retreat to the safety of a closed and protected world.
5. A closed and protected world may give us safety, but will spell the end of the Internet as a fount of innovation.

The Internet, like capitalism, contains the seeds of its own destruction. But Zittrain is a digital reformer, not a revolutionary. Inspired by the continuing success of Wikipedia in the face of similar problems, he favours a combination of light regulation (like health and safety standards for the Net) and popular community action (neighbourhood watch). He believes that these, combined, can preserve the creative spirit that has led to so many innovations, while staving off the worst of the security and other problems. And he makes a solid case, buttressed by broad research (50 pages of notes and references) and a careful, undogmatic and pragmatic attitude.

The book is in three parts. Part I is a history of the Internet, told to highlight two design principles that were present right in the original TCP/IP architecture. The first is the "procrastination principle", which says the "the network itself should not be designed to do anything that can be taken care of by its users" [p31]. Most features of a network should be implemented at its computer endpoints (the end-to-end principle) rather than "in the middle". The second principle is trust; the Internet is "a bucket-brigade partnership in which network neighbors pass along each other's packets", and the assumption of co-operation and fair dealing is present in its design. So the Internet has no built-in security or identification mechanism; anyone can join the network; and there is no quality of service guarantee for packets it delivers. These two principles have led to what Zittrain calls the "generative dilemma" [36]. "The idea of a Net-wide set of ethics has evaporated as the network has become so ubiquitous" [45]. So how do we regain security while maintaining the ability to be creative?

Part II outlines what Zittrain sees as some of the dangers facing the Internet. One danger is the rise of Internet appliances, such as many of today's mobile phones, X-boxes, and Kindles. These devices promise a secure environment, but at the cost of restricting the ability of programmers to be creative. The second is at the other end of the network, where "Web 2.0" platforms such as Google Maps, Facebook, Salesforce and other hosted environments offer "contingent" environments for programming, where the prospect of unilateral changes to terms of service or agreements inhibits creativity.

Zittrain is enthusiastic about much of what the Internet has wrought, but he parts company from current digital orthodoxy on these issues. When it comes to Web 2.0, influential commentators from Shirky to Lessig to Tapscott and even to Benkler gloss over the differences between commercial web sites and non-profits, in an attempt to highlight a unified Internet culture. Lessig, for example, talks optimistically of a hybrid economy in which these motivations muddle along next to each other, and while Benkler does see an opposition of interests between the market economy and the collaborative network economy, he never makes much of it. I haven't seen as clear-minded an analysis of why commercial Web 2.0 platforms threaten creativity as Zittrain provides and I agree with him wholeheartedly.

The mobile Internet is only beginning to get the kind of attention that Web 2.0 has received. Today's smartphones and yesterday's desktop computers are similar in terms of computational power (see this claim of Windows 3.1 running on a Nokia N95 if you don't believe me). But whereas Microsoft got hauled in front of the courts to keep the desktop computing environment open for non-Microsoft applications and opposing the bundling of Internet Explorer, there are no such worries for mobile phone vendors as they keep their proprietary app stores and their managed devices in the name of security. There are layers of the iPhone, the BlackBerry - yes, and Android devices too - that are open only to the operating system, and which third-party applications cannot access.

A natural response to Zittrain's worry that we face an Internet of closed appliances (or, as Margo Seltzer calls them, "gizmos") and closed services is that these can exist in parallel with open, "generative" devices and communications. But Zittrain rejects this particular compromise:

even in a world of locked-down PCs there will remain old-fashioned generative PCs for professional technical audiences to use. But this view is too narrow. We ought to see the possibilities and benefits of PC generativity made available to everyone, including the millions of people who give no thought to future uses when they obtain PCs, and end up delighted at the new uses to which they can put their machines. And without this ready market, those professional developers would have far more obstacles to reaching critical mass with their creations.[165]

The end-to-end principle, argues Zittrain, has had its day, as "'middle' and 'endpoint' are no longer subtle enough to capture the important emerging features of the Internet/PC landscape" [167].  In its stead he advocates a more general principle that seeks explicitly to maintain "generativity", so that ISP filtering of viruses may be worth considering, for example.

Zittrain sees an inspiration in Wikipedia, whose shambolic, after-the-fact, bits-and-pieces way of fixing problems has been one of its strengths. Zittrain gives two initiatives he has been involved with, that could drive a similar approach for security while maintaining generativity. One is herdict, a browser plugin that collects the input of people from around the world to assess web site accessibility. Another is stopbadware.org , which helps to maintain a list of web sites that may host viruses and other badware. These initiatives extend the now commonplace collection of user experience by commercial software providers and service providers by making the collected data universally available. The distinction between, on the one hand, Google's use of our searches and Microsoft's collection of Windows crash reports and, on the other hand, herdict's goal of making the collected data public is a big one. It shines a light both on the closed nature of the major corporate collectors (no matter how unevil they may claim to be) and on the possibilities of open data. Public access to current databases of virus reports, spam outbreaks, and so on provides a basis for innovative solutions to the worst damages inflicted by malware.

Maintaining a healthy digital environment - both security and generative - is a commons problem, and while dictatorship is an appealing route to take to solve the security half of the issue, real transparency provides a second possibility that has more chance of solving both. But it needs to be real transparency, not just the half-hearted efforts of Web 2.0 companies as they dance along the line between community and profit.

There's a lot more in the book. On the technological front Zittrain sees virtualization technologies as reason for optimism and I agree; I'm getting a new computer at work next week and I anticipate running everything in virtual machines. If I get a virus or my registry gets screwed up, I'll roll back to yesterday's state and it will be gone. Well, that's the plan. Let's see how it works. On the social and political side, the book's final pages tend to wander around issues of privacy, reputation, and behavioural norms without saying a whole lot that's new. It's a shame the book finishes on this topic, because it fizzles out a bit. But don't let that deter you - there is a lot of subtle insight and a real breadth of knowledge in these pages that you rarely see in books about the cutting edge of technology, and Zittrain's efforts deserve to generate a big discussion.

Lawrence Lessig's Remix: a rambling review

Remix: Making Art and Commerce Thrive in the Hybrid Economy,  Lawrence Lessig, Penguin 2008.

---

Lawrence Lessig's latest book is about how "a regime of copyright built for a radically different technological age" [xvi: numbers in square brackets are page numbers] is inhibiting art, culture, and individual expression in a digital world. His inspiration, as a parent of two youngsters, is the "copyright wars" and the effect they are having on a generation of children and young adults. Copyright does have an important role to play, he argues, but "Criminalizing an entire generation is too high a price to pay for a copyright system created more than a generation ago" [xviii].

Surprisingly, given the author, the legal portion of the book is a relatively short final section, in which he outlines some common sense and practical suggestions for change in the regulatory environment of the USA. Instead, most of Remix is a description of what Lessig sees as the nascent and booming digital R/W culture that is under threat; of what kind of new activities are appearing and how they work economically. Lessig is mainly concerned with amateur activities like playing, hobbies, and gossip, and with how digital technology has changed them. Unfortunately, the picture he paints is less full of insight than his legal views. He paints a familiar heavily-photoshopped picture of reality that has now become almost orthodoxy in business and technological circles.

Here Comes Everybody

Here Comes Everybody: The Power of Organizing Without Organizations, Clay Shirky, Penguin, 2008.

---

"We are living", says Clay Shirky, "in the middle of a remarkable increase in our ability to share, to cooperate with one another, and to take collective action, all outside the framework of traditional institutions and organizations" [p. 20]. Digital technologies are now part of our social fabric, and all "the phones and computers, the e-mail and instant messages, and the webpages are manifestations of a more fundamental shift. We now have communications tools that are flexible enough to match our social capabilities and we are witnessing the rise of new ways of coordinating action that take advantage of the change" [20].

Here Comes Everybody is an accessible and challenging introduction to these changes - to the many ways every Shem, Shaun and Issy can now share, collaborate, and act together. It combines some great stories with non-technical introductions to some of the key ideas (a little game theory, a little network theory, a few power laws) and is well worth reading -- but it should be read with caution.

The reason for this mixed verdict is the dual nature of the book itself. Here Comes Everybody has two voices. One (let's call him 'Shirky') is a perceptive and creative interpreter of the ways that digital technology is changing society. I like and respect Shirky. He is blunt and provocative enough to cut through the mess of questions that come up when tackling something as far-reaching as the Internet, while being even-handed and reserved enough to respect the complexity of his subject. Here is Shirky on the erosion of journalism and photography:

There is never going to be a moment when we as a society ask ourselves, "Do we want this? Do we want the changes that the new flood of production and access and spread of information is going to bring about?" It has already happened; in many ways, the rise of group-forming networks is best viewed not as an invention but as an event, a thing that has happened in the world that can't be undone. As with the printing press, the loss of professional control will be bad for many of society's core institutions, but it's happening anyway. [p73]

I've struggled with this message for a few days because I don't like its determinism, but he has convinced me. Shirky separates what is happening from what is desirable (not that everything digital is undesirable) and that's an important separation to make if we are to be at all clear-headed.

Shirky is not a techno-inevitabilist in a broad sense. Nuclear power, for example, "is a technology that society can, for the moment, make a decision about" [299]. As with driving a car, we "have a good deal of control over both the route and the speed with which nuclear power progresses, including the option to simply pull over" [299]. But when it comes to digital technologies we are steering a kayak: "We are being pushed rapidly down a route largely determined by the technological environment... Our principal challenge is not to decide where we want to go but rather to stay upright as we go there." [300]

The other voice (let's call him 'Clay') is a techno-enthusiast and an inveterate story-teller. When Clay looks at the Internet he sees no reason for worry - he sees freedom and unlimited potential. It's "the largest increase in expressive capability in the history of the human race" [106], bringing with it a world in which, when "people care enough, they can come together and accomplish things of a scope and longevity that were previously impossible; they can do big things for love." [142] Shirky may warn that there are losses from social changes, but Clay is breezily dismissive: "The spread of cheap and widely available creative tools is sad for people in the advertising business in the same way that movable type was sad for scribes -- the loss from this kind of change is real but limited and is accompanied by a generally beneficial social change." [209] This is the logic of The Lottery - the short story in which one person is stoned so that others can be better off.

I blame Clay and his enthusiasms for the two major flaws of this book.

The first is a fallacy of composition. Clay looks at the Internet and sees lots of groups forming (and things are easy to see on the Internet because even our most casual utterances get stored on someone's servers for posterity to investigate) and he concludes that the world is alight with a new groupiness, the likes of which we have never seen. From time to time Shirky pipes up to remind his alter ego that this is not enough, that "treating the internet as some sort of separate space... was part of the problem" [194]. "The internet augments real-world social life rather than providing an alternative to it" [196]. One implication of Shirky's caution is that, to evaluate the state of groups in our world, we also have to look at how our use of the Internet may have displaced other forms of group building. But while Shirky knows all about Bowling Alone, Clay is too busy running off to tell us a story about Meetup.com to take a really close look at how those positives and negatives are adding up.

While Clay is telling us all about the use of digital technology to spark innovative forms of protest in Belarus, which is a fascinating story, we really need Shirky to ask why, with all these group-forming tools at our disposal and despite the documented disillusionment with the war in Iraq, there is so little coherent protest happening compared to previous wars? Is it really the case that society now is becoming, thanks to the internet, more democratic, more collaborative, and more cooperative than before? I am not convinced. Clay is in danger of making the same mistakes that William Greider made in One World, Ready or Not, and which Paul Krugman demolishes - of finding lots of examples of groups and inferring that the world must have more groups in it than it used to - but that logic is flawed.

Now Shirky is far too well-informed to fall into this kind of trap. Shirky recognizes that, just as removal of a bottleneck at one point in a highway may prompt a new bottleneck to form a few miles down the road, so the "removal of technological limits has exposed a second set of social ones" [91]. But Clay's enthusiasms mean that the book is unbalanced, and this second set of limits never really gets investigated closely. Clay the enthusiast wins out over Shirky the dispassionate observer. If you are going to argue that groups are forming as never before, and if you are going to use minor events like angry airline passengers protesting about being trapped on runways to claim that "Consumers now talk back to businesses and speak out to the general public, and they can do so en masse and in coordinate ways" [179] then you really have to think about consumer activity before digital technology. Here is "talking back to business" with a vengeance:

August [1800] - Notwithstanding that the last day of this month was a Sunday, it was marked as the commencement of a serious riot. A great increase in the price of provisions, more especially of bread, had roused the vindictive spirit of the poorer classes to an almost ungovernable pitch. They began late in the evening, by breaking the windows of a baker in Millstone Lane, and in the morning proceeded, with an increase of numbers and renewed impetuosity, to treat others of the same trade in the same unwelcome manner. Granaries were broken into at the canal wharfs, and it was really distressing to see with what famine-impelled eagerness many a mother bore away corn in her apron to feed her offspring. [link]

Do consumers have a stronger voice now than in the past? I don't know, but I do know that a story or two about American Airlines passengers is not going to convince me. If Clay wants to tell us about a student group using Facebook to protest about British Bank HSBC's cancellation of interest-free loans then perhaps he should think about the longstanding student boycott of Barclays Bank during the 1970's and 1980's that contributed to the end of apartheid. But he doesn't, and that's disappointing.

The second flaw - also a common one - is a reluctance to follow the money. Between them, Clay and Shirky convince me that the Internet marks a change; that a society with an Internet is different from a society without one. Also, they convince me that the Internet is not a single model of sharing/collaboration/collective action; it's many models. So let's talk about these models, about which ones have legitimacy and longevity and which don't. To do so requires Shirky to take the broad brush that Clay is using and to start to make smaller, more detailed points.

Perhaps we no longer need books telling us that the Internet is a big thing. It is time to treat that fact, as Shirky sometimes does, as the starting point for a discussion rather than the conclusion. The questions then become ones of what kind of structures will form and persist in the online world, and if you are going to talk about these questions then you have to address the economics of the problem.

Money matters.

People's willingness to contribute to Wikipedia, Shirky points out, is tied to the non-profit status of Wikipedia. A threat from Spanish participants to start an alternative version convinced founder Jimmy Wales to "formally forgo any future commerical plans for Wikipedia, and to move the site from Wikipedia.com to Wikipedia.org, in keeping with its nonprofit status. Similarly, he decided to adopt the GNU Free Documentation License for Wikipedia's content [which] assured contributors that their contributions would remain frerely available" [274]. There is an ongoing tension between contributors to sites and the "owners" of those sites that is visible on MySpace (see Billy Bragg's efforts to gain musicians rights over their content), on Facebook (the failed "Beacon" program as one example) and even Google and the line it walks with advertising content on its search results. Contributor-driven movie database IMDB was taken over by Amazon, but Amazon does not put its name on the pages of the site - does it fear that people would not contribute so freely if they realized they were just making Jeff Bezos even richer? The book ends with a story by Clay about a successful revolt by participants at digg.com - but once again, a story is not enough to support the thesis he is trying to make. 

There are many issues of democracy, ownership, privacy and control that, depending on how they are resolved, will influence our chances of staying upright as we get carried along by this technological flood. Clay Shirky shows in this book that he has many insights that we can use, but to really help steer this kayak we need more Shirky and less Clay.

The Anarchist in the Library

The Anarchist in the Library, Siva Vaidhyanathan, Basic Books 2004.

---

Siva Vaidhyanathan (see here and here) sees the central problem of the Internet and of society as a whole, as the tension between decentralization/freedom/anarchy and centralization/control/oligarchy.

The great challenge in the new century is to mediate between two divergent trends -- anarchy and oligarchy. In the war between distribution and concentration of information, the issues and conflicts seem intractable. [xvii]

In the world of the Internet, at least in the years leading up to 2004 when the book was published, these poles of anarchy and oligarchy manifest themselves technologically. Anarchy is the "ideology of peer-to-peer systems" such as the file-sharing and media-sharing networks that have followed Napster; it is characterized by a fluid, decentralized architecture in which "all the 'thinking'.. happens at the end point" and in which there is "no discernible command-and-control system" [17]. On the other side, digital rights management is the technology of oligarchy, imposing controls on what you can and cannot do with the software and media that you buy (or, increasingly, license). I don't usually buy the technology-determines-behaviour line, but his discussion of what it means for a technology to "have an ideology" is the best I've read, and is well illustrated by the distinction between technologies that operate on the basis of protocols (handshakes, conventions) and those that work on the basis of controls.

When I first picked up this book (from my library) I assumed that the author was another techno-utopian and that he is firmly on the side of freedom and anarchy over control and oligarchy (as who but the most unromantic of us could not be)? Not unreasonable, given chapter titles like Hacking the Currency and The Peer-to-Peer Revolution and the Future of Music, but wrong. In fact, although his heart is with the anarchists, Siva Vaidhyanathan is best described as a librarian. Not as in somebody who works in a library, but as in someone who is a fan of libraries:

Librarians should be our heroes. The library is not just functionally important to communities all over the world; it embodies Enlightenment values in the best sense. A library is a temple devoted to the antielitist notion that knowledge should be cheap if not free -- doors should be open. Supporting libraries -- monetarily, spiritually, intellectually, and legally -- is one of the best things we can do for the life we hope to build for the rest of the century. [Page 119]

Libraries seem anachronistic -- if they did not already exist, they could not be created now (in North America anyway) -- and yet they are among the most popular of institutions. People make jokes (deserved or not) about the service of the post office, the inflexibility of government, and the so on, but no one (well, apart from Seinfeld) pokes fun at libraries. Even some libertarians, who see any path laid by the state as a road to serfdom, love libraries. Our local libraries are always busy, and after a bad patch some time ago appear to be thriving with their mix of Internet access, DVD and CD rentals and, of course, loads of books. The popularity of the library surely comes from its nature as patron-focused but not commercial (no "consumers" here) and state sponsored but not monopolistic. It is populist and yet highbrow.

In the end Vaidhyanathan takes a dialectical stance and rejects both anarchy and oligarchy in favour of the library - a civic, noncommercial, open and public model. I didn't see this coming until well into the book, which after the first couple of chapters is more a set of essays around a common topic than a sequential argument. I thought he was heading towards one more simplistic agenda ("information wants to be free" anyone?) but he pulls himself back from the brink and ends up writing [185] "The heart of my argument in this book is a call for modesty and patience" and this:

The urge to break heads, to do the bidding of oligarchy by any means necessary is intimately linked to specters of anarchy. The urge towards anarchy depends on oligarchic abuses. Each creates the conditions that allow the other to thrive. The question for us in the twenty--first century should not be choosing anarchy or oligarchy by constructing and maintaining systems that discourage both. Anarchy is a reaction, not a vision or solution that can produce the best society and the best human future.  [187]

One of the reasons he pulls back is a debate with Randy Cohen of the New York Times, which he describes with admirable honesty [63].

"The history of popular culture is a continuous struggle on the artists' part not to get robbed... it seems to me that what MP3 [digital music] does is democratize the ability to rip off an artist," Cohen wrote to me. "And what's particularly galling is that you not only want to do it, you want to be praised as a social progressive when you do."
He got me. That's my schtick. By the guiding principles Cohen deployed in our peer-to-peer debate, I had no escape. He considered copyright to be an artists' right and concern; I consider the chief player in the copyright system to be the corporation.

The artists' struggle continues. The commercial oligarchs do whatever they can to avoid paying artists, as the Hollywood writers' strike showed and as recent press about royalty awards not being passed on to artists makes clear. But the free-use techno-anarchists exploit artists too. Greatest Living Englishman Billy Bragg writes in the New York Times about how he advised Michael Birch, the founder of social networking site Bebo.com, on how to handle artistic content on the site (via Nicholas Carr). Bragg was angry that while artists contributed to the site, Michael Birch sold it to AOL for $850million and the artists got nothing. Here is part of his op-ed:

He was hoping to expand his business by hosting music and wanted my advice on how to construct an artist-centered environment where musicians could post original songs without fear of losing control over their work. Following our talks, Mr. Birch told the press that he wanted Bebo to be a site that worked for artists and held their interests first and foremost.

In our discussions, we largely ignored the elephant in the room: the issue of whether he ought to consider paying some kind of royalties to the artists. After all, wasn’t he using their music to draw members — and advertising — to his business? Social-networking sites like Bebo argue that they have no money to distribute — their value is their membership. Well, last week Michael Birch realized the value of his membership. I’m sure he’ll be rewarding those technicians and accountants who helped him achieve this success. Perhaps he should also consider the contribution of his artists.

The musicians who posted their work on Bebo.com are no different from investors in a start-up enterprise. Their investment is the content provided for free while the site has no liquid assets. Now that the business has reaped huge benefits, surely they deserve a dividend...

If young musicians are to have a chance of enjoying a fruitful career, then we need to establish the principle of artists’ rights throughout the Internet — and we need to do it now.

Instead of cheering for one side or the other, Vidhyanathan's book foresees and acknowledges both these problems. And problems they are. He does not offer clear or simple solutions, but he points us thoughtfully in the right direction, and that's a very valuable contribution. So I definitely recommend the book, even though I wish it were a bit less scattered than it is (and that it had a better index). It's an exploration, not a recipe for the future, and has the rough edges that come with that territory, but there is a lot of food for thought in the pages.

The Anarchist in the Library, written in 2004, is already a bit outdated. Not the author's fault of course, but it is already odd to read a book about the Internet that has no mention of Wikipedia and no index entry for Google. These recent developments have changed the nature of the Internet. Peer to peer networks are still around, but I don't think they are the defining feature of the Internet. If we think of music we think of iTunes, not Napster; if we think of books we think of Amazon; if we think of social networks we think of Facebook and MySpace. None of these are peer-to-peer: they are centralized technologies built on the basis of controls, not protocols. Software architecture is often described as a stack, and while the low-level plumbing of the Internet remains a peer to peer protocol, the higher levels of the stack are "platforms" or client-server models of request and response. The techno-capitalist digirati have moved happily onto such platforms, and the centralization of ownership that they carry with them. In the era of utility computing, peer-to-peer networks appear to be on the wane.

There is one other place in which I think Vaidhyanathan gets it wrong. He underestimates the role of information asymmetries and transaction costs. He says that "Major record labels perform four basic tasks: production, distribution, price fixing, and gatekeeping" [48]. But there is a fifth, which is promotion. Simply putting a record on the Internet is hardly more effective than playing your music on your front lawn - the problems of finding something no one knows exists are far greater than he credits. One major function of libraries, after all, is to match readers and books ("Every reader his/her book" in one of Ranganathan's five laws of library science -- thanks John). When Vaidhyanathan tells a journalist that African musicians don't need record companies because "The artists can do it all themselves for less than $10,000" he is naive. Billy Bragg's friends on Bebo did the same, and it got them nowhere. Others have proclaimed this line before - for example Chris Anderson in my least favourite book talks about the band Birdmonster who eschewed labels

Label were calling with deals, but Birdmonster turned the offers down. As [lead singer Peter] Arcuni put it, “We’re not anti-label in principle, but the numbers (risk vs. reward) didn’t add up.”

A music label exists primarily to fulfill four functions: 1) talent scouting; 2) financing (the advances bands get to pay for their studio time is like seed capital invested by a venture capitalist); 3) distribution; 4) marketing.

From Birdmonster’s perspective, they didn’t need that.

Well, apparently they do now. The problem is that so much of culture is governed by asymmetric information, information cascades, and network effects. You don't know what a book is going to be like until you buy it, so simply knowledge of the existence of a book is insufficient - you need recommendations and reliable ones at that. I've gone on about this in various ways here and here and here. His neglect of these forces is one of the reasons I thought he was heading down the techno-utopian path, but as I say he ends up, thankfully, rejecting the "California Ideology" [155] in favour of something less catchy, less simple, but more hopeful.

The Big Switch

The Big Switch, by Nicholas Carr, is published by W.W.Norton, January 2008. Quotes and page numbers are from an advance copy.

Unlike most technology commentators Nicholas Carr knows that if you want to predict what's happening next, you've got to follow the money. And he does so very well, which makes this book (and his weblog) recommended reading for anyone interested in where  technology is taking us.

---

Google is everywhere in The Big Switch and the reason is simple: cost.

No corporate computing system, not even the ones operated by very large businesses, can match the efficiency, speed and flexibility of Google's system. One analyst [Martin Reynolds of the Gartner Group: see here ] estimates that Google can carry out a computing task for one tenth of what it would cost a typical company.

That means, if you are a company and you have a computing task to be done that Google already does, you can save a bunch of money and you can now start outsource your CPU cycles just as you previously outsourced other tasks. And that means that the computing landscape will get shaken up. Not in a matter of months, but over the next decade or so. It's amazing how quickly we get used to a landscape and many of us are now so accustomed to PC's and the basic layout of corporate computing systems that they seem almost natural. But Carr warns us that this is going to change and, as if to confirm his claims, last week Sun Microsystems, supplier of many of the computers that make up corporate data centres, announced that by 2015 it won't have a single data centre.  Information Technology is not sacrosanct.

Google's cost advantage comes partly from a built-in inefficiency of corporate computing: capacity underutilization. Many applications demand their own servers, and those servers must be able to handle the peak load that the application will experience even if that peak load happens only rarely. As a result most corporate computers, most of the time, do nothing except consume electricity and produce heat. This inefficiency was unavoidable until recently, but now high-speed Internet availability makes it possible for companies that have the resources (Google and a few others) to build warehouses full of servers that look like power stations (see Google's The Dalles centre in Oregon, below, with two football-stadium-sized buildings full of perhaps 60,000 servers). And then they can supply CPU cycles over the Internet just like electrical utilities supply electricity. The demand on Google's CPU cycles is smoothed out, being balanced among many consumers in different timezones with different needs, and that only helps their efficiency. It's what Carr calls utility computing.

The first half of The Big Switch is given over to convincing us that utility computing is the wave of the future, and the second half of the book explores the implications - many of them disturbing - of this switch. The good news is that both are thought-provoking and open up a lot of questions. The less good news is that to cover all this ground Carr has to skim and, at 250 pages, this short book can't delve very deeply into any of the questions.

The argument for the switch to utility computing is made by drawing an analogy between the history of electricity supply and the history of computing. A century ago factories generated their own electricity and many of the fears that people have of outsourcing computing were faced by the nascent electrical industry. Would companies trust their lifeblood to an external source? Can they the get guarantees they need to go over to this new model? History shows that they did, and quickly. Will computing go the same way? Carr says yes.

I broadly agree (in the long run) but there are reasons to think the Big Switch may be less than complete. First, there is an intermediate solutuion that companies are already adopting, which is "virtualization software" that allows them to run many "virtual computers" on a smaller number of real computers, so making better use of the real CPU cycles. The technology is booming and the savings are huge. And while this may still be more expensive than the full-fledged utility model, to the extent that many software packages are customized for individual companies (not so much word processors and so on as the ERP systems that many companies run their business on) a utility model may not be applicable. Whether the benefits of custom applications will win out over the cheapness of commodity software is open to debate. Second, while electricity is a relatively simple thing, computing is complex. Carr only scratches the surface of what forms utility computing will take. Will companies and consumers pay bills for complete applications (the salesforce.com model), for storage (Amazon's S3, for example), for virtual computers on which they can install their own applications (now also starting to be offered by Amazon), or what? And third, from the consumer side, the flip side of the unused CPU cycles efficiency argument is that we may be prepared to pay for a computer to run the latest games and then, well, the marginal cost of using those CPU cycles is very low.  In fact, they are starting to be used (Seti @home and others) by central applications - rented back (or donated back) to others. It is clear that the utility model is making ground in the consumer space (how much of our time do we spend in the browser) but the reasons are, I think, different from the electricity analogy Carr pursues.

But this is splitting hairs, because the broad trend Carr identifies is surely largely correct, even if it takes a decade or two for the switch to be made. We'll still have a lot of computing cycles happening everywhere, but perhaps it may turn out to be true that the world really does need only five computers. And if there are limitations to the expansion of the megacomputers because of the complexity of computing as a utility, well there are other limits to growth (national ones, legislative ones) that electrical utilities have always faced that don't constrain Google, Amazon and our other suppliers.

So that's part one. Part two is a welcome counterweight to the techno-utopian fluff put out by some prominent commentators. Carr covers the centralization of control (how mass participation ends up with just a few people getting loads o' cash), what he calls "the great unbundling" - the change from buying whole albums/newspapers to viewing individual stories and how that may affect production - and the darker side of the web: spam, identity theft, loss of privacy, and so on. Of these, the best chapter is on unbundling because it's just not clear to me how or why that phenomenon will work itself out. The argument is that (to take newspapers as an example) when advertizers sell by the click and when content cannot be sold for its own sake, newspapers lose the ability to fund such expensive endeavours as investigative reporting and foreign news desks. Cross-subsidization of different parts of a newspaper, Carr argues, is the only way that quality content has managed to keep being produced, and once that model goes then so does the quality.

There's a lot of possibile futures here, and I wish Carr's book was twice as long so he could explore some of them more seriously. In the end, the book is a survey more than a deep investigation, but it is a survey that asks the right questions and we could use more books like this to chart, and perhaps help alter the course, of technological change and its many social impacts.

Independent Bookstores in the UK

I didn't mean to write about things like book sales and so on, but one thing leads to another, so here I am.

The Guardian has an article about the shape of the UK book market and the surprising health of independent bookstores. Here is a graph based on numbers taken from the article (and taken, in turn, from the consumer research group Books Marketing). Click it to see it bigger.

It shows that from 2003 to 2006 the big winners are the supermarkets and the online retailers - the twin jaws of the digital vice. No surprises there.

It also shows that the big losers are the chain bookstores (Waterstones in particular) and the direct mail book clubs. Again, no surprises.

The green line is the independent bookstores.  They are still bigger (for a few years anyway) than online retailers and have actually improved their share from 15.6 to 15.9% of the market.

There are some other statistical nuggets in the article. See if you can make sense of these:

Figures vary for the growth of book sales in Britain, but each source points to at least small but steady growth. The Publishers Association says that 459m books were sold in the UK in 2005, slightly fewer than in 2004, but ahead of the three previous years. Nielsen Bookscan, which only counts books sold at retail level (not including sales to schools for example), says that 225m books were sold last year in Britain with a value of £1.7bn. Nielsen draws a straight line of gains from 2001 when its records show 163m books sold with a value of £1.2bn. Waterstone's reckons the market is growing at a more sedate 2% a year...
If not exactly thriving, the independent book store is not in as dire straits as many fear. According to the Booksellers Association, a trade group for retailers, the number of independents has fallen from 1,700 in 2000 to 1,400 today. But that figure now appears to have stabilised. There are an increasing number of small publishers targeting the independents...
"Publishers are falling over backwards to sell to supermarkets at very large discounts," he says. "The focus on celebrity memoirs and potential bestsellers is relentless and the proportion of book sales by authors in the top 50 is going up and up. The middle band of authors is finding life increasingly tough. The spread has diminished and that trend will continue in the short term at least as competition gets more intense. It is having a substantial impact on what is published."