Whimsley

Predicting hits may be like predicting the weather

In the New York Times sociologist David Card physicist turned sociologist Duncan Watts writes about how cultural hits may be, like the weather, impossible to predict. It comes down to how much we like stuff because it's good and how much we like it because other people like it. If it's the latter, then it becomes impossible to predict at some point. What's nice is that they did some experiments to demonstrate some of this in the lab too - you'll have to click through to see it.

A side effect of this would be that there is a strong limit to the effectiveness of recommendation schemes from Amazon and Netflix and so on. Maybe our preferences are just too damn quirky to be captured, no matter how fancy the algorithm.

Here's a few paragraphs from the article.

Conventional marketing wisdom holds that predicting success in cultural markets is mostly a matter of anticipating the preferences of the millions of individual people who participate in them. From this common-sense observation, it follows that if the experts could only figure out what it was about, say, the music, songwriting and packaging of Norah Jones that appealed to so many fans, they ought to be able to replicate it at will. And indeed that’s pretty much what they try to do. That they fail so frequently implies either that they aren’t studying their own successes carefully enough or that they are not paying sufficiently close attention to the changing preferences of their audience.

The common-sense view, however, makes a big assumption: that when people make decisions about what they like, they do so independently of one another. But people almost never make decisions independently — in part because the world abounds with so many choices that we have little hope of ever finding what we want on our own; in part because we are never really sure what we want anyway; and in part because what we often want is not so much to experience the “best” of everything as it is to experience the same things as other people and thereby also experience the benefits of sharing.

There’s nothing wrong with these tendencies. Ultimately, we’re all social beings, and without one another to rely on, life would be not only intolerable but meaningless. Yet our mutual dependence has unexpected consequences, one of which is that if people do not make decisions independently — if even in part they like things because other people like them — then predicting hits is not only difficult but actually impossible, no matter how much you know about individual tastes.

The reason is that when people tend to like what other people like, differences in popularity are subject to what is called “cumulative advantage,” or the “rich get richer” effect. This means that if one object happens to be slightly more popular than another at just the right point, it will tend to become more popular still. As a result, even tiny, random fluctuations can blow up, generating potentially enormous long-run differences among even indistinguishable competitors — a phenomenon that is similar in some ways to the famous “butterfly effect” from chaos theory. Thus, if history were to be somehow rerun many times, seemingly identical universes with the same set of competitors and the same overall market tastes would quickly generate different winners: Madonna would have been popular in this world, but in some other version of history, she would be a nobody, and someone we have never heard of would be in her place.

Why Europeans Work Less Than Americans

Steven Landsburg in Forbes Magazine misinterprets what a union is (or could be) -- seeing it as a source of coercive rules rather than as a mechanism for coordinating actions, but does have some interesting things to say about how many hours people work in different countries, and why there is a pressure to work a similar amount to everyone else.

Compared to Europeans, Americans are more likely to be employed and more likely to work longer hours--employed Americans put in about three hours more per week than employed Frenchmen. Most importantly, Americans take fewer (and shorter) vacations. The average American takes off less than six weeks a year; the average Frenchman almost twelve. The world champion vacationers are the Swedes, at 16 and a half weeks per year.

This raises more than one interesting question. First, why do Americans choose to work so much? (Or, if you prefer, why do Europeans choose to work so little?) Second, who's happier?

... One trio of economists (Ed Glaeser of Harvard, Bruce Sacerdote of Dartmouth and Jose Scheinkman of Princeton) offers this explanation: When your wages are cut 20%, you take more vacations. But when your friends' wages are also cut 20%, you'll take even  more vacations, because vacations are more fun when you've got friends to share them with. So a 20% across-the-board tax hike, which affects both you and your friends, yields a more dramatic response than a 20% cut in your own wages.

An overlapping trio (Glaeser, Sacerdote, and Harvard's Albert Alesina) give some credence to this "social multiplier" theory...but claim to have identified the primary culprit in European labor regulations. While 20% of American workers are unionized, more than 80% of workers are unionized in France and Sweden. And while American unions have spent the past few decades fighting for higher wages, European unions have fought for shorter hours.

Is that a good thing or a bad thing? According to traditional economic theory, workers are best off when they can decide for themselves how much to work. Some workers might choose long hours and big paychecks; others might choose less income and long vacations.

Union-inspired regulations, by restricting individual choice, can only make those workers worse off. But here's what traditional theory overlooks: If you live in Europe, then union regulations not only force you to work less; they also force your  friends to work less--which can be a blessing if you're trying to coordinate a camping trip.

So at least in principle, coercive union rules could be good for everyone. So, for that matter, could high marginal tax rates, and for exactly the same reason: The more your friends are taxed, the easier it is to coax them into taking time off to share your vacation time.

...If European and American paths continue to diverge, we'll soon learn a lot more about whether well-rested Frenchmen are happier than wealthy Americans.

Steven Landsburg is an economics professor at the University of Rochester and the author of The Armchair Economist  and Fair Play.

(Link: Why Europeans Work Less Than Americans - Forbes.com.

Chain reaction: UK Bookstores

In a story that will have a familiar ring to it for anyone from Canada, the Grauniad reports on HMV (aka Waterstone's) being given the go ahead to take over Ottakars: another example of how free markets fail to preserve diversity once economies of scale kick in.

Link: Chain reaction from Guardian Unlimited: Culture Vulture.

"Waterstone's choose about 5,000 books a year and promote them so that they sell tremendously - at the expense of other books," he said. "If a book isn't taken up within a month, it is replaced. Ottakar's, on the other hand, gives books more time to take off. There are two categories of books - the tortoises and the hares. If this deal goes ahead, we will end up with all hares and no tortoises." And it's not just authors who are apprehensive; figures from across the industry have voiced their concerns. Just last week I spoke to a representative from the Booksellers' Association, who told me that in his opinion, an Ottakar's takeover would have a devastating effect on publishers' chances of introducing debut novelists - or even the lesser-known works of popular authors - to new readers.

And the Ottakar's story is just the visible face of what is happening to independent bookshops across the UK. Although Ottakar's is a chain, it shares many of the strengths and weaknesses of the independents. Its branches are locally run by staff who are sensitive to the needs of their local communities and choose their stock accordingly (in contrast with Waterstone's, which nowadays operates a policy of centralised buying), but it has neither the revenue nor the advertising clout to compete on a pricing basis with the bigger shops. As a result, Ottakar's have seen their sales and profits falling, making a buyout ever more likely.

FootBinding: Search for the Three Inch Golden Lotus

So the other day LS was watching FootBinding: Search for the Three Inch Golden Lotus.  Part of the show was about whether footbinding was really oppressive, given that the choice to bind was usually made by mother, who had herself been bound.  In particular, "Columbia University Professor Dorothy Ko mounts her show Every Step a Lotus and contends that footbinding was not the tragedy modern thinkers make it out to be"

In Dorothy Ko's book of the show "she contends that footbinding was a reasonable course of action for a woman who lived in a Confucian culture that placed the highest moral value on domesticity, motherhood, and handwork."

There is no contradiction between reasonable courses of action and tragedy. It is quite possible for people to make choices and still be oppressed: what matters is the system of incentives. In this case, the origin of the exploitation is in the value given to bound feet (by a patriarchal culture), not in any coercion to bind: once the right incentives are set, coercion is no longer needed. Given that a woman with bound feet would have a better chance at a stable or prosperous future than one without, it made sense for women to choose to bind their daughters' feet. The cost to the decision was high (and higher, of course, for those of lower status who had no others to do work for them, and for whom the binding process started later, if at all), but not as high as the cost of failing to secure a good marriage.

This idea -- that there can be no exploitation as long as people are given choices -- is present today in a lot of free-market arguments. Why not give children the opportunity to work in sweatshops? It gives them a chance to be better off than they would otherwise be. Or, as Gary Becker argued recently, why not let people sell their internal organs for transplants?

Footbinding is a compellingn example of how oppression is perfectly compatible with individual choice (albeit made by mothers, but with their daughters' best interests at heart).

India creates database to halt theft of lore

A fine initiative reported in today's  Globe and Mail

For thousands of years, Indian villagers have used an extract from seeds of the neem tree as an insecticide. So when a U.S. company patented a process for producing the substance in 1994, India reacted with outrage.

After spending millions of dollars in legal fees to successfully overturn the patent, India's government is now creating a 30-million-page database of traditional knowledge to fend off entrepreneurs trying to patent the country's ancient lore.

India is not alone in worrying about "bio-prospectors" profiting from the genetic resources of its plant life with no benefit to its people.

It joined with China, Brazil and nine other countries a few years ago to begin pushing for international protections. 

The database project already has caught the interest of others. A South African team recently visited and a Mongolian mission is coming in January, said V.K. Gupta, chairman of India's National Institute for Science Communication and Information Resources.

The database, called the Traditional Knowledge Data Library (TKDL), will make information available to patent offices around the world to ensure that traditional remedies are not presented as new discoveries.

"If societies have been using it for centuries why should it be patented?" said Shiv Basant, a senior official at the Health Ministry's Department of Ayurveda, Yoga, Unani, Siddha and Homeopathy, India's traditional health and medical disciplines.

The government has also successfully challenged patents on the use of the spice turmeric to heal wounds and rashes and a patent on a rice strain derived from India's famed Basmati rice.

But that is a tiny fraction of the problem.

A 2003 study by Mr. Gupta's institute estimated that about 7,000 patents worldwide are based on indigenous Indian knowledge, far too many for India to challenge in expensive legal fights.

Officials hope the database will head off future battles.

"If we have all the data in TKDL, we will not have to spend all those millions of dollars," said Ajay Dua of the Commerce Ministry's Department of Industrial Policy and Promotion.

Currently it is difficult for overseas patent office researchers to prove purported innovations are really based on old lore because, while the information is widely published in India, it is often in ancient languages such as Sanskrit or modern regional languages like Tamil.

"We decided we have to break the language and access barrier," Mr. Gupta said.

He convened a group of 150 experts in traditional medicine, scientists, doctors, patent lawyers and computer programmers to put together the database of traditional knowledge.

Instead of laboriously translating the manuscripts, the scholars structured the texts into classifications widely used by patent examiners.

The texts are then entered in the database, where specially developed software translates them into Hindi, English, German, French, Japanese and Spanish.

"We created knowledge conversion software that converts local names of diseases and plants into modern names," Mr. Gupta said.

More than 1,500 yoga poses have been catalogued, too.

That's because yoga poses also have been patented, often by Indians who are living abroad, Mr. Basant said.

A patent researcher can search the database using key words or phrases.

So if the plant aloe vera is entered, the traditional term Kumari will come up with a list of its known medicinal uses.

More than 10 million pages already have been loaded into the system and 20 million more will be available by the end of 2006, Mr. Gupta said.

Several international patent offices have applied for access to the database and it will be made available to them as soon as the group finishes establishing technological and legal safeguards to prevent the knowledge from being wrongly exploited, he said.

Vandana Shiva has written a lot about Biopiracy, and this story is a next chapter to some things in her book of the same name. I have not followed the Indian and Pakistani governments' activities in challenging patents on long-known non-innovations, such as Neem fungicides, Basmati rice and Nap Hal wheat. The use of the patent system as a way of enclosing the intellectual commons (as Shiva describes it) illustrates just how distinct the notions of competitive markets and capitalist markets (which are commonly conflated) really are.

Winner Take All in Books

Two depressing things for any fiction writers out there.

The wonderful Miss Snark (literary agent) More more more! describes how:

publishers demand that writers sell increasing quantities of subsequent books they publish. Book one can sell five thousand copies, book two six thousand but by book five you've got to sell forty thousand. If you don't, the publisher doesn't renew the contract. It's very similar to get promoted or get out in the military. If you don't make a certain rank by a certain age, they ask you to retire.

The reason is increasing returns (in the form of fixed costs) in the book publishing industry:

The reason publishers want zillion copy sellers from one writer instead of ten writers each selling one-tenth of a zillion is because of unit cost.

Unit cost is the cost of each book sold. Add up all the cost for printing, editorial time, design time, and a percentage of the fixed cost like heats light and water and voila and voila: what it costs to make a book happen. That cost is almost same if you sell 3000 books or 30000 books.

The end result of increasing returns is a winner-take-all market. It is the search for the next blockbuster rather than the building of a set of steady sales. It is wealthies woman in Britain if you write Harry Potter and zilch otherwise. And it is not healthy for book readers.

The point is confirmed by a piece in the December edition of Quill & Quire, in which agent Denise Bukowski argues that "There has been a sea change in the fiction marketplace" in which life is getting tougher for most mid-level and even good-selling authors.

Fact: Foreign rights to American novelist Edward P. Jones's bestseller The Known World, which last year won the Pulitzer Prize and the National Book Critics Circle Award, were not sold until after it had won those awards.

Fact: U.K. star Andrea Levy;s Small Island, which not only won the Orange Prize but also was recently declard "The Best of the Best" of the first 10 years of the prize, was turned down all over New York. Picador finally picked it up for a pittance as a paperback original, after it won the prize.

The reason is partly the big chain structure of today's book industry.

And in the U.K. one bookselling chain rules the market, and one man does all the buying for it: he has refused publishers' sales reps access to his stores. Consquentuly, many boks are available in the U.K. over the Inernet.

I presume she is referring to Waterstone's. Is there a chance that online book sales will counter this trend to fewer bigger books? Let's hope so.

A new eugenics?

Interesting letter by the always-worth-reading Margaret Somerville in the Globe and Mail this morning, following an article by the rarely-worth-reading Mararet Wente on abortion and disability. She says:

Re Margaret Wente's Disabled Kids Are The Abortion Debate No One Wants To Have (Nov. 17): Whatever our view of the ethics at the individual level of parents-to-be screening their embryos and fetuses for genetic and developmental abnormalities and discarding those that are genetically or developmentally "defective," these decisions at a societal level will have the effect of wiping out certain groups of people, such as those with Down syndrome or achondroplasia (dwarfism), or who are profoundly deaf or manic-depressive.

And as genetic knowledge expands, other groups are likely to be reduced. Female embryos and fetuses are already being eliminated, and some fear a similar fate if the genes for homosexuality are identified. The cumulative effect of individual decision-making is wiping out certain groups, a situation that would never be tolerated as public policy.

Are we creating a "new eugenics" -- deliberately eliminating genetically undesirable people from society? Those favouring screening finesse this question by arguing that an individual's choice regarding the nature of their child is not a eugenic decision and comes with their "absolute right to reproductive autonomy". They say eugenics is practised only when a choice is made in relation to a group or class or by someone who's not the future parent. But is that sophistry?

And where the government supports screening programs to avoid, as a recent article in the medical journal The Lancet euphemistically puts it, "live-born children with preventable physical or metnal handicaps," is this really the adoption of public policy of eugenics?

I wouldn't pretend to know how to balance this particular tension between individual reproductive rights and the collective consequences of individual choice, but I can't imagine there is a clean solution to this messy problem. Somerville has written on this stuff a lot (see The Ethical Canary, for example) and Thomas Schelling (Choice and Consequence) has some insights into this, as into so much else.