Mr. Amazon's Bookshop: Recommending the Big Sellers

[This is the eleventh episode of Mr. Amazon's Bookshop. A list of all episodes is here; the previous episode is here. In the previous episode Kylie began to think that Mr. Amazon might be "just like the bastard publishers", pushing big sellers at customers rather than promoting books like her own novel The Adventures of Wazzock. Meanwhile, Whimsley was becoming increasingly befuddled by the swirl of activity around him. His befuddlement even interfered with his sleep....]

I had trouble sleeping that night. It wasn't just the heat mixing with the cat-urine-induced damp of the carpets to produce the acidic tang that so characterizes Whimsley Hall in the summer; I find that odour reminds me of my own childhood and is surprisingly comforting. No, Kylie's virulence had quite upset me and my usual tonics did not seem to relax me as they usually do. But finally I slept fitfully, only to dream...

I looked out of my bedroom window to see a book lying in the middle of the vegetable garden. As I watched, the book opened and a vine grew from its spine, each page becoming a leaf. The vine sprouted pods periodically along its length, and each one erupted to vomit a new book onto the ground. These new books sprouted vines in turn, and the vegetable garden was soon a mass of twisting green creepers, writhing ever closer to the walls of Whimsley Hall. Before I knew what was happening, they were appearing at the window, and each leaf bore the face of Mr. Amazon. "If you like vegetables", said one, "maybe you would be interested in this," and the leaf rotted on the stalk, emitting a stench of rotten turnip. "If you like books", said another "maybe you would like a phonograph, or some toys, or some gourmet groceries", and an avalanche of bread, mechanical devices, soup, toy soldiers, and other contrivances flooded the floor of my bedroom. "It's a new economy of abundance!" crowed a face to my left; "It's a world of choice!" cheered one to my right. More and more, faster and faster, the faces wriggled and wormed their way over the windowsill, in through the ceiling, up through the floor, until I could hardly move

I woke suddenly, tangled in the bedclothes, sweating and shaking. The room was empty, the window closed, the night quiet. I squirmed out from the sheets, ran to the window and stared out, but everything was peaceful. Reassured but exhausted, I put my nightcap back on my head and returned to my bed, and drifted back to slumber. But just as I was about to lose consciousness, I thought I heard Mr. Amazon's voice again. "If you want a vision of the future, imagine a human trying on a new boot - forever."

The following day dawned grey and wet. The rain battered the windows, and I felt a mixture of relief and disappointment as I realized Kylie was unlikely to venture all the way from the village to Whimsley Hall today, but before I had even started in on my kippers there was a knock at the window.

"What're you doing Mr. W? Me and the imbecile had been busy for hours. Come and have a look."

I ran to the door, and sure enough, steam was billowing from the stable chimney. The differ was running. I stood straight, called for an umbrella, and set out to join the urchins. Jennie the one-legged housekeeper carried the umbrella for me so that I could finish my kippers and tea on the way over. The umbrella kept knocking against my head as she bounced up and down. It really is most inconvenient to keep her on sometimes, but one does what one can.

"So tell me," I demanded as I strode - impressively I'm sure - into the stables.  "What have you been doing with my differ?"

"Well," started Kylie, "the undersized lamebrain here was watching you when you set up your dials to send the questions to Mr. Amazon. So we worked out how to send our own questions. And we chose another book, and we grew a shop from it. We did the same as you did: choose one book as the seed; visit about a thousand times. Each visit, we look at summat between 0 and 20 books, and the shortarse wrote down all the output." She pointed at Edmund, asleep in the corner. "This time we chose something called The Shack by William Paul Young. It's a one of a kind invitation to journey to the very heart of God, and a lot of people seem to like it. So we've got the whole bookshop down in your notebook. You'll be needing some new cats by the way." She gestured to a damp pile of fur on the floor opposite the dials.

"And what did you find?"

"In short, a lot. Here, why don't I pour you a drink and you can take a seat while I tell you all about it." She poured what looked like a stiff gin into a cup and handed it over.

I took the drink with thanks, and sat attentively.

"First things first. In the Shack shop, we spent half the time looking at books in the top 2204 sellers, which is pretty much just the same as the Special Topics in Calamity Physics book shop. And four fifths of the time is spent in the top 22,700, which is five thousand less than yesterday's bookshop, and there's only one in ten views that are outside the top one hundred grand. So that's slanted even more to the best sellers than the Special Topics shop."

"You've obviously been busy," I was having a hard time following all these numbers, but Kylie ploughed on remorselessly.

"So dummkopf here, he's pumping the bellows like mad and writing it all down and he says 'Let's do some more Kylie!', so we did. He's quite the bundle of energy this kid", She punched him amiably in the stomach and Edmund stirred in his sleep. "I'll show you the results here."

A sheet of paper in my notebook had a summary of each of the shops they had created, the median sales rank, and the time spent in browsing around the top 100,000 sellers. Here it is.

Store Median  Top 100K   Seed Book
 7     1742    87.5    Testimony
 4     2007    91.4    The Lucky One
 9     2113    89.4    The Shock Doctrine
10     2113    87.6    Special Topics in Calamity Physics
12     2191    86.5    Our Mutual Friend
 1     2192    89.8    The Shack
29     2238    98.3    Bones
20     2276    88.2    The Shock Doctrine
22     2388    88.5    The Long Tail
17     2471    92.5    Our Mutual Friend
 8     2650    86.6    Conservatism That Can Win Again
21     2808    90.4    The Conscience of a Liberal
24     2888    90.1    The Long Tail
27     3135    88.8    The Long Tail
 5     3158    96.0    Seven Habits of Highly Successful People
19     3380    89.1    The Long Tail
 6     3438    91.2    No One Makes You Shop at Wal-Mart
14     3487    88.2    The Conscience of a Liberal
25     3610    88.3    The Long Tail
23     3627    93.3    The Long Tail
26     3681    91.5    The Long Tail
18     4060    84.8    The Conscience of a Liberal
13     4101    88.0    Surrender is Not an Option
16     5326    90.0    Riddley Walker
30     5750    87.9    Graph Drawing
28     6117    88.3    The Secret History
 2     6222    86.2    How To Eat
15     7231    84.6    Our Mutual Friend
31     7583    100     The Secret History
 2    17794    76.3    Last Best Gifts
11    51992    58.5    The Dilbert Principle

I pored over the numbers, unsure what to make of them. "So explain, snotnose" I grumbled. Somehow this venture did not feel entirely mine any more.

"It's like this," the vixen explained, slightly patronizingly I felt, "We did a number of books as seeds to see how things went differently depending on your starting point."

"But some have the same book name."

"That's because each bookshop may go different ways, depending on what choices the cat tells you to make, and we wanted to see if the results depended a lot on what book you start with. So would you always get the same shop if you start with one book."

"And do you?"

"Sometimes yes, sometimes no. Like, we did Our Mutual Friend twice because we both did Charles Dickens at school and hated it. Can't that man say anything in less than twenty pages? And you can see that one of them ended up with a median of 2471 and one with a median of over 7,000. That's quite a difference on this list. And we did The Long Tail quite a few times because Mr. Amazon told me about that when I asked him how he could help The Adventures of Wazzock to get what it deserves. But in the end, with a few exceptions it doesn't make much difference what book you start with. You'll spend at least half your time in the top 10,000 books, which is like browsing around the heavy traffic area of a regular bookshop. And you'll spend almost all your time in the top 100,000, so you don't really see much that you wouldn't see in a normal bookshop."

"What about these exceptions you mention?"

"There are three that are different, and all for the same reason. Look at these bottom three in the list. The Secret History, Last Best Gifts, and The Dilbert Principle."

"What about them?"

"Well The Dilbert Principle is the book that spends most of its time out in the outsider part of the shop, beyond the 100 grand mark. Any idea why?"

"Well I have heard that Dilbert is an humorous illustration of some kind. Isn't that right?"

"Yeah, so what you get is, you bounce around a lot of other comic books. Every one of the top 14 books is a Dilbert book and from what I can tell all but two of the top 80 are comics. There's not really any one big bestseller in comics, so that's why you spend so much time in the outsider books."

"And what about Last Best Gifts?"

"A bit different. It's basically an academic book and you end up going round a lot of other academic books. You don't always do that - like Graph Theory is pretty much like a bestseller - but Last Best Gifts seems to just wander the academic ghetto. What a bunch of losers!"

"And The Secret History?"

"Now that one is very strange." Kylie frowned and scratched her stomach, contemplating the oddities of the world. "This shop has the least number of books of all. Ten thousand different views and there's only 14 different books in the lot! The reason is that early on it gets into books by this bloke Christopher Moore who, it says here, writes offbeat humour. And then all the recommendations are for other Christopher Moore books. And the really odd thing is that none of those books appear in any of the other bookshops. You either like Christopher Moore or you don't I guess."

"So what have we learned?"

"Well, first that it's really easy to get distracted by these little stories of what makes one bookshop different from another."

"And second?"

"That'll have to wait until tomorrow. I'm hungry and I'm going to steal some lunch." And with that she stalked, hands deep in pockets, down the drive towards the village.

Comments

[John-Lee H]

I'm really enjoying this series. Thanks Tom!
I do hope you named your scripts after your characters.

[Russell]

Compellingly, roll on installment #12!

[Ruth]

Hmmm... "And you'll spend almost all your time in the top 100,000, so you don't really see much that you wouldn't see in a normal bookshop." That would be a very big bookshop. I tried googling how many books are in a book store and actually came on someone who estimates 1,000-2,000. (I couldn't get TrueKnowledge to understand the question. In May, when Wolfram Alpha arrives, we may have an easier time answering stuff like that.)

In any event, when I was trying to find the one book I didn't have in an obscure and out of date series, and found it quickly on Amazon for about 14 cents from a used vendor (since they so nicely return used books with new books when you search), I didn't much care if the upselling ads were "only" touting the top 100,000...

Not to quibble too much though. Let's have more of the one-legged housekeeper!

[tomslee]

Independent bookshops typically carry around 30,000 titles, I've heard from sources that I think are reliable (though I cannot quite recall who now - Dave from Words Worth?) My rough count at home is that a 5-shelf, 2' wide bookshelf holds about 250 books, so 30,000 titles would take 5 shelves times 240'. I think most bookshops would have 250' of shelving.

The figure Chris Anderson uses for big chain stores is 100,000.

The biggest bookstore I have seen is Blackwell's in Oxford. I asked them how many distinct titles they stocked and they told me 200,000.

Your experience actually fits with Whimsley's. If you know what you want you can go to Amazon and get it, and much of the diversity of sales comes from exactly the model you are using. But the current orthodoxy is that recommender systems help you discover obscure books that you can't find elsewhere, and my experiments (the numbers I give do come from real tests, code available on request) show that the recommender system doesn't do that - it directs you to other books that are generally already fairly popular.

As for Jennie the housekeeper, it is embarrassing to realise that Mr. Amazon's bookshop fails the Bechdel/Wallace test, so I think she will have to reappear.