Romance, Historical, Contemporary, Paranormal, Young Adult, Book reviews, industry news, and commentary from a reader's point of view

publishing industry

Dear Author

Is genre fiction creating a market for lemons?

In 1970 George Akerlof (Nobel Prize winner in Economics 2001) wrote a scholarly article called “The Market for Lemons: Quality Uncertainty and the Market Mechanism.” In true academic fashion, the article was deemed trivial by the top journals and he finally landed it in a highly-ranked but not flagship publication. It has since gone on to become one of the most cited economics articles and his insights about the used-car market in the US have been shown to have a much wider application.

Briefly put, a market for lemons can be created when buyers and sellers have asymmetric information. In the olden days, sellers of used cars would withhold information about the cars’ past performance and repair history in order to fetch the highest prices. Sometimes this was borderline or outright unethical behavior, and sometimes they just didn’t know. As a result, people approached buying a used car with the enthusiasm they generally reserved for root canal surgery.

“Wow, I got a great deal on this used car! It is better than I thought it would be!” said almost no one, anywhere, ever, in those days. If you did get a good deal and a good car, it was basically a miracle. This was mostly because people who had good cars to sell knew they wouldn’t get the price the car was worth, because there was often no way to convince the seller that it was a good car. So they found other ways to offload the car (sell to friends, trade it in, etc.) and many good cars were kept out of the market. That turned the used car market into a market for “lemons,” slang for a bad car.

It wasn’t that good cars weren’t around, or that no one could buy them, it was that the market didn’t have a reliable way to signal to sellers which were the good and which were the bad cars. Today we have Carfax and other computerized, easily available information about used cars, and the market thrives for both buyers and sellers. The information asymmetry has basically been erased.

What does this have to do with books, you might ask? Well, thanks to the brouhaha over the Author Earnings report and the influx of new commenters to my DA post, I wound up reading a number of threads at the Kindleboards. Most of the time I forget that the Kboards community exists, given I don’t share their interests, and since I gave up Amazon discussion groups and Goodreads I don’t notice mentions of them in my various Twitter and blog streams.

While I was immersed in the Kboards threads, my Twitter stream happened to be featuring a conversation about self-published books that purportedly featured purchased reviews (here is a roundup by Mike Cane of the blog posts they were discussing). And then Jane wrote a column asking readers about their price-sensitivity in reviewing books, and it started to come together for me.

The phrase “Gresham’s Law” had been bouncing around in my head, but I knew that wasn’t quite the right concept. Gresham’s Law is about currency circulation, where everyone knows that two currencies that have the same official value may differ on other value dimensions (e.g., gold coins and base-metal coins that have the same value when used as currency). But the book market seemed to me to be an asymmetric information situation.*

The result of the two processes is the same: the bad product drives the good product out of the market. But the way it occurs and why it occurs are different. In the case of Gresham’s Law the good money leaves because its official valuation is lower than its real valuation (people take the good money out of circulation). In the case of lemons it’s because the better product cannot be sold for more than the worse product, since there is no transparent way to guarantee to buyers that a quality difference exists.

Two decisions have converged during the rise of ebooks to make low-production-value, low-priced books much more commonplace. The first is the original decision by the Big 6 publishers to conspire illegally to set and maintain high prices for their ebooks. The second is the decision of authors to self-publish their books and price them much lower than Big 6 books in order to gain market share. The second decision was facilitated on a large scale by Amazon’s willingness to provide a retail portal for these author-publishers.

By the time the Big 6 (now the Big 5) were forced to stop price-fixing, the average price of ebooks had decreased because of the influx of self-publishers and the willingness of readers to take a chance on these books. This trend was especially true in genre categories like romance and SFF. As a result, a $7.99 book not only looked awfully expensive to the average high-volume romance reader, it was competing with many lower-priced and even free books in the same genre. (An aside: this is Jane’s point about anchoring. $7.99 wouldn’t look nearly as bad if it weren’t near the top end of a price scale that begins at $0.00.)

Worse, the price points didn’t necessarily reflect quality differences, or even easily measurable criteria like length. Consider the following examples:

  • A “self-published” rerelease of a NY-published author’s backlist
  • A new self-published book from a previously NY-published author
  • A self-published debut from an unknown, newbie author
  • A loss leader from a NY publisher trying to drum up readership

Four very different types of provenance, but all four books could have price points of $3.99. With that price confusion, what rational consumer is going to take a chance on a $7.99 book unless she has additional information about the book or the author?

So the primary differentiator, price, is becoming a very noisy signal. At $3.99 you can get a very good book or a very bad book or something in between.

Price, of course, is not the only signal. A second signal is review rating. Amazon has review ratings for everything from hair dryers to Montblanc fountain pens to The Great Gatsby (1-star review: “I was really glad when the story was over. I felt no depth in any of the characters and was not intrigued by his writing style.”). Amazon loves reviews of the products they sell, and all you have to do to write one for any product is buy one thing from Amazon, once. Reviews range from careful, well-thought-out critiques to “this book had a torn cover when I received it: 1-star.” This is not surprising, since practically anyone can post a review, and in my opinion it’s fair, because the consumer is buying a product and has the right to review the purchase as a product.

Book reviews exhibit an interesting pattern: reviews of popular and best-selling genre books are systematically higher than reviews of corresponding general fiction and literary fiction. Some have hundreds of ratings with very, very few in the 1-star category. Looking at the list as I’m writing this, I see that Donna Tartt’s The Goldfinch has a 3.8 rating, while H.M. Ward’s Stripped (The Ferro Family) has a 4.7. In addition, well-known self-published books often have very high rankings, especially compared to Big 5 and other major publishers’ releases. For example, compare the positive (4- and 5-star reviews)  percentage of Wool to the percentages of other well-known SFF books:

Wool (Omnibus): 93.6%
Harry Potter #1: 94.4%
Neuromancer: 71.7%
Cryptonomicon: 77.6%
Ender’s Game: 90.1%
Ready Player One: 89.4%

Only Rowling’s first Harry Potter book has a higher reviewer rating than Wool, and of the rest, only Ender’s Game reached 90%. Neuromancer arguably established a genre, Cryptonomicon and Ender’s Game are both critically and popularly acclaimed, and Ready Player One was a word-of-mouth hit. All have substantially lower approval ratings, and the three books that are clearly targeted at not-young adults have far fewer ratings than the YA-friendly novels. Neuromancer, which is 30 years old this year, has 726 ratings, which is less than a tenth of the total numbers racked up by Howey, Rowling, and Card.

It’s entirely possible that readers of the Ward and Howey books were more satisfied with their reading experience than readers of the Tartt, Gibson, etc. (I tend to think of Rowling as being in a category of her own).  I have more trouble with the idea that the Ward and Howey books are better books. And herein lies the problem with Amazon reviews. They’re only partially about quality.

Quite apart from whether reviews are genuine or fake, written by people with a stake in the book/author or by an unconnected reader, I don’t find it believable when a best-selling book has a 93.6% positive rating. But even if I were convinced of the overwhelming love for it, when so many books get 4.5+ star averages, two things happen to me, the unconnected reader:

  1. I have trouble knowing which book to pick;
  2. When I read it and find out it is not a 4.5 star book in objective terms, i.e., technique/production issues, I start to mistrust reviews. A lot.

And when I mistrust reviews, there goes my second signal. Rather than having helpful signals, I now have two measures that provide noise. So how do I choose among the vast numbers of books Amazon offers me? I can flip a coin, read a lot of samples, or throw up my hands and decide to stream something on Netflix or Amazon Prime instead.

Because the ultimate competition to books isn’t other books. It comes from other forms of entertainment. If sifting through the book recommendations becomes too difficult, a lot of people will just turn to other media. I’ll keep looking because I have a lifelong investment in being a reader. But for younger people who don’t, why should they bother?

Right now we have ways of sorting amongst the enormous slush pile that is the Kindle bookstore. We can follow NY authors to their self-published efforts and hope they have kept their same standards. We can feast on the backlist releases (there are luckily many, many of those). We can follow word-of-mouth and reviews from people we trust to discover new writers.

Two of those three options involve people with connections to traditional publishing, that hoary old dinosaur that self-publishing is supposed to render irrelevant. Right now, the majority of self-publishing success is almost certainly built on the foundation provided by New York. You won’t find that information in most evangelical tracts about self-publishing, including the Author Earnings report, because it undermines the “all self-publishing is great and we are all in this together!” message. But I haven’t seen any evidence to disprove it.

If you vanquish New York, though, you undermine self-publishing. What will be left, in overwhelming numbers, will be the inexperienced, still-working-on-the-writing-thing authors, and that will make good authors wary of entering, for fear of being lost in the slushpile.  And that’s when readers will realize they’re shopping in a market full of lemons.


*ETA: I’m not the first person to use the Akerlof paper to talk about the book market. See this post by Balder Bjarnason written last year. We have slightly different foci but lots of overlap as well.

How (not) to lie with statistics

How (not) to lie with statistics

Fortune Teller CatI’ve been looking at the Author Earnings Excel spreadsheets (the “raw data”) for the last few days. Many people have become very, very excited by the conclusions the authors draw from this data, and critics have consistently been shouted down as being pro-publisher, anti-self-publishing, or having various other axes to grind.

I have no dog in this hunt. I’m not a fiction author and very little of my income comes from royalties or other direct remuneration from writing. I have to write and publish in order to earn promotions and salary increases, and to be respected in my field, but that writing is judged on quality, reliability, and academic contribution. No one really cares about sales.

I do, however, care about how data are collected, analyzed, and reported, and this report doesn’t pass my smell test for reliability and validity. As a political scientist trained in both political science and sociological methods, I’ve conducted qualitative and quantitative studies and I’ve hand-built and analyzed a quantitative dataset comprised of over 3000 observations. I’ve also taught research design and methods at the university undergraduate and graduate levels.

My concerns aren’t about math or statistics (there is very little of either being used here). My concerns are about (1) how the data were selected; (2) inferences from one data point to a trend (you literally cannot do this); and (3) inferences about author behavior that are drawn entirely from data about book sales. I would be less concerned if the authors of the report showed more awareness of the limitations of the data. Instead they go the other way and make claims that the data cannot possibly support.

I could write 10,000 words on this pretty easily (I started to), but I’ll spare DA’s readership that and try to be succinct.

(1) This is Amazon-only data. We call this a “sample of convenience,” i.e., the authors figured out a way to scrape Amazon so they did. They justify extrapolating from Amazon to the entire book selling market by saying that Amazon has been “chosen for being the largest book retailer in the world.” This is analogous to me studying politics in the state of Alaska and then claiming my study provides an accurate representation of politics in the entire USA because Alaska is the biggest state. Or studying politics in China and then claiming my study explains politics all over the world because China is the most populous country. In other words: No.

Amazon is the biggest, but it’s not the only or even the majority bookseller. And it’s not a representative bookseller statistically, i.e., there is no reason to believe that Amazon provides a representative snapshot of all book sales. It probably sells the most ebooks, it might sell the most titles, and it is where a lot of self-published authors sell their books. But it does not sell the most total units of (print and e-) books, which means comparisons across categories will most likely be skewed and unreliable. If the authors’ conclusions were limited to Amazon-specific points, I would be less bothered. But they are making claims about general author behavior based on partial, skewed data. No. No. No.

(2) This is one day’s worth of data, one 24-hour period of sales and rankings. We call this a cross-sectional study. Cross-sections are snapshots, which can be very useful to give you an idea of relationships between variables. But they have their own biases, and these biases can be irrelevant (good for your study) or relevant (bad for your study). In this case the 24-hour period comprised parts of the last Tuesday and Wednesday of January 2014. I can think of at least two relevant biases: (1) Books for the next month are frequently released on the last Monday-Tuesday of the previous month; and (2) people buy textbooks at the beginning of school semesters (January is the beginning of spring quarter/semester). This latter condition might create substitution effects (people spend their money on non-fiction books) which are not the same across all publishers and categories. I don’t know if these biases matter, but the point is that the authors don’t even tell us that they considered that there might be bias issues in picking this particular time period. I mistrust studies where limitations aren’t at least discussed.

(3) Cross-sections cannot give you trends. Trends need more than one data point. You cannot determine a trend from a single observation. If a book is #1 today, that doesn’t mean it will be #1 tomorrow. You cannot infer anything about the past or the future from a single data point in the cross-section.

(4) Nevertheless, the authors do try to infer from this. In fact, they do a lot of inferring that is analytically indefensible. Let’s take the inferences in turn.

  • They infer sales numbers from rankings (because Amazon does not publicly report sales), based on their own books and information from other authors. In asking around I have been told that the sales numbers correspond to rankings fairly well. I’m willing to believe this, but Amazon itself points out that rankings can change without sales figures changing and vice versa. This may be a bias that is sufficiently general that it doesn’t compromise the inferences. But it’s something to keep in mind.
  • They take the sales numbers for one day (an inference) and combine it with the publisher data, which gives them the gross and net sales figures for each book. They then take the author’s net revenue and multiply that number x365 to get the author’s earnings for that book for the entire year. This is insane completely absurd. According to this “formula,” a book that sells zero copies on January 28/29 nets the author zero dollars for the year. A book that sells 7000 copies and is published by Amazon nets the author over $4 million. Not only is this unbelievable (99% of books move around the list over the year, unless they’re stuck at 0 sales), it casts doubt on every other data and inference decision the authors make. Why on earth would you accept information, let alone take advice, from someone who thinks this is a good way to calculate author earnings? You should be running far, far away. It is very bad analysis. It is horrifically bad inference. This criticism doesn’t even take into account the difficulty of estimating author earnings without including advances, but frankly, in my estimation it’s a sufficiently disabling criticism on its own.
  • Author behavior is being inferred from data on books. There is too much author behavior that is simply missing and can’t be inferred in any legitimate way. Authors make choices about editing, packaging, writing quality, etc. that affect reader decisions to purchase books. In addition, anecdotal evidence consistently points to the importance of a backlist, a backlist that can come from self- or publisher-published books. None of these variables are captured in this data. So what we are at risk for having is “omitted variable bias,” where the correlations are inaccurate because not everything that matters is present in the dataset.

(5) Many of the top selling books are Amazon imprints. Thomas & Mercer is the top-selling imprint in this dataset. That lucky author is on course to make $4 million, according to the report. (Or not, depending on which planet and universe you inhabit. Here’s hoping she lives in the alternate one.) It makes sense that Amazon does so well at Amazon, since they have many ways of boosting visibility and they naturally use those techniques to sell their own books. And since the NYT and USA Today don’t include exclusive-vendor books on their bestseller lists we can’t see how Amazon books do when we include other rankings. It’s a classic problem of comparability: Amazon doesn’t include pre-order sales in its rankings and NYT/USA Today don’t include Amazon-only books in their rankings. so you can look at apples at one and oranges at the other, but you can’t look at apples and oranges together.

The authors attempt to bolster their existing data by looking at Bookscan numbers, but because Bookscan doesn’t break down data between ebook and print sales and instead includes overall sales, the information revealed to us in this new report doesn’t help us situate the Amazon data in a larger context. At best the Bookscan numbers might reveal the proportion of books sold at Amazon relative  to the larger marketplace captured by Bookscan (although Bookscan doesn’t account for all sales). But instead, the Bookscan numbers are used to compare e-book sales to print sales, which are a completely different issue.

(6) The authors provide the data in an Excel spreadsheet format so that the rest of us can analyze it. I appreciate this, and I’m happy to work with flat files, although there are a lot of advantages to relational databases (e.g., MySQL and Access). But when I downloaded the files I realized that (a) this is not “raw data” and (b) important information is uncollected or removed. In particular:

  • When the data were scraped, if they had picked up the release date it would let us know where the book was in its life cycle. Then we could apply a survival model, i.e., one that estimates a rate of sales decay over time, and the date would also help us identify whether the price in the cross-section is permanent or temporary. These data aren’t perfect, because publishers can change release dates (and there are different release dates for different editions, including self-pub updates). But being able to use even imperfect release date information would allow revenue projections to approximate something that isn’t prima facie absurd.
  • The “author data” sheet in the file combines all of each author’s books into one observation (one row in the spreadsheet) and labels them with one publisher category. This potentially conflates self- and publisher-published books, books across genre categories, and top-selling and lesser-selling books (case in point: #1 Book is 7000 sales and #1 Author has two books at 7000 sales, so one of #1 Author’s books has 0 sales). I would like to decompose this data but I can’t because the author info has been “anonymized,” as has the title info. Therefore I can’t combine the info provided in the two sheets into one dataset. This is apart from the main problem, of course, which is that there is very little point to running more rigorous statistical analysis because the underlying data have essential reliability and validity problems.

There is a post at Digital Book World which provides descriptive statistics for the data (something the report’s authors did not, which is also a breach of data analysis norms). The data look to be skewed, and also to be non-normally distributed. I’m betting there are correlation issues that will wipe out at least some of the results in the pretty charts if we subjected those bivariate relationships to proper controls in a multivariate analysis. There is also an excellent criticism of the report’s discussion of star ratings here.

A sentence in the report has been making the rounds:

“Our data suggests that even stellar manuscripts are better off self-published.”

No. That conclusion is writing a check that the data can’t cash.

As an empirical researcher who respects the limits inherent in all data collection and analysis, my strongest advice is to read this report as you would read any interesting tidbit about the publishing industry. Treat it as entertainment, not information. If you’re interested in data analysis more generally, think of this as a stellar example of What Not To Do.

If you pushed me for a recommendation based on what I see in these data, I would say, after reminding you of the insurmountable shortcomings contained within it: If you plan on selling ebooks solely or primarily at Amazon and the opportunity cost of your time is greater than zero, you might want to sign up with submit to (and hope you are offered a contract by) an Amazon imprint. Because Amazon books do extremely well and the cut they take may well be worth the time you save doing all your own production and promotion. Somehow I don’t think that’s the takeaway the authors intend, but that’s an obvious one for me. But remember, I don’t have a dog in this hunt. I’m just looking at the data.