The Future Arrived in 2006 — The Publishing Apocalypse.

IAM JSONEssaysChanges / databases / future / Hope / Tech Sunday9 Comments

Starting out With the Rough Stuff

This week’s post is going to get technical. As ever, I am leaving out some details in order not to write a textbook.

Updated to add a link (see bottom of post) to a very apropos Venture Beat article)

Some acronyms up front:

1. SQL:  Structured Query Language
2. NoSQL: Despite the name, the major NoSQL databases generally use Postgres SQL for parts of their underlying SQL engine.
3. MySQL: Open source database system, bought by Oracle some years ago, and now forked into dbs like Maria etc.
4. RDBMS: Relational Database Management System — Such as SQL Server or Oracle
5. OLTP: On Line Transactional Processing. A database that processes serially with all parts of a related set of actions in the same transaction such that if any part fails, the entire transaction is rolled back. DBAs like to talk about transactional consistency. Because they get fired if their OLTP databases are not transactionally consistent or cannot be returned to a consistent state. Databases that support sales to a consumer are OLTP, for example.
6. OLAP: On line Analytical Processing. A data warehouse. OLAP systems are for reporting on data. The containers that hold the data are designed differently so as to quickly slice and dice data from OLTP databases.

Test in 20 minutes!

Just kidding.

Here’s some general information that I think is useful for non-DBAs to know. RDBMS’s replaced mainframe systems and quickly proved themselves far less expensive and, in some ways, better systems for managing data. But they’re still not cheap. If you need a robust, fast, database server then you could easily be looking at $100,000 in hardware and the same or more in licensing costs.

Your typical SQL Server is a series of disk drives talking to the same database engine and can be talked about as a single server without being too horribly wrong. One server can host multiple databases, by the way. And very often, though we talk about “a server” we’re usually really talking about many servers with many drives acting as a single entity.

NoSQL databases solved some speed and cost issues that arise when your database is beyond huge. The NSA, by the way, collects and stores its information in Hadoop, a NoSQL system.

Getting to the Good Stuff

There are three companies that transformed the database and storage world and lead to NoSQL systems going mainstream. You will recognize all the names. If there were Nobels given out for computer science, two of these companies (or their “two guys in a garage” predecessors) would have been short listed. The three companies are:

YouTube
Amazon
Google

The two short-listers would be YouTube and Amazon. YouTube built the practical (and on-the-fly) implementation of federating disks and servers such that the data on those disks was not transactionally consistent at a given point in time but rather, eventually consistent. Amazon invented massively shared structures (Simple Storage Service, or S3).

Amazon was doing this development at the time everyone thought of Amazon as the online book store. Anybody paying attention in the database and storage business around this time knew about Amazon’s development work in this area. This matters. Hold that in your head for a bit. It matters a lot to the publishing industry because publishing appears to be fundamentally ignorant about Amazon’s history and what that means for the future and specifically their future.

In the early days of Web 2.0 when we began to see internet applications that grew to serve millions of users uploading their own content and, in turn, delivering that content to millions of other users, the plain fact is that the companies/people behind those startup applications could not afford even a cheap server and a SQL server or Oracle license. Those systems were not priced such that they could be used by a startup. (Hold that in your mind, too.) Those startups invented new ways of meeting these new challenges.

Let’s talk about the fact that Amazon was developing its shared storage solution in 2003. (Here). S3 debuted in 2006. (here) A startup or even an individual, no longer had to figure out how to get a server and enough money to develop a consumer application and bring it to market. They could use Amazon’s web services and S3. This enabled rapid, inexpensive application development.

The Big Bang

Suppose you are a traditional publisher today, talking about how horrible Amazon is. Set aside corporate ideologies for a moment and consider the fact that Amazon was inventing transformative storage technologies before it ever got around to Kindle Direct Publishing. Ask yourself why Amazon would develop S3 and give everyone anywhere access to those servers and that technology, from a 1-person outfit to large companies. What do those facts tell you about how Amazon thinks about the future?

Compare that to how traditional publishers think about the future. The words and deeds we see from them are all about how to preserve the status quo even as that status quo crumbles around them. No matter what happens in contract negotiations between Amazon and traditional publishers, it’s already too late. The future arrived in 2006 but it started before that. Amazon debuted as an online book store. Sure. But what we know from all the evidence is that this was never Jeff Bezos’s end game. It was his opening move.

Traditional publishers should be thinking hard about why the first move was books, and why books are still a core part of Amazon’s website business. They should be thinking about why Amazon opened up S3 to developers and said, come do stuff here! They should surely reflect on what it means that they were not among those companies.

Here’s a few of the companies today using AWS and S3: (here):

NASA
wunderkinder (Germany)
Airbnb
8securities (Hong Kong)
Pinterest
Casa & Video (Brazil)
Bristol-Myers Squib

What has Amazon said to authors? They said, here’s KDP. Come do stuff here! What happened when they did that? Authors reinvented themselves. Authors did an end-run around traditional publishing. It was inevitable that an author (Barbara Freethy) would sign a direct deal with Ingram. She may be the first, but she won’t be the last.

It does not matter one bit that there are authors who don’t do well in this new world. Let me remind everyone that there are authors who didn’t and don’t do well in the traditional environment.

No, I didn’t Forget About Readers

What matters is that this new open infrastructure ensures there will be authors who do well without a traditional publisher. This will continue to happen and it will continue to benefit readers as they tweet about the books they’re reading and buy the books of NYT bestselling authors 30 seconds after reading someone’s tweet about it. Book sales, my friend, that did not earn a single penny for a traditional publisher. In 2006, that would have been impossible.

Traditional publishers have already lost.

I do think there’s hope, by the way. New hope. Not old hope.

Today, Venture Beat posted this article: “The Race to Zero” about AWS in which they say this:

When Cisco announced it would spend $1 billion on the cloud last summer, the executive leading the project, Nick Earle, was quick to declare, “Our strategy is not to follow AWS on the race to zero.”

Next week: What might a new publisher look like?

IAM JSON

Json is a longtime technology geek who has been, more or less in order, desktop support, Netware Admin, Network Administrator, web developer, lead web developer, data architect, database administrator. Json has a strong interest in network and computer security and currently works on the database end of software development. On the side, Json has set up a DocBooks workflow and would be done with the Hadoop install if Mongo db weren't so shiny.

→ IAM JSON