The Future Arrived in 2006 — The Publishing Apocalypse.
Starting out With the Rough Stuff
This week’s post is going to get technical. As ever, I am leaving out some details in order not to write a textbook.
Updated to add a link (see bottom of post) to a very apropos Venture Beat article)
Some acronyms up front:
- SQL: Structured Query Language
- NoSQL: Despite the name, the major NoSQL databases generally use Postgres SQL for parts of their underlying SQL engine.
- MySQL: Open source database system, bought by Oracle some years ago, and now forked into dbs like Maria etc.
- RDBMS: Relational Database Management System — Such as SQL Server or Oracle
- OLTP: On Line Transactional Processing. A database that processes serially with all parts of a related set of actions in the same transaction such that if any part fails, the entire transaction is rolled back. DBAs like to talk about transactional consistency. Because they get fired if their OLTP databases are not transactionally consistent or cannot be returned to a consistent state. Databases that support sales to a consumer are OLTP, for example.
- OLAP: On line Analytical Processing. A data warehouse. OLAP systems are for reporting on data. The containers that hold the data are designed differently so as to quickly slice and dice data from OLTP databases.
Test in 20 minutes!
Just kidding.
Here’s some general information that I think is useful for non-DBAs to know. RDBMS’s replaced mainframe systems and quickly proved themselves far less expensive and, in some ways, better systems for managing data. But they’re still not cheap. If you need a robust, fast, database server then you could easily be looking at $100,000 in hardware and the same or more in licensing costs.
Your typical SQL Server is a series of disk drives talking to the same database engine and can be talked about as a single server without being too horribly wrong. One server can host multiple databases, by the way. And very often, though we talk about “a server” we’re usually really talking about many servers with many drives acting as a single entity.
NoSQL databases solved some speed and cost issues that arise when your database is beyond huge. The NSA, by the way, collects and stores its information in Hadoop, a NoSQL system.
Getting to the Good Stuff
There are three companies that transformed the database and storage world and lead to NoSQL systems going mainstream. You will recognize all the names. If there were Nobels given out for computer science, two of these companies (or their “two guys in a garage” predecessors) would have been short listed. The three companies are:
YouTube
Amazon
Google
The two short-listers would be YouTube and Amazon. YouTube built the practical (and on-the-fly) implementation of federating disks and servers such that the data on those disks was not transactionally consistent at a given point in time but rather, eventually consistent. Amazon invented massively shared structures (Simple Storage Service, or S3).
Amazon was doing this development at the time everyone thought of Amazon as the online book store. Anybody paying attention in the database and storage business around this time knew about Amazon’s development work in this area. This matters. Hold that in your head for a bit. It matters a lot to the publishing industry because publishing appears to be fundamentally ignorant about Amazon’s history and what that means for the future and specifically their future.
In the early days of Web 2.0 when we began to see internet applications that grew to serve millions of users uploading their own content and, in turn, delivering that content to millions of other users, the plain fact is that the companies/people behind those startup applications could not afford even a cheap server and a SQL server or Oracle license. Those systems were not priced such that they could be used by a startup. (Hold that in your mind, too.) Those startups invented new ways of meeting these new challenges.
Let’s talk about the fact that Amazon was developing its shared storage solution in 2003. (Here). S3 debuted in 2006. (here) A startup or even an individual, no longer had to figure out how to get a server and enough money to develop a consumer application and bring it to market. They could use Amazon’s web services and S3. This enabled rapid, inexpensive application development.
The Big Bang
Suppose you are a traditional publisher today, talking about how horrible Amazon is. Set aside corporate ideologies for a moment and consider the fact that Amazon was inventing transformative storage technologies before it ever got around to Kindle Direct Publishing. Ask yourself why Amazon would develop S3 and give everyone anywhere access to those servers and that technology, from a 1-person outfit to large companies. What do those facts tell you about how Amazon thinks about the future?
Compare that to how traditional publishers think about the future. The words and deeds we see from them are all about how to preserve the status quo even as that status quo crumbles around them. No matter what happens in contract negotiations between Amazon and traditional publishers, it’s already too late. The future arrived in 2006 but it started before that. Amazon debuted as an online book store. Sure. But what we know from all the evidence is that this was never Jeff Bezos’s end game. It was his opening move.
Traditional publishers should be thinking hard about why the first move was books, and why books are still a core part of Amazon’s website business. They should be thinking about why Amazon opened up S3 to developers and said, come do stuff here! They should surely reflect on what it means that they were not among those companies.
Here’s a few of the companies today using AWS and S3: (here):
NASA
wunderkinder (Germany)
Airbnb
8securities (Hong Kong)
Pinterest
Casa & Video (Brazil)
Bristol-Myers Squib
What has Amazon said to authors? They said, here’s KDP. Come do stuff here! What happened when they did that? Authors reinvented themselves. Authors did an end-run around traditional publishing. It was inevitable that an author (Barbara Freethy) would sign a direct deal with Ingram. She may be the first, but she won’t be the last.
It does not matter one bit that there are authors who don’t do well in this new world. Let me remind everyone that there are authors who didn’t and don’t do well in the traditional environment.
No, I didn’t Forget About Readers
What matters is that this new open infrastructure ensures there will be authors who do well without a traditional publisher. This will continue to happen and it will continue to benefit readers as they tweet about the books they’re reading and buy the books of NYT bestselling authors 30 seconds after reading someone’s tweet about it. Book sales, my friend, that did not earn a single penny for a traditional publisher. In 2006, that would have been impossible.
Traditional publishers have already lost.
I do think there’s hope, by the way. New hope. Not old hope.
Today, Venture Beat posted this article: “The Race to Zero” about AWS in which they say this:
When Cisco announced it would spend $1 billion on the cloud last summer, the executive leading the project, Nick Earle, was quick to declare, “Our strategy is not to follow AWS on the race to zero.”
Next week: What might a new publisher look like?
This was an really interesting post. Thank you for writing it! :)
Excellent post. Enjoyed the Tech Talk explanation along with an entertaining style. Informative.
I wish I understood more of this post. But I’m still looking forward to the next one, even with my limited grasp of what’s going on.
I was uncertain about how much detail to provide, so if you have questions, please request clarifications. I’ve been in tech for long enough that it can be difficult to see where I’ve assumed certain knowledge that I shouldn’t have.
I’m happy to clarify or add!
Well, something I don’t know and can’t quite work out from the post is how much control/influence owning/developing the internet infrastructure gives Amazon. Is it just an example of them doing something new or is it the first step to world domination?
@Ros: Hmm. Technically, this is not ownership of the internet infrastructure. This is ownership of an infrastructure (that runs on the actual internet infrastructure) that was offered to the public at a very low cost.
At this point there are other companies, Microsoft, for example, that offer competing environments. If Amazon were to close down AWS, developers would have other places to move to.
In 2006, it was completely new. It’s the reason we have this astonishingly diverse application environment today.
Hopefully that answers your question!
@IAM JSON: I think so, thank you!
I may not have grasped much at the beginning but I did appreciate where you ended up. I want Amazon to have competition but publishers have had plenty of opportunities to get in the game (Jane has been writing about this for years now) and they have yet to do so in any significant way.
I remember being surprised a couple of years ago when I found out that Netflix uses AWS for their streaming service since Amazon streaming is in direct competition but AWS obviously offered an affordable solution and continues to do so.
I really appreciate the technical insight even when I understand so little of it. (Also, I’m visiting family for Thanksgiving this year and maybe I can pick up some things to impress my uncle who works for Oracle. ;-))
@Lada: Lada: Thanks for your comment! Many many large companies use AWS, including companies you’d think of as direct competitors. This is a competitive space, with Google and Microsoft as excellent company.
When you see your uncle, over delicious pie ask him, “So. Written any good pl-SQL lately?”