Every big company has an origin story that sounds inevitable later and is anything but at the time. Databricks starts here, when seven academics from UC Berkeley take a research project and a relatively small amount of venture money and decide to build a company around it. The research project is Apache Spark. The reluctant money is a 14 million Series A led by Andreessen Horowitz. And the thing they are actually selling, though almost nobody understands it yet, is a bet that the world is about to drown in data and will pay handsomely for a better way to process it.
Where This Came From
To understand Databricks, you have to understand Spark, and to understand Spark, you have to understand what came before it. In the late 2000s, the dominant tool for processing enormous datasets was Hadoop MapReduce. It worked, in the sense that it could chew through petabytes of data spread across hundreds of machines, but it was slow and clumsy for anything that required repetition. Here is a useful way to picture the problem. MapReduce wrote its intermediate results to disk after every step. Imagine doing your taxes but being forced to put every sheet of paper back in the filing cabinet after each calculation, then walking back to retrieve it for the next step. That constant trip to the disk is fine if you only do it once, but machine learning and interactive data analysis involve looping over the same data hundreds of times, and the trips add up into hours of wasted waiting.
Matei Zaharia, then a PhD student at Berkeley’s AMPLab, built Spark in 2009 to fix exactly that. The core idea was in-memory processing: keep the working data in RAM across all those machines instead of writing it back to disk after every step. To continue the tax analogy, Spark lets you keep all your sheets spread out on the desk while you work through them, only filing them away at the very end. For the iterative workloads that are becoming important, machine learning especially, this is not a small improvement. It is often 10 to 100 times faster. Spark also adds fault tolerance, so if a machine in the cluster dies mid-job, the system can rebuild the lost work without restarting.
The seven founders, Ali Ghodsi, Ion Stoica, Matei Zaharia, Patrick Wendell, Reynold Xin, Andy Konwinski, and Arsalan Tavakoli-Shiraji, released Spark as free open-source software and watched it gain academic interest but very little commercial traction. Over a series of meetings at Indian restaurants, they decide that the way to change that is to build a company that makes Spark actually easy to use in production.
Here is the detail worth savoring about the founding. The team is not even trying to raise as much as it ends up with. Ben Horowitz of Andreessen Horowitz, the story goes, pushes them to take more money than they ask for, because he can see a bigger company in this than the founders, cautious academics by training, are picturing. They walk in modest and walk out with 14 million dollars and an investor who believes in the scale of the opportunity more than they do.
What It Means for the Market
The data warehouse world belongs to Oracle, Teradata, and IBM, expensive proprietary systems that large enterprises have been locked into for decades. The newer big-data excitement belongs to the Hadoop companies, Cloudera and Hortonworks, who are commercializing the on-premises Hadoop ecosystem. Into that crowded field steps a tiny startup whose core technology is free and whose product does not yet exist. On paper, it looks like a long shot.
But the founding bet contains the seeds of a real market shift. The Hadoop companies are selling complexity: their products require specialized engineers and long deployment cycles. Databricks bets that the cloud makes all of that obsolete, and that companies prefer a managed service they can spin up in minutes over a sprawling on-premises cluster that takes months to stand up. If that bet is right, the incumbents built around on-premises Hadoop are not competitors so much as future roadkill. The benefit Databricks offers from day one is speed in two senses: faster data processing through Spark’s in-memory model and faster time to value through eventual cloud delivery. Whether that actually disrupts the entrenched players or simply becomes a nice niche tool is, at present, a completely open question. The 14 million dollar bet is that it will be the former.
It will take another year before there is even a product to sell. But the company starts here, with seven academics, a faster way to process data, and a venture capitalist who believes in it more than they do.
Sources: Contrary Research, MicroVentures, Bigeye, Dtyped, Forge Global, CanvasBusinessModel. Founding and Series A details as reported.