Much of the data story so far has been about data at rest, information that has already been collected and is sitting in a warehouse or lake, waiting to be analyzed. But a great deal of the world’s most valuable data is not at rest at all. It is in motion: the continuous torrent of events streaming off websites, mobile apps, sensors, financial markets, and connected devices, every click, swipe, payment, and reading happening right now. The company doing the most to make that moving data usable is Confluent, built around an open-source technology called Apache Kafka, and it has become one of the most important and well-funded players in the data infrastructure world. This is the story of data in motion.
Batch Versus Streaming, Explained
The traditional way of handling data is called batch processing. You collect data over some period, an hour, a day, then process the whole batch at once. Think of it like doing all your laundry once a week: it piles up, and at a set time you run it all together. Batch is fine for many purposes, monthly reports, overnight number-crunching, but it has an obvious limitation: the insights are always a little stale, because you are analyzing what happened earlier, not what is happening now.
Streaming, sometimes called real-time data, is the opposite. Instead of piling data up and processing it in batches, you process each event the instant it arrives, continuously, as a never-ending flow. The laundry analogy would be washing each item the moment it becomes dirty. For some uses this is not just nicer, it is essential. A bank detecting credit-card fraud cannot wait until tomorrow’s batch to notice a stolen card being used; it needs to catch the suspicious transaction in the moment, before the next one goes through. A streaming service recommending what to watch next, a rideshare app matching drivers and riders, a factory monitoring equipment for failures, all of these depend on reacting to data the instant it is created.
What Kafka Does
Apache Kafka, originally built at LinkedIn and released as open-source, is the technology that has become the standard backbone for handling streaming data. The clearest way to picture Kafka is as a central nervous system for a company’s data. Every system that produces data, the website, the mobile app, the payment processor, the sensors, sends its events into Kafka as a continuous stream. Every system that needs to react to that data, the fraud detector, the recommendation engine, the analytics dashboards, reads from those streams in real time. Kafka sits in the middle, reliably carrying the flow of events from the things that produce them to the things that consume them.
A useful image is a vast, high-speed conveyor-belt system in a building. Anyone can place items on a belt, and anyone elsewhere in the building can take items off the belt they care about, all simultaneously, without the senders and receivers needing to know about each other directly. Kafka is that conveyor system for data events, able to handle millions of them per second without losing any. This decoupling, where producers and consumers of data do not have to be wired directly together, is what makes Kafka so powerful for building large, complex real-time systems.
Confluent: A Business Around the Belt
Confluent was founded in 2014 by the same engineers who created Kafka at LinkedIn, following the now-familiar pattern of this era: take a powerful open-source technology that is hard to operate, and build a company that makes it easy, reliable, and enterprise-ready, then sell that as a managed service. Confluent has raised substantial funding and established Kafka as the de facto standard for streaming data. Like Databricks with Spark and the Hadoop vendors before them, Confluent monetizes the convenience and support around free software rather than the software itself.
The same logic applies: the open-source project drives adoption and makes Kafka skills common across the industry, while Confluent captures revenue from the enterprises that want Kafka without the operational headache of running it themselves.
What It Means for the Market
Streaming data is adding a crucial dimension to the data landscape and reshaping what a complete data platform needs to offer. For now, the world splits roughly into two camps: batch systems, the warehouses and lakes where data sits and is analyzed after the fact, and streaming systems, the Kafka-based pipelines where data moves and is acted on instantly. But customers increasingly want both, and they want them to work together. This is pushing the major data platforms to absorb streaming capabilities. Databricks has built streaming directly into Spark, so the same platform can handle both data at rest and data in motion. Snowflake is adding features to ingest and query streaming data. The line between batch and streaming is beginning to blur as every serious platform races to handle both.
For the competitive picture, Confluent occupies a distinct and valuable niche, the movement of data, that complements rather than directly opposes the warehouses and lakes, though as the big platforms add streaming, the territories begin to overlap. Confluent’s success also reinforces the dominant business model of the era, commercializing open-source infrastructure, the very model that produced Databricks and so many others. The benefit to customers is the ability to act on data the instant it is created, unlocking entire categories of applications, real-time fraud detection, live personalization, instant operational monitoring, that simply are not possible when every insight arrives a day late. Confluent and Kafka are teaching the data world that some of the most valuable data is the data that is still moving, and that lesson is working its way into every major platform.