(AI-created at Shutterstock)
Databricks today announced its intent to buy Neon, a database startup founded by Nikita Shamgunov that develops a serverless and infinitely scalable version of the open source Postgres database. Databricks, which is reportedly paying $1 billion for Neon, says it plans to incorporate the database startup’s scalable Postgres offering into its big data platform, thereby eliminating the need to scale separated server and storage components together in response to AI workload spikes.
We have come a long way from the bad old days of co-located compute and storage in big data architectures. Nobody wants to be forced to buy additional compute they don’t need just to support a growing database or file system, just as nobody wants to be forced to buy additional storage they don’t need just to support temporary increases in CPU or GPU power.
The separation of compute and storage in modern cloud architecture has addressed these concerns. Customers can dial up compute and storage separately. The big cloud providers also provide customers with an abundance of big data, analytics, and AI services to run on that data, which has spurred an incredible amount of growth in cloud platforms.
Postgres is all you need
Despite the huge success, cracks have emerged in the new stack, particularly around the need for human effort to scale up storage to support burgeoning compute needs. This is what is driving Databricks’ interest in Neon.
Databricks cites a statistic from Neon that says 80% of Neon database instances are provisioned by software as opposed to humans. “Agents operate at machine speed and traditional database provisioning often becomes a bottleneck,” Databricks says in its announcement of its intent to buy Neon.
The solution, according to Databricks, is a serverless relational database that is capable not only of being provisioned in 500 milliseconds or less, but also can be provisioned programmatically in response to the needs of AI agents, as opposed to being provisioned manually by human administrators.
The separation of compute and storage still is maintained with Neon’s serverless Postgres database. Databricks spells out that fact in its announcement: “Neon’s full separation of compute and storage keeps the total cost of ownership for thousands of ephemeral databases proportional to the queries they actually run.”
Instead, Databricks is looking to Neon to help “remove the traditional limitations of databases that require compute and storage to scale in tandem,” Databricks states in its release.
Nikita Shamgunov is the CEO and co-founder of Neon
It’s all about responding to the changing demands of the database market, which is being driven by the proliferation of AI agents, Databricks CEO and Co-founder Ali Ghodsi says.
“Neon proves it: four out of every five databases on their platform are spun up by code, not humans,” Ghodsi stated. “By bringing Neon into Databricks, we’re giving developers a serverless Postgres that can keep up with agentic speed, pay-as-you-go economics and the openness of the Postgres community.”
Shamgunov co-founded Neon with Postgres contributor Heikki Linnakangas and Stas Kelvich in 2021 to push the state-of-the-art in what Postgres could do. As Shamgunov explained to BigDATAwire in a 2024 interview, the company focused its efforts on solving the hard engineering tasks, such as separating compute from storage in Postgres and developing its own storage engine for Postgres that is 100% compatible with Postgres while enabling it to use S3 as backend network storage.
“What we’ve done is we’ve separated that storage and moved it into network attached storage that is custom built for Postgres,” Shamgunov told us. The Neon storage engine plugs into Postgres at “an incredibly low level,” which is a key factor enabling full Postgres compatibility.
Other cloud vendors, like Google Cloud and AWS, also have serverless Postgres offerings. However, those serverless Postgres instances are not open source, Shamgunov told us in 2024. Maintaining that 100% compatibility with Postgres open source eliminates the need to make any code changes to applications to run them on Neon, which Shamgunov said was the main driver in creating Neon.
Terms of the deal, which is subject to customary closing conditions, were not disclosed (although the deal was reported by the Wall Street Journal to be valued at “about $1 billion”). Databricks says it expects the Neon team to join it soon. More details will be shared at Databricks upcoming conference, the Data + AI Summit, which is taking place in San Francisco June 9 -12.
Related Items:
Neon Looks to Light Up Cloud Postgres Market
Databricks to Raise $5B at $55B Valuation: Report
Neon Raises $30M for Its Postgres as a Service
