Lakebase: 19,000 Transactions Per Second, 4.56ms Latency

0:09Hello. Uh, I've been given the job to manage inventory levels for a drinks company, making any last minute adjustments to stock, but also to share data between analytics teams. This job is tricky at crudtime, but fortunately, I have some new tools from Databricks to help me. And today I'll be sharing how Lakebase powers Databricks apps

0:31works at scale and can use data from delta tables all in real time all in one platform. So I'm going to switch to my demo and in front of me I have a data bicks app that's bringing together both operational and analytical data. Whenever I'm changing these filters these are live queries. Nothing is cached in the app. I can see here orders that I need to review but I am not going

0:53to do that. uh instead I am getting some intense requests to place some orders for cherry burst for 90 units. I'm going to select my store, click through the app and go. This app will go and update Postgress in the back end where the orders table sits. Now I'm going to double check that this has gone through correctly and I'm switching to the brand new Postgress SQL

1:15editor natively integrated to the Databricks platform and I can tell it's Postgress because I've got this icon on the top right here. Um, and when I select my compute, it's got this Postgress tag next to it. I'm also not going to do this on production. Uh, so I am going to use dev, which is an isolated test branch. And if I look at the left, I can see some neon details. Um, I'm also going to check the settings

1:37of the version. And I can see that yes, this is Postgress version 16.6. And to query the data, I would just query it like I would any other table. And I can see here that when I run this that yes, my order has gone through. But that's just one record and that's not very impressive. So earlier today I set up a little program to simulate lots of users to run in the background. Um I'm going

2:00to switch to the native performance monitoring in Databricks. I'm going to head to this metrics tab here. Um and I can see how this has ramped up to thousands of transactions a second uh and tens of thousands of rows per second. And I've got all sorts of interesting operation types happening here. One thing this doesn't have is our app latency as it scales. And fortunately, uh, I bought my little

2:23program, uh, so it can measure latency. So, I'm just going to run a simulation and see how well we're doing. And, oh, I can see my performance is almost 19,000 queries a second. Uh, with the median at 4.56 milliseconds and the 95th percentile at 5.6 milliseconds. So, I'm pretty pleased with that.

2:46Thank you. Lakebase can also access data in delta and I can combine them together for snappier responses in the UI. So I head to the table in Unity catalog. I can create a synced table. I go ahead and I can name the table. I select which instance I want to use. I want to use production. I can also choose the database as well. And then the primary

3:08key. Now I can either run this as a one-off snapshot triggered at a certain interval or have it continuously updating. And I can either group this with other updating items in a pipeline or I can have it as a standalone. Now that that's set up, uh when I go to it in Unity catalog from the database view, uh I can see here that I've got this

3:30little synced icon next to my table. And so now when I head back to my app, oh here it is, and I head to my insights tab, I've got this insights tab, and that is including data from a delta table for my forecast. And of course, I can sync all of my orders uh data back to the lakehouse for historical analysis and the other way around, too. So in

3:54this demo, we've shown how Databricks is bringing together both operational and analytical data closer together. And I've hope I've shown you how powerful this is when combined with other Databricks features like apps, delta, and unity catalog. And it's inspired you to give it a go on your next project. And as for me, I don't think I like being an inventory manager, and I probably should have made this an agent.