Real-Time Analytics on Lakehouse with Reyden

0:04You mentioned nice to meet you again. >> Good to see you again. >> Yeah. Tell us more. What are you doing at Databricks? >> What am I Well, it's my sixth year at Databricks, and this year I'm busy with Photon and Lakehouse RD. We also have a lot of other exciting announcements. We just GA'd managed Iceberg. Uh we have now gated public preview of disaster recovery. A lot of

0:26customers have been very, very excited about that. Uh but uh yeah, on the DBC wallet on the data warehousing side, the biggest thing we are very excited about is Photon, our new engine. And then Lakehouse RD is the new compute form factor that Photon has now. Yeah. >> So, tell us more about Photon. Like, how is it It's It's so good, it's unbelievable. How like, can you give us some insight? Like, what is it? What is

0:49How is it working like a little bit under the hood? >> Yeah, sure. Okay, yeah. So, yeah, it is It sounds like magic, right? Like, it's so good, yeah. So, a couple of things. I think the first principle was really like rewriting everything from scratch and starting from first principles in a very Databricks manner. Uh so, you know, like taking advantage of the modern hardware. Over the years, instances and the hardware from EC2,

1:11AWS, Azure have improved significantly, right? But a lot of these engines and systems that were written, let's say even as long as 5 years ago, right? They don't take advantage of the latest SIMD innovation, for example, or the latest bandwidth. The pipes between the instances and object storage has become much bigger. So, they don't >> optimize for all of that. >> So, we just rewrote the entire engine

1:33from scratch, took advantage of the modern hardware. Um no Java overhead. It integrated natively with Photon. >> It's still running on Photon? >> It's still running on Photon, runs in Photon. So, it still uses Photon, but it's natively integrated. So, no overhead anymore, right? Like, extremely fast query performance. And then, we are caching everything and anything that we can cash. And so, it's like from the

1:55data to metadata to query, everything which are cached. Um so yeah, those were the things that made us uh really fast. The other piece of innovation we are doing is on the compute side as well. So up until now, any warehouse, when you scale, it will scale in t-shirt sizes, right? So it goes from one warehouse to second warehouse to third warehouse, so 2x, 3x,

2:174x, right? And with Redshift, and what we are doing is we're doing incremental auto scaling. So we are adding nodes, so you go add one more node, and two more node, and three more node. And then we scale um out and and out, you know, like so we de-scale also in the same way. So when we are going back, we remove one node, another node, another node. >> So you guys improved the cuz we used to

2:39have like in Spark like the driver and the set of, you know, nodes to run the computes. So you improved all of that, and now you can have like within one cluster you can spin everything dynamically. >> Exactly. So it's like a lot of innovation happening, yeah, on all these on the engine side, on the caching side, on the compute side. So that's what makes Redshift fast, and I think you saw the results from Redshift today. 10 milliseconds, customer themselves are

3:02seeing like 10x better performance, uh 12,000 queries per second. >> That's crazy. >> Um you know, and at very low latency. So yeah, super excited about >> I know the like the next question, and I know folks are asking that like it's how is it compared to like base like to Postgres because like it's getting so good like which should I which one should I be using? >> Yeah, so that's a very it is a question that has come up often which is "Hey,

3:25when should I use lake base and when should I use uh Redshift, right?" And it's funny, so let me set the premise of like what it what what what is happening today. Today, the OLTP databases are being used in two scenarios, right? The first scenario is they are actually being used as >> Let me just >> Oh.

3:47Sorry, sorry. The first scenario is they are actually being used as a system of record. >> Yes. >> the OLTP for the T piece of piece, right? The transactional piece. Where you are reading and writing lot of data and your app you're running an app which is like for example updating a record. You just updated a phone number and you want to immediately go back and read

4:09that phone number, right? That's the use case for which OLTPs are made of, right? What made for? That's the use case. The second use case where customers are using OLTP a lot is they're using it as actual a serving layer on top of the analytical warehouse because analytical warehouses are failing to meet the latency demands that

4:32application needs. >> right. >> Right? >> Yeah. >> So a lot of the customers what they are doing is it's so funny that they are taking first they bring the data from the OLTP into the lake, they do some ETL and they do some, you know, processing and then they bring it back to the OLTP to serve it, right? >> Yeah. >> And they're purely using it as a serving layer. And OLTP is only one example of this serving layer, right? The other example of serving layer is extracts,

4:55right? Customers are doing extracts and they're putting this data in the Tableau or Power BI. Right? That's the extract. And then the third version of this is like customers are using these real-time systems. There are a lot of open source projects like Apache Druid, Apache Pinot, there's ClickHouse, all these open source projects, right? It's all the same thing. They're trying to solve a problem

5:19because current warehouses do not meet the latency requirements. These are all the solutions which requires you to create a side stack, right? Now you have created another stack. And that was the idea. So now with Raiden, you don't need to create a side stack. If your use case is you are doing analytics and you are doing aggregations or averages or median, right? Any kind

5:41of analytical query and you are running it and you all your requirement is low latency, Raiden is the answer. >> And if you are doing transactions >> and you keep all the data at one place and you can serve it to >> We just use lakes. The data is in Delta or Iceberg, right? If you are If you are doing OLTP transaction kind of workload, system of record where you are reading and writing

6:03and you're updating your name or information or whatever and you need to read it immediately, credit card transaction, that OLTPs are made for, right? Yeah. Or for example, agents, agents storing their states. >> Agents spin up lot of lake-based databases, right? >> Yeah, agent memory. That's also because it's not massive amounts of parallel data. It's not that and the requirement there is very fast access to this

6:26row-based data. And that's what the OLTPs are best for. So, to summarize, Rayden is for analytical data to serve at very, very low latency, [music] aggregations, averages, any kind of analytical query. And OLTP lake-based is best suited for serving row-based data for very, very high read and write throughput. >> All right.

6:48So, can we try Rayden now? >> You can try Rayden now, yeah. If you are a customer, please go and come and talk to your account team. We have the beta going on right now. We'll be opening it up very, very quickly and public preview is planned for end of August. So, yeah, you can try it now and please let us know your feedback. >> All right. Well, thank you so much. >> In lake house, Rise data sees lake flow

7:10strings the flow. [music and singing] Iceberg and Delta lake tables aglow. Unity catalog governs lake-based OLTP strong. Genie takes to SQL MOSAIC AND KEA PROLONGS. YOU sipping Quip [music] and co-host like the fire. Genie dreams scale higher and higher. [music] Next gen. Drop it.