Databricks DAIS: Top 3 Announcements—Lakebase, Agent Bricks, One

0:04Hey guys, today I have the pleasure to host. I don't think I need to introduce him. He's already a superstar. You can see his post everywhere on LinkedIn, on YouTube as well. And he's back again. Hi, Jose. >> Great, great, great to be here. I appreciate you having me here, man. >> Always glad to have time to discuss. I don't think we didn't have much time

0:27to talk uh during the data and AI summit despite having this incredible presentation together where we talked about AIBI dashboard and GI and remember we had almost 500 people full room >> that was it was a pretty packed room. Yeah. I mean and honestly I think you you you you nailed that that you did such a great job on on the Genie portion. And of course you handled the

0:50AIBI part and also dashboarding talked about Sigma and other dashboarding tools. You're here because we wanted to talk announcements and which advantage they can have for developers, engineers, data scientists and I wanted to have your opinion on what was the key announcement for if you couldn't choose only one and then of course we will talk

1:12about the others. I know this one is tough is a tough question. >> Yeah. Well, tell do you mean to start off with uh with which with which one was the biggest announcement or that from my perspect >> so in that moment I think it was Databricks one because I've worked before at places where not having a business

1:34userfriendly interface for a lot of the hardcore developer work that the team was working on was a big pain and it led to like oh we have to have this uh front-end portal we have to do this we have to do that we're sometimes doing that would be a little bit overkill and with Databricks one the premises hey you can have build your genie spaces you can have your AIBI dashboards you can

1:57have your applications serve for your business users without really having to invest time in that uh businessfriendly interface so I think that was a big win for that I will say interestingly enough lakebase is probably the second one except at the moment that lakebase was announced I wasn't really that excited. I thought it's like oh great we got

2:19Postgress and which is a good thing but I was a little bit like ah a little bit like I'm not I don't care so much about it. But as I got to spend more time researching and really understanding okay this is this is the potential that it has and how does it work with the Unity catalog overall I I'm I'm excited for for for for lakebased probably. Uh that's the second spot right there.

2:41>> And okay, let's go for the third one as well since like you know whenever I have a podium it's like three places. >> Well, I think third one I think agent bricks except to be honest that thing should be called Databricks agents not agent bricks. So I'm going to keep on calling it Databricks agents but I think that it's an area where Databricks needed to catch up on and because

3:03you know you have the Databricks playground which is great for building if you feel very comfortable around coding and I think that agent bricks or Databricks agents uh makes it easier to build with without feeling overwhelming uh without the code feeling

3:25overwhelming. >> Yeah. And I think for this year, again, this is my my personal opinion, Databricks on the um edge of pushing for simplicity and ease of use. You mentioned the datab bricks one which is a very cool and simple interface where you can interact with your gen dashboards apps bunch of other things are coming there but also you mentioned

3:47the agent bricks because we know that bricks has a lot of awesome features like vector search putting them together requires some knowledge. >> Yes. >> Why for agent bricks it's mainly just drag and drop and you're good to go. like you just choose the right uh agent whe they're like going to extract some analysis or maybe do a lot of things. This is I think cool plus of course the

4:10announcement of like flow designer as well. That's why I wanted to mention like this ease of use like you can build your pipeline quickly. I've tried it few few days ago and it was just amazing like you just need to drag and drop give like it will build some SQL code for you filter changing branches the experience is just amazing this is a post gray

4:33manage pos because we're we're decoupling the storage from the compute adding the branches things you can't do easily if you're using just regular postgra and also this uh instant startup this This is super cool because at the end you'll be only paying for uh for what we will be using and integration of Apache iceberg within the platform like since

4:56the acquisition of tabular we were waiting what would Databricks announced and we had it finally. I'm just so happy about the two things. I'm very happy about like the whole iceberg thing mainly because I wanted that whole conversation about Delta versus iceberg to just die. I'm so over it. My my official stance is that most businesses don't care about ice, whether it's

5:19Iceberg or Delta. Some do, and I have respect for that, but most don't. So, I'm very happy for that to just be settled. So, it's less of a vendor A or vendor B saying, "Hey, we support this, but we have more support, but we have more support." I kind of I'm I'm over that conversation. So, I'm excited for that. I will say Lake Flow 2, um maybe that's perhaps my fourth. And the

5:42reason is like you know with lake flow obviously I was a longtime critic of delta life tables and I think lakeflow has addressed about 85% of the things that I didn't like about delta life tables uh now called spark declarative pipelines so I'm happy about that and designer too I was part of the technical council uh on lakeflow and so I got to

6:07see even some of the early drafts of designer so I'm very excited to see it all go through through the end and just see see that shift from Databricks from being known as just the platform for highly technical people to being the platform that's both for highly technical people as well as less technical but still knowledgeable folks do so >> and since you mentioned lakeflow

6:30pipelines or what you call lake ldp so I think one of the first or mo most important things is how uh ldp or like uh spark declarative pipeline is now open source and will be part of Apache Spark 4.0 if I'm not wrong. And the second thing is that as you mentioned the experience has been completely revamped. I've tried DT since it was

6:53released uh maybe 3 years ago if I'm not wrong and the new experience is completely different. First of all, you have this deep integration with uh with Unity catalog. You can have files that can define whether you will you're going to build like be building a pipe python pipeline or SQL pipeline whether it's going to be inest or transformation and

7:15while building your pipeline like those tables are created for you and you can see the pipeline in the being populated for you. You can click on the table it will show you uh a sample of the data the constraints the quality you did built on the top of this specific data. we can uh deploy uh like data brick asset bundles from the workspace within

7:39the same pipeline. >> Yeah. And it really and really I think it's about that developer experience because that's what the Delta live tables lacked and when I say 85% of the things that I had problems with have been addressed and a lot of it was yeah that you know you have oh you had to have with Delta Life tables you had to have multiple screens to get anything done really. So what was supposed to be

8:02a unified experience was really a disperate experience. I think Lakeful has made great strides in improving that um you know with the error handling with really the organization. So I'm I'm you know I I went from telling people don't don't do anything in Delta Life Tables unless you really have to saying hey I think Spar declarative pipelines is the future. There's still some things

8:25that I have my disagreements on, but it's good enough now that I think that it's going to continue to get better in the future. So, I'm excited about it. >> Yeah. And for those who are interested there, we will like leave this des in the description of this video. There is a session that was done at the data and summit going through all those uh like innovations explained in 40 minutes by

8:49the PM of uh lakeflow declaration pipeline. And and what about uh DBSQL? What was your uh maybe key takeaways? >> Well, as as you know, I like to do a lot of testing uh on performance. Uh I mean uh honestly there was there was little to me that was new about data be that about DBSQL uh at summit just because

9:11you know I do so much testing. I've seen the progress over time in terms of performance. You know, I had recently done some testing that I published the week of uh Databricks summit. I I I was surprised in a good way. I was pleasantly surprised as to the cost to performance that Databricks SQL

9:34offered. Not so much at the big data size or you know like the 5 billion rows and and more. Uh I have seen that before from from Databricks. So I knew that Databricks was probably I expected Databricks to be the most affordable option there but when it came to the smaller data sets Databricks did very very well and it it was a very pleasant surprise so I know there were what like a 20%

9:58reduction in in in performance cost that was announced >> about AIBI maybe let's first start with with dashboard then we can move to chini >> yeah so I think there are two sides I think it was very important for me to See, one is obviously Genie I think was the show stealing thing. As I spoke

10:20to many many folks, there were two things that stood out to them in terms of excitement. One was Databricks one especially for those that have been around Databricks for a while. For those folks that were newer to the Databricks ecosystem and even some that are just beginning to try out a few more capabilities, Genie was overwhelmingly one of the topics that came up the most

10:43and it was and it was very nice to see a lot of people really excited as to what it can do, wanting to understand what it can't do and I think you know as I would have conversations with folks as to hey this is what it can do the reception was very very good. Um, I think again I always go back to Ken Wong's uh statement as to he calls Genie an intern

11:07that you slowly train to know more and more and more as to how I should present the data, what types of data you care about and you know you go from intern level knowledge to junior analysts or a little bit more than that. So it's pretty awesome stuff and AIBI dashboards. I've been a hard critic before of dashboards and I've seen it grow over the last year,

11:29year and a half and I think that just a few days ago I think there were some new announcements that I haven't gotten to try out yet and I see the team continued continue to develop that still hungry to make it a better product and I recently put out a post to B you know saying hey this is where G this is where AIBI dashboards is at today this

11:51is what it needs to do to compete with some of the other BI tools out there. And his this is what it needs to do to set itself aside or be the creme of the creme when it comes to dashboards. Again, is it a full-blown replacement for other BI d for other dashboarding tools? Not in every scenario, but can it be a replacement

12:15for half of your dashboard needs? Yes. If you're a smaller organization perhaps that you know you don't really have that many externally facing dashboards I would say it could potentially replace most of your dashboard. It can be the right tool for most of your dashboarding needs. So and also this is part of what I was saying at the beginning. Simplicity is the key word because you

12:37know that we did uh or like Databricks did announce UC metrics where you can define your your semantic at the at the catalog level and then you can integrate those metrics with Genie and that's this is the cool thing is now you can define a lot of things using the UI which means no longer need to go and ask how I can join those data together you can now just go and declare the joins on your

13:01table to enhance genie you can add synonyms. You can do a lot of things just using the UI and all your business users and you will also be able to build the dashboard and integrate them in the datab bricks one and again u there is no additional cost just the compute you're sure that they won't be doing anything wrong with the data because they will only have the read only permission

13:25>> thanks to unity catalog since we mentioned the catalog what was your main takeaway from the announcement >> really the biggest thing was I was looking for that iceberg integration to be better with the unity catalog because you know you bought what's the name of the company you bought tabular one year ago right and obviously you won't

13:47implement everything overnight and but I think that's what a lot of people were expecting but obviously that's an unrealistic expectation so I it was good to see iceberg becoming more and more integrated for those folks that care about that iceberg integration because you know for a little bit right iceberg was like the that you know second cousin that distant cousin that you had had you

14:10had it inside of inside of Databricks but it still didn't have necessarily that first class citizen experiences Delta because obviously Delta has been part of Databricks for a long time in iceberg more recently and you want to make sure all the pieces work together right so I think seeing it become more and more of a first class citizen I think that that was a big win and I

14:32think there were there were a few pieces on interoperability that I don't remember off the top of my head, but sounds like Databricks is spending a lot of time and resources in making it more interoperable from my conversation that I had with James Malone uh who is a product I think I forget his official title but he's a product leader uh as well as even even with David Meyer right

14:55who is SVP of product one of the things that they they both said to me uh in some of the interviews that I did with them is that they want Databricks to win your workloads so it has to be less of you can only use our compute, but how can we help you have the right governance while still allowing you to use the compute that is best for the workloads that you want to run? We want to earn

15:19the your we want to earn your money on our workloads. Basically, >> you know that they announced a lot of features with the Unity catalog, but I thought about the scenario that I think few years ago would take us maybe a lot of time to do it while now everything is automated. And I thought about like what if I have some customers data with

15:41sensitive PII, you can just enable like the data classification that will go and automatically scan your whole table. Right. Yeah. >> Exactly. and tag tag the PII and because of aback you had already implemented some masking functions with a specific tag you know for sure once your data is inserted automatically tagged and which means if I go and query the data this

16:04data will be masked for me because I'm not allowed to view those sensitive data and again because I keep maybe doing merges update drops on this data the prediction optimization will go and automatically scan my table optimize it, vacuum it, run the analyze to know when I query this data, I will have the most optimized query execution plan. And if I

16:27am not doing the partitions but clustering my table, I will find the right clustering keys. I I think I may have listed seven steps that I had to do in the past and now the system is handling those steps for me on my behalf. >> You can see is this really working for me for my benefit or not. >> Exactly. And this is amazing because I had to do maybe handle them maybe do at least those tasks maybe once per month.

16:49Now I can maybe have those seven hours back to do something else. >> It's it yeah I think again like like I shared it I think you you're seeing data bras move from the hey we're just pure technical we're only for technical folks to hey we will meet you where we're at. Um but you know we will still cater to the technical folks. We're still cater to the less technical folks and we

17:12will try to meet we'll try to create a middle ground as well for for everyone. >> DBSQL is getting better and better but I think uh data is also adding more features to match what people used to have on the legacy data warehouses and here I'm talking about store procedures maybe half of the people will hate me. I was going to say I I don't care for stored procedures, but yeah, sure.

17:36>> I mean, when you're doing migrations, don't want to reange your entire code. You just want to make sure that you can keep some of the functions you had and like having sort pursuers can be super super important. Cursive CTS can also be interesting. Temp tables, collations, like bridge. Yeah, I have one for you. As some may know, I

17:59have my misgivings with uh observability, right? And I know there were a little bit of uh announcements around observability. And really my misgivings with observability is that there can be a lot of fluff sometimes and sometimes there's a lot of very good real business value things around observability. So I'm curious to hear from your take. What are your thoughts

18:21on the observability related announcements at the data brick summit? Good question. I mean I'm happy because you know that I delivered the session on observability during the data and AI summit and I will put the link in the description. Over the past two years we did announce system tables and for me that was a game changer like in the past you had to enable something called

18:43verbos audit logs that was like a bunch of JSON file that you have to parse and push then store them on a on a delta table query them. It was like a lot of you had a lot of steps to do. Now they are just available for you a bunch of tables and in order to simplify this part Databricks they release a bunch of tables tables not tables ashboards that you can import and start

19:07tracking the consumption understanding the percentage of usage of your compute. You can find what is the most expensive workflow or the workflow that did that fails a lot or the one that runs the most. Those dashboards are not the only one. They also released some features which is super super important and tightly linked to system tables. It's called serverless budget policies

19:30because we know that people will now more often use serverless. But you need to make sure that those people are tagging their resources to understand at the end of the month or the end of the week how much this project or this use case cost me. And in order to do this, you had to have a tool because people tend to forget to add taxs with but but with the budget policies you can just say hey this group of people like

19:53engineers whenever they're going to create like or use a serverless instance it will be automatically tagged with their business unit. So whether you're using I don't know SQL warehouse a workflow serverless notebook or maybe a model serving it will be automatically tagged with with with your business unit. You you know what's funny is when when when you're talking about for example like budget policies I don't

20:15think so much of it as uh observability I I don't know if I if I would maybe call it governance which probably has related to observability but I I would say cost management uh versus the data preservability but sorry me picking here but I'm like okay for me that's a little bit of a topic anything cause governance I think you know that I'm very much of a

20:37hey cause know your cost, manage your cost, control your cost, which by the way, I'm I'm very excited about all the changes on budget policies, and I did a video on on that. I think it's very exciting stuff. So, >> what you mentioned is completely, it's completely true. You need to put those tags, then use system tables where you're going to have those tags being used and then with the dashboards,

20:59you'll be able to track uh to track the cost. If you want, you can build your own dashboard if you have time to do X the extra mile that maybe Databricks is not doing because you have some specific needs that those dashboards are not fulfilling. In the past two years, a lot of things has been added. And the last feature I want I will talk about is the alerts like it's called budget alerts.

21:23Again, we keep we keep always keep budget. And basically here what you can do using the tags that you have probably added you can go to uh your account console and specify an alert. You you will specify the budget you will put for the specific team. You're going to specify the name of the tag. Then you're going to specify the people that will receive an alert once you're above. So

21:45we have a bunch of guardrails to protect you from overspending and also if you're overspending how you can optimize >> some stuff. >> Did I convince you a bit? >> Yeah. I I I again I my I have just different perspective on observability. All the stuff that you're saying that's awesome stuff. I love all that. Uh but anyway that that's just going to be a mute point. I you know it just in you

22:08know in in data space in the data space you have so much you you can be using the same word and you can have 10 definitions depending on the vendor. One time I was actually talking with uh with with Franco and I think we're talking about semantic modeling and we had a

22:30brief conversation as to well it depends on who you're asking what that means and I think I even even a conversation that I had with Joe Ree too. We were also saying like what what does that really mean? >> Yeah. Yeah. I I I totally I totally agree with you. I think we did summarize almost all the announcement. >> Yeah. No. Well, yeah. I feel I feel like we cover good good good stuff. I think those are areas I think we feel very

22:53comfortable talking through it and I think it's good stuff. >> And for the folks who are watching us, make sure to subscribe to Jose YouTube channel. He does publish a lot of interesting things. I'm pretty sure you've seen them, but just put a thumbs up and subscribe. That would be that will be boost him to make maybe more content. >> Thank you. >> Thank you, Jose. It was a real pleasure

23:15to talk with you. looking forward to meeting you very very soon. >> Awesome. Thank you so very much again.