Databricks Serverless: Cost Control + 5-Second Warm-Up

0:01hey host way I'm pleased to have you back again always a pleasure I think now we well known for the uh the orange hat you're wearing yeah yeah it's it's it's it's now it's now my uh my trademark apparently so it's become my trademark so so keep it as well as your your

0:24awesome s data uh yeah my t-shirt that that is cor yeah no thank thank you so much for for having this uh conversation with me man yeah I mean when I posted about like on the hour on LinkedIn that's I'm going to have a chat with you about seress workflow I had a bunch of like message like okay when it's going to be released

0:46and play any inside like make sure to ask like many many questions to hos I mean I think you're like aw well known now there is I mean you can't we can't miss your post in LinkedIn at all they're everywhere so awesome job thank I appreciate that um no it's again yeah it's great great to be here and

1:08obviously uh you know I I write a lot about Databricks Sometimes good sometimes heavy criticism but like I said uh before data breaks generally speaking listens to what I have to say and appre and I appreciate that which makes me want to help data brakes make a better product so

1:30it it's related to what I've said once that Databricks is built by the customer for the customer yeah because they help shape in the product we hear the feedback and then they like the product team and the engineering team do the their magic and make things happen you know I've

1:53criticized different companies in the past uh data breaks is one of the few that publicly taking on the feedback instead of trying to tell me hey keep it down they've said like no when you're writing this stuff it lets us know how customers are feeling about our product

2:16and if there's something that they're not feeling good about our product we should make it better so so yeah I appreciate that a lot yeah now you got you I've seen I've I've been following your post for I think for more than a year and I've seen like like how you provided feedback uh to many PMS I think I've seen so many

2:39posts to chiao hi Chia if you're watching this recording yeah chiao to Miranda the aiit I discussed about phobs with uh with Greg uh you discussed about AI functions with uh Seattle guy oh yeah with with Ben yeah he with Ben uh like I've seen like you talked with the

3:02with Jason merer about notebooks and how they can be be improved so I I I've seen you almost everywhere so uh kudos to you thank thank you man thank you man appreciate it and today let's talk about big our our topic uh let's say s SP clo and I don't know if you have tried them

3:25um yes what how was your first let's say user Journey uh have you noticed any change of behavior yeah so so two pieces that so in terms of like the experience with it I think has overall been good I haven't had to spend time setting compute up when running a

3:49workflow which feels kind of nice to me uh it's a lot less painful to get up and going not that the previous way was very painful but it just a little bit less knobs for me to have to deal with and I can just work on orchestrating things in itself so the journey to it has been pretty

4:11smooth um in terms of How It's progressed Over time I want to say I haven't seen performance improvements over time uh over time I mean since I've been running a job every now and then for the last three months and I haven't seen any

4:33performance gains costs have stayed just the same on those which that's a good thing so performance uh time to execution has been consistent costs have been consistent so that's that's that's good right that's on the server L side where I have seen improvements hasn't been so

4:55much on serverless jobs but more on the interactive experience I think when when it was released the warm up time was about 30 seconds I want to say then it got down to about 15 seconds and you know I tested it again

5:18like three weeks ago or so and it was just like I think 5 seconds I want to say 5 six seconds to warm up which is a pretty great improve I would say so a lot faster than I expected in interactive workflow so experience I will say and perhaps I'm getting ahead of some of the questions you might ask here is I am looking forward to having

5:44the option to optimize for cost as well as optimize for performance because I've seen a mix I've seen Mi mix feedback as to people's experiences with the cost tied to serverless jobs so I think that will be a good cost control for customers to

6:07have yeah yeah that makes sense because and I've discussed last time with one of the PMs and what they have released right now is the as fast as possible and it's the cost optimize will be here very very soon and I think this will be also a game changer because for the cost tracking what they are working on right

6:30now and again it's going to also be released I think very very soon it's called serverless policies where we can automatically put some tags on the clust on the uh serverless jobs to make sure that we have the right cost attribution and I don't know if you have seen it but at the same time we have built a dashboard called job uh tracking you can

6:53track the cost of the jobs not only the sance one but also the uh classic one it's going to tell you the most expensive runs because sometimes the job like workflow can be cheap but one run I don't know for some reason yeah something went wrong and then it can go I don't know $1,000 so you can see the most expensive runs you can see the most

7:15expensive workflows so this is this is awesome because you can go and try to optimize the code just for the most expensive one because this where you can have the main uh let's say impact and also you can see the jobs that are running with interactive clusters and you can fix this and say okay I no longer need to use interactive clusters I can switch to job cluster at least the

7:37the workflows that are not tagged which means you need to go and tag them to have this cost cost propagation and I think this is also something very very good to be able to track the cost in in I think hourly basis and so I think you want to have a surprise at the end of the month thing oh I don't know I did something wrong here no if you the right

8:00guardrails with the with the system tables with the U uh serverless policies with the budget aler team I think you have all the all this the features to help you have this maybe this observability on the platform yeah and yeah and you know I had had that conversation with with Greg that for

8:23those who don't know him he is the money guy uh in terms of cost observability with h a Databricks very awesome guy he you know him and I had very good conversation that we recorded and shared or let's to say we split it up into a few different videos and yeah

8:46I think data breakes is on the right path to being able to give that cost visibility just getting there is a matter of time and when you're the customer that time feels like forever right and when you are the one creating the product and modifying the product and building all this features making

9:09sure you're not breaking anything you feel that pressure right but going back to policies which um you know I got a little bit of sneak peek at that and I'm excited to see the direction that they're going to go go with it I think it's a matter

9:31of uh would you rather wait for the perfect solution to be here or would you rather start getting some value out of it immediately and I think their the ability to force tags is going to be a very important thing just honestly to help educate customers uh around topics such as fups because at the end of the day a lot of

9:53customers just think of how much am I paying they don't stop and think where is the money that I'm paying going to what's costing me money and really I think that's a problem that you really experience with any cloud provider and if databas is able to kind

10:15of push that education to the customers I think it's going to lead to happier customers with with the platform lead to less attrition of customers and honestly be a very valuable selling tool when you know you're out there prospecting against some of your competitors you can say hey do our competitors have this tools that help

10:38you track your cost that help youate your cost do have they taught you how to use it I think those are some of areas that data brakes really has a good potential to become known as the company that is truly working to help you deliver the best total cost of

11:00ownership and I think it comes with giving a powerful platform and giving the right controls and giving the right education to customers so that they feel like they're getting that whole package uh and that they feel that they are really in control of their bill instead of thinking hey here's my credit card and just going to run it as much as you want

11:22right no one wants to do it I would not give you my credit card be like hey here go I trust you um so I think again that's an area that the more cost controls and cost awareness that data breaks builds into the platform in a way that kind of helps us uh forceful education that's just going to be very

11:44impactful for for data brakes and its customers sorry no but that yeah but that that makes sense I mean who's better than data breaks to monitor data breaks cost I think it's that that is correct yeah and and it's and and I think there's there's a few different angles to right I think

12:06that that leaves the space for Niche Solutions uh outside of data brecks that can connect Databricks cost ultimately to company's bottom lines which you know cuz data breakes is going to try to keep the scope within everything that directly affects dater

12:29bricks so then the next step for other companies doing their own builtin reporting or organizations that help build reporting is how do they connect your Cloud cost to their p&l to their general ledger how how does this happen right

12:51and I think that's that that's a good Niche that I don't think that there's a few companies that I think are tackling that's going to be very complimentary to the work that data brakes is doing but if your business model is just doing just cost reporting you know on data brakes cost uh you know that that's not going to

13:13survive well and you know the pro the project that excites me the most is the um so they're working on Genie so it's on Improvement right now to make sure that you can use Genie with system abled to ask you mean a i i Genie too a hi sorry chiao yes

13:35AI yes aibi Genie because you know with system tables uh so to be honest it's sometimes I I try to be in shoes in of like a product manager Databricks and it's so hard because as you know system tables are shared using Delta sharing and in Delta sharing just I think six months

13:57ago we couldn't share metadata now it's possible we can share primary key foreign key comments which means now we are able to share the comments from system tables and add some comments on on the uh tables we have in within system tables and those comments are going to be very comments yeah and and this one can be very useful because gen

14:20cannot access the data within your tables but you can understand the the comments and based on those comments then you can ask questions like I don't know uh last month uh for uh model serving uh how many dbus did I consume and now it know that okay the skew is the different SKS are in this

14:42specific column so and this is how I can query it this is the this is the the filter I should I should do like go to select star from I don't know count uh from I don't know table usage and and then filter by skew model serving or model Ser like model suring and it's going you will have the answer so it's really not only democr not only making

15:05sure making like the tracking of the cost easily but you can also use Genie to ask those kind of questions if the dashboard that we are sharing with you or the dashboard you have customized is not good enough or you still need have some Niche question to ask yeah and and I and I love that I think yeah because I think the just giving customers here's

15:28the system state have fun uh that can be a little bit overwhelming for those that are not very familiar with those system tables right and ultimately you're trying as a company you're trying to make things easier for your customers so if you're saying hey you guys have to explore a lot of this stuff uh that means you're taking away from your customer spending

15:51time on their data and instead spending the time on understanding your platform which I don't think you want to be that you want to make things as seamless as possible for customers to focus on what really drives value to their business okay so go going back to one of the things that I had actually

16:13written about some time ago was just around this right like help customers focus on their data not understanding the systems that you have because you know Databricks knows the data very well you probably have essays

16:35that have built this and solutions Engineers that have built this tables and reporting over and over again so why not share some of that with customers so I'm glad that that's the direction that datab bricks is going with just hey just just ask the question through aibi so Genie yeah 2.0 and and and yeah so just and so this is something they're working

16:58on it's not there yet but you can customize it if you want like they can go to instructions and give some details to help Genie understand those tables so you can do it on your own right now or wait quite some time before having something maybe similar and yeah also some like you mentioned like sharing like on on the Shelf dashboard so that's now we have the if you go to the um budget tracking you can see like you can

17:22click on a button that's automatically deploy a dashboard for you as I mention like to track the costs and now we are building as I mention like job dashboard already available in GitHub so I think they will find a way to add it in the in the UI and I think they are they are building more and more uh dashboards but there is something cool like I've seen some customers using uh

17:46you know system tables to track the tables that are updated but not used and then based on this they can track the jobs that update those tables and see maybe some some some I don't know how to say how to say this but uh Phantom jobs you know jobs that are not doing anything but just see if anyone notices that this the data is on exactly and

18:10then you can just stop those those jobs and base on the job ID and the uses you can track the cost and then you can see how many debut or how many money you did save because of those system tables so so that's that's that's the beauty of data you have something built on the shelf for you or if you want to build some custom stuff based on those tables you can do like a lot of interesting things and this is this example was

18:33really amazing I've seen like people doing some cool stuff like okay how many tables were built I don't know last week uh the size and we have bunch of other system tables coming like model serving and others that I that I think will be unveiled within next quarter so the goal is really to have everything monitored

18:55through the uh our system tables now this exciting I'm sure our customers to it at cath LA to I think they're going to appreciate that too so yeah and and one more thing have you tried the um uh no I think it's the same thing I was I was about talk about DLT sever but technically the same thing so there's no

19:17I I I I don't think you want me talking about the LTS okay so okay so it's okay we can talk about it DT serverless it seems to work well which is a good thing the like the concept works well dlts are still broken in my opinion

19:44and by broken I mean just for clarity they're not a good option for batch work that requires mid level of complexity or higher they are very difficult to develop with and the documentation is not so great but that said using DLT serverless

20:09at least the compute part feels nice and simple which I appreciate that just to like to reply what what you said so this is this is my point of view regarding DLT so the with we had some ideas and and they have been de developed in such way but

20:32now they are they're fixing let's say some issues with with by let's say building a unified experience like with this streaming tables and materialized views and unifying this because you had like a lot of keywords now the goal is really to have just two keywords if you want to inest data streaming table if you want to update stuff then you have

20:56uh materialized views and with this enzyme this incremental um just modifying some incremental rather than doing a full recompute it's going to be let's say amazing and and then just something because in the past you have some like live keyword you need to add live keyword now we are getting rid of this and we are having something called publish direct publish mode which means

21:18you can select the catalog and the schema where you can write the data because in the past you were obliged to stay within one schema now things are changing so we are really removing all those uh all those limitations and same thing for the da experience like uh in the dev experience you need you needed to run the pipeline on uh Dev mode and then run it wait for

21:42this wait for and we know it was a little bit hard so then they fix this and now you can you can test the pipeline within uh within DLT if you are writing for example with SQL you can validate it and then s you exactly exactly and then you can publish it straight from the validation to create a pipeline from the UI so we're really

22:04fixing uh all those tiny tiny pain points and and just for information like most of the systems like tools we are using are basically backed by DLT if you can think about s like Delta like system tables mainly it's Delta life tables that are behind scenes if you could think about I don't know in materialized views this is DT if you're talking about

22:28um uh how do you call it Vector search when you do the inje again it's DLT if you talk about Lake flow the connectors you can to have it's DLT so all the everything like the data BR is using backed by DLT so this is a product I think that data breaks believe a lot on it and for those who haven't watched the talk yet we had an awesome talk with

22:51armburst I think few weeks ago who and where he talks about the future of DLT and and where we are going and he talks he talks a bit about the points that H mentioned mentioned yeah which by the way I've heard the name so many times I know who he is I've actually never gotten to meet him he's like one of one one one of the guys that I know is always just heads down building stuff

23:15but uh maybe someday I'll actually get to get to meet him which next dayit yes exactly hopefully y I was going to say okay so going back you know at the beginning you said oh how my content has grown and all that fun stuff your channel and everything you're doing um has grown too I mean you've

23:37had um you with am Herz uh you've I think you've had it with rain too right yeah you've also done mate yes who by the way he still has never accepted my LinkedIn request and feels a little personal but that's okay uh probably for his sanity it's really

23:59good thing that I haven't that he hasn't accepted it um and I mean you have the wne up coming with Adam and some on on in December right which I don't know if it's something you're publicly listing out yet but I know that's com not not but you the point is you have more than a few of them that uh I think

24:21are very high profile so congratulations on on that part too man no to bonus but really just trying to so the goal of data braks is democratizing access to data and AI my goal is really democratizing access to data breaks like uh point of view or Vision to

24:43everyone because I've had this chance because I work for the company to um to be amazed by this folks like who are shaping the future of data brakes with what the customers are representing and I want to just share their Vision with the folks say okay you can now have the opportunity to ask questions about some stuff and like be

25:05connect like like feeling connected with with those with those brilliant mind and that was the and to be honest I if you have told me like one year ago that's maybe I would have interviewed ma Reynolds or I don't know having PMS it would have been I would have told you no it's impossible you know so I'm

25:27I'm really happy to be here and having this this chance to talk not only with break sters but we're opening to other uh non databas folks like like you hosway but we talk also with many partners with I think DBT with did with did Salesforce uh um uh Rivery and a bunch of other partners so the goal is really to democratize say Hey listen you

25:50can use data brakes but you can use also other tools within not only with with databas but other tools that can interconnect and the goal is really to make sure you're getting the mo the value from from your data and that's really the most important thing that makes that makes sense man so tell me so so when so when are

26:12you interviewing a Lee that is the question good question I don't know let's see what the future is holding for us but I I have one question for you if you had maybe one tiny tiny thing to modify in the platform what what would you what you

26:34prefer to have something something you think may be missing or maybe need to be improved a little bit besides dlts uh I I'm am I'm hoping that aibi dashboards continues to

26:56grow at first I don't think I was very convinced that it had really a future to be honest but just given how the data landscape has played out especially like you know with powerbi doing you know Microsoft doing fabric L powerbi And then trying to bundle all this other stuff into on

27:20the data engineering side uh I think it makes sense as a move for data breaks to have a strong dashboarding experience and I think that right now you have the basics right now but taking it to the next level when you're competing against

27:42so many other companies that do a very good job with it too such as powerbi Sigma and I'll I'll even say Tableau I'm not a fan of Tableau but still it's in a better place than compared to aibi dashboard so I'm I'm I'm excited to see what the future holds on those

28:05regards and I'm hoping that Databricks invest in um making those better and I would say making sure that it tailors that experience not with the mindset of we're doing this for data Engineers but doing it for the bi developers that are not used to some of this data engineering

28:27tool tools cuz those are the ones that are going to really embrace it if the development experience is awesome and I think the text to chart or however it's called I think they do a very good job I think the team does a very good job with using that feature which I think is hopefully going to enable adoption and the other point with that is how can you build a data modeling

28:51experience inside of a IBI dashboards that just works well with everything else in in data breakes too I will say uh one more thing that I'm excited to see is Lake flow I'm excited to see how Lake flow plays out and seeing if especially how it plays with dlts and how how if at all is able to simplify

29:15the the you know the development experience especially L flow transform which I'll I'll I'll be part of this technical Council on Lake flow uh so that's I'm excited for that to help with that part but I I'm I'm excited and eager just to see how it helps Drive adoption especially from those individuals coming

29:37from like an analytics background that are not data Engineers um so let's see how how that plays out I think Bilal Aslam will answer this question maybe next month we're gonna interview him about Lake flow and the future of Lake flow and the uh where he going to talk about the upcoming connectors we're going to have how how potentially they have been implemented how it's going to

30:00work and what's the future of this uh this new let's say features within the platform I I I feel bad for below I'm always bugging him all the time with stuff I think but it's always it's always um interesting to have those kind of feedback from the from the customers to make sure that we are building the

30:22best product that's awesome yeah thank you thank you Jos was a a real pleasure to have you again thank you so very much man appreciate you having me here