Cut Your Databricks Costs: The 3-Step Cost Control Guide

0:05hi folks we're back with Quinton and we're going to be talking about a very important subject observability and cost control in data breaks yep so it's super easy and we'll show you how to make sure you have everything to be able to track and check all your cost within the bricks at the workspace and a level so the first step is super simple you might

0:27not have to do it but I'm still going to show you what you have to check you want to make sure your system table especially the building one is enabl in the workspaces and the well at the account level so how to do that super easy you can just log in in any of the workspaces that you have so here I'm in my workspace you want to check in the catalog Explorer on the left you want to select the system catalog and then you

0:51want to make sure you have all the system tables being available especially the billing one because that's the one we'll be interested in so by default you might not have all the schema there in the catalog you just have to enable them and it's super easy I'm going to show you how to do it so you have a bunch of option my easiest option is just to use the small DB demo and system table that we have which is going to give you like

1:14a notebook prepackage with everything on top of that we will have nice dashboard and we'll show you the dashboard at the end so super easy just go with P install DB demos you can install the system table demo and then you will have a books being installed including the enable system Sy table notebook that we have here that one can just click on run all what it's going to do it's going to

1:36go and you know check all the system table and make sure they are being enabled within the account level so you have to run the notebook as a mean so that you have the permission to enable the system level the system schema and that's going to make sure you have all the system table being available and you will see all the billing table within your workspace

1:59that's it that's the only thing you have to do so go and log into the catalog menu check if you have all the billing if you don't have all all the billing just install theb demo run the notebook and you are good to go and step one done and now let's move to the uh to the Second Step the Second Step would consist on adding tags over all the

2:22computer the tags are very very important why because that's how you're able to track those costs the first option is to go and for and of course tag manually the compute whether you're using an interactive clusters going to be for example here tags going to be engineering so that's that's the first

2:44example like when you create an allpurpose cluster you can also tag the workflows like create a job then select your Tas and just go just go to tag and again same thing can do team engineering of course you can add as many tags as you want you can you can break down by Department by business

3:08units by cost center we have many options I talked about workflows but you can also tag SQL warehouses so going to create a secret Warehouse Advanced options team engineering you can also tag the model serving one so just come here again you have tags you are able to track the cost of all the Computer

3:31Resources but I know this can be sometimes overwhelming because you need to make sure that every time whenever a developer is creating a cluster or compute it needs or she needs to add the TX so that's why you can rely on for the classical compute something called cluster policies you can create them and

3:53you can for example make sure that you create this policy it can be either a custom or inter active or a job and as you can see here I added a custom tag called team and the value is data engineering and then I can just go and attach this policy to a group of people for example this policy will be attached

4:15to data Engineers so whenever someone from the data Engineering Group try to create a cluster this cluster will be automatically tagged and let me show you a quick example so I'm going to create a compute of course here I am an admin mean so I have all the policies but by default I would have have assigned this specific policy and if I go to automatically tagged you see team data

4:40engineering but you're going to ask me a question what about the new resources the new computer resource I'm talking here about serverless notebooks serverless workflow model serving SQL Warehouse the answer is budget policies and this one go beyond the uh Beyond the cluster policies why because when

5:02you using a cluster and you're tagging this cluster for example data engineering maybe this cluster is being used by three people how can you for example see the cost by for for every single person it's it's it's a little bit hard with this one you can just go and you need to go to settings you need to be admin of the workspace need to go

5:24to compute and then go to budget policies click on manage create the policy let's call it for example data engineering you can add the tag for example project data engineering and then I going to create this policy after creating this policy I

5:47need to assign it to a specific group of people so just need to go to the permission and then you grant access to a specific group whether it going to be a manager or a user once it's done so now if I go for example to a notebook and maybe I am using a serverless uh a serverless notebook what will happen if

6:11I go to the environment I will see that I have the budget policy data Engineering in case I have multiple policies attached to me I can select the one that I that needs to be to be applied here then of course I know at the end if I'm running some code then this cost will be assigned to my business unit which is or my team which is data engineering and same thing will

6:34be applied if I am creating a workflow using serverless compute so let me just select I don't know a random notebook uh let's call it test it's going to be serverless once I create the task you're going to see that automatically there is a tag added budget policy data engineering and the tags is data

6:56engineering and of course same thing will be applied to model serving again if you go here we have this budget budget policies data engineering and it's also coming to SQL warehouses where we can see budget policy engineering youf attached which means I

7:18can which this will tag all the different compute resources within my environment and by this I would have completed the second task yeah and the good thing with the budget policies is that it's going to track like all the comput at the user level so for example if you SE is running some some kind of comput using serverless comput in the notebook will'll be able

7:41to say okay that that run was specifically on yourself and you can track at the user level which is really powerful inde and that's the beauty of serverless all right so that was for step two so again classic computes you can go and add manual tagging and you can INF these manual tagging with cluster policies and for all the serverless computes you can use the new

8:04budget policies which are much more powerful and give you user level detail so once you have that you have the tagging all the tracking is going to be done automatically by theck for you and everything will go in the system table that we enabled and so now we can just use this information which is the last step to track the cost and just check what's happening in the like at the

8:26level I level or workspace level whatever you want to do so you have a couple of options and thing you can do first I'm going to show you the new budget alerts which is really cool and that's something you can have at the account level so let me log at the account level as like an admin of my of my accounts so you see all the information that you have on the

8:48left and the one we want to go is the usage menu and in the usage menu you have the new budgets path so what's a budget well you can create a budget and give it a name name so let's say for example to use use example data engineering so you say that you only want to get spend maybe 1,000 um USD in

9:10data engineering you want to apply that to all the workspaces that you have and will track the key uh was project so you have to make sure you match the tags you have in the policies and then the value you want to track data engineering and then you can say okay if something goes wrong send me an

9:32email so if I go above the limits and the amount that I have here yeah I'm going to receive an email saying hey something is wrong like you are over the amounts you prescribed in the budget please check it out something is not going as expected and that's it once the tracking it's going to fill up you will have nice datagram to show you

9:54the consumption and see what's going on apparently I don't have anything because it's a brand new budget but you know as as user for example is going to start some computes and the comp is tagged I will see that appear here I will be able to track everything so you can create as many budget as you want link the budget with multiple policies like you can be flexible based on how you want to organize your team so that's it for the

10:18budget and the aler thing yeah it can be for example just the head of each project as the person who will receive this email or maybe the phobs uh uh Team which will help them track the cost for every single project yeah and you can also plug the email to for example slack you know get a slack notification I something is wrong dat inuring is over spending please have a

10:41look and fix what's going on if you want to go into details and track you know what your teams are actually doing you can also use the pre-baked dashboard that you have on the consumption so you have the you know the button visit the new dashboard you can click on the button uh and that's going to land into the brand new dashboard within one of your workspace so the dashboard is just

11:04using the system table the one we enabled just before to track everything and show you what's going on at the level so you have a bunch of dashboard for example that one is showing you you know how much you spend we split that by skew so you can see that the main skew the pinkish one is the SQL my workspaces are consuming a lot of TB SQL and then

11:26you have you know a lot of information you can use and aage and of course you can also filter on the tags that we defined before so that you can go into more details track which team is using what it depends of your use case but you have all the information you want to be able to drill down and see you know what's going on on top of that one using TB demos we also package for you a bunch of dashboards so I'm going to show you that

11:49so the these are the more in know Community kind of dashboard that we have so again if you use DB demo you install the UC system table one you go down after the installation and you see a bunch of dashboard that we have so that one is the one I show you just before at the account level and then we also give you example on how you can track um the cost for your ml model the serving endpoints the warehouse the workflow the

12:13job everything so I'm going to show you a bunch of them so that you see what you can do all right so that one is to track the model suming points and see which endpoint and which user is potentially consuming what so you can have an like a

12:35view of all the model sing end points very interestingly we can see that someone created like really big end points recently so you definitely want to have a look and check that and and see what's going on here so giving you you know a bunch of information which endpoint are using foundational model track the cost track the cost per end points track potentially the user so if someone create a big end point you can go back

12:58to the user and say hey what going on here and you can also filter of course by tags and you have the same one for the warehouse so same thing you can track all the workspaces you can track which Warehouse is spending what potentially which user created the warehouse you can drill down and see what's going on and and you know get more information and avoid having the cost going too high we also have one like an amazing

13:22one for jobs and workflow so you can track everything you know which user is triggering which job the cost which work who is triggering the most and get all the information from this dashboard so you can really use that to drill down and see what's going on who is consuming what and which potentially workflow workspace model saving end point basically which resources is consuming the most and where you can go and check what's going

13:46on yeah and and those are like just pre-built prebuilt dashboards and we have a bunch of other dashboards we can I put the links in the description of the recording and you can also use system tables to build literally anything like use your imagination you have the schemas you can for example track the cost for every single query see the longest running workflows the

14:09workflows using I don't know interactive clusters you can do literally everything and now you no longer have excuses you can only understand from where the costs are coming yep and again super simple three steps enable the time table and first the tagging for classic and server L and then you can just set up your

14:31alerts and track the cost through all the nice nice dashboard we made for you super easy perfect thank you